Running a test you can trust

How to find out whether a change to your protection offer really sold more plans: change one thing, compare fairly, wait for a clear result, and know what to test first.

Most opinions about what sells more plans are guesses until they are tested on your own shoppers. A good test is simple: change one thing, compare fairly, and wait until the result is clear. This guide shows you how, with or without a testing tool.

What you can do#

You want toDo this
Compare two designs on live shoppers, on ShopifyRun a design test in Studio
Test on another platform, or without StudioAsk your SureBright contact to set up the test for you, or use a before and after comparison
Test a pricePublish the new price and compare versions. See Pricing your plans
Decide what to test firstWhat to test first
Know when to stopWait for a clear result

Four rules for a fair test#

Change one thing. If you change the layout, the colours and the pre-selected plan together, you will not know which one made the difference. Copy your live design and change a single setting.

Compare at the same time. Two designs shown to two groups of shoppers in the same weeks is a fair comparison. This month against last month is not, because a sale, a season or an ad campaign changes who is shopping.

Decide what you are measuring before you start. For most changes that is attach rate: the share of eligible orders that added a plan. For price changes it is what you earned, because a lower price can sell more plans and still earn less.

Wait until the result is clear. Early in a test one side is often far ahead by luck.

Wait for a clear result#

A design test tells you where it stands in one line:

You seeDo this
No winner yetKeep the test running
Early signalKeep the test running. Do not decide on this alone
Clear winnerMake the winning design live. If your live design won, keep it

Three things to expect:

  • Stores with fewer orders wait longer. A result needs enough orders on both sides. Sending half your shoppers to each design gets you there fastest.

  • Your current design can win. That is a real result. It tells you not to make the change.

  • A tie is useful too. If two designs sell the same, pick the one you like and test something else.

Try not to start or end a test in the middle of a big sale. Shoppers behave differently in those weeks, and what wins then may not win in a normal month.

What to test first#

Start with the changes that tend to matter most, one at a time.

TestThe question it answersRead
A pre-selected plan against noneDoes a default sell more plans, and do orders hold up?Pre-selected plans and conversion
Which plan is pre-selectedDoes a shorter or a longer default earn more?Default plan selection
With and without a No coverage choiceDoes an easy way to decline change what shoppers pick?Default plan selection
Cards, stacked rows or a dropdownWhich layout do your shoppers read best, especially on phones?Studio
How the price is shownDoes a monthly amount read better than the full price?Studio

Colours and corner styles are worth matching to your brand, and they rarely need a test.

Compare before and after when you cannot split#

Some changes cannot be shown to half your shoppers, such as a new place for the offer or a new price. You can still learn from them:

  1. Note the date of the change and make no other change for that period.

  2. Use periods of the same length on each side, covering the same days of the week.

  3. In Sales, set the comparison to Previous Period and read Attachment Rate.

  4. Check nothing else happened in those weeks: a sale, a new theme, a product launch or an ad campaign.

  5. Treat a small difference as no difference. Only a large, lasting change tells you something.

Keep a test log#

Write down each test: the date, what you changed, the result and what you decided. In Studio, the Designs tab keeps the result of each design's latest test. A simple list of your own covers everything else, and it stops you testing the same idea twice.