Design tests
Show your live offer design to some shoppers and a new design to others, then see which one gets more orders to add a protection plan. Start a test, pause or end it, and know when the result is real.
Small changes to how the offer looks can change how many shoppers add a protection plan. A design test shows you which design works better in your store, using real shoppers instead of guesses.
A design test (also called an A/B test) splits your shoppers into two groups. One group sees your live design. The other group sees a new design. SureBright then counts which group added more plans. Tests are in Studio, in the Merchant Portal, under the A/B Testing tab.
Good to know before you start:
Only one test can run at a time.
You need a live design first. Every test compares something against it.
On the test screens, your live design is called control. The design you compare it with is called the challenger.
What you can do#
Start a test#
Pick the design to test
Click New test. Under Pick what to test against, choose a design from Drafts (designs you saved but never made live) or Previously live (designs you used before). You can also click New version to copy your live design and change it first.
On the right, What changes lists every setting that is different between the two designs, side by side. If a design was live before, you also see how it did in its last 30 days.
Choose how many shoppers see each design
Under Traffic split, pick 50, 70 or 90 percent for your live design, or type your own number. The other number fills in so they add up to 100 percent. The new design can go to anywhere from 10 to 90 percent of shoppers.
Start it
Click Start test. Studio asks Start this A/B test? and tells you what share of shoppers will see the new design. Confirm to start.
A test has no end date. It starts straight away and runs until you stop it or pick a design.
Pause or end a test#
The test card shows Running or Paused. Your two designs are labelled Live · control and Challenger. For each one you see:
Shoppers: how many shoppers saw it.
Sold: how many plans it sold.
Revenue: how much money those plans brought in.
Bars underneath show each day. Use the buttons above the figures to switch between Attach rate (the share of orders that added a plan), Policies sold (plans sold) and Revenue, or to look at one kind of plan only.
After you stop a test, a shopper who already had a page open may still see their design for up to two hours.
While a design is in a running test, you cannot edit it in Studio. Edit a copy instead, or pause and end the test first.
Know when you have a winner#
The better design is the one where a bigger share of orders added a plan. Only orders with a product that a plan can cover are counted. These numbers are measured on Shopify only.
The results page tells you where the test stands in one line:
Only trust a result that says Clear winner. It shows up when both designs have had enough orders, and the gap between them is too big to be luck. Early in a test, one design can look far ahead just by chance. That is why it is shown as an Early signal and not as a result.
Your live design can win too. If the new design does clearly worse, the result is that your current design stays the better one. In the Designs tab, each design shows the result of its latest test, for example Won 4.6% vs 4.2%, Lost … or Tie …. While a test runs, it shows Testing · … so far.
You also see the plan money brought in per order that could have had a plan. Two designs can get the same share of orders to add a plan, while one sells more expensive plans. See Design performance for that view and for what each result message means.
Which design each shopper sees#
Each shopper keeps seeing the same design, on every page and every visit, for as long as the test runs. Each order counts for the design that shopper saw, so the two sets of results never mix.
See results and past tests#
Click View results to open a test's own page. It shows:
the one-line result
a box for each design with its share of orders that added a plan, plans sold and revenue
a chart over time, which you can switch between those three figures
the Pause, Stop and Promote winner buttons
When a test has ended and the new design won, the card shows Publish design so you can make it live straight away.
Older tests are listed under the current one. Each design's card keeps the result of its latest test. If a card says A test was set up but never started, nothing is running and every shopper sees your live design. Starting a new test clears it.