Design tests

Show your live offer design to some shoppers and a new design to others, then see which one gets more orders to add a protection plan. Start a test, pause or end it, and know when the result is real.

Works onShopify

Small changes to how the offer looks can change how many shoppers add a protection plan. A design test shows you which design works better in your store, using real shoppers instead of guesses.

A design test (also called an A/B test) splits your shoppers into two groups. One group sees your live design. The other group sees a new design. SureBright then counts which group added more plans. Tests are in Studio, in the Merchant Portal, under the A/B Testing tab.

Design tests come with Studio and are turned on when you ask for them. They work on Shopify only.

Good to know before you start:

  • Only one test can run at a time.

  • You need a live design first. Every test compares something against it.

  • On the test screens, your live design is called control. The design you compare it with is called the challenger.

What you can do#

You want toGo to
Try a new design on some of your shoppersStart a test
Take a break, end a test or pick a designPause or end a test
Know which design is better, and when to trust itKnow when you have a winner
Understand which shoppers see which designWhich design each shopper sees
Look at results or older testsSee results and past tests

Start a test#

  1. Pick the design to test

    Click New test. Under Pick what to test against, choose a design from Drafts (designs you saved but never made live) or Previously live (designs you used before). You can also click New version to copy your live design and change it first.

    On the right, What changes lists every setting that is different between the two designs, side by side. If a design was live before, you also see how it did in its last 30 days.

  2. Choose how many shoppers see each design

    Under Traffic split, pick 50, 70 or 90 percent for your live design, or type your own number. The other number fills in so they add up to 100 percent. The new design can go to anywhere from 10 to 90 percent of shoppers.

  3. Start it

    Click Start test. Studio asks Start this A/B test? and tells you what share of shoppers will see the new design. Confirm to start.

A test has no end date. It starts straight away and runs until you stop it or pick a design.

Pause or end a test#

The test card shows Running or Paused. Your two designs are labelled Live · control and Challenger. For each one you see:

  • Shoppers: how many shoppers saw it.

  • Sold: how many plans it sold.

  • Revenue: how much money those plans brought in.

Bars underneath show each day. Use the buttons above the figures to switch between Attach rate (the share of orders that added a plan), Policies sold (plans sold) and Revenue, or to look at one kind of plan only.

You want toClickWhat happens
Take a break and keep your resultsPause, then later ResumeWhile the test is paused, every shopper sees your live design. When you resume, the split starts again. Results so far are kept.
End the test and keep your live designStopStudio asks Stop this test? Every shopper sees your live design. You cannot restart a stopped test, but you can still make the new design live later.
End the test and use one design for everyoneMake design liveThat design goes live for every shopper. Studio first asks End the test and publish design?, or End the test and keep design live? if your live design is the one ahead.

After you stop a test, a shopper who already had a page open may still see their design for up to two hours.

While a design is in a running test, you cannot edit it in Studio. Edit a copy instead, or pause and end the test first.

Know when you have a winner#

The better design is the one where a bigger share of orders added a plan. Only orders with a product that a plan can cover are counted. These numbers are measured on Shopify only.

The results page tells you where the test stands in one line:

You seeWhat it meansWhat to do
No winner yetNeither design is far enough ahead.Keep the test running.
Early signalOne design is ahead, but it could still be luck.Keep the test running. Do not decide on this alone.
Clear winnerOne design is ahead and it is not luck. The lead is shown in points.Make it live. If your live design won, keep it.

Only trust a result that says Clear winner. It shows up when both designs have had enough orders, and the gap between them is too big to be luck. Early in a test, one design can look far ahead just by chance. That is why it is shown as an Early signal and not as a result.

Your live design can win too. If the new design does clearly worse, the result is that your current design stays the better one. In the Designs tab, each design shows the result of its latest test, for example Won 4.6% vs 4.2%, Lost … or Tie …. While a test runs, it shows Testing · … so far.

You also see the plan money brought in per order that could have had a plan. Two designs can get the same share of orders to add a plan, while one sells more expensive plans. See Design performance for that view and for what each result message means.

Which design each shopper sees#

Each shopper keeps seeing the same design, on every page and every visit, for as long as the test runs. Each order counts for the design that shopper saw, so the two sets of results never mix.

See results and past tests#

Click View results to open a test's own page. It shows:

  • the one-line result

  • a box for each design with its share of orders that added a plan, plans sold and revenue

  • a chart over time, which you can switch between those three figures

  • the Pause, Stop and Promote winner buttons

When a test has ended and the new design won, the card shows Publish design so you can make it live straight away.

Older tests are listed under the current one. Each design's card keeps the result of its latest test. If a card says A test was set up but never started, nothing is running and every shopper sees your live design. Starting a new test clears it.