A/B tests — compare two popups honestly

An A/B test shows half of your visitors the running popup (Control A) and the other half a variant (B). It measures which version brings more email signups per view. A winner is only declared once the data really supports it.

What is compared#

• Split: a fixed 50/50 of eligible popup views.
• Winner metric: the email-submit rate — signups divided by popup views. Not revenue, and not an estimated probability of winning.
• What stays shared: theme, reward, code delivery, targeting and integrations apply to both variants. B differs in its content: copy, structure, design — or an entirely different mechanic.

By default both variants share one reward, so different discount levels (say 10 % versus 15 %) can't be tested against each other. Compare copy or design with the same offer.

Reward per variant (where it is enabled for your shop): in the Variant B editor, the Reward tab lets B use one of your existing rewards instead of A's. No new code is created. What changes:

• Different offers: the test compares sign-ups, not profit, and the winner is always chosen manually.
• Own code per variant: a static code that only one variant uses makes revenue per variant measurable. A code shared by A and B counts for the popup, not per variant.
• Frozen at start: each variant keeps the reward it started with, even if the reward is edited later. A winner keeps its own reward.

Create a test#

Requirements: the popup is published, not paused, and has no unpublished changes — Control A needs one unambiguous live version. Then use Add test variant on the popup's detail page, or go through A/B Tests in the navigation.

1. Starting point — what B starts from: a copy of your published popup (Edit current popup), another popup from your shop (Use existing popup, published or a saved draft), or a new starting point.
2. What to change — a focused Test headline or button text (one text element, A versus B) or Edit freely in the full editor.
3. Review variant B — check B and look at it in the preview. Up to here everything is a draft; the live popup stays unchanged.
4. Start 50/50 test — from now on both variants are locked until a winner is chosen.

Variant B can edit its copied popup freely, but it can't be turned into a different mechanic later (say from a prize wheel to a scratch card). If you want that, discard the draft and create B again.

When there is a winner#

PopUplift only reports a measured winner once both variants have at least 1,000 views and the evidence threshold of the statistical method is crossed. Before that — and on “The measured result does not show a clear difference” — the test simply keeps running.

Apply the winner: manually or automatically#

In the test, Winner selection decides what happens:

• Choose a winner manually — the test runs until you apply a winner yourself (End test and apply a winner). Once there is enough data, PopUplift shows the measured winner as a recommendation.
• Apply a statistically confirmed winner automatically — PopUplift applies the winner itself once the minimum sample and the evidence threshold are reached. The results then say Applied automatically.

Choosing by hand stays available at any time in both modes — earlier too, though results may be inconclusive. An automatic switch only happens for a popup that is published and active at that moment; paused or unpublished popups are skipped.

Applying makes the winning version the only live version and ends the test. This cannot be undone.

Tests with more than two variants#

Tests with several variants can also be started from prepared drafts. They split views evenly but have no automatic winner: you review the results per variant and choose yourself.

Reading results#

Results are on the test page (View test results, also reachable through A/B Tests or Reports → A/B test performance): views, signups and rate per variant, plus the current verdict. Code revenue is shown alongside — but it does not decide the winner.

While a test is running, it also manages the Smart Trigger of that popup; its switch is locked in the meantime.

Autopilot#

The autopilot keeps testing your popups for you, one round after another. It is being rolled out and not yet enabled for all shops; where it is, you find the switch PopUplift improves my popups automatically on Home.

• One switch for the shop. Once it is on, every live popup with at least 1,000 views in 14 days, no open test and no unpublished changes joins by itself, new popups included.
• What it tests. Your own prepared variants first: the oldest one marked Ready that keeps the original's reward. A variant with a different reward is never taken, and neither is one whose text names a different offer (for example 20% off on a 10% popup); Home tells you which one to adjust. A popup without such a variant waits; Home shows it with Add a variant as the next step. Once all your variants have run, the same step appears again, and a new ready variant joins within the hour. A popup whose own setup blocks a test (for example a reward without an element that shows the code) is listed under Needs a fix before testing.
• How a round ends. A measured winner is applied. Without a difference (3,000 views per variant or 28 days) or when the signup rate drops by 40%, the round ends and the original stays.
• What never changes. Offers, discounts and rewards. Only one test per popup; any change you make to the popup stops the autopilot for it.
• Stops and pauses. After 3 rounds in a row without a winner the autopilot pauses that popup for 14 days, then tries again.
• Your control. Switch it off on Home: no new rounds start, a running test continues as a normal test. On a popup's page, Turn off takes just that popup out for good, and you can switch off the automatic ends there.

Every step is listed in the popup's timeline.