Most Shopify merchants run A/B tests wrong. Not because the tools are bad, but because the math gets misread. A test hits 95% confidence after three days and the team ships the winner. A month later, revenue is flat or down. That is what happens when statistical significance gets treated as a finish line instead of a tool.
What Statistical Significance Actually Means
Statistical significance answers one question: how likely is it that the difference you observed happened by random chance?
When a test shows 95% statistical significance, it does not mean the variation is 95% likely to be better. It means: if there were no real difference between the control and the variation, you would only see a result this large 5% of the time just by chance.
Put differently, you are accepting a 5% chance of a false positive. One in twenty tests that reach 95% confidence will show a winner that is not actually a winner.
Statistical significance tells you whether a result is real. It does not tell you how large the effect is, whether the effect matters for your business, or whether it will hold over time.
Why 95% Confidence Is the Standard
The 95% threshold (p less than 0.05) comes from academic research and has carried over into industry practice. It is a reasonable default, not a law of nature.
What matters more than the threshold you pick is that you pick it before the test starts and stick to it. Changing the threshold after you see the data is called p-hacking. It is the single fastest way to produce meaningless results.
Some teams use 90% confidence for low-stakes copy changes where the cost of a false positive is minimal. For checkout flow changes, pricing tests, or any change that is difficult to reverse, 99% confidence is more appropriate. The threshold should match the business risk of acting on a false positive.
The Math Behind the Threshold: What p-Value Actually Measures
The p-value is a probability statement about your data, assuming there is no real effect. A p-value of 0.05 means: if variant B were truly identical to variant A, there is a 5% chance you would see a difference this large or larger just from random sampling.
This is where most teams get confused. The p-value does not measure the probability that your variation is better. It measures the probability that your data is this extreme under the null hypothesis (no effect).
Practical implication: if you run twenty tests and all of them are truly neutral, you should expect one of them to appear significant at 95% confidence by chance alone. That is not a testing program failure. That is statistics. The antidote is a large enough test volume with disciplined pre-registration of your hypothesis and your stopping criteria.
Sample Size: Why Most Shopify A/B Tests End Too Early
Before you launch a test, you need to know the minimum detectable effect (MDE): the smallest lift that would actually matter to your business.
The smaller the MDE, the more traffic you need. This is non-negotiable math.
Use any sample size calculator (Evan Miller has a free one) and input:
- Baseline conversion rate: your current CVR for the page or flow being tested
- Minimum detectable effect: the absolute or relative lift you care about
- Statistical power: typically 80%
- Significance threshold: 95% or your chosen level
The output is the number of visitors per variation, not total visitors. A test with two variations at 10,000 visitors each needs 20,000 total visitors.
A Concrete Example
Your product page converts at 3%. You want to detect a 15% relative improvement, meaning a lift from 3.0% to 3.45%. At 80% statistical power and 95% confidence, you need approximately 8,600 visitors per variation, or 17,200 total visitors before calling the test.
If your page receives 4,000 visitors per month, that test will take over four months to reach a valid conclusion. The options are: run it anyway and wait, accept a higher MDE (which means only detecting larger effects), or accept a lower power (which means a higher rate of false negatives).
Most teams do none of these. They call the test at day 14 with 2,000 visitors per variant and declare a winner. That is not a result. That is a guess with a confidence percentage attached to it.
The Peeking Problem
Looking at your test results before you hit your target sample size is called peeking. Every peek is a chance to call the test early. A test designed for one analysis at 95% confidence but peeked at ten times has an effective false positive rate closer to 30 to 40%.
The fix: calculate your required sample size before launch and do not call the test until you hit it. Turn off the daily Slack notifications from your testing tool if they are creating pressure to call tests early.
How to Calculate When Your Test Is Ready to Call
A test is ready to call when all three of the following are true:
- You have reached your pre-calculated sample size in each variation.
- You have run the test for at least one full business cycle (seven days minimum).
- Your significance threshold has been met at the planned analysis point, not during a mid-test peek.
The seven-day minimum exists because Shopify stores have predictable day-of-week patterns. Traffic and purchase intent on Monday look different than on Saturday. A test that runs only three days may have captured a skewed sample, regardless of what the confidence meter says.
The Difference Between Significance and Practical Impact
A result can be statistically significant and practically meaningless.
If you run a test and detect a 0.05% absolute lift in conversion rate at 99% confidence, that result is real. But 0.05% on a 3% CVR baseline may not cover the cost of the engineering time to implement the change permanently.
Practical significance asks: does this lift move the needle for the business?
The metric for practical significance is not p-value. It is revenue impact. Take your monthly unique visitors, multiply by the detected lift in CVR, multiply by average order value. That is your projected annual revenue impact. If that number does not justify implementation cost plus opportunity cost, the test result is worth recording as a learning, not shipping as a permanent change.
Calculating Revenue Impact From a Test Result
Here is the formula we use for every test we run:
Annual revenue impact = (detected CVR lift) x (monthly sessions) x (average order value) x 12
Example: a test detects a 0.4% absolute CVR lift (from 2.0% to 2.4%) on a product page receiving 8,000 monthly sessions with a $75 AOV.
0.004 x 8,000 x $75 x 12 = $28,800 in projected annual revenue.
That number passes the practical significance threshold for almost any store. A 0.04% lift on the same page produces $2,880 annually. That may or may not justify implementation depending on your dev costs and roadmap.
Common Mistakes That Lead to False Positives
Stopping tests based on the dashboard confidence meter
Most A/B testing tools show a running confidence percentage. The meter updates in real time, which means it is calculating significance continuously. This is peeking by design. That 97% confidence reading you see on day four is almost certainly an overestimate of the true signal at that sample size.
Testing multiple metrics without correction
If you test a variation and measure conversion rate, add-to-cart rate, AOV, and revenue per visitor, you are running four hypothesis tests simultaneously. The probability that at least one shows p less than 0.05 by chance is roughly 19%. Designate one primary metric before the test starts. Track secondary metrics as diagnostic signals, not decision inputs.
Running too many variations at once
An A/B/C/D test with four variations gives each variation roughly 25% of the traffic. You now need four times as long to reach significance for each pairwise comparison. For most Shopify stores, multivariate tests are not viable because traffic volume is too low to sustain them. Stick to one control and one variation unless you are running more than 50,000 monthly sessions on the page being tested.
Ignoring novelty effects
A new design often gets a short-term lift simply because it is new. Returning visitors notice the change and engage differently, at least temporarily. Running tests for at least a full week, ideally two, helps filter this out. If your test shows a large lift in week one that partially reverses in week two, novelty effect is likely a factor.
Seasonal contamination
Running a test that spans a major sale event, a holiday, or a product launch contaminates the results. External traffic quality and purchase intent shift dramatically during these periods. The difference you observe between control and variant may reflect the timing, not the change. Pause tests during major promotional events and restart them cleanly afterward.
How Statistical Power Affects Your Test Program
Statistical power is the probability that your test will detect a real effect if one exists. The industry default is 80%, meaning your test has a 20% chance of missing a real effect (a false negative).
This matters at the program level. If you are running a testing roadmap with ten tests per quarter, an 80% power setting means two of those tests may fail to detect a real positive effect. You call them inconclusive or losses when they were actually wins you did not have enough data to see.
Increasing power to 90% requires more traffic per test but reduces that miss rate. For teams running fewer tests per period and operating at lower traffic volumes, 90% power is worth the longer test duration. For high-velocity programs at high traffic, 80% is a reasonable tradeoff.
Segment Your Results Before Calling a Winner
A test that shows a 12% lift overall may be hiding important variation underneath. Before shipping any winning variant, segment the results by:
- Device type: a change that lifts desktop CVR 18% but hurts mobile CVR 6% is not a clean win given that mobile typically drives 60 to 70% of Shopify traffic.
- Traffic source: paid traffic and organic traffic often respond differently to the same change because buyer intent and familiarity with the brand differ.
- New vs. returning visitors: returning visitors have prior brand context. A trust-signal test may lift new visitor CVR significantly while having no effect on returners who already trust you.
- Geographic segment: if you sell internationally, a test may perform very differently across markets due to cultural differences in purchase behavior.
Segmented results tell you not just whether a variant won, but who it won with. That shapes how you implement it and what you hypothesize next.
What to Do With a Losing Test
A losing test is not a failure. It is data. The question is what it tells you.
A test that shows the variant is definitively worse eliminates a hypothesis with confidence. That is valuable. You now know your visitors do not respond to that particular change the way you predicted. Update your model of customer behavior and form a better hypothesis.
An inconclusive test (neither winner nor loser at your confidence threshold) is more ambiguous. It may mean the effect is smaller than your MDE, the test ran short, or the change simply does not matter to your visitors. Before discarding it, check whether the test was adequately powered. If sample size was the issue, consider whether the traffic will ever support a longer run.
Document every test result, including context: what was the hypothesis, what did the data show, what is the likely explanation, and what does this mean for the next test. A testing library that only records winners is a library that learns nothing from half its data.
How We Apply This at Scalo
Every A/B test we run follows a pre-registration protocol. Before a single line of variation code is written, we document:
- The hypothesis and the research supporting it
- The primary metric and any secondary diagnostic metrics
- The required sample size per variant
- The confidence threshold for this specific test
- The minimum test duration in days
- The planned analysis date
This documentation gets locked before the test launches. No one on the team can see live results until the planned analysis date, which eliminates peeking entirely.
After the test completes, we run a segmented analysis before calling the result. We check device split, traffic source, and new vs. returning visitors. If the results diverge meaningfully across segments, we flag it and may run a follow-up test targeted at the segment where the effect was strongest.
The result is a testing program that produces reliable learnings instead of a dashboard full of questionable wins that never show up in revenue.
If your store is running tests but not seeing compounding revenue improvements, the issue is almost always process. Our Shopify CRO service covers the full testing infrastructure, from research through analysis, and our CRO audit is the right starting point if you are not sure where your current testing program is breaking down.