Statistical Significance in Plain English
When you run a test, one version usually appears to win. Statistical significance tells you whether that result is trustworthy or just chance — the kind of fluke you would see from flipping a coin a few times.
Understanding this concept stops you acting on noise. Declaring a winner too early is one of the most common and expensive mistakes in optimisation.
What the Numbers Mean
A confidence level of 95% means there is only a 5% chance the result happened by luck. Most teams treat 95% as the threshold before they trust a test, because it balances caution against the time spent waiting.
Why Sample Size Matters
- Small samples produce wildly swinging results.
- Larger samples settle towards the true figure.
- Low-traffic pages simply need longer to reach confidence.
A Practical Rule of Thumb
Decide the required sample size before you start, then let the test run to that point regardless of how tempting early results look. Peeking and stopping early is the surest way to fool yourself into a false win.
Frequently Asked Questions
Is 95% confidence always enough?
For most commercial decisions it is a sensible standard. Higher-risk changes may justify waiting for 99%.
What if my test never reaches significance?
That often means the difference is too small to matter, which is itself a useful finding.
If you need a hand with any of this, your Progressive Robot delivery team is ready to help. Raise a ticket from the Support area of your client portal or speak to your account manager and we will guide you through the next steps.