How to get statistical confidence from your tests with small amounts of data.

Sep 28, 2011

tl;dr Split-testing doesn’t verify your numbers, it only verifies which option is better. If one of the options tested is a clear winner, you’ll know with small amounts of data. So, I use split-tests to look for big winners. If my test doesn’t show a big winner quickly, I move on quickly.

Myth - Split-testing requires a large sample size to be accurate

Don't get distracted by the numbers. We're used to thinking of statistical significance as only being possible with a large number of tests, known as the sample size in stats. In certain types of statistical tests, your sample size needs to be thousands or more because those tests establish the likelihood that a specific number will happen. When you are split testing, you are not learning about specific numbers, just which option is better.  You only need a large sample size if there's a tiny difference between the two. If there's a big difference between the two, you get confidence with small sample size.

What does split-testing actually tell you?

I'll share a split-test I did on Unbounce that will make your brain jump out of your head and slap your face. I started with a waiting list page for Leancamp - It had a respectable 10% conversion rate, but I had launched the page really quickly and wasn't happy with it, so I made a change and tested that change as a split-test.

 

[caption id="attachment_1547" align="alignnone" width="300"] Version A: Crappy starting version.

[caption id="attachment_1546" align="alignnone" width="300"] Version B: Blatant copy of Buffer version

30 visitors later, Unbounce was telling me:

Version	Conversion rate	# of visitors
A	10%	100 visitors
B	25%	30 visitors	Winner with 99% confidence!

What, you cry out? 99% statistical confidence in just 30 visitors?!

Ask yourself, what was it so confident about?

That Option B was better. Maybe only slightly better, but better.

The test did not tell me that Option B would continue to perform at 25% or would be 15% better than Option A - just that Option B is very likely to outperform Option A in the long run.

Split testing only tells you which option is better, not how much better.

Get it? In a split-test, the only number you can really act on is the statistical confidence of which option is better. The conversion rates, impressions and click-through rates are not reliable as predictions, just the winning option. That's why you don't necessarily need big numbers to get confidence.

Using split-testing for quick, actionable learning.

Split testing is a tool to learn and improve quickly, giving you confidence of one option over the other. You can use it to evolve quickly and with confidence.

If you have a big winner on your hands, split-testing will tell you this quickly.  So, especially when I’m starting, I look for big wins quickly.  If my first test, say about a picture or a headline, doesn’t give me statistical confidence after 100-200 visitors, I usually scrap the test.

I would rather quickly abandon a version that might have worked better if I ran the test longer, because I can better invest that time in testing other things that might yield a big win. (There’s a balance to be found with sampling error here, but since I’m testing frequently and moving forward with so many unknowns, I accept false negatives in the interest of speed, and address sampling error when I’ve found a hit.)

This is how split-testing gives you actionable results fast.

Thanks Tendayi Viki and Andreas Klinger for reviewing this post.