tl;dr Split-testing doesn’t verify your numbers, it only verifies which option is better. If one of the options tested is a clear winner, you’ll know with small amounts of data. So, I use split-tests to look for big winners. If my test doesn’t show a big winner quickly, I move on quickly.
[caption id="attachment_1547" align="alignnone" width="300"]![]() |
[caption id="attachment_1546" align="alignnone" width="300"]![]() |
Version | Conversion rate | # of visitors | |
A | 10% | 100 visitors | |
B | 25% | 30 visitors | Winner with 99% confidence! |
Ask yourself, what was it so confident about?
That Option B was better. Maybe only slightly better, but better.
The test did not tell me that Option B would continue to perform at 25% or would be 15% better than Option A - just that Option B is very likely to outperform Option A in the long run.
Split testing only tells you which option is better, not how much better.
Get it? In a split-test, the only number you can really act on is the statistical confidence of which option is better. The conversion rates, impressions and click-through rates are not reliable as predictions, just the winning option. That's why you don't necessarily need big numbers to get confidence.If you have a big winner on your hands, split-testing will tell you this quickly. So, especially when I’m starting, I look for big wins quickly. If my first test, say about a picture or a headline, doesn’t give me statistical confidence after 100-200 visitors, I usually scrap the test.
I would rather quickly abandon a version that might have worked better if I ran the test longer, because I can better invest that time in testing other things that might yield a big win. (There’s a balance to be found with sampling error here, but since I’m testing frequently and moving forward with so many unknowns, I accept false negatives in the interest of speed, and address sampling error when I’ve found a hit.)
This is how split-testing gives you actionable results fast.
Thanks Tendayi Viki and Andreas Klinger for reviewing this post.
A tough experience triggered me to reconsider by goals, so switched from educational services and NGOs to a monastic, creative life.
Mentor Impact - from the startup mentors that get results.
Decision Hacks - early-stage startup decisions distilled.
Peer Learning Is - education for fast-changing topics.