Statistical Methods in Online A/B Testing
The book to reach for when you want to actually understand the math behind your test calculator, not just trust it.
What it is
Georgiev runs Analytics-Toolkit and has spent years writing about where online A/B testing statistics go wrong. The book is a careful, example-driven walk through the inference behind experiments: significance, confidence intervals, power, the assumptions of the tests marketers use every day, and the consequences of breaking them.
It is more technical than most marketing books and more applied than most statistics texts - a useful middle ground for people who run tests but were never formally trained in the methods.
What's strong
Its treatment of the peeking problem is the clearest in print. Georgiev quantifies how repeatedly checking a test inflates the false-positive rate well beyond the nominal five percent, and makes a strong, evidence-based case for sequential testing - methods that let you stop early without paying the error penalty. If your team has ever called a winner because the dashboard crossed 95 percent on a Tuesday, this is the corrective.
He is also good on the difference between statistical and practical significance, and on why observed power is a trap.
Where it stops
The writing is dense and occasionally reads like the blog posts it grew from. It is narrower than Kohavi, Tang and Xu - this is a statistics book, not a program-building book, so you will not find much on organizational change or platform design. Read it for the methods, not the management.
Who should read it
Analysts and CRO specialists who want to defend their numbers, and anyone considering sequential or always-valid testing. It pairs naturally with our minimum detectable effect and statistical significance tools.
How RGM uses it
This is the reference we open when a result looks too good and we need to explain, precisely, why early stopping created it. Georgiev's quantification of peeking error gives us the language to set a stopping rule before a test launches rather than negotiating one mid-flight under pressure to ship. For programs running many concurrent tests, his case for sequential methods is the practical path to faster decisions without surrendering rigor. We pair its lessons with our sample-size and significance tools so the theory in the book is never more than a click from the number a client actually needs to plan a test.