Ronny Kohavi

Ronny Kohavi is the computer scientist who scaled online experimentation from a tactic to an institution — director of data mining and personalization at Amazon, then founder of Microsoft's Experimentation Platform (ExP), which grew to run tens of thousands of experiments a year across Bing and Office, then VP at Airbnb. Trustworthy Online Controlled Experiments (2020, with Diane Tang and Ya Xu) is the field's standard text — known simply as "the experimentation book."

The Overall Evaluation Criterion (OEC) — one agreed metric combination that defines success before any test runs; Twyman's law, his favorite warning — any figure that looks interesting or unusual is usually wrong, so investigate stunning wins before celebrating them; A/A tests and sample-ratio-mismatch checks to validate the instrument itself; small percentage changes are the honest norm (Bing's famous title-spacing win was rare); and institutional memory — most ideas fail, so a platform that records everything beats a hero analyst.

Every growth team that ships by experiment inherits his guardrails, usually without knowing their source. The uncomfortable statistics he publicized — that the majority of well-designed experiments at sophisticated companies fail to move their target metric — is the strongest argument for testing at all: intuition at Microsoft and Amazon scale was wrong most of the time, and yours probably is too.

Born in Israel; Stanford PhD in machine learning (1995) under a lineage that produced the MLC++ library and the C4.5-era decision-tree work. The experimentation chapter began at Amazon (2003) and became Microsoft's ExP platform in 2006.

Etymology: source.

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Areas of marketing where ronny kohavi is a core concern: