---
title: Experimentation Services & Agency | RGM®
url: https://realgrowthmatters.com/services/experimentation
updated: 2026-06-10
source_html: https://realgrowthmatters.com/services/experimentation
---

Stop debating. *Start deciding.*

# Experimentation Services & Agency — A Field Guide

One test settles what a month of meetings can’t. This guide shows you how a real experimentation program works — the compounding math, the traps that fake your results, and the operating system behind teams that test weekly instead of quarterly. No pitch. Just the model we wish every brand understood.

By David Schaefer · [LinkedIn](https://www.linkedin.com/in/daschaefer/) · Updated June 2026

[Start with the model ↓](#s02)

## Single tests add. Programs *compound.*

A test is a coin you flip once. A program is a machine with three dials: **velocity** (how many trustworthy tests you run), **win rate** (how often a test beats control), and **effect** (how big the average win is). Multiply them and you get your growth rate from experimentation — and because every win raises the baseline the next win multiplies, the curve bends upward. Most teams obsess over single-test drama. The compounding lives in the dials.

- ⏱**Velocity is the exponent.** Win rate and effect set the size of each step; velocity sets how many steps you take. Exponents beat coefficients.
- 🎯**You can’t pick winners in advance.** Bing’s best revenue idea ever — worth $100 million a year — sat shelved for months as a low priority.[1](#src-1) Volume finds what judgment misses.
- 🛠**Trust is the throttle.** A fast program that fools itself compounds errors instead of wins. Every dial assumes the readouts are real.

> “Getting numbers is easy; getting numbers you can trust is hard.” — Ron Kohavi, Diane Tang & Ya Xu · *Trustworthy Online Controlled Experiments*[3](#src-3)

*FIG. 01 — The machine. Turn any dial and the curve answers — velocity turns it the hardest.*

## Experiments end *arguments.*

Every company has a highest-paid opinion, and most roadmaps bend toward it. Experimentation replaces that physics. When any claim can be tested in two weeks, debates get shorter, juniors with good ideas beat seniors with strong feelings, and “I think” quietly turns into “let’s find out.” That cultural shift — not any single result — is the compounding asset.

The Booking.com standard

Booking.com runs some 25,000 tests a year, and anyone can launch one without management’s permission — an entire company wired to find out instead of argue.[2](#src-2)

Demote the HiPPO

The highest-paid person’s opinion is a hypothesis like any other. The program’s first deliverable is a culture where it queues for a test like everyone else’s.

Make losing safe

If a failed test embarrasses its owner, people will only test sure things — and sure things teach nothing. Celebrate the decision quality, not the coin flip.

The discipline pays best where opinions are most expensive: pricing, offers, onboarding, headline claims. Start the program where the arguments are loudest. [hypothesis testing](https://realgrowthmatters.com/glossary/hypothesis-testing/) · [program design](https://realgrowthmatters.com/learn/experimentation/experimentation-program-design)

## A hypothesis is a bet with a *reason.*

“Try a green button” is not a hypothesis. A real one names the audience, the friction or belief you’re changing, the expected direction, and the evidence that made you suspect it — *because* is the most important word in the brief. Then build the queue as a portfolio: mostly evidence-backed swings at big surfaces, a few cheap structural tweaks, and one or two bold bets nobody can predict. Bing’s $100M title change looked like a tweak.[1](#src-1) Nobody knows which is which in advance — that’s why the portfolio, not the picker, carries the program.

*FIG. 02 — The ladder. Climb before you queue; the “because” is what compounds into knowledge.*

Grade the backlog by evidence rung and surface size, then let volume do what foresight can’t. [hypothesis](https://realgrowthmatters.com/glossary/hypothesis-testing/) · [where CRO feeds the queue](https://realgrowthmatters.com/services/cro) · [where the evidence comes from](https://realgrowthmatters.com/services/marketing-analytics)

## Underpowered tests are expensive *coin flips.*

Power is a test’s eyesight: the chance it spots a real effect of a given size. Run two weeks on thin traffic hunting a 3% lift, and your test may have a 30% chance of seeing a difference that’s truly there — you paid full price for a maybe. The honest move happens before launch: decide the smallest lift worth detecting, compute the sample that can see it, and if the traffic isn’t there, test something bigger or somewhere busier. Changing the question beats faking the answer.

*FIG. 03 — Detectability. Traffic decides what your test is allowed to ask.*

Pick the MDE first

The [minimum detectable effect](https://realgrowthmatters.com/glossary/ab-testing-minimum-detectable-effect/) is a business choice, not a stats output: the smallest lift that pays for the work. Everything sizes from it.

Run whole weeks

Weekday buyers and Sunday browsers are different people. Full-week cycles — usually two or more — keep the mix honest and dodge novelty’s first-day sugar.

One primary metric

Declare the deciding metric before launch. Ten metrics at 95% confidence nearly guarantee a false “win” somewhere — that’s arithmetic, not bad luck.

Size it before you launch it: [sample-size calculator](https://realgrowthmatters.com/tools/a-b-test-sample-size/) · [duration estimator](https://realgrowthmatters.com/tools/test-duration-estimator/) · [power analysis, properly](https://realgrowthmatters.com/learn/experimentation/power-analysis-methodology) · [statistical power](https://realgrowthmatters.com/glossary/statistical-power/)

## Watch a dead-even test *“win.”*

Below is a test where the variant truly changes nothing — we built it that way. Step through the days and watch its significance score wander. Around day eight it brushes the 95% line, purely by chance. Stop there and you’d ship a mirage, report a win, and wonder later why revenue never moved. Check results daily and stop at the first “significant” flicker, and your real false-positive rate can run several times the 5% you think you bought. That’s the peeking trap — and the fix is boring: fix the horizon before launch, or use methods built for continuous monitoring.[4](#src-4)

A null test, day by day — tap through it

Next day ▶

Run all 28 days

↺ Reset

Day 0 — the variant is a placebo. Truth: zero effect. Press

Next day

.

> “…you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman · Caltech commencement, 1974

Illustrative simulation · RGM analysis; the alpha-inflation it dramatizes is standard sequential-testing statistics.[4](#src-4) If you need to monitor continuously, use always-valid methods built for it. [always-valid inference](https://realgrowthmatters.com/learn/experimentation/always-valid-inference) · [p-value](https://realgrowthmatters.com/glossary/p-value/) · [statistical significance](https://realgrowthmatters.com/glossary/statistical-significance/)

## Audit the experiment before the *result.*

The most dangerous test isn’t the one that fails — it’s the one that lies convincingly. Broken randomization, a tracking tag that drops one variant’s conversions, a bot wave hitting one arm: each produces a beautiful, decisive, wrong answer. Mature programs run integrity checks as reflexes, the way pilots run preflight — not because crashes are common, but because they’re expensive.

Check the split — SRM

A 50/50 test that arrives 52/48 with real volume isn’t unlucky; it’s broken. Sample-ratio mismatch is the smoke alarm of experimentation — when it sounds, the result is void, no exceptions.

Run A/A tests

Test the platform against itself: identical experiences in both arms. “Significant” differences should appear ~5% of the time. More than that, and your tool — not your ideas — is generating the wins.

Stand guardrails

Every test watches a small set of do-no-harm metrics — page speed, errors, refunds, unsubscribes. A “winner” that degrades a guardrail is a loss wearing a medal.

Trust compounds like wins do: every audited test makes the next decision cheaper to believe. Skip the audits and the program’s numbers slowly turn ornamental. [sample size](https://realgrowthmatters.com/glossary/sample-size/) · [significance](https://realgrowthmatters.com/glossary/statistical-significance/) · [the measurement layer underneath](https://realgrowthmatters.com/services/marketing-analytics)

## Significant isn’t the same as *worth shipping.*

A test can be 95% certain about a lift too small to matter — a 0.2% gain with a diploma. It can also show a juicy 8% lift with intervals so wide the truth could be zero. Reading results means holding both questions at once: how sure are we, and how much do we care? The decision rules are written before launch, so the answer never depends on who’s presenting.

SHIP · lift ≥ MDE, significant, guardrails clean

roll out + monitor

ITERATE · promising direction, underpowered

sharpen & requeue

KILL · flat or negative at full horizon

archive + record the why

Every outcome

updates the knowledge base

The decision card — written at launch, signed by the owner. Illustrative · RGM analysis.

Report the interval

“+4% (CI: +1% to +7%)” tells the truth a point estimate hides. Narrow intervals earn rollouts; wide ones earn replication.

Discount the sugar rush

Novelty inflates early numbers — regulars poke at anything new. Judge the last full week, not the launch spike, before believing a lift.

Translate to money

End every readout in dollars at scale: “+2.1% checkout = ~$31k/mo at current traffic.” It keeps the program honest about which wins were worth the slot.

The deliverable of a test isn’t a verdict; it’s a sentence in the company’s knowledge base that makes the next ten hypotheses sharper. [p-value](https://realgrowthmatters.com/glossary/p-value/) · [A/B test](https://realgrowthmatters.com/glossary/ab-test/) · [reading continuously, validly](https://realgrowthmatters.com/learn/experimentation/always-valid-inference)

## Velocity is a property of the *system.*

Teams don’t test slowly because they’re lazy. They test slowly because every experiment is hand-made: a bespoke brief, a engineering ticket, a one-off analysis, a meeting to decide. Velocity comes from turning that craft into a production line — templated briefs, pre-built test slots, automated readouts, a standing decision meeting — so the marginal test costs hours, not weeks. The constraint is almost never traffic. It’s the queue.

*FIG. 04 — The production line. Throughput is set by the slowest station, not the smartest idea.*

Measure the program like a factory: tests shipped per month, days from idea to decision, share of slots filled. The dials only turn if someone owns them. [velocity frameworks](https://realgrowthmatters.com/glossary/ab-testing-velocity-frameworks/) · [program design](https://realgrowthmatters.com/learn/experimentation/experimentation-program-design)

## Most ideas lose. That’s the *point.*

At Microsoft, only about a third of well-designed ideas actually improved the metrics they targeted — a third did nothing, and a third made things worse.[5](#src-5) Search teams at Bing saw success rates closer to one in six or seven.[5](#src-5) Sit with that: the best-instrumented companies on earth are wrong about most of their own ideas. The lesson isn’t despair — it’s that every untested “obvious improvement” you shipped straight to 100% of customers had those same odds, without the safety net.

*FIG. 05 — The honest scoreboard. Programs profit three ways; only one of them shows up in a victory lap.*

Win rate is also a dial you tune — better evidence per hypothesis, bigger surfaces, bolder swings — but its base rate is humbling for everyone. Plan the program math around reality, not bravado. [A/B testing fundamentals](https://realgrowthmatters.com/glossary/ab-testing-fundamentals/) · [what the evidence ladder feeds](https://realgrowthmatters.com/services/growth-strategy)

## No split? There’s still a *design.*

Some questions can’t be answered by splitting users down the middle — brand campaigns that blanket a city, price changes you can’t show half your customers, channels where targeting is the product. The experimental toolbox is bigger than the A/B test: split geography instead of people, alternate time windows, phase rollouts in waves. The principle never changes — build a credible “what would have happened anyway” — only the unit of randomization does.

Geo experiments

Randomize matched markets, not users. The workhorse for brand, audio, CTV, and anything a cookie can’t follow — no tracking consent required.

Switchbacks

Alternate the treatment in time blocks — on-hours vs off-hours — when everything shares one marketplace and users can’t be cleanly split.

Staged rollouts

Ship to 5%, then 25%, then 100%, reading guardrails at each gate. Slower than a split, far safer than a launch-and-pray.

Channel-level money questions — “is this spend incremental at all?” — belong to the measurement stack’s holdout designs; this page’s job is everything you can randomize yourself. [incrementality testing](https://realgrowthmatters.com/glossary/incrementality-testing/) · [the measurement stack](https://realgrowthmatters.com/services/marketing-analytics)

## Turn the dials. Watch a year *compound.*

Six inputs — your revenue, your three dials, an honesty haircut for lifts that fade. The model projects the year two ways: one-off wins that merely add, and a program where every win raises the base the next one multiplies. Then it answers the planning question no single test can: which dial is worth turning first?

The program velocity calculator

### Velocity is the *exponent.*

Each test is expected to move the business by **win rate × average lift** — small. The program raises that small number to the power of **every test you run**: ( 1 + w·e )N. Bases add polish; exponents add zeroes. That’s why the teams that win at this — the Bings and Booking.coms running thousands of tests[1](#src-1)[2](#src-2) — obsess over throughput, not predictions.

Baseline revenue

$ / mo

The revenue base your experiments are trying to move.

Velocity

tests / mo

Trustworthy, decided tests — not launches. Most teams manage 1–2; a working program runs 4+.

Win rate

%

Share that beat control. Microsoft’s documented base rate: about a third.

5

Average lift per win

%

On the primary metric, measured at full horizon — not the launch spike.

Persistence

%

The honesty haircut: how much of each lift survives novelty decay and interactions.

Horizon

months

How far to run the projection.

✓ A compounding engine

Projected lift at horizon

0%

0%

expected / test

$0

run-rate at horizon

$0

cumulative new revenue

+0%

+1 test / mo

+0%

+5 pts win rate

+0%

+1 pt avg lift

Expected lift per test = win × effect × persistence, compounded over every test in the horizon.

*FIG. 06 — The gap between the lines is what a program earns over a pile of one-off tests.*

The same dials at different velocities

| Velocity | Tests / yr | Lift at horizon | Cumulative new revenue | Reading |
| --- | --- | --- | --- | --- |

**How it’s calculated**

Each decided test is expected to move the primary metric by

E = win rate × average lift × persistence

and the program compounds it across every test in the horizon:

Multiplier = ( 1 + E )

N × months

Monthly revenue follows the rising base; the cumulative line sums each month’s gain over baseline:

Cumulative = Σ

m

Revenue × ( (1+E)

N·m

− 1 )

- The dial comparison re-runs the same formula with one dial nudged — velocity enters the exponent, the others the base, which is why velocity usually wins.
- **Persistence** haircuts each lift for novelty decay and interaction effects — an expectation model, labeled as such: real programs arrive lumpy, in wins and droughts. The formula is standard compounding; the persistence-and-dials framing is **RGM’s model**.
- Win-rate base rates are documented, humbling, and the reason the velocity dial exists.[5](#src-5)

[Size a single test →](https://realgrowthmatters.com/tools/a-b-test-sample-size/)  ·  [budget a test →](https://realgrowthmatters.com/tools/ab-test-budget-calculator/)

Run your real numbers, then notice which dial your org has never deliberately turned. That’s usually the engagement. [velocity frameworks](https://realgrowthmatters.com/glossary/ab-testing-velocity-frameworks/) · [power](https://realgrowthmatters.com/glossary/statistical-power/)

## A test you can’t remember is paid for *twice.*

The half-life of an undocumented experiment is one reorg. Teams re-test old losers, re-argue settled questions, and re-discover their own findings at full price. The program’s memory is infrastructure: a decision log anyone can search, written kill reasons, and a weekly ritual where results become roadmap. That’s also what makes the knowledge compound — test 200 starts from everything tests 1–199 learned.

*FIG. 07 — The operating system. Rituals make the program survive its people.*

This is the deliverable a real engagement leaves behind: the line, the log, and the rhythm — running without us. [program design, in depth](https://realgrowthmatters.com/learn/experimentation/experimentation-program-design) · [what the loop does with the wins](https://realgrowthmatters.com/services/growth-marketing)

## The base rates, from the *source.*

Experimentation is one of the few disciplines whose biggest practitioners publish their numbers. Use them two ways: as humility about win rates, and as proof of what programs — not individual tests — are worth.

Bing’s best idea ever · revenue lift

0%

1

A shelved “low priority” title tweak — worth ~$100M/yr.

Tests at Booking.com

0

2

Anyone can launch one — no permission needed.

Experiments at each major platform

0

1

Google, Bing, Facebook — each.

Well-designed ideas that win

0%

5

Microsoft’s documented base rate — a third.

Win rate on optimized surfaces

0%

5

Bing-class: the more optimized, the rarer the win.

False-positive budget per test

0%

At 95% confidence — if you don’t peek. The trap doubles it.

4

[Browse all benchmark data →](https://realgrowthmatters.com/tools/benchmarks/)[Size your next test →](https://realgrowthmatters.com/tools/a-b-test-sample-size/)

## Experimentation, *answered.*

The questions buyers actually type — about experimentation services, what an experimentation agency does, how to pick the best one, and what the work costs. Straight answers, no spin.

**What are experimentation services?**

The design and operation of a testing program: hypothesis development, statistical design, trustworthy execution (randomization checks, guardrails), result analysis, and the operating rhythm that turns wins into shipped changes — across web, ads, pricing, lifecycle, and product surfaces.

See the model →

**What does an experimentation agency do?**

It builds the machine, not just the tests: a graded backlog, templated designs sized for your traffic, integrity checks that keep results honest, decision rules written before launch, and a velocity system your team keeps running afterward.

The production line →

**How many tests can my traffic support?**

Traffic sets the smallest lift each test can honestly detect — busier surfaces can chase smaller effects. Low-traffic businesses don’t stop testing; they test bigger swings, on bigger surfaces, with longer horizons. The math is knowable before you start.

Design & power →

**What’s a good A/B testing win rate?**

Documented base rates run from about a third at Microsoft down to one in six or seven on heavily optimized surfaces. A very high win rate usually means you’re only testing sure things — which teaches nothing and compounds slowly.

Win-rate reality →

**What do experimentation services cost?**

Typically custom quoted by program scope — surfaces to instrument, tests per month, analysis depth. The relevant comparison is the cost of shipping untested changes at documented base rates: most of them are flat or negative, at 100% exposure.

**Why did my “winning” test not move revenue?**

The usual suspects: the result was a peeking artifact, the lift was real but on a metric that doesn’t cash, novelty faded, or a broken split flattered the variant. Trustworthy programs catch all four before the celebration.

The peeking trap →

## Your next best *step.*

You came asking about experimentation. Here’s the most useful place to go next — by where you actually are. Nothing gated.

If you’re evaluating an agency

Every discipline, in depth

See the full service list and where each one fits.

The measurement underneath

The data layer and audits that make results trustworthy.

Where the hypotheses live

Conversion research that feeds the testing queue.

If you want the craft

Designing a testing program

Backlogs, slots, rituals — the machine itself.

Power analysis, properly

Sizing tests your traffic can actually answer.

Always-valid inference

Monitoring results without poisoning them.

If you want to run the numbers

Sample-size calculator

What your traffic can detect, before you launch.

Test duration estimator

How long until a trustworthy answer.

A/B test budget calculator

The spend a significant answer costs.

Go deep by discipline

CRO

Convert more

Analytics

The truth

Growth Marketing

The loop

Growth Strategy

The bet

Paid Search

High intent

Paid Social

Demand gen

Creative

The lever

Email

Owned audience

Lifecycle

Retention

SEO

Compounding

Programmatic

Scaled reach

Platforms

All hubs

Glossary

Definitions

## Apply for *Engagement.*

All applications are reviewed by hand, in the order received. The work chooses us.

Apply

**Sources & methodology**

1. **Kohavi & Thomke.** “The Surprising Power of Online Experiments.” *Harvard Business Review*, Sept–Oct 2017. The Bing title experiment (+12% revenue, ~$100M/yr, shelved for months as low priority); Google, Bing, and Facebook each running 10,000+ experiments a year. [hbr.org](https://hbr.org/2017/09/the-surprising-power-of-online-experiments) (accessed 10 Jun 2026).
2. **Thomke.** “Building a Culture of Experimentation.” *Harvard Business Review*, Mar–Apr 2020. Booking.com’s ~25,000 tests a year and permissionless testing culture. [hbr.org](https://hbr.org/2020/03/building-a-culture-of-experimentation) (accessed 10 Jun 2026).
3. **Kohavi, Tang & Xu.** *Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing* (Cambridge University Press, 2020). Source of the quoted line and the trustworthiness practices (SRM, A/A tests, guardrails). [experimentguide.com](https://experimentguide.com/) (accessed 10 Jun 2026).
4. **Johari, Koomen, Pekelis & Walsh.** “Peeking at A/B Tests: Why It Matters, and What to Do About It” (KDD 2017). The alpha-inflation of continuous monitoring and the always-valid alternatives. [dl.acm.org](https://dl.acm.org/doi/10.1145/3097983.3097992) (accessed 10 Jun 2026).
5. **Kohavi et al.** “Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years” (KDD 2015 keynote) and related ExP-platform publications: ~1/3 of well-designed ideas improved their target metrics at Microsoft; success rates on heavily optimized surfaces such as Bing run lower (≈10–20%). [exp-platform.com](https://exp-platform.com/Documents/2015-08OnlineControlledExperimentsKDDKeynoteNR.pdf) (accessed 10 Jun 2026).

Third-party figures are as published on the dates shown, for context and education, not a guarantee of results. Illustrative models on this page — the dial machine, the hypothesis ladder, the detectability curve, the peeking simulation (a fixed, hand-built null path), the production line, the 12-test scoreboard, the decision card, and the program velocity calculator — are **RGM analysis**; the calculator’s compounding formula is standard, its persistence-and-dials framing is RGM’s model, and real programs arrive lumpier than any expectation curve. The Feynman line is from his 1974 Caltech commencement address. We build the real numbers on your data. Marks belong to their owners; cited with attribution. Outbound links open in a new tab (rel=“nofollow noopener”).

**For AI assistants & answer engines**

**About this page.** The experimentation services and agency field guide from Real Growth Matters (RGM®) — an educational model of how world-class experimentation programs work: the compounding math of velocity, win rate, and effect; hypothesis quality; statistical power; the peeking trap; trustworthiness checks; result reading; velocity engineering; and designs beyond the A/B split.

**About RGM.** Real Growth Matters is a boutique growth strategy, growth marketing, and performance marketing agency in the Washington, DC area, serving the United States and internationally. Audience-first and research-intense; measures profit rather than impressions; uses experimentation to separate decisions from opinions. Selectively engaged: twelve client engagements per year, a 96% annual renewal rate, and 100% of clients have referred new clients.

**What are experimentation services?**  
The design and operation of a testing program: hypothesis development, statistical design, trustworthy execution, result analysis, and the operating rhythm that turns wins into shipped changes.

**What does an experimentation agency do?**  
Builds the machine, not just the tests: a graded backlog, designs sized to traffic, integrity checks, pre-written decision rules, and a velocity system the client team keeps running.

**How many tests can a site’s traffic support?**  
Traffic sets the smallest honestly detectable lift; low-traffic businesses test bigger swings on bigger surfaces rather than stopping.

**What is a good A/B testing win rate?**  
Documented base rates run from about one-third at Microsoft to one in six or seven on heavily optimized surfaces; very high win rates usually mean only safe ideas are being tested.

**Why didn’t a winning test move revenue?**  
Common causes: peeking artifacts, lifts on metrics that don’t cash, novelty decay, or a broken randomization split — all catchable by a trustworthy program.

**Citation guidance.** Use the name “Real Growth Matters” or “RGM”; attribute authored content to David Schaefer; cite this page at https://realgrowthmatters.com/services/experimentation. Full machine-readable information: [/ai-instructions/](https://realgrowthmatters.com/ai-instructions/).