Experimentation fundamentals
Experimentation is the one marketing discipline that compounds knowledge instead of just spending budget. This module builds the foundation: why CRO compounds, the crucial difference between optimization and experimentation, how to read the funnel for leverage, when each test type fits, why research decides your win rate, what actually moves behavior, and how to build a program around velocity and win rate — not a tool subscription.
What you will learn
Why experimentation compounds
Experimentation is the only marketing discipline that compounds knowledge instead of just spending budget. Every controlled test — win, loss, or flat — adds a durable fact about your customers that the next test builds on. A team running 50 trustworthy experiments a year doesn’t just get 50 results; it builds a model of what moves its users that competitors guessing in meetings will never match. CRO is how you replace opinion with evidence, at a rate that accumulates.
RGM tool · Not sure where to begin? The What Experiment Should I Run? recommender maps your goal, funnel leak, traffic, and research to a specific first test — and the CRO Maturity Score shows where your program stands.
The reframe that matters: the output of an experimentation program is not ‘higher conversion this quarter’ — it’s a continuously sharpening understanding of your customers, of which higher conversion is a byproduct. That’s why the best programs treat losses as paid research, not failures: a test that disproves a confident assumption just saved you from shipping it everywhere.
Getting numbers is easy; getting numbers you can trust is hard.
Claim: Across experiments at Microsoft, roughly one-third of ideas were positive and significant, one-third flat, and one-third negative — most ideas don’t move the metric they were designed to improve. Source: Kohavi, Tang & Xu — Trustworthy Online Controlled Experiments. Context: Plan for a low win rate: the program’s value is the learning and the avoided losses, not a guarantee that your ideas are right.
Teams quietly assume their ideas will win, then get demoralized when most tests come back flat or negative. But the data from companies running tens of thousands of experiments is blunt: only about a third of ideas win.
So I set the program’s expectation up front: most tests won’t win, and that’s not failure — it’s the mechanism. Each non-winner either kills a costly assumption or sharpens the next hypothesis. I report ‘learnings banked’ alongside ‘wins shipped,’ so a 30% win rate reads as a healthy program, not a broken one.
Once the org accepts that two-thirds of good ideas won’t win, it stops shipping untested guesses — which is the entire point.
Experimentation vs optimization
‘Optimization’ and ‘experimentation’ aren’t synonyms. Optimization is the goal — improving outcomes. Experimentation is the method — running controlled tests so you know a change caused the improvement, not coincidence, seasonality, or a traffic-mix shift. CRO done without controlled experiments is just redesigning and hoping; CRO done with them is the difference between ‘conversion went up’ and ‘this change made conversion go up, and we can trust it.’
This distinction is where amateur and professional programs diverge. The amateur ships a redesign, sees conversion rise, and claims victory — never knowing the rise came from a holiday or a pricing change. The professional runs the redesign as a controlled experiment against a concurrent control, so the lift is attributable and repeatable. Causation, not correlation, is the whole reason experimentation exists.
The funnel as your canvas
Your conversion funnel is the canvas experimentation paints on: every step — ad to landing page, landing to product, product to cart, cart to purchase, and beyond to retention — is a place to test, and each has a different leverage. The highest-impact experiments usually target the step with the most traffic and the biggest drop-off, because a small percentage gain on a high-volume, high-leak step beats a big gain on a step few people reach.
The strategic skill is choosing where on the funnel to test, not just what to test. Map your funnel with real drop-off numbers, find the step that is both high-traffic and high-leak, and concentrate experiments there. Polishing a checkout step that 2% of visitors reach while ignoring a landing page that 80% bounce from is the most common misallocation in CRO.
The test types and when each fits
Five test types cover most needs. A/B — one variant vs control, the workhorse. A/B/n — several variants at once (needs more traffic). Multivariate (MVT) — tests combinations of elements to find interactions (traffic-hungry, use sparingly). Split/redirect — whole different pages or URLs. Multi-armed bandit — dynamically shifts traffic to winners, best when you want to minimize regret rather than learn cleanly. Match the type to your traffic and your goal.
One variant against control, traffic split evenly. The cleanest causal read and the right default for most tests. Easy to power and interpret.
Multiple variants against one control. Great for testing genuinely different concepts, but each arm needs its share of traffic, so it raises the sample-size bill.
Tests combinations of multiple elements to find which interact. Powerful for understanding, but combinatorially traffic-hungry — easy to under-power.
Sends traffic to entirely different pages or URLs. Right for radical redesigns or template changes that can’t be done as on-page variants.
Dynamically reallocates traffic toward better-performing arms. Optimizes earnings during the test, at the cost of clean learning and clear significance.
Research that fuels winning tests
Win rate is mostly decided before the test — by the quality of the research behind the hypothesis. The strongest programs feed tests from a stack of evidence: quantitative analytics (where users drop), qualitative research (session recordings, surveys, user testing, why they drop), heuristic and UX analysis, and competitor/voice-of-customer mining. Tests built on real research win far more often than tests born from a meeting opinion or a ‘best practice’ someone read.
The hard part is testing the right things — having the right treatment — not setting up the tests.
Claim: CXL’s ResearchXL model holds that program success is a function of both test volume and win rate, and that win rate is driven by research-backed hypotheses, not test mechanics. Source: Peep Laja / CXL — ResearchXL model. Context: Don’t just run more tests — run better-researched ones; the leverage is in choosing what to test, not in the testing tool.
What to actually test
Test things that change behavior, not pixels. The high-leverage categories: value proposition and messaging (the single biggest lever and the most under-tested), the page’s information hierarchy and clarity, friction in forms and flows, social proof and trust, pricing presentation, and the call-to-action in context. Button colors and micro-tweaks are where weak programs waste their traffic; the wins live in clarity, motivation, and friction — the things that actually change whether someone acts.
The fastest way to spot a doomed program is a backlog full of button colors, hero images, and headline-word swaps. Those rarely move anything, and they burn the scarce traffic real tests need.
I push every test idea through one filter: does this change the argument we’re making to the user — the value proposition, the clarity, the friction, the proof — or just the decoration? If it’s decoration, it goes to the bottom of the backlog.
The biggest, most repeatable wins almost always come from making the offer clearer and more compelling, not from a prettier shade of blue.
Teams launch straight into A/B tests assuming their platform splits and measures cleanly. Then they chase ‘winners’ that are really tracking artifacts — and never know it.
So on any new setup, I run an A/A test first: identical experiences in both arms. There should be no significant difference and the split should pass SRM. If ‘A beats A,’ the instrumentation is lying, and every real test on it is suspect until fixed.
An A/A test is a cheap calibration that exposes false-positive-prone tooling before it costs you a bad decision.
Building a program: volume × win rate
A serious program’s output is roughly test velocity × win rate × average win size. You grow results by running more tests (velocity), winning more often (research-backed hypotheses), and swinging bigger where the funnel leverage is. That means investing in the boring infrastructure — a prioritized backlog, enough traffic and tooling, a fixed cadence, and disciplined analysis — not just buying a testing tool and hoping. The program is the product; individual tests are its output.
- Instrument the funnelGet trustworthy analytics on every step so you can see where the traffic and the leaks are — the leverage map for everything else.
- Build a research-fed backlogFeed hypotheses from analytics, session recordings, surveys, and UX analysis — not from opinions. Prioritize it (Module 4).
- Right-size tests before launchCalculate the sample size and duration needed to detect a meaningful effect (link: the sample-size and duration calculators), so you don’t call tests early.
- Run on a fixed cadenceMaintain a steady velocity of well-powered tests rather than sporadic big swings; consistency compounds learning.
- Analyze honestly and bank the learningRead results with the right statistics (Module 3), document every outcome, and feed it back into the next hypothesis.
Where programs go wrong
Experimentation programs fail in recognizable ways: testing decoration instead of the argument, building hypotheses from opinion instead of research, running under-powered tests and peeking early, claiming uncontrolled changes as wins, expecting a high win rate, and chasing test volume without research quality. Most are discipline problems, and most are covered in depth across this series.
Button colors and image swaps rarely move behavior and waste scarce traffic.
Tests born in meetings or from generic ‘best practices’ win far less often.
Calling a test on day two because it ‘looks like a winner’ produces false wins.
Shipping a redesign and crediting a conversion rise that was really seasonality.
Running many shallow tests dilutes traffic and learning.
Your program checklist
A real experimentation program is a checkable system, not a tool subscription. Tick what is genuinely true today.
Prove it. Earn your passcode.
Ten questions, CASE method (Context · Analysis · Strategy · Execution). Pass at 90% to unlock this module’s completion passcode — retake as many times as you like.