A/B experiment deliverables

Running an A/B test is easy. Running one whose result you should believe is mostly a documentation problem, and the documents have to exist in a particular order.

The reason is uncomfortable but simple. Every experiment involves dozens of small analytical choices — which metric counts, which users are included, when to stop, which segments to examine, what size of effect is worth shipping. Made before the data arrives, those choices are neutral. Made after, every one of them is quietly informed by which option produces the answer somebody wants. Nobody involved experiences this as dishonesty, which is exactly why it needs a procedural fix rather than a cultural one.

So the deliverables split cleanly. Six are written while you are still ignorant of the outcome, and they are the ones that give the other six their authority. The rest describe what happened and what you did about it.

A useful test of any experimentation programme: pick a shipped result at random and ask whether the primary metric and the ship threshold were written down before launch. If nobody can produce that document, the experiment measured something, but it did not decide anything.

Deliverable nine is where most of the damage happens, because a readout that reports only significance invites two opposite errors: shipping a real effect too small to matter, and abandoning a promising change because an underpowered test failed to detect it.

The fix is to lead with the interval rather than the verdict. Four experiments, all of which a significance-only readout would compress into two outcomes:

Four things a readout must not do

• Report significance without effect sizeA p-value says an effect probably exists. It says nothing about magnitude, and magnitude is what the ship decision was supposed to turn on.
• Present unplanned segments as findingsSlice enough ways and something crosses the threshold. Segments declared in advance are results; segments discovered afterwards are hypotheses for the next test, and should be labelled as such in the document itself.
• Omit the guardrailsReporting them only when they moved means the reader cannot tell the difference between checked-and-fine and not-checked.
• Treat a null as proof of no effectWithout the achieved power and the minimum detectable effect alongside it, a null result is uninterpretable — and will be interpreted anyway.

Twelve documents per experiment is not a realistic ask for a team shipping several tests a month, and a process that heavy is abandoned rather than followed. The way it works in practice is that most of these are fields in a template, not standalone documents.

• Collapse the pre-launch six into one form. Hypothesis, primary metric, guardrails, MDE, sample, runtime, stopping rule, ship criteria. It fits on one page and is the only artifact that genuinely must be complete before launch.
• Make the platform enforce what it can. Sample ratio checks, minimum runtime, and blocking the readout until the planned sample is reached are far more reliable than asking people to remember.
• Keep the decision record separate from the readout. They answer different questions — what happened, and what we did — and merging them is how the reasoning gets lost.
• Write nulls up to the same standard. The temptation to document only wins is strong and it corrupts the archive within a year.
• Give the archive an index someone would actually search. By surface and hypothesis. Nobody looks up an experiment by its ticket number.

The honest summary is that experimentation infrastructure is cheap now and experimentation discipline is not. Almost every organisation that concludes A/B testing does not work for them has a platform that runs tests correctly and no document, anywhere, written before the data arrived.