Experimentation & CRO
A/B Test Hypotheses That Get Results
An A/B test is worth only as much as the hypothesis that generates it. Most of the CRO backlogs I have inherited were lists of nice ideas with no behavioural basis behind them: a variant got tested because a competitor used it, not because a real friction in the user's journey had been understood. Traffic spent on a test like that is almost always wasted, because even a positive result teaches you nothing about what to replicate elsewhere. In this article I share the format I use to write a hypothesis that gets results, the one I used to earn real lifts on the Whirlpool EMEA e-commerce.
Where a hypothesis worth testing comes from
A solid hypothesis does not start from a solution ("let's try a green button") but from an observation about real behaviour. My approach predates my work in marketing, coming from an MSc in Cyberpsychology, where I learned to ask why people behave the way they do online long before trying to solve it with a test. In practice it means looking at session data, heatmaps and drop-off points in a funnel to understand where the user is lost, and only then formulating what I believe addresses it.
The difference is clear. "Users abandon checkout when shipping costs appear late: showing them earlier reduces abandonment at that step" is a hypothesis. "Let's change something in checkout and see" is not.
The format I use: observation, change, expected metric
Every hypothesis I write holds three elements together. What I observe in real behaviour, which is the basis and not an intuition. What change I believe addresses that observation. And which metric I expect to move, and by how much. The third element is the one that goes missing most often, and the one that turns a test into a learning: without a prediction, any result is acceptable in hindsight, and a test that could not be disproven measured nothing.
Why the psychological basis changes the quality of the hypothesis
A hypothesis anchored to a specific psychological mechanism is stronger than one born of imitation, because it explains why the change should work, not just what to change. If I know that loss aversion weighs almost twice as much as an equivalent gain, I can form a hypothesis about how I communicate a free-shipping threshold, instead of trying phrases at random. The move from a behavioural insight to a testable hypothesis is the heart of the craft, and I have devoted a separate article to how you build that bridge between psychology and CRO.
Writing the hypothesis with the metric and threshold already decided
The last step, before launching, is to fix the primary metric and the threshold that will decide whether the test won, before seeing any data. Changing the metric after seeing partial results is the most common and least visible way to fool yourself with an A/B test. The primary metric must be a single one, flanked by a few guardrail metrics that check you are not worsening something else while optimising the main number.
Deciding the threshold in advance ties directly to how you read the result: looking at the data too early and stopping the moment the variant seems to win produces false winners with surprising frequency. I go deeper into this in the article on statistical significance and peeking. The full method, from hypothesis to experimentation program, is in the guide to CRO and A/B testing.
Let's talk about your project
Found this useful? Let's see how it applies to your context.
Related articles
Did the Test Really Win? Significance and Peeking
How to tell whether an A/B test really won: statistical significance, sample size, and the peeking that declares nonexistent winners.
CRO & Experimentation: The Complete A/B Test Guide
A complete guide to CRO and experimentation: from hypothesis to A/B test, prioritisation and significance. The method behind Whirlpool EMEA's +57%.
+57% Conversion: Anatomy of a CRO Program
The anatomy of the experimentation program behind the +57% conversion lift on Whirlpool EMEA: how it started, what we tested and what worked.