How to write an A/B test hypothesis worth testing

“Let's try a shorter form” is an idea. An A/B test hypothesis explains who has a problem, why that change might help and what result would challenge the explanation.

That extra work matters because traffic is limited and experiment results are easy to overinterpret. A variation can improve a metric without confirming the story behind it. A negative result can tell you little if the original hypothesis was too vague to examine.

If you work in CRO, product or digital marketing, use the hypothesis as a bridge between customer evidence and a release decision. It should guide design, measurement and interpretation. It should also let a colleague challenge the logic before you spend time building the variation.

The goal is a useful test, not a statement that sounds scientific. Start with an observation and follow it all the way to the decision you need.

Begin with an observation you can describe

Record what you saw and where it came from. Examples include repeated support questions, an error pattern in a form or a difficulty observed during a usability session. Keep the description separate from your explanation of it.

“Several participants could not find the delivery conditions” describes an observation within a particular research exercise. “Customers do not trust our delivery promise” is an interpretation. The interpretation might be worth investigating, but it requires more evidence.

Note the limitations. Support contacts represent people who chose to ask for help. Usability participants may not represent all visitors. Funnel data can locate a drop-off but cannot, by itself, reveal the reason. These limitations should shape the confidence you place in the hypothesis.

Look for related evidence that can change your view. If people find delivery details easily but reject the available dates, making the details more visible addresses the wrong problem. Good research can remove an idea from the backlog before it becomes a test.

Identify the audience and decision point

Specify who faces the problem. New visitors may need different information from returning customers. A compatibility concern may apply only to a product group. A form error may occur only on mobile or in a particular language.

Define the point where the proposed change can act. Showing useful information after the decision has already been made may miss the problem. Showing too much information earlier can distract people who do not need it.

Choose eligibility using stable criteria where possible. A population defined by behaviour the variation itself can influence may complicate the comparison. Be clear about whether you are analysing assigned users, exposed users or a pre-defined eligible group.

Also consider who could be harmed. A shorter form may help some users while removing information needed for service delivery. The hypothesis should include those consequences instead of treating every increase in submissions as success.

The CRO programme guide explains how these choices fit into the wider customer journey.

State a plausible mechanism without pretending certainty

The mechanism explains how the change might influence behaviour. Reducing a recurring error could help people finish a task. Providing relevant evidence could reduce uncertainty. Clarifying a cost could prevent an unpleasant surprise later in the journey.

Use behavioural explanations with care. You cannot diagnose a cognitive bias from a click alone. A familiar psychological term can give an idea a persuasive name without making the causal story more credible.

Write the mechanism in ordinary language first. If it is difficult to explain without jargon, the problem may not be well understood. You can introduce a research concept when it adds meaning, but the concept should not replace the observation.

The article on psychological insights and test hypotheses covers this boundary. In a commercial experiment, the expected mechanism remains a hypothesis unless the design and measures genuinely establish it.

Specify a change you can interpret

Describe what the variation changes and what it leaves comparable. Include the content, placement and behaviour that matter to the proposed mechanism. “Improve the product page” does not define an intervention.

Prefer a change whose result can inform a clear decision. If you need to know whether people understand a particular condition, a broad redesign may be too blunt. If the business decision concerns an entire redesign, testing the combined experience can still be appropriate.

Microsoft's pre-experiment guidance connects clear hypotheses with measurable success criteria. That connection helps avoid building a variation first and searching for a favourable metric afterwards.

Single-variable testing is useful when it supports interpretation, but it is not a universal requirement to alter one visual property. A coherent change may involve several elements. Be honest that the test estimates the effect of the package, not every component independently.

Choose an outcome that matches the question

Identify one primary outcome tied to the decision. If the hypothesis concerns helping eligible visitors complete a purchase, an increase in panel clicks alone is insufficient. The clicks may help diagnose exposure, but the main decision concerns a later outcome.

Define numerator, denominator and attribution window. A visitor-level purchase outcome differs from purchases per session or average order value. You need the definition before calculating sample requirements or discussing an expected effect.

Choose guardrails for important trade-offs. A form simplification may raise submissions and reduce lead quality. A promotion may raise orders and reduce margin. A new interaction may increase engagement and harm page performance.

A metric can also fail because it is difficult to measure consistently. Check the instrumentation before promising what the experiment will establish. If the meaningful outcome is unavailable, either improve measurement or narrow the claim the test is designed to support.

Make the hypothesis falsifiable

Use a plain structure: for this audience, we observed this difficulty; we expect this change to affect this behaviour, because of this proposed mechanism; we will evaluate this outcome while protecting these constraints.

This is a writing aid, not a universal sentence to paste into every brief. A hypothesis may need several sentences to define the important conditions. Clarity matters more than fitting a slogan.

Write what would count against the idea. A lack of clear evidence is not always proof that the mechanism is wrong: the test may be too imprecise or the change may be poorly delivered. Still, specifying those possibilities before launch prevents convenient explanations after the result.

Record rival explanations. If a result could follow from better visibility, a new discount or a different layout, say so. You can then decide whether the experiment answers the business question despite that ambiguity or whether a more focused design is needed.

Hypothetical example: a B2B enquiry form

Imagine a business where usability research suggests visitors struggle to identify what information an enquiry form needs. This is an illustrative scenario, not a client experiment. The team proposes clearer field instructions and examples.

The hypothesis is that eligible visitors will complete the enquiry with fewer avoidable errors because the requested information is easier to understand. The primary outcome is a valid completed enquiry, defined in advance. Guardrails include sales acceptance and the ability to deliver the requested service.

The team does not treat every submission as equally valuable. It also does not promise a lift because “clarity always converts”. It checks whether the new instructions are visible, understandable and compatible with assistive technology.

If errors fall but completed enquiries do not clearly change, the finding still informs the next decision. The team may retain a usability improvement while acknowledging that the experiment did not establish a commercial lift. It should report those conclusions separately.

Use a worksheet before prioritising

Complete this with evidence rather than marketing language. Ask a colleague to find the weakest link.

The last row often reveals whether the hypothesis is useful. If any result leads to the same recommendation, the test may be serving as approval theatre rather than decision support.

Store the completed brief with the implementation and analysis record. Later teams should be able to see the original question and whether the experiment answered it. That continuity prevents a positive secondary metric becoming the remembered purpose of the test.

Once the hypothesis is complete, compare it with other candidates through the experiment backlog guide. A well-written idea still needs a useful decision, feasible measurement and a credible place among the team's priorities.

Check traffic and statistical feasibility

Estimate the information needed to detect an effect worth acting on. Consider outcome frequency and the eligible audience, not only overall site traffic. A large site can still have little usable traffic for a narrow hypothesis.

Use the rules for the statistical method configured in your experiment. Optimizely's methods overview distinguishes fixed-horizon, Bayesian and sequential approaches. A stopping rule borrowed from another method can undermine the intended analysis.

Avoid setting the expected effect purely to make the sample calculation convenient. If only a very large change would be detectable within the available period, ask whether the intervention plausibly could produce it. If not, choose a different research route or decision.

Read significance and peeking for the interpretation. Statistical feasibility should shape the test before launch rather than explain an inconvenient result afterwards.

Interpret a negative result constructively

Separate several possibilities: the estimate may be precise enough to rule out a useful effect; it may remain too uncertain; the intervention may not have reached the intended audience; or the data may be unreliable.

These are different conclusions. A well-run test can weaken the commercial case for a change without proving the proposed psychological mechanism false. An implementation failure may say little about the mechanism at all.

Return to the brief. Which link did the evidence challenge? Update that link and explain the next step. Do not automatically add a more aggressive variant to make the original idea work.

The article on e-commerce conversion improvement shows how to separate a reported outcome from unverified explanations. The same discipline applies to your test archive: keep observations, interpretations and decisions distinct.

Start with one hypothesis the team can challenge

Select a real customer difficulty and write the full chain from observation to decision. Check that the change is deliverable, the outcome measurable and the guardrails meaningful. Then ask what you would do if the result disappointed you.

A good hypothesis does not guarantee a winner. It makes the experiment valuable whichever result appears, because the team knows what question it was trying to answer.

Discuss your next A/B test hypothesis if the idea is clear but the evidence, mechanism or measurement is still uncertain. Bring the observation first; it is the strongest place to begin.

Apply this to your project

Tell me the decision you are facing, the constraints and what you have already tried.

Complete all three fields to send your message.

Name

Email

Message

Send message

Keep exploring

Experimentation & CRO

A/B test significance and peeking: read results properly

Understand A/B test significance, uncertainty and peeking. Learn how fixed-horizon and sequential methods differ before calling an experiment a winner.

May 25, 2026·9 min read

Experimentation & CRO

CRO and A/B testing: build a useful experiment programme

Build a CRO programme from customer evidence to a trustworthy A/B test. Learn how to choose hypotheses, measure outcomes and make release decisions.

May 15, 2026·9 min read

Experimentation & CRO

A/B test guardrail metrics: protect the whole journey

Choose A/B test guardrail metrics that protect quality, performance and downstream outcomes. Define thresholds and decisions before launching a test.

October 3, 2026·8 min read