Experimentation & CRO
Low-traffic A/B testing: choose a question you can answer
Plan low-traffic A/B testing around useful effect sizes, eligible audiences and uncertainty. Learn when to test, research, simplify or defer.
Low-traffic A/B testing starts with a constraint: your site may not generate enough eligible observations to distinguish a useful change from random variation within a reasonable period. A new statistics label cannot remove that constraint. The decision is whether to change the question, improve the design, use another research method or defer the comparison.
You can still make progress. The aim is to spend scarce traffic on questions the experiment can answer, while using qualitative evidence and quality checks for other decisions. This guide connects sample planning with practical alternatives. The wider approach is in the CRO and A/B testing guide.
Count the audience that can actually see the change
Start at the exposure point. Total site visitors are not the sample for a checkout-step test if only a small part reaches that step. Define eligibility before treatment and identify the unit you will randomise.
If you assign people, repeated visits are part of their experience rather than new independent people. If you assign accounts, the account count matters even if each account generates many sessions. Choose an analysis appropriate to the assignment; more events do not automatically supply more independent evidence.
Check the quality of identifiers and tracking. Cookie loss, missing consent where required, logged-in transitions and device changes can complicate exposure measurement. Describe the limitations rather than claiming that an analytics dashboard contains a perfect census of people.
Use a recent baseline that reflects the intended journey and relevant operating period. A site-wide conversion rate may be inappropriate for a specific audience. Keep sample availability separate from outcome frequency: both affect whether the comparison can answer the question.
Decide what effect would change your decision
A minimum detectable effect describes the sensitivity planned for a test. A minimum worthwhile effect describes what matters operationally. They should be discussed together, but they are not identical. A calculator can tell you what is detectable under assumptions; it cannot decide what makes the change worth maintaining.
For a hypothetical baseline of 2%, a change to 2.4% is an increase of 0.4 percentage points and 20% relative. These figures are illustrative arithmetic, not a forecast. State both scales so stakeholders do not mistake a relative change for percentage points.
Ask what improvement would justify delivery, maintenance and any trade-off. If only a tiny change would matter, your low-traffic setting may not support the required evidence. If a substantially larger improvement is needed, testing a stronger intervention may be a more coherent question than polishing a minor detail.
Microsoft's research on metric sensitivity connects detectable effects with power, sample size and metric variance. Use those inputs to explore scenarios rather than selecting whichever calculator output gives the shortest duration.
Turn sample requirements into a credible calendar
Plan the test using the chosen outcome, baseline, effect size, significance approach, power and allocation. Use a method suitable for the metric and design. A formula intended for one proportion or independent observations should not be copied into every experiment.
Then translate the requirement into eligible units per period. Use a range for traffic rather than one optimistic estimate. Include the follow-up required for delayed outcomes and a period that covers relevant behavioural cycles. A numerical sample target alone does not establish generalisability across different operating conditions.
If the estimated duration is longer than the journey can remain stable, say so. Offers, campaigns, products and implementation may change before the sample matures. Running for months through unrecorded changes can make the resulting comparison difficult to interpret even if the final count is large.
Do not solve the problem by lowering standards after seeing favourable results. Any changes to the planned inference method, effect or decision rule need a defensible reason and appropriate treatment of the data already observed. The guide to significance and peeking explains the stopping problem.
Keep a dated log of changes that could affect the journey: campaigns, availability, pricing, audience targeting and service disruptions. A simultaneous change affecting both variants does not automatically invalidate randomisation, but it can change what the result represents. Record whether the intervention's effect appears relevant only under a particular operating condition, without turning exploratory subgroup findings into a new confirmatory result.
Also check whether the traffic estimate includes people who will be excluded by the eligibility rule. A campaign that doubles visits may contribute almost nothing to a test aimed at returning customers. Build the calendar from the actual assignment rule and maintain that distinction when reporting feasibility.
Use a feasibility sheet before building variants
The worksheet below keeps the business question and the statistical constraint visible together. It is an editorial planning aid, not a fixed sample-size rule.
Swipe or use the arrow keys to see every column.
| Question | What to record | What a weak answer suggests |
|---|---|---|
| Who is eligible? | Audience and pre-treatment rule | Clarify exposure before testing |
| What is the assignment unit? | Person, account or another defined unit | Revisit independence and analysis |
| What is the baseline? | Metric definition, period and uncertainty | Improve measurement |
| What effect matters? | Absolute and relative scales | Clarify the release decision |
| What can be detected? | Planned sensitivity under assumptions | Change design or acknowledge limits |
| How much traffic is available? | Eligible units per period, as a range | Recalculate duration |
| What could deteriorate? | Guardrails and required precision | Check complete decision feasibility |
| Will the journey remain comparable? | Expected changes and follow-up | Defer or narrow scope |
| What is the alternative? | Research, repair, simpler rollout or no change | Choose the method that answers the question |
Include the guardrail metrics in feasibility planning. A test may measure its main outcome adequately while remaining too imprecise to protect a relevant consequence.
Hypothetical example: a small B2B enquiry journey
Imagine a hypothetical specialist service website with a limited number of qualified enquiries. This example illustrates decisions, without claiming a measured client outcome. The team wants to test a different button label against completed enquiry submissions.
The available eligible audience is small, and only a minority completes the form. The change is modest, while the decision would require confidence about a small improvement. The planned sample would take longer than the team expects the offer page to remain stable.
Instead of launching the test immediately, the team investigates the task. It observes target users trying to determine fit and prepare an enquiry. People struggle with missing information about the engagement process. This observation creates a more substantial hypothesis about clearer qualification and expectations.
The team prototypes that explanation, fixes visible usability defects and improves instrumentation. It may later compare a coherent revised page when there is enough traffic. It does not report a quantified conversion lift from a handful of research sessions.
The GOV.UK guidance on moderated usability testing explains how to observe people completing tasks. That evidence serves a different purpose from estimating a population treatment effect.
Improve the design without changing the question in secret
Reduce unnecessary variants. Every additional treatment shares available traffic and creates more comparisons. A focused two-arm test can be more useful than a broad multivariate design when resources are limited.
Improve metric sensitivity where the method is justified. Pre-experiment information can sometimes reduce variance, but it requires suitable data and an estimator that preserves the causal comparison. Do not assume that a variance-reduction feature creates a fixed multiplier for your particular site.
Use eligible audience targeting only when it can be defined consistently in treatment and control. Selecting people because they clicked a treatment-only element creates a post-treatment selection problem. You cannot treat that subgroup as if it were randomised at the start.
A larger coherent change may yield a more detectable effect, but it also answers a broader question. Document the intervention as a package and avoid claiming which component caused the result. Break the package into later tests only if that distinction matters and can be measured.
Use qualitative evidence for the decisions it can support
Research can identify confusing language, missing information, task barriers and implausible assumptions. It can improve a proposed intervention and help decide what deserves traffic. It cannot, by itself, quantify a conversion uplift across the whole audience.
The qualitative research guide for CRO helps separate observations from explanations. Include people relevant to the task and account for accessibility needs. A convenient sample of colleagues may reveal defects without representing customers.
For behavioural ideas, use the psychology-to-hypothesis guide. A plausible mechanism gives you a better question; it does not establish an effect size. Keep the evidence trail from observation to proposed change.
When the issue is a clear defect or missing required information, correct it through the relevant quality process. A reversible rollout with monitoring may be appropriate for some design choices, but describe it as a rollout. Before-and-after movement does not isolate causation.
Read an inconclusive result without forcing a winner
A non-significant result can still be informative if its interval rules out effects large enough to matter. It can also be uninformative if the uncertainty includes worthwhile benefit and unacceptable harm. Read the estimate and interval in relation to the decision, rather than treating a label as the answer.
Bayesian methods likewise require assumptions, data and an agreed decision rule. They can express uncertainty differently; they do not make a weak sample automatically decisive. Report the prior and sensitivity where relevant, rather than suggesting a method guarantees faster truth.
Keep inconclusive outcomes in the experiment backlog. Record whether the question should be reframed, repeated under better conditions or closed because its decision value is too low.
Start with one proposed test and complete the feasibility sheet. If the available traffic cannot support the decision, choose an alternative that can. To design an experimentation approach around your audience and constraints, tell me about the journey.
Apply this to your project
Tell me the decision you are facing, the constraints and what you have already tried.
Keep exploring
A/B test guardrail metrics: protect the whole journey
Choose A/B test guardrail metrics that protect quality, performance and downstream outcomes. Define thresholds and decisions before launching a test.
Experiment backlog prioritisation: choose better tests
Prioritise your experiment backlog using evidence, traffic and decision value. Build a practical queue that turns observations into useful A/B tests.
Personalisation tests: measure the value of relevance
Design personalisation tests that compare a clear targeting rule with a useful default. Measure incremental value, guardrails and uncertainty.