Experimentation & CRO
Experiment backlog prioritisation: choose better tests
Prioritise your experiment backlog using evidence, traffic and decision value. Build a practical queue that turns observations into useful A/B tests.
Your experiment backlog is growing faster than your ability to run tests. One idea comes from customer research, another from a competitor and another from the most senior person in the meeting. Giving each idea an impact score does not make those sources equally credible.
Experiment backlog prioritisation should help you choose which uncertainty deserves the next portion of traffic and team time. The objective is a decision you can trust, rather than a calendar full of variants. This guide builds a queue around evidence, testability and learning. For the full programme, use the CRO and A/B testing guide.
Make the backlog a record of problems and predictions
An entry should begin with an observation. “Change the hero” is a proposed solution. “People cannot tell whether the service fits their situation after reading the hero” is a problem that you can investigate. Keep the observation, interpretation and proposed intervention separate.
State where the evidence came from and how broad it is. An interview can reveal a plausible difficulty without establishing how common it is. An analytics pattern can show an exit point without explaining the reason. A sales request can reveal a frequent objection, but its frequency still needs a defined source and period.
The GOV.UK guidance on planning research recommends turning assumptions into research questions. Apply that distinction to the queue. Some entries need another round of research before an A/B test can answer a useful question.
Write a falsifiable prediction for entries ready to test. Explain who should behave differently, what change should cause it and which outcome would reveal the difference. The guide to A/B test hypotheses develops that record in more detail.
Separate fixes, investigations and experiments
A broken form does not need an A/B test to establish whether it should remain broken. Correct defects through the appropriate quality process. Check the repair after release, but do not let obvious fixes compete against genuine experiment ideas for scarce traffic.
An investigation addresses a missing explanation or an uncertain design. Interviews, usability work, data checks or a prototype may be the next activity. The resulting evidence can strengthen, change or remove a future experiment entry. The backlog should preserve that connection.
An experiment compares outcomes under a controlled intervention. It needs a coherent hypothesis, instrumentation, sufficient eligible traffic and a decision that could change after the result. If no result would alter the plan, call the work a rollout or validation activity instead of dressing it as an experiment.
Keep these lanes visible in the planning view. They can share delivery resources but should not share a single success metric. A repaired defect, a resolved uncertainty and a reliable treatment estimate create different kinds of value.
Check testability before scoring impact
Confirm the audience and exposure point. A site may receive substantial traffic while the relevant step receives little. Count eligible randomisation units, not total page views. Repeated visits from one person do not automatically create independent observations.
Then establish metric reliability. Can you measure the intended outcome in both variants? Does the change alter event collection or the denominator? An idea is not ready if its apparent effect may be an instrumentation difference. Repair that uncertainty before assigning a slot.
Microsoft's pre-experiment guidance links hypotheses, metrics and statistical power. Use those elements as readiness checks. A large expected impact written by an enthusiastic stakeholder does not replace a defensible test design.
Identify technical and operational constraints. Include design, development, QA, analysis and follow-up. An easy interface change may create a difficult analysis if it affects account-level collaboration or several linked steps. Estimate the complete experiment effort rather than the time required to draw a variant.
Compare decision value, evidence and effort
A prioritisation sheet should show assumptions instead of burying them inside one number. The fields below are an editorial worksheet. They organise discussion without claiming that a score predicts an experiment's result.
Swipe or use the arrow keys to see every column.
| Field | What to record | Why it changes priority |
|---|---|---|
| Problem | Observable friction in a defined journey | Keeps the test tied to real work |
| Evidence | Source, date, coverage and uncertainty | Distinguishes observation from opinion |
| Prediction | Intervention, mechanism and expected behaviour | Makes the hypothesis contestable |
| Eligible audience | Randomisation unit and exposure | Shows whether a test can answer the question |
| Primary outcome | Metric definition and practical effect | Connects the result to a decision |
| Guardrails | Outcomes that should not deteriorate | Prevents a narrow local win |
| Decision value | What changes if the result is positive or negative | Makes learning actionable |
| Complete effort | Delivery, QA, analysis and maintenance | Reflects the actual capacity cost |
| Dependency | Required data, design or policy decision | Exposes blocked entries |
| Next state | Investigate, prepare, run, defer or close | Gives the entry a concrete action |
If you use an impact–confidence–effort score, keep the evidence and assumptions beside it. Confidence should mean confidence in the rationale, not confidence that the variant will win. A team that assigns high confidence only to expected winners discourages useful questions.
Check the sensitivity of the order. If small changes to uncertain scores reverse two priorities, treat them as a comparable tier. Use capacity, dependencies and strategic coverage to choose within that tier, with the reason recorded.
Hypothetical example: three checkout proposals
Consider a hypothetical retailer with capacity for one checkout experiment. This scenario is illustrative, with no claimed performance result. The proposals are a new button colour, earlier delivery information and a simplified address form.
The colour proposal comes from a competitor screenshot. It has little evidence about the retailer's users. The team keeps it as an observation to investigate rather than automatically awarding “quick win” status because development looks easy.
The delivery proposal has relevant support questions and usability observations suggesting that people discover restrictions late. The team still needs to check how common the affected journey is and whether earlier information changes the outcome it wants to measure. The proposed mechanism is clearer expectation-setting.
The form proposal has a visible technical defect affecting keyboard navigation. That defect is moved to the fix lane. A separate hypothesis about removing optional fields can remain in the experiment lane after the defect is repaired.
The next experiment tests earlier delivery information if readiness checks pass. The colour idea is deferred and the defect receives a repair task. The planning decision comes from different evidence and work types, rather than three arbitrary numbers pretending to be comparable.
Reserve capacity for research and measurement
A queue filled entirely with implementation leaves no time to improve the evidence feeding the next tests. Reserve practical capacity for user research, instrumentation checks and analysis. These activities prevent the programme from repeatedly testing vague ideas because they happen to be easy to build.
Use the qualitative research guide for entries whose mechanism is unclear. Research may reveal that a proposed variant addresses the wrong problem. Closing that entry is useful progress because it preserves traffic for a better question.
Similarly, an A/B test guardrail plan can expose a measurement gap. If returns or support demand are relevant but unavailable, the next task may be instrumentation rather than a variant. Do not quietly ignore a consequence because it is inconvenient to measure.
The right mix depends on the programme's maturity and bottleneck. There is no universal percentage to reserve. Review whether weak evidence, slow delivery or poor measurement is limiting decisions, and allocate capacity to the actual constraint.
Track learning across related tests
Link tests that address the same problem. A negative result can challenge the intervention, the proposed mechanism, the target audience or the measurement. Preserve that interpretation so a new colleague does not repeat the same idea under a different title.
Record both the estimate and the limitations. “Lost” does not explain whether a test showed meaningful harm, a near-zero effect or insufficient precision. “Won” does not explain guardrail movement or the scope in which the result applies.
Use the guide to statistical significance and peeking to distinguish a planned result from an early favourable fluctuation. A backlog that rewards wins can encourage stopping rules that manufacture them.
For behavioural explanations, connect the record with the psychology-to-test guide. A psychological mechanism is a hypothesis to test in context, rather than a guarantee that a familiar persuasion technique will improve conversion.
Keep the queue current and limit work in progress
An old idea may no longer fit the current journey. Review entries when pages, audiences, offers or instrumentation change. Keep dates and source links so reviewers can see whether the rationale still applies. Archive with a reason instead of preserving an ever-growing list of undecidable items.
Limit preparation and active experiments to available capacity. Too many half-ready tests create context switching and make it harder to investigate problems. A small prepared queue can be more useful than a large ranked backlog whose top entries all lack one prerequisite.
Watch for recurring failures: scores inflated to win attention, effort reduced by excluding analysis, evidence copied from unrelated audiences and negative results discarded. The worksheet does not remove these behaviours; it gives the team a visible basis for challenging them.
Start by reviewing the next five proposed tests. Identify their problem, evidence, decision and missing prerequisites. Move each to the correct lane and prepare one that can answer a worthwhile question. To connect that queue with your team's goals, tell me about your experimentation programme.
Apply this to your project
Tell me the decision you are facing, the constraints and what you have already tried.
Keep exploring
A/B test guardrail metrics: protect the whole journey
Choose A/B test guardrail metrics that protect quality, performance and downstream outcomes. Define thresholds and decisions before launching a test.
Low-traffic A/B testing: choose a question you can answer
Plan low-traffic A/B testing around useful effect sizes, eligible audiences and uncertainty. Learn when to test, research, simplify or defer.
Personalisation tests: measure the value of relevance
Design personalisation tests that compare a clear targeting rule with a useful default. Measure incremental value, guardrails and uncertainty.