Experimentation & CRO
Personalisation tests: measure the value of relevance
Design personalisation tests that compare a clear targeting rule with a useful default. Measure incremental value, guardrails and uncertainty.
A personalised page can perform better than the current page because its content is better, because its targeting is useful or because the comparison selected different people. A personalisation test needs to distinguish those explanations. Otherwise you may pay to maintain a complex rule when a good default would deliver the same benefit.
The question is whether a defined targeting policy creates incremental value for an eligible audience. This guide explains the comparison, assignment, segmentation and release decision. Begin with the CRO and A/B testing guide for the wider programme.
Describe the policy as a rule someone can inspect
Write what the experience does, for whom and on which information. “AI-powered relevance” does not define a treatment. “Returning visitors who previously viewed installation guidance receive that guidance beside the product details” is inspectable.
Record the eligibility rule, available input, content choices and fallback. Explain when the rule is evaluated and whether it can change during a visit. The policy includes those operational choices, rather than only the algorithm or the personalised text.
Choose inputs appropriate to the context and permitted use. Avoid collecting additional sensitive information merely because a model can consume it. Involve the relevant privacy and security specialists where needed. The experiment should not assume that a possible data connection is an authorised one.
Identify the user problem. A different page for each person is not automatically a useful experience. Personalisation may help someone find relevant information; it can also hide alternatives or make the service inconsistent. State the expected task improvement before building a targeting mechanism.
Compare against a useful default
A weak default makes any new experience look attractive. Improve the basic clarity and usability of the default before claiming a need for targeting. For content experiments, check whether the proposed personalised information would help everyone.
A simple comparison can assign eligible people to the current policy or a fixed personalised policy. If you need to isolate targeting from content quality, a design may compare the current default, a generally improved default and the targeting policy. That additional arm costs traffic and needs planned comparisons.
The right comparator depends on the release decision. If the question is whether to maintain the targeting system, compare with the best credible simpler alternative. If the question is whether a particular targeting rule beats the current policy, keep that narrower interpretation in the report.
Research on personalised treatment selection distinguishes population-wide selection from selection using heterogeneous effects. It supplies a methodological reason to examine variation, rather than a guarantee that personalisation will improve your particular journey.
Assign people before the policy changes their behaviour
Randomise at a stable unit appropriate to the experience. A person receiving inconsistent versions across visits may encounter a different treatment from the one intended. In B2B workflows, collaborators may share information, requiring attention to interference between assigned units.
Establish eligibility consistently in both arms. If the personalised content is shown only after a qualifying event, record who would qualify under the comparison policy as well. Do not analyse only people who clicked the personalised component: that click is affected by treatment.
Keep assignment and policy eligibility separate in the records. You need to know which policy someone was assigned to, whether it could act and whether the experience was delivered. These distinctions help diagnose implementation and exposure failures.
Inspect allocation and event quality before interpreting effects. A targeting rule that fails on some devices can create an apparent audience difference. Use the guardrail metrics guide to separate data validity from product consequences.
Decide which segment questions are planned
An overall average can hide variation. It does not follow that every observed subgroup movement is real. Looking through many segments after a test gives you many opportunities to find an attractive pattern by chance.
Choose a small set of segment questions before launch, based on the policy and business decision. Use characteristics available before treatment. If you need a formal claim that effects differ between groups, plan an appropriate interaction analysis and its precision, rather than comparing two separate significance labels.
For exploratory findings, label them exploratory. Preserve the observation as a hypothesis for independent evaluation. A segment discovered because it looks favourable in the current data should not be treated as a proven target in that same data.
Microsoft's research on ranking heterogeneous effects shows how assumptions enter targeting analysis. For a practical programme, retain independent evidence rather than turning a promising ranking into a claim of certain benefit.
Write a policy-test specification
The worksheet below is an editorial planning aid. It connects the targeting idea with the comparison that can support a release decision.
Swipe or use the arrow keys to see every column.
| Item | What to specify | Failure to avoid |
|---|---|---|
| Task | What information or action becomes easier | Personalisation without a user problem |
| Policy | Inputs, rule, content choices and timing | A treatment that changes during analysis |
| Default | Credible alternative and fallback | Winning against a deliberately weak page |
| Eligibility | Consistent rule before the relevant effect | Selecting on treatment behaviour |
| Assignment | Unit, persistence and interference risks | Mixing experiences across visits |
| Primary result | Outcome tied to task and business value | Optimising only a component click |
| Planned segments | Pre-treatment characteristics and questions | Hunting through many subgroups |
| Guardrails | Performance, quality, accessibility and relevance | Ignoring the cost of targeting |
| Analysis | Comparisons, uncertainty and multiplicity plan | Choosing the result after the data |
| Release | Who decides, for which scope and duration | Expanding beyond evaluated users |
Link the policy to a falsifiable test hypothesis. If the team cannot say what a negative result would change, the experiment's decision value needs clarification.
Hypothetical example: guidance for returning shoppers
Imagine a hypothetical retailer considering installation guidance for returning shoppers. This scenario is illustrative and does not report conversion results. The policy uses an authorised prior page interaction and always offers a default when that information is unavailable.
The team suspects that returning visitors need installation details before choosing a product. It first examines the task through qualitative CRO research. Some participants need the guidance; others are comparing specifications or checking delivery.
The proposed experiment compares a clear default with a fixed targeting rule. The team considers whether a generally visible guidance link is a sufficient simpler alternative. The primary outcome concerns a suitable completed task, with relevant performance and order-quality protections.
If the personalised component receives more clicks, that is useful diagnostic evidence. It does not establish incremental business value. If the task outcome improves only in a segment discovered after the result, the team treats the segment as a hypothesis requiring independent evidence.
The decision may be to improve the default, retain a narrow policy or stop targeting. The test does not owe the team a successful personalisation story; it owes a reliable basis for the next choice.
Keep learning data separate from evaluation data
When a policy is learned from data, distinguish training, tuning and evaluation. Choosing a model and judging its value on the same outcomes can overstate performance. Reserve independent data appropriate to the policy and task.
A fixed policy can be simpler to evaluate than one that continually adapts. If it learns during the experiment, document the adaptation and choose an analysis suited to that design. A conventional fixed-allocation interpretation may not describe an adaptive system correctly.
Evaluate the policy as deployed, including missing inputs, fallbacks and implementation latency. An offline ranking can look useful while the live experience fails to retrieve the required context. Include those cases in the operating assessment rather than discarding them as inconvenient.
If AI generates personalised copy, preserve approved claims and content constraints. The AI marketing guide addresses that application. More relevant wording still needs factual support and a review process appropriate to its consequences.
Measure the costs of relevance
Targeting adds maintenance: input quality, rule ownership, content variants, monitoring and fallback behaviour. Include those costs in the release decision. A small uncertain gain can be hard to justify when the rule creates persistent operational complexity.
Monitor loading, errors and accessibility. Check that the personalised route does not hide necessary information or create misleading differences in terms. Evaluate relevant downstream outcomes with appropriate follow-up, rather than assuming a component click represents satisfied intent.
For the behavioural interpretation, the consumer psychology guide helps frame relevance and uncertainty as questions. Do not infer that a person prefers a topic permanently from one observed interaction.
Where traffic is low, segmentation can make estimates even less precise. Reduce unnecessary variants, narrow the decision and consider whether a simple default is the more defensible implementation.
Release the policy within the evaluated scope
Report the overall estimate, planned segment results, exploratory findings, guardrail status and uncertainty separately. Do not blend them into one claim that “personalisation works”. Explain which policy, audience and operating period the evidence concerns.
Keep a comparison group or renewed evaluation where practical and justified. A policy's value can change as audience behaviour, content or inputs change. Define review triggers and who owns the fallback if data becomes unavailable.
Record inconclusive or negative findings in the experiment backlog. They can reveal that the default is strong, the targeting signal is weak or the decision needs more evidence.
Start with a policy that can fit in one paragraph and a credible default. Define the outcome and the comparison before building complexity. To test relevance within your actual journey, tell me about the personalisation idea.
Apply this to your project
Tell me the decision you are facing, the constraints and what you have already tried.
Keep exploring
A/B test guardrail metrics: protect the whole journey
Choose A/B test guardrail metrics that protect quality, performance and downstream outcomes. Define thresholds and decisions before launching a test.
Experiment backlog prioritisation: choose better tests
Prioritise your experiment backlog using evidence, traffic and decision value. Build a practical queue that turns observations into useful A/B tests.
Low-traffic A/B testing: choose a question you can answer
Plan low-traffic A/B testing around useful effect sizes, eligible audiences and uncertainty. Learn when to test, research, simplify or defer.