AI programme management
Evaluating AI output: a practical quality scorecard
Evaluate AI output with task-specific criteria, representative cases and independent review. Build a scorecard that informs a clear release decision.
A fluent answer can be wrong. A correct answer can omit the fact the reader needs. A technically valid action can update the wrong record. Evaluating AI output means deciding what success looks like for a task, then checking the complete result against that definition.
A generic quality score cannot make the release decision for you. You need a representative task set, criteria that reviewers can apply and a record of failures important enough to change the scope. This guide builds that evaluation from the business task. It connects with the AI programme management guide and the production KPI guide.
Define the result the task requires
Start with the person receiving the output and the action they need to take. For a document summary, useful quality may include accurate decisions, named owners, dates and retained qualifications. For a customer answer, it may include supported statements, relevant scope and a clear escalation when the source is insufficient.
Write criteria as observable questions. “Professional” is difficult to score consistently. “States the requested next step without adding an unsupported commitment” gives reviewers something to inspect. Include examples that pass, fail and require judgement.
Separate factual quality, task usefulness, style and permitted behaviour. A weighted average can hide a serious failure under attractive formatting. Identify criteria that must pass independently, such as an action staying within its authorised boundary. Agree that rule before viewing the results.
The People + AI guide on data and evaluation connects evaluation with real user conditions. Apply that idea to the task set: polished examples alone will not tell you how the workflow handles incomplete inputs, mixed formats or ordinary ambiguity.
Build distinct sets for development and decisions
Use development cases to improve the workflow. Use a separate evaluation set to assess a candidate configuration. Add challenge cases for known boundaries, and retain examples of failures as regression cases. These sets answer different questions.
If every weak evaluation result leads to prompt tuning on the same examples, the score gradually measures familiarity with the test. Preserve unseen cases where possible and record when a test case has influenced development. A small independent set can be more informative than a large set used repeatedly without that distinction.
Keep expected answers where a clear answer exists. For open tasks, define required content and unacceptable behaviour instead of demanding one exact phrasing. A correct answer may express the same fact in several ways. Exact wording checks can reject useful alternatives.
Protect the data used for evaluation. Work within the permissions and retention rules appropriate to the material. Synthetic examples help test particular situations, but do not automatically represent real task frequency. Describe what the set covers and what it cannot establish.
Use a scorecard with independent quality dimensions
Build a short scorecard around the task. The example below concerns a document-based internal answer assistant. It is an editorial template, not a universal benchmark or a claim about a particular model.
Swipe or use the arrow keys to see every column.
| Dimension | Question | Evidence | Failure treatment |
|---|---|---|---|
| Support | Are material statements supported by approved sources? | Statement-to-passage comparison | Reject unsupported consequential claims |
| Accuracy | Does the answer preserve meaning, numbers and qualifications? | Source and task context | Record error type and severity |
| Coverage | Does it include the facts needed for the task? | Agreed required elements | Diagnose missing elements |
| Relevance | Does it answer this question rather than a nearby one? | Input and intended action | Review task fit |
| Uncertainty | Does it handle missing or conflicting evidence honestly? | Challenge cases and source gaps | Reject invented resolution |
| Action boundary | Does it stay within authorised permissions? | Tool result or verified state | Treat as an independent gate |
| Usability | Can the reader understand and use the answer? | Reviewer task observation | Inspect wording and structure |
| Operating effort | How much checking and correction does it require? | Complete handling-time record | Include in the value decision |
Keep the dimensions separate in the report. A single composite can support a comparison, but reviewers still need the underlying failure distribution. Explain the scoring scale in ordinary language so a score means the same thing across people.
Match the evaluator to the criterion
Deterministic checks work for concrete constraints: a required field exists, a total matches, a prohibited action did not occur or a record reached the expected state. They are less useful for nuanced relevance or style unless the criterion has been made sufficiently precise.
Human reviewers can assess domain meaning and whether the result helps with the task. Give them the source, the rubric and a way to mark uncertainty. Use independent scoring on a subset to identify disagreement. Discussion should improve the rubric, rather than force agreement by ignoring difficult cases.
Model-based judges can help flag unsupported claims or compare open-ended responses. Their output still needs calibration and monitoring. In its guide to agent evaluations, Anthropic distinguishes code, model and human graders. The point for a business workflow is to choose a check that fits the question.
A judge agreeing with another model does not establish ground truth. Compare judge decisions with qualified human review, inspect systematic disagreements and preserve examples of missed failures. If the judge is given a flawed reference, it can confidently reward an incorrect answer.
Hypothetical example: two configurations, different failures
Imagine a hypothetical team comparing two configurations of an internal answer assistant. The example is illustrative; it does not report measured client results. Both candidates answer the same questions using the same approved source collection.
Configuration A produces concise answers but omits exceptions. Configuration B includes exceptions but sometimes adds statements absent from the documents. A broad “helpfulness” rating might make them look similar. The separated scorecard reveals different remediation tasks.
For A, the team examines required coverage and how qualifications are represented in the source. For B, it examines unsupported claims and whether the workflow should refuse when evidence is missing. The team does not choose the more polished candidate while leaving those failures hidden.
After changes, it runs the fixed evaluation set again and checks previous failures. It also adds independent cases to examine whether the improvement generalises. A candidate can pass the examples that motivated a change and still fail a closely related task.
The release decision may restrict the assistant to a task type with adequate evidence and route other questions to a person. That is an evaluation outcome. Choosing “no release yet” or a narrower scope can be more useful than publishing a headline average.
Measure variability and preserve the configuration
Generative systems can produce different outputs for the same input. For important task types, repeat relevant cases and record the variation. Choose the number of trials in proportion to the decision, cost and observed variability. One successful attempt cannot establish reliable performance.
Preserve the configuration: model identifier, relevant settings, prompt version, source snapshot, tools and permission scope. Keep task identifiers, outputs and grading decisions. Without this record, a later comparison may combine a changed model with changed sources and changed scoring criteria.
When evaluating agents, inspect the final state as well as the written response. A message claiming that a task was created is different from a verified task record. Likewise, an answer can look correct while the system used an unauthorised tool or accessed an inappropriate source.
Record failed attempts and retries separately. A final pass after repeated correction answers a different operational question from a first-attempt pass. Include cost, latency and human intervention so the report does not conceal the effort needed to obtain the result.
Connect quality with the release decision
Decide acceptance criteria before running the candidate evaluation. Specify independent gates, acceptable performance ranges and what happens when results are uncertain. The AI governance guide identifies who can make that decision and what they are accepting.
Use uncertainty rather than disguising it with decimals. For a small set, describe the limited coverage and show counts. If you estimate population performance, use an appropriate statistical method and explain its assumptions. Avoid presenting a task set assembled for challenge testing as a random sample of production.
Weight cases transparently if task frequencies differ. A rare high-consequence failure may deserve a dedicated gate even when it barely changes the average. The risk register helps keep those scenarios visible.
Evaluate the assisted workflow against a meaningful alternative. This may be the current manual process, an ordinary search interface or a simpler automation. A new model beating an old model does not establish that either improves the business task. Include reviewer work using the human-in-the-loop design guide.
Continue evaluation after release
Production monitoring can reveal changed inputs, new task types and source problems absent from the test set. Create a route for users and reviewers to report specific failures. Connect those reports to evaluation updates, source corrections and decisions about scope.
Do not add every incident directly to the main score without considering what it represents. Some cases are regression checks, some reveal a new class of task and some are isolated data issues. Keep the ordinary-task estimate and the challenge suite interpretable.
Re-evaluate material changes. New tools, permissions, documents or model versions can invalidate earlier evidence. A passing test is evidence about a configuration and task set at a particular time; it is not a permanent guarantee.
For marketing output, the same discipline applies to claims, relevance and reader usefulness. The AI marketing guide provides the application context. Start by writing the criteria for one real task and grading a small authorised set independently. To turn those results into a programme decision, tell me about your evaluation challenge.
Apply this to your project
Tell me the decision you are facing, the constraints and what you have already tried.
Keep exploring
AI governance: assign responsibility before release
Make AI governance operational with clear decision rights, release gates and incident ownership. Use a charter that your team can apply to real work.
AI project risk register: turn concerns into controls
Build an AI project risk register around failure scenarios, owners and tested controls. Use a worked example to review residual risk before release.
AI roadmap prioritisation: choose what to build first
Prioritise an AI roadmap around evidence, dependencies and measurable value. Use a practical decision sheet to choose the next project with confidence.