Do you turn AI quality disagreements into a review inbox?

During a support team's weekly quality review, Northwind Traders' example dashboard labels 14 AI–human disagreements as wrong answers and recommends a prompt change. Four answers differ only in format; three quote a retired returns policy. The headline hides different problems that need different fixes.

Review disagreements against the evidence before changing the prompt. Use an inbox to turn a quality rate into specific, owned repairs.

These Northwind dashboards use example data.

❌ Figure: Bad example - Fourteen disagreements become fourteen alleged hallucinations without anyone checking the evidence

Separate agreement from correctness

A person's answer is a comparator, not automatically ground truth. The accuracy metric compares predictions with reference labels; its meaning depends on those labels being correct. Name the measure the evidence supports:

• Strict agreement - AI and human answers match under a documented comparison.
• Normalised agreement - A secondary comparison accepts specified equivalences, such as equivalent date and number formats. Preserve original answers and version the comparator.
• Reviewed outcomes - A person checks the source and written rubric, then records whether AI is right, wrong or still unclear, including cases where both answers are wrong.

Northwind reports strict agreement at 47/61 (77%) and normalised agreement at 51/61 (84%) separately. Review confirms that the four formatting mismatches were already-correct AI answers. Normalisation fixes their comparison; it neither changes those answers nor proves the remaining agreements correct. For uncertainty and comparisons between periods, show counts, a period and an appropriate range.

Two answers can agree and both be wrong. Adjudicate a random sample of agreements too; reviewing only disagreements cannot establish overall accuracy. Choose the sample size before reviewing, based on the risk and precision the decision needs, and publish the reviewed count and uncertainty.

Northwind's separate agreement audit reviews 10 of the 47 strict agreements and finds one where both answers are wrong. Report Agreements audited: 10/47; both wrong: 1/10, with a 90% Wilson range of about 2–35% for that error rate. These example counts are separate from the 11 reviewed disagreements below and do not establish overall accuracy; ten is not a recommended sample size.

✅ Figure: Good example - Auditing agreements finds an error the disagreement inbox cannot reveal; the small audit leaves wide uncertainty

Give every disagreement evidence and an owner

Put unclassified cases first, newest first within each status. Show the question, both answers, source version, cause, owner and review status, with a link to the full context. Apply the original records' access controls and minimise personal data in the preview.

AI may suggest a cause, but a reviewer confirms it against the source. An ambiguous case stays unresolved or goes to a second reviewer; don't force it into the model-error count. Show how much reached review beside the rate. Set a review cadence and an owner for overdue cases. Northwind reviews new rows each working day and flags any unassigned row older than one review cycle.

Fix the cause, then verify the repair

Northwind reviews 11 of 14 disagreements; three remain new. The cause and outcome counts agree:

• AI right: five - Four format-comparison bugs and one reviewer error
• AI wrong: five - Three answers retrieved from the retired source and two model/prompt errors
• Unclear: one - An ambiguous question that needs further review

The three wrong-source cases need a knowledge-base or retrieval repair. They do not establish a prompt problem.

✅ Figure: Good example - Readable answer and source evidence separates comparator and source repairs from the two confirmed model or prompt errors

Fix and recheck the comparator and retired article before rewriting the prompt. Own the AI output: close a row only after a reviewer verifies the repair. Add confirmed model errors to the eval set and rerun it before release.

Keep review rates in context

Show each rate's numerator, denominator, period and sampling frame. In Northwind's example, 61 random checks cover 718 answers in five topics, out of 870 overall. Label the three unchecked topics and report Random-review coverage: 61/718 (8.5%) separately from Coverage of all answers: 61/870 (7%). Keep targeted checks separate; use Not measured when no reviews finish.

A Below 80% target badge can coexist with No clear change yet. Show counts and uncertainty; compare like-for-like samples before claiming a change. Withhold rankings when checks are too few. Release markers show timing, not cause; verify a proposed repair with evals.