Why a low AI error rate might be measuring fear, not fewer failures

Last updated: August 5, 2026

A hospital study on medical errors has a direct parallel to how product teams handle their own error data. Researcher Amy Edmondson expected teams with more trust, clearer communication, and stronger leadership to make fewer errors. The data showed the opposite: teams with better teamwork reported more medication errors, not fewer.

What turned out to be true: the better teams weren't making more mistakes. They just weren't afraid to report the ones they made. Teams where a mistake got held against the person who made it kept making mistakes at the same rate — those mistakes just never got logged. The error count wasn't measuring how many mistakes each team made. It was measuring how safe each team felt admitting one. Nothing about an AI model changes this dynamic, because the fear in the hospital study can't attach to the model itself. It attaches to the people around it.

For leadership: your dashboard might be measuring the wrong thing

If your AI error rate looks low, it might mean the system is working. It might also mean it's not safe for anyone to tell you when it isn't — and most dashboards have no way to distinguish between the two, because the metric is built entirely from what people were willing to report. Other than great leadership and culture to embrace the failure mode, there are three changes you can design to fix the measurement.

Change 1: Make the human report cost nothing

An engineer who reports that a shipped AI feature is giving wrong answers can trigger a rollback, a postmortem, or a hit to their own numbers. That has a cost, and the cost is paid by one identifiable person. Instead, you could auto-populate incident reports from logs, monitoring, and chat history so the record doesn't rest on one person's confession.

In 2022, a customer asked Air Canada's website chatbot about bereavement fares, and the bot described a refund policy the airline didn't actually offer. Nobody caught the gap before it reached a customer, then a tribunal: in February 2024, Canada's Civil Resolution Tribunal ruled Air Canada liable for what its own chatbot told that customer. Somewhere between the chatbot shipping and the ruling, the wrong answer existed for over a year without getting fixed — and it's worth being honest that an automated check might not have caught this one either. A chatbot confidently misstating a real, written policy isn't always a "hallucination" in the pattern a detector is tuned for; it can look exactly like a normal, well-formed answer. That gap is precisely why the human channel has to stay open, not why it can be replaced.

Tools like Rootly, FireHydrant, and incident.io can help by auto-reconstructing the incident timeline from monitoring alerts, Slack threads, and command history, so the report isn't built entirely from one person's account of what happened — it's corroborated by evidence that exists whether or not that person speaks up. The engineer can still flag it, quickly, and often does; they just aren't the sole source of truth exposing themselves by doing it, and the postmortem template has no field asking whose fault it was.

What to watch for: if opening an incident requires the person closest to the failure to narrate what went wrong from scratch, with their name attached to a cause, expect underreporting no matter how the form is worded. Pull the evidence automatically from logs, monitoring, and chat history first, so a report already has legs before anyone adds their name to it. Keep the reporting option open and easy — this doesn't replace it, it just stops the report from resting on one person's nerve.

Change 2: Look for inexplicit failure signals

A user who gets a bad answer from an AI feature rarely files feedback. Most assume they asked the question wrong and just try again.

This isn't specific to AI. Research on customer complaints — most commonly cited to Lee Resources International and TARP's earlier work — puts the ratio at roughly one complaint for every 26 customers who stay quiet and simply leave. Applied to an AI feature, the pattern is the same: a bad result reads as user error, not system error, and the person who hit it never tells anyone. The 25 who didn't try again are the ones a dashboard can't see at all.

What to watch for: a feedback button alone only catches the 1 in 26 who bother to use it. Keep the reporting option — a thumbs-up/down attached to the response — but don't treat it as your only error signal. This is already standard practice in LLM observability tools, not a hypothetical: Langfuse and LangSmith both let a thumbs score attach directly to a trace, and both also capture signals that don't require a click — retries (someone regenerating instead of accepting an answer) and escalation rate (handed off to a human agent right after the AI reply). A rising escalation rate right after an AI response is treated as a leading indicator of quality decay in its own right, before a single explicit complaint shows up. The traces that score low or get retried get routed into an annotation queue automatically, so review doesn't depend on someone deciding to go looking.

Change 3: Track the detected rate next to the reported rate

This case isn't about whether any one person feels safe reporting — it's about whether the number on the dashboard has any signal in it that doesn't depend on someone reporting at all. A team whose targets are tied to a low error rate has a reason not to go looking for a new category of failure.

At Wells Fargo, employees who called the company's own ethics hotline to report the sales practices behind its fake-accounts scandal were fired, in at least one case within eight days of the call. Sales volume targets gave the organization a structural reason not to look too closely at how the numbers were being hit.

What to watch for: ask which of the two instruments your "error rate" actually is. If there's no automated detector running independently of user reports — no Sentry-style exception tracking, no groundedness check, no LLM-as-judge flagging bad outputs — you don't have an error rate, you have a complaint rate. Build the automated side as a second, independent instrument, then track it next to the user-reported rate so a quiet quarter has to prove it's quiet on both counts, not just one.