pipette
ESEspañol

Planted Errors Cannot Measure What a Clinician Misses

K. C. Kotabhattara, A. Siroha, P. Pal, S. K, V. S. Dhara, P. Prabhakar, N. Honkote, M. Marreddy, N. Namita, A. K. K, P. Saha, J. M. Agris

PreprintReal-world use

In the authors' words

A safety certificate for a clinical AI promises that no more than a stated fraction of the answers it releases carry a defect that would change care. That promise is computed from past cases a clinician marked safe, and clinicians miss things. A missed defect is not filed as a disagreement, it is filed as a safe case, so the certificate rests on a record cleaner than the ward ever was and has to be widened to compensate. Widening it correctly takes one number: how often that clinician misses a defect the model produced. We show that number cannot come from the only test a hospital can run on a working reviewer, planting a known defect and counting the catches. A planted defect is as hard to spot as whoever planted it made it, a real defect is as hard to spot as the model happened to make it, and comparing the two needs the very number being sought. No safety factor recovers it, because one wide enough for every miss rate releases nothing. The failure also runs the dangerous way, because corrections that turn noise into one rate need the mistakes to be unrelated to the case, whereas a defect is missed because it read well, so the risk on record sits below the true risk at every threshold. Across four simulated reviewers that condition never holds, and testing it on a separately drawn sample is affordable only where the answer was never in doubt: a reviewer missing 85% of defects costs 185 adjudicated cases against 147,938 for one missing 3%. Adjudicating the reviewer's own cases instead removes that test and most of that cost, 498 cases for the careful reviewer, so modelling the noise costs more than measuring it. Our library therefore refuses the widened certificate unless the condition is confirmed on adjudicated data, and fails closed otherwise.

Main resultLimitation the authors admit

In plain words AI summary

This certificate is based on cases that a clinician marked as safe, but clinicians sometimes miss real defects. To fix this, the certificate must be widened using the rate at which clinicians miss defects the AI makes. The authors show this missing rate cannot be found by planting known defects and counting catches.

Written by Kimi K2.5 using only the abstract above. Jev checked that every sentence is supported by the abstract sentences it cites, and removed any that were not. Point at a sentence to highlight its source. 2 sentences were removed for not being supported by the abstract.

Appeared: Saturday, October 3. medRxiv. Preprint, not yet peer-reviewed.

DOI: 10.64898/2026.10.01.26364465