Hospitals are judged by numbers that begin with a person reading a chart. Surgical complications, cancer outcomes, cardiac and stroke measures, and many of the quality scores that shape public rankings all trace back to clinical registries. Those registries don’t fill themselves. Trained staff comb through each patient’s record and enter the details by hand. It’s slow, meticulous work that most people never see, but it’s how healthcare performance gets measured.
To check that work, registries run an audit. A second abstractor re-reads a case someone already completed, and the two sets of answers are compared. The rate at which the two agree is called inter-rater reliability, and each registry sets a minimum rate programs are expected to meet. But the audit only re-reads a sample of cases. It says nothing about the ones nobody reopened.
The Audit Was Always a Sample
Verifying registry data has always meant re-abstracting a fraction of it. The Commission on Cancer requires accredited cancer programs to review at least 10 percent of their analytic caseload each year. Surgical-quality programs flag any variable that disagrees more than 5 percent of the time so it can be retrained, which shows where they diverge, not what share of cases were re-abstracted. The industry didn’t adopt sampling because it was the right way to measure quality, but because re-abstracting every case by hand costs more than registry budgets or time allow.
That would matter less if abstraction were uniform work. It isn’t. Manual abstraction is among the largest sources of error in clinical data, and in most studies that error is never quantified at all. When researchers pooled accuracy data across 93 studies, reported error rates ran from roughly 2 to 2,784 per 10,000 fields.
Two abstractors can read the same chart and code it differently. When investigators compared trauma registrars across hospitals, coding and scoring varied widely enough to challenge the accuracy of the aggregated registry. A reliability study of medical record abstraction found learning effects across every variable it tracked, meaning one abstractor applied the same definitions differently as the work went on. The reviewer on Tuesday isn’t the reviewer on Friday, and neither one is the reviewer from six months ago.
The audit is more fragile than it looks. A small sample is being used to estimate the reliability of a process whose own error is large and, in most settings, never directly measured. The score is accurate for the cases it reviewed. For the rest, it’s an educated guess.
The Comparison Problem
In one multicenter audit, center-level error ranged from 0.4 to 5.2 percent across hospitals working the same registry, showing how a record’s accuracy depends partly on where it was abstracted, not on how the patient was treated. That score feeds benchmarking, accreditation, public reporting, and research. Ranking one hospital against another assumes they all capture and read the data the same way, and when they don’t, a hospital’s ranking can rise or fall without any change in the care it delivered. A fair comparison needs consistent data, and the audit never checks for it.
This isn’t news to the people who study measurement. A quarter-century ago, researchers comparing chart abstraction against a gold standard warned that quality measures drawn from the record have real limits as a basis for policy and management decisions. Ranking hospitals against each other is exactly that kind of decision.
Complete Review Is Possible
Checking every case used to mean checking it by hand, which is why programs settled for a sample. Automation changes that. When a software system surfaces candidate answers with links to where they appear in the record, the abstractor’s work shifts from searching to validating, and reviewing a full caseload becomes practical.
A sample turns up mistakes one at a time, while full coverage shows whether a mistake is a one-off or a pattern, so a team can fix a recurring problem at its source instead of correcting the same error chart by chart. The abstractor still holds the judgment. The tool changes what they spend time on, not who signs off.
Stop Settling for a Spot Check
Healthcare leaders might never open an audit report, but they act on its results every day. Registry scores help decide where improvement resources go, and patients and payers use them to compare one hospital against another. Decisions that big deserve more than a spot check.
Leaders don’t need to become abstraction experts. They can ask how many cases the last audit re-read and whether the disagreements it found were isolated or part of a pattern. Those two answers say more about the data than the score does.
Reviewing every case is now within reach. Until the abstraction process does that, the audit that clears the benchmark is still just an audit of a sample.
About Brent Dover
Brent Dover is the CEO at Carta Healthcare, the leader in enterprise clinical data management. With a deep commitment to improving healthcare outcomes, Brent leads Carta Healthcare’s mission to unlock the power of clinical data for hospitals, health systems, and life sciences organizations. His strategic vision drives the company’s focus on leveraging advanced artificial intelligence and expert clinical professionals to streamline data abstraction, enhance operational efficiency, and generate actionable insights that ultimately elevate patient care. Under his leadership, Carta Healthcare has experienced significant growth and recognition for its innovative approach to solving complex healthcare data challenges.
