Every healthcare organization has been through the ordeal of a major software deployment: months of prep work, weeks of testing and calibration during development, hours of user training, and a big sigh of relief at rollout. Along the way, they’ve also set up regression testing to guarantee updates don’t break anything, as well as monitoring to make sure everything continues to work in production.
Development testing, regression testing, and monitoring, it’s a tried-and-true formula that should work just as well for AI, right? Not necessarily, because there are two basic problems when you try to apply this formula to AI:
- The tasks AI performs often resist effective quantitative controls, allowing only subjective quality measurements.
- AI systems are stochastic by nature and more unpredictable than most software systems, which means they can be (in)correct in multiple ways.
Both of these issues are similar to other intelligent systems that we also struggle to evaluate: humans. Anyone who has had to evaluate the performance of employees or grade non-multiple-choice student exams knows the advanced thought required to execute such evaluations.
This is the twist. AI systems require intelligent systems, such as human or AI judges, to evaluate them. Why? Because they tend to resist mechanistic tests that are usually effective for non-AI software systems.
That’s why in order to ensure continuous safety, healthcare organizations need to add an intelligence twist to the three-tiered trust strategy: development testing, regression testing, and continuous monitoring.
Tier 1: Establishing Initial Trust With Deep Development Testing
Development testing involves rigorous upfront evaluation to ensure baseline accuracy and clinical safety before deployment. AI, however, introduces potential wrinkles into this process that require special attention, such as:
- Representative data challenges: Training and test datasets capture “common cases,” but may miss the long tail of rare or edge scenarios. Overcoming this issue involves culling through enough data to unearth the uncommon cases, which requires intelligence.
- Clinical faithfulness issues: Small linguistic shifts from AI can create major clinical risks, highlighting the importance of being able to evaluate summarizations in a medical setting. For example, AI converting a note about “GI issues” into “nausea and vomiting” creates a false specificity that may negatively impact a patient. Overcoming this issue likewise requires intelligence.
For a one-time cost, humans could be deployed to address both of these issues. But the large amounts of data and clinical training required could make even that one-time cost an expensive proposition.
A more scalable approach is to manually annotate a small amount of data and then use it to develop an LLM judge – or better yet, an LLM jury – to evaluate AI output.
Tier 2: Protecting What Already Works With Regression Testing
Deep development testing is essential, but it’s not the endgame. In fact, a software rollout is really just the beginning. Software systems don’t remain static: bugs need to be fixed, features need to be added, and upgrades need to be made to improved versions of supporting software packages.
AI-enabled systems are similar, with underlying models continually being improved and APIs being changed. That means re-running an evaluation every time an AI model or system component changes. This can be challenging to keep up with, however, since development tests are often not easily repeatable and manual annotation is out of the question due to the frequency and speed required.
This is why regression testing for AI can benefit from techniques, such as:
- Side-by-side comparisons of old and new system outputs, using an LLM jury to validate that the new system outputs are at least as good if not better than what they’ve replaced.
- A gold data set, where a data set is manually annotated with optimal output and the new system’s output is compared to this gold standard; often an LLM jury is still required to do this comparison. The manual annotation is performed only once, often leveraging the development testing work (Tier 1).
Like almost all software, AI systems exist within a broader ecosystem, where changes in data pipelines, user interface, or upstream processing can introduce unintended failures. Regression testing helps ensure that an organization’s progress doesn’t come at the cost of hidden degradation.
Tier 3: Managing the Unpredictable With Continuous Monitoring
Unlike the occasional bugs of software, AI systems regularly produce unexpected behavior due to several factors, including…
- Real-world data that introduces long-tail edge cases not seen in development.
- Surrounding systems – including user interfaces, pipelines, and integrations – that evolve independently.
- Inherent unpredictability, which amplifies both of these factors.
This unpredictability is constant. And it requires monitoring that keeps up – namely, real-time (or near-real-time) evaluation of live outputs to detect issues early. The challenge is that human review is simply too slow and expensive to maintain a realistic level of real-time detection. As with regression testing, an automated LLM jury approach can be the answer.
Of course, using an LLM jury for every output would be prohibitively expensive. Thus, the LLM jury is best applied only to a random sample of outputs at a rate that doesn’t break the bank, but can nevertheless quickly catch a systematic degradation in quality, safety, and faithfulness.
And while an LLM jury can be calibrated with periodic human review to make sure it continues to align with current standards, the bottom line is that automation is essential to monitoring AI systems.
Three Tiers With a Twist
Healthtech AI has all the same risks as traditional healthtech software, which is why it too requires the three standard software system testing tiers:
- Development testing that builds initial confidence.
- Regression testing that preserves reliability through change.
- Continuous monitoring that ensures safety in real-world conditions.
The twist with AI is that it requires added intelligence to properly execute each of these tiers. While human evaluation can be employed for development testing, it is too slow and expensive for regression testing and monitoring.
Which means AI must be employed to evaluate itself: that’s the twist.
About Marc Light
Marc Light is the director of AI and innovation at Zus Health, a healthcare data platform that enables longitudinal care by giving providers a real-time, comprehensive view of each patient’s care history in the form of a common patient record. Before Zus, he worked for Analog Devices, Facebook, and Bitsight Technologies.
