What You Should Know
- A report by the Center for Connected Medicine (CCM) at UPMC and KLAS Research reveals that while 93% of surveyed health system leaders have deployed third-party AI, strategy and testing infrastructure lag far behind.
- Clinical documentation and ambient scribing lead deployment at 52%, followed by revenue cycle and coding (36%), medical imaging (32%), and EHR-embedded clinical decision support (32%).
- While 92% of organizations conduct pre-deployment testing, less than half (44%) possess a dedicated data platform or sandbox environment to validate model accuracy, safety, and drift.
- 63% of health system AI strategies remain “developing” or ad hoc, with only 4% described as “advanced”.
- Top operational pain points include reliance on manual spreadsheets (17 respondents), inconsistent data definitions across teams (14 respondents), and severe talent or resource constraints (11 respondents).
CCM and KLAS Research Expose Health IT’s Governance and Testing Deficit
The enterprise health IT, clinical transformation, and healthcare analytics sectors face a structural operational challenge: health system AI procurement is significantly outpacing the internal governance, data architecture, and testing pipelines required to validate algorithms safely.
While executive boards face immense pressure to adopt clinical and administrative AI, deploying algorithms without dedicated testing environments exposes health systems to algorithmic drift, unvetted bias, security vulnerabilities, and unvalidated ROI.
To quantify this operational gap, the Center for Connected Medicine (CCM) at UPMC and KLAS Research released a comprehensive report titled “Validation and Trust: How Health Systems Are Testing and Governing Analytics and AI Solutions“.
Surveying C-suite, clinical informatics, and IT executives across 27 health systems, the research demonstrates that the primary bottleneck in healthcare AI has shifted from tool procurement to operational validation and infrastructure.
The research outlines key technical and structural realities across current health system deployments:
- Widespread Deployment vs. Infrastructure Deficit: 93% of health systems have deployed third-party AI, yet only 44% maintain a dedicated data environment (such as cloud lakehouses, production clones, or real-world data platforms) to test models before clinical integration.
- Primary Use Case Distribution: Ambient clinical documentation leads adoption (52%), followed by revenue cycle management and coding (36%), diagnostic imaging (32%), and EHR-embedded clinical decision support (32%).
- Data Quality Friction: Reliance on manual workarounds/spreadsheets (17 respondents), inconsistent metrics across departments (14 respondents), and unstructured clinical data (10 respondents) represent the greatest obstacles to model reliability.
- In Silico Sandbox Validation: Advanced organizations are deploying dedicated platforms—such as UPMC’s real-world data engine, Ahavi—to evaluate third-party algorithms against de-identified patient populations without disrupting live care.
“The health care industry has moved remarkably quickly from discussing the potential of AI to actively deploying solutions across the enterprise,” stated Dr. Rob Bart, Chief Medical Information Officer at UPMC. “Health systems are now focused on building the governance structures, testing capabilities and organizational strategies necessary to ensure AI delivers meaningful and measurable value.”
