Ashok Benial is right. His argument in these pages last week — that most hospitals are buying AI faster than they can govern it, that “the tools go live while the guardrails are still on a slide deck” — is the most useful thing anyone has said about healthcare AI governance this quarter. His three controls are the correct ones: validate locally against your own population, monitor for drift after go-live, and fund the human review layer as a control system rather than overhead. Every health system deploying AI should adopt all three.
But notice what those three controls have in common. Local validation asks whether the model is accurate on your patients. Drift monitoring asks whether it is still accurate. The review layer asks whether a human checked the output. All of them guard against the same failure: a wrong answer. That was the right failure to guard against while healthcare AI meant prediction and documentation — a risk score, a draft note, a suggested code that a person would read before anything happened. It is not the failure that will define the next two years, because the systems going live now do not just answer. They act.
Look at where the agentic wave is actually landing in health IT: claims, coding, prior authorization, and revenue cycle — transactional systems of record. Last week an autonomous coding company raised $25 million on a platform that reads the chart, assigns the CPT and ICD-10 codes, and creates direct-to-billing claims with, in its own words, zero manual human retyping. That is not a suggestion a coder reviews. It is an action a billing system executes.
Healthcare should be better prepared for this than any industry. It has the most mature access-control culture in enterprise computing: minimum necessary, role-based access, break-the-glass, entitlement review, audit controls — decades of discipline, all of it built to answer one question. Who may see what?
That question was sufficient as long as the actor holding the credential was a person, whose actions were bounded by working hours, judgment, and the speed of clicking. It cannot express agent risk. What turns an authorized agent action into an incident is not what the agent could read. It is what the agent could break: whether the action was reversible, whether it touched a production system of record, and how far it propagated before a human saw it. A data-sensitivity analysis cannot even state the comparison that matters. For an agent, a read-only credential pointed at the live EHR is less dangerous than a write credential on a billing sandbox — and no minimum-necessary review in existence can produce that sentence.
The published agent-security incidents keep returning the same uncomfortable detail: the agent was authorized. The clearest documented case comes from outside healthcare. In June, Microsoft Threat Intelligence disclosed that a widely used AI coding agent, running inside automated workflows, could be steered by untrusted content until its file-reading tool “was eventually authorized” to read the credentials available to its own environment. Nothing was jailbroken and no control was defeated; the vendor fixed it promptly once it was responsibly disclosed. The agent held permissions and used them exactly as granted.
For a compliance function, that finding is worse than a breach. A breach means a control failed. This means the control worked as designed — and the design was answering the wrong question. In the workflows health systems are automating right now, an agent authorized to submit prior authorizations is authorized, by the same grant, to submit them wrong — at machine speed, across a payer panel, before anyone reads the first denial. Role-based access cannot distinguish those two activities, because they are the same permission.
So here is the fourth control, in the language a governance committee already uses. Classify agent tasks by reversibility, not by data tier. Make “irreversible” its own permission category rather than an implication of “write.” Before any credential is issued to an agent, require a written answer to the question the risk register never asks: what can this system do that cannot be undone? A coding suggestion a human reviews sits in one tier. A claim transmitted directly to a payer sits in another. Anything that overwrites state in the system of record sits in a third, behind the strongest gate you have. Scope credentials per task and per run — the standing service account that lives for years, holding every permission its agent might someday need, is precisely the design the incident reports keep finding. And treat the reversal path as a designed, tested control: for anything irreversible, the governance question is not “will we detect it?” but “can we put it back?”
The objection writes itself: healthcare already controls write access, logs everything, and reviews entitlements. True — and audit logs are detection after the fact, which is a weak control against an actor that operates faster than review. The stronger answer is that healthcare has already solved a problem exactly this shape. Break-the-glass exists because the who-may-see-what question fails in an emergency, and the profession built a formal exception category rather than pretending the old question covered it. Reversibility deserves the same honesty: a distinct category, formally defined, separately gated.
Benial closed by saying the algorithms are ready and the question is whether our oversight is. I would sharpen it: oversight is only as good as the question it asks. Validation, monitoring, and human review will catch the wrong answers. None of them will catch the wrong permission — because that flaw is already in place before the agent is asked anything at all.
About Chase W. Hughes
Chase W. Hughes is an AI product leader and three-time founder who built and sold ProAI, one of the first commercialized GPT products, and has been building multi-agent AI systems since early 2023.
