The Short Answer
A sepsis model can score well where it was built and miss most cases elsewhere, and ONC's HTI-5 rule no longer makes certified EHRs disclose model cards for predictive AI. Health systems now have to validate models on their own patients, and on ibl.ai you own all the code and the data that validation runs on.
The independent check found the model missed 67% of sepsis cases at Michigan Medicine. HTI-5 was finalized on October 26, 2026.
What did the independent validation of the Epic Sepsis Model find?
In June 2021, Wong and colleagues published in JAMA Internal Medicine an external validation of the Epic Sepsis Model, a proprietary model they described as in use at hundreds of US hospitals, across 38,455 hospitalizations of 27,697 patients at Michigan Medicine.
At the alert threshold of 6, the model did not flag 1,709 of the patients with sepsis, 67%, while alerting on 18% of all hospitalizations. Its positive predictive value was 12%.
Its area under the curve was 0.63, which the authors called "substantially worse than the performance reported by its developer." Epic's internal documentation had reported 0.76 to 0.83.
Did the sepsis model fail outside the hospital that built it?
Close, with one correction to how the finding is often retold. The model was built by Epic, an EHR vendor, not by a hospital. Michigan Medicine was an independent site checking a vendor's model on its own patients.
The finding is also from 2021, not a new result. What is new is the regulatory context around it, and the point holds: a model's reported accuracy describes the data it was developed on, and it can move sharply on a different population.
What did HTI-5 change about AI transparency in certified health IT?
ONC's HTI-1 rule, released in December 2023, required certified health IT to support 31 source attributes for predictive decision support interventions, a model card, plus risk-management practices, from January 1, 2025.
The HTI-5 final rule, issued October 26, 2026, narrowed the decision support interventions criterion to remove those AI model card and risk-management requirements. It removed 14 certification criteria, not the 34 ONC had proposed.
Two points are often missed. HTI-5 is a certification rule: it changes what EHR software must support, not what hospitals may do. And it did not remove privacy and security criteria; the AHA notes those stay and may be updated later.
Why does local validation of clinical AI matter more after HTI-5?
Because the one federal mechanism that put a predictive model's development data and validation details in front of clinicians, inside the EHR, is no longer a certification requirement. Vendors may still publish them, but nothing in certification asks them to.
The Michigan result shows what that disclosure could not do on its own anyway. A model card states how a model performed on its developer's data. Only a test on the health system's own patients shows a gap like 0.63 against 0.76.
So the durable control is local: a fixed set of the health system's own cases, scored every time a model or its version changes, with the results kept where the health system can audit them years later.
What does a health system need to validate AI models on its own patients?
Four things, none of them exotic:
| Need | Why | On ibl.ai |
|---|---|---|
| A local evaluation set | Reported accuracy describes someone else's patients | Evaluations, with LLM-as-judge scoring and human annotation |
| A re-run on every change | A new model version is a new model | The same evaluation set, re-scored per model |
| A record of every run | Reviews happen long after the encounter | Audit logs and transcripts |
| Data that stays put | Validation needs PHI | The stack runs in the health system's environment |
Evaluations, Audit and Transcripts are existing ibl.ai capabilities. They apply to agents and models running on ibl.ai; a model embedded inside an EHR still has to be validated through that EHR's own tooling and data exports.
For the observability side of the same argument, see you cannot govern a clinical model you cannot observe.
How does owning the stack help a health system govern clinical AI?
Validation is only as durable as the place its evidence lives. If the evaluation sets, the logs and the model versions sit with a vendor, the health system's governance depends on that vendor's retention and terms.
On ibl.ai you own all the code and the data, run it model-agnostic across any LLM, and pay with no per-seat pricing, so you can deploy anywhere, from your own cloud to a fully air-gapped network. Swapping a model means re-running your own evaluation set first.
ibl.ai is family-owned and operated from New York, NY — a U.S.-headquartered, domestically-owned long-term partner, not a vendor that sells licenses and moves on. Book a 30-minute demo and we will set up an evaluation set on one of your agents.
Sources: sepsis model figures from Wong et al., "External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients," JAMA Internal Medicine, 2021; HTI-1's 31 source attributes and compliance date from Arnold & Porter; HTI-5's date, model card removal and criteria counts from the American Hospital Association, and the risk-management removal from Akin Gump. The brief said the model failed "outside the hospital that built it"; Epic, a vendor, built it, and the study dates from 2021. It also described HTI-5 as dropping "AI transparency rules from EHR certification"; it removed the predictive AI model card and risk-management requirements, a narrower change.