--- title: "HTI-5 Dropped Model Cards for Predictive AI. The Sepsis Lesson Says Validate Locally Anyway" slug: "sepsis-model-local-validation-hti-5-model-cards" author: "Jaione Amigot" date: "2026-11-05 10:00:00" category: "Premium" topics: "HTI-5 final rule, predictive decision support interventions, AI model cards healthcare, Epic Sepsis Model, sepsis prediction validation, local validation clinical AI, ONC certification, clinical AI governance, healthcare AI transparency, own the code" summary: "An independent validation found a widely used sepsis model missed 67% of sepsis cases at a hospital that had not built it. ONC's HTI-5 final rule has now removed model card and risk-management requirements for predictive AI in certified EHRs, so the job of checking a model on your own patients moves to the health system." banner: "" thumbnail: "" linkedin: | In 2021, University of Michigan researchers checked a widely used sepsis prediction model on 38,455 hospitalizations at their own hospital. It missed 67% of the patients who had sepsis, while alerting on 18% of everyone admitted. Its AUC was 0.63, against the 0.76 to 0.83 its developer reported. On October 26, 2026, ONC finalized HTI-5. Among other changes, certified health IT no longer has to support model cards or risk management for predictive AI tools. HTI-1 had required 31 source attributes for them. Read together, the lesson is simple. A model's published numbers describe the place it was built. Whether it works on your patients is something only you can measure, and the federal checklist no longer asks your EHR vendor to help. So the evidence has to live with the health system: its own evaluation sets, its own logs, on infrastructure it controls. #iblai #HealthcareAI #ClinicalAI #AIGovernance #Sepsis --- ## The Short Answer **A sepsis model can score well where it was built and miss most cases elsewhere, and ONC's HTI-5 rule no longer makes certified EHRs disclose model cards for predictive AI. Health systems now have to validate models on their own patients, and on ibl.ai you own all the code and the data that validation runs on.** The independent check found the model missed [67% of sepsis cases](https://pmc.ncbi.nlm.nih.gov/articles/PMC8218233/) at Michigan Medicine. HTI-5 was [finalized on October 26, 2026](https://www.aha.org/news/headline/2026-10-26-onc-finalizes-certain-deregulatory-actions-related-health-data-technology-interoperability). ## What did the independent validation of the Epic Sepsis Model find? In June 2021, [Wong and colleagues](https://pmc.ncbi.nlm.nih.gov/articles/PMC8218233/) published in *JAMA Internal Medicine* an external validation of the Epic Sepsis Model, a proprietary model they described as in use at hundreds of US hospitals, across 38,455 hospitalizations of 27,697 patients at Michigan Medicine. At the alert threshold of 6, the model did not flag 1,709 of the patients with sepsis, 67%, while alerting on 18% of all hospitalizations. Its positive predictive value was 12%. Its area under the curve was 0.63, which the authors called "substantially worse than the performance reported by its developer." Epic's internal documentation had reported 0.76 to 0.83. ## Did the sepsis model fail outside the hospital that built it? Close, with one correction to how the finding is often retold. The model was built by Epic, an EHR vendor, not by a hospital. Michigan Medicine was an independent site checking a vendor's model on its own patients. The finding is also from 2021, not a new result. What is new is the regulatory context around it, and the point holds: a model's reported accuracy describes the data it was developed on, and it can move sharply on a different population. ## What did HTI-5 change about AI transparency in certified health IT? ONC's HTI-1 rule, released in December 2023, required certified health IT to support [31 source attributes](https://www.arnoldporter.com/en/perspectives/advisories/2024/01/onc-final-rule-focuses-on-ai-and-algorithm-transparency) for predictive decision support interventions, a model card, plus risk-management practices, from January 1, 2025. The HTI-5 final rule, [issued October 26, 2026](https://www.aha.org/news/headline/2026-10-26-onc-finalizes-certain-deregulatory-actions-related-health-data-technology-interoperability), narrowed the decision support interventions criterion to remove those AI model card and risk-management requirements. It removed 14 certification criteria, not the 34 ONC had proposed. Two points are often missed. HTI-5 is a certification rule: it changes what EHR software must support, not what hospitals may do. And it did not remove privacy and security criteria; the AHA notes those stay and may be updated later. ## Why does local validation of clinical AI matter more after HTI-5? Because the one federal mechanism that put a predictive model's development data and validation details in front of clinicians, inside the EHR, is no longer a certification requirement. Vendors may still publish them, but nothing in certification asks them to. The Michigan result shows what that disclosure could not do on its own anyway. A model card states how a model performed on its developer's data. Only a test on the health system's own patients shows a gap like 0.63 against 0.76. So the durable control is local: a fixed set of the health system's own cases, scored every time a model or its version changes, with the results kept where the health system can audit them years later. ## What does a health system need to validate AI models on its own patients? Four things, none of them exotic:
Need Why On ibl.ai
A local evaluation set Reported accuracy describes someone else's patients Evaluations, with LLM-as-judge scoring and human annotation
A re-run on every change A new model version is a new model The same evaluation set, re-scored per model
A record of every run Reviews happen long after the encounter Audit logs and transcripts
Data that stays put Validation needs PHI The stack runs in the health system's environment
Evaluations, Audit and Transcripts are existing ibl.ai capabilities. They apply to agents and models running on ibl.ai; a model embedded inside an EHR still has to be validated through that EHR's own tooling and data exports. For the observability side of the same argument, see [you cannot govern a clinical model you cannot observe](/blog/clinical-ai-governance-observability-hospitals-own-the-stack). ## How does owning the stack help a health system govern clinical AI? Validation is only as durable as the place its evidence lives. If the evaluation sets, the logs and the model versions sit with a vendor, the health system's governance depends on that vendor's retention and terms. On ibl.ai you own all the code and the data, run it model-agnostic across any LLM, and pay with no per-seat pricing, so you can deploy anywhere, from your own cloud to a fully air-gapped network. Swapping a model means re-running your own evaluation set first. ibl.ai is family-owned and operated from New York, NY — a U.S.-headquartered, domestically-owned long-term partner, not a vendor that sells licenses and moves on. [Book a 30-minute demo](https://cal.com/iblai/30min) and we will set up an evaluation set on one of your agents. *Sources: sepsis model figures from Wong et al., "External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients," [JAMA Internal Medicine, 2021](https://pmc.ncbi.nlm.nih.gov/articles/PMC8218233/); HTI-1's 31 source attributes and compliance date from [Arnold & Porter](https://www.arnoldporter.com/en/perspectives/advisories/2024/01/onc-final-rule-focuses-on-ai-and-algorithm-transparency); HTI-5's date, model card removal and criteria counts from the [American Hospital Association](https://www.aha.org/news/headline/2026-10-26-onc-finalizes-certain-deregulatory-actions-related-health-data-technology-interoperability), and the risk-management removal from [Akin Gump](https://www.akingump.com/en/insights/ai-law-and-regulation-tracker/onc-issues-hti-5-final-rule-removing-model-card-and-risk-management-requirements-for-predictive-decision-support-interventions). The brief said the model failed "outside the hospital that built it"; Epic, a vendor, built it, and the study dates from 2021. It also described HTI-5 as dropping "AI transparency rules from EHR certification"; it removed the predictive AI model card and risk-management requirements, a narrower change.* ## Why does owning the AI stack matter? **ibl.ai is the agentic AI platform where you own all the code and the data. You self-host the entire stack inside your own perimeter, run it model-agnostic across any LLM and switch anytime, and pay by usage with no per-seat pricing — so you can deploy anywhere: your cloud, on-premise, GovCloud, or fully air-gapped.** - **You own all the code and the data.** Full source code under a perpetual license, running on your infrastructure. Not API access to someone else's platform — the stack itself is yours. - **Model-agnostic.** Run any LLM — Claude, GPT, Gemini, Llama, Command, or your own fine-tune — and switch providers without rewriting the platform. - **No per-seat pricing.** Usage-based billing against a budget cap you set. Cost tracks what your organization actually uses, not how many people you employ. - **Deploy anywhere.** Your cloud, your VPC, on-premise, GovCloud, or a fully air-gapped network with no outbound connectivity. 1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University. ibl.ai is family-owned and operated from New York, NY — a U.S.-headquartered, domestically-owned long-term partner, not a vendor that sells licenses and moves on.