Health systems are demanding post-launch accountability from AI vendors
The first hard lesson in clinical AI is simple: go-live is not the finish line. Once a tool enters daily practice, it meets shifting patient populations, changing workflows, new documentation habits, and the messy reality of front-line care. Those forces can erode performance long after a polished pilot has ended, leaving a system with software that still appears active but no longer behaves the way clinicians need it to.
Nirav Shah, associate CMIO for AI and innovation at Endeavor Health, describes that problem in biological terms. AI, in his view, behaves less like a fixed product and more like a living system that adapts, drifts, and changes under pressure. The concept is more than rhetoric. Model performance can slide when demographics shift, service lines expand or contract, clinical protocols change, or staff members develop workarounds to bypass a tool that no longer fits the workflow.
One sentence captures the operational risk:
A model can look deployed and still be failing at the bedside.
That failure often surfaces first through a quiet signal rather than a formal alert. A nurse may route around a recommendation. A physician may stop trusting a prompt. An administrative team may build manual steps to compensate for poor automation. Those responses can preserve throughput in the short term, but they also reveal that the AI no longer earns its place in care delivery. When workarounds become routine, the system is absorbing the burden of a tool that has fallen out of alignment with practice.
Shah said continuous validation has to remain a vendor obligation after launch. If accuracy degrades or if clinical staff begin building manual detours around the product, the vendor should be in the trenches fixing it. His team at Endeavor Health watches for concrete workflow problems and performance degradation, then gives the vendor a defined period to respond. If there is no real movement, the health system pivots.
Continuous validation is now part of the clinical contract
The timeframe matters because validation itself is not abstract. A few months can be enough to determine whether a model still performs as expected in a live environment. A vendor that cannot show measurable improvement within that window is effectively asking the health system to fund product development with clinician time and patient-facing effort. Shah rejected that arrangement outright. Health systems should not underwrite a vendor’s research and development on the backs of frontline teams or the patients those teams serve.
This approach reflects a broader shift in how health systems manage AI procurement. The old model of software delivery assumed that once a product was installed, the main work was over. Clinical AI breaks that assumption. Data inputs change, care pathways change, and the model itself can drift as it encounters patients who are not identical to the training population. In practical terms, the tool must keep proving that it still belongs in the workflow, not just in the contract.
Dr. Shah’s framing also points to a deeper clinical truth: when AI becomes part of diagnosis, documentation, triage, or patient communication, deterioration is not a software problem alone. It becomes a care-delivery problem. A poorly calibrated model can slow clinicians, obscure useful information, or push staff back toward manual processes that the technology was supposed to reduce. If the tool creates more friction than relief, the safety and efficiency promise quickly erodes.
Johns Hopkins wants vendors engaged after the contract is signed
Danny Lee, MD, chief medical informatics officer at Johns Hopkins Medicine Community Physicians, describes a similar expectation from the vendor side. The relationship does not end when the contract is signed; it begins there. Vendors are expected to stay engaged, gather feedback, and keep improving the product rather than treating launch as a final milestone.
His team is particularly interested in clinical-facing tools that can make everyday care easier for clinicians while supporting safer decision-making. One current example involves real-time clinical decision support that synthesizes information as a patient encounter unfolds. In that setup, the tool helps assemble the relevant data before the physician walks into the room, then offers suggestions during the visit itself. If questions arise in the conversation, the system can retrieve answers without forcing the clinician to leave the EMR.
That workflow matters because it removes friction at the exact moment clinicians feel it most. Decision support that requires multiple clicks, context switching, or separate log-ins can become just another burden. When integrated well, the tool can provide small, timely nudges that support clinical reasoning without interrupting the encounter. The promise is not replacement. It is compression of cognitive load.
Dr. Lee also described a practical filter for adoption: the technology should reduce burnout, make work easier, and help clinicians function more effectively. Patient safety is part of the same equation. If the product improves those dimensions, pilot enrollment becomes an easier decision. If it does not, it never gains traction.
AI tools in medicine now face a standard that looks less like installation and more like stewardship. Vendors are being asked to prove they can adapt after deployment, remain accountable when models drift, and respond quickly when workflow gaps appear. Health systems are no longer treating AI as a one-time purchase. They are treating it as an operational relationship that must be maintained, audited, and repaired when it starts to fail.
The unresolved issue is not whether AI will keep changing after launch. It will. The open question is whether vendors will accept continuous clinical accountability as part of the product itself, or whether health systems will keep walking away when the tool stops earning trust in real time.