Healthcare AI guardrails begin long before deployment

Healthcare organizations are racing to adopt artificial intelligence, but the real constraint is often hiding in plain sight: the data feeding those systems. During a Sunday panel at HLTH 2026, executives and data leaders argued that health systems cannot safely scale AI by treating data as an afterthought. The operational message was blunt. If the inputs are inconsistent, incomplete or poorly governed, the outputs will be unreliable no matter how polished the interface looks.

Charlie Harp, CEO of Clinical Architecture, drew a sharp line between organizations that simply buy technology and those that manage the information infrastructure required to support it. He said healthcare has a long habit of deploying new tools before they are ready, then dismissing the category when the results disappoint. In his view, AI only works responsibly when organizations move from being “data orderers” to data stewards—a shift that means treating information quality as a clinical and operational responsibility, not a back-office technical detail.

The stakes are especially high in healthcare because AI systems are being asked to support decisions that affect diagnosis, treatment, utilization, patient communication and administrative workflows. Poor data quality can distort every one of those functions. A model trained on fragmented records, mislabeled diagnoses or outdated medication histories may appear intelligent while quietly amplifying error. In practice, the danger is less about a dramatic malfunction than a steady accumulation of small mistakes that erode trust, waste staff time and create avoidable clinical risk.

Nasim Eftekhari, chief AI and analytics officer at City of Hope in Duarte, California, said every healthcare application needs dozens of checkpoints to confirm that AI tools are functioning as intended and to trace problems when they arise. Her description reflects a basic engineering truth: a model should not be treated as a black box that simply gets approved once and then left alone. Health systems need monitoring layers that can detect drift, flag unusual outputs and document where a failure began.

Guardrails in this setting are not decorative policy statements. They are the difference between a tool that can be audited and one that cannot be trusted.

Eftekhari also pushed back on the alarmist narrative that AI agents are inherently dangerous because they may “escape” and behave unpredictably. In her framing, most of the high-profile failures people fear can be traced to bad engineering rather than some mysterious autonomy inside the software. That point matters for health systems eager to separate solvable implementation problems from genuine biological or clinical uncertainty. When AI behaves badly in a healthcare workflow, the cause is often traceable: weak validation, limited testing, bad data pipelines or a lack of oversight.

The panel’s caution extends to staffing as much as software. Ian Blunt, vice president of advanced analytics at Pittsburgh-based Highmark Health, said health systems should hire people who understand AI and, just as importantly, understand its limits. That is a practical workforce issue, not a branding exercise. Hospitals and insurers need employees who can ask whether a model was trained on the right population, whether its predictions generalize across settings, and whether its recommendations are being used in ways that stay inside clinical and regulatory boundaries.

In healthcare, the confidence placed in an algorithm can rise faster than the evidence behind it. A system may look efficient on day one and still fail under real-world pressure if no one is checking the data stream, the workflow fit or the downstream effects on patients and staff. The panelists’ comments pointed to a common weakness across the industry: organizations often buy capability before they build governance.

That is especially risky now, as AI moves from narrow administrative tasks into more clinically consequential settings. Once a tool begins influencing triage, documentation, prior authorization, care navigation or risk stratification, data quality becomes inseparable from patient safety. Better guardrails mean better lineage, stronger validation and faster escalation when the system behaves unexpectedly. It also means resisting the temptation to equate adoption speed with progress.

Health systems that want durable returns from AI will need more than vendor promises. They will need internal expertise, explicit checkpoints, continuous monitoring and a culture that treats data as an asset requiring stewardship. Without those basics, the industry risks repeating a familiar cycle: rapid rollout, poor performance and a public backlash that blames the technology more than the process built around it.

The remaining question is not whether AI can be useful in healthcare. It already is in many settings. The harder operational test is whether organizations are willing to build the data discipline, technical oversight and human expertise needed to keep those tools from failing in the places where failure matters most.


Reporting by Dr. A.R. Athira, MBBS for Medicine World Foundation, with additional data provided by Modern Healthcare.