Agentic EHR workflow reaches high accuracy on pediatric orthopedic queries
Electronic health records hold the clinical detail investigators need, but extracting it at scale remains slow, inconsistent, and heavily dependent on analyst time. In this preprint from Boston Children’s Hospital, investigators built an agentic large language model pipeline designed to turn investigator-authored questions into executable data requests across enterprise EHR systems. The workflow did more than search charts. It decomposed each request into source-specific tasks, generated and executed SQL against structured EHR data, integrated structured and unstructured evidence, and then applied study-specific clinical note adjudication to sharpen cohort identification.
The target problem is familiar in pediatric research. Orthopedic studies often depend on details buried in operative notes, follow-up documentation, and problem lists that do not map neatly onto a single coded field. When those data are dispersed across the record, even experienced analysts can miss eligible patients or include patients who do not meet the exact definition. An automated system that can interpret research intent and then move across both tables and notes has obvious appeal, especially in a hospital environment where query volume is high and the data model is complex.
The investigators evaluated the workflow against cohorts identified by data analysts across 50 pediatric orthopedic clinical data query requests. Of those, 43 were included in manual-review-adjusted performance analyses. The primary performance metrics were strong: the median adjusted positive predictive value was 0.93, with a 95% confidence interval of 0.88–0.98. Median sensitivity among patients identified by either approach was 0.92, with a 95% confidence interval of 0.90–0.95. The corresponding median F1 score was 0.90, with a 95% confidence interval of 0.87–0.94.
Those numbers matter because they speak to balance, not just precision. A query system that is highly selective but misses eligible patients will fail researchers who need complete cohorts. A system that captures everyone but floods the output with false positives creates a different kind of workload, shifting effort from query building to manual cleaning. Here, the pipeline performed well on both fronts, which suggests it was not simply memorizing common query patterns but was actually coordinating multiple data sources in a clinically meaningful way.
The architecture reflects how EHR evidence is used in practice. Structured data can identify diagnoses, procedures, encounters, and medication exposure quickly, but many research definitions require context from narrative notes. Clinical note adjudication is the difference between a coarse approximation and a defensible cohort definition. In orthopedic research, where laterality, surgical intent, postoperative course, and complication status can be described in free text rather than coded fields, that step can change who is counted and why. By combining SQL generation with note-level review, the pipeline aimed to reduce the gap between a clinician’s question and the database objects needed to answer it.
One sentence captures the operational implication: the system translated natural-language research intent into autonomous multi-stage EHR workflows.
The study also speaks to a broader shift in clinical informatics. Large language models are often discussed as tools for drafting, summarizing, or retrieving text, but this project positions them as orchestrators of a research pipeline. That is a more demanding role. The model had to decompose a request, identify which source held the needed evidence, create executable queries, and then reconcile heterogeneous outputs into a final cohort. Each of those steps introduces failure modes, especially when the request is ambiguous or when chart language does not align cleanly with database structure. High performance in this setting suggests the approach may be able to absorb some of the friction that usually slows retrospective studies.
There is another practical advantage embedded in the design: the pipeline was evaluated in an enterprise environment rather than an artificial benchmark. Real EHR systems are messy, and pediatric orthopedic data are not arranged for the convenience of machine parsing. Queries often span procedure histories, follow-up intervals, injury descriptions, and documentation styles that vary by clinician. A workflow that can navigate that variability without collapsing into either over-inclusion or under-capture has direct value for research teams trying to move faster without sacrificing traceability.
Still, the evidence here remains tied to a single institutional setting and a specific clinical specialty. The data were drawn from 50 pediatric orthopedic query requests, and 43 were used in the manual-review-adjusted analyses. The performance profile is encouraging, but the scope is narrow enough that generalization to other specialties, other institutions, and other kinds of research questions cannot be assumed from these results alone. Orthopedics may be a particularly favorable test bed because many questions combine structured procedures with narratively documented context; other domains may strain the system differently.
The clinical question now is not whether language models can produce text about EHR data. It is whether they can safely and reproducibly support research-grade cohort construction across real hospital data, where every missed inclusion criterion and every false-positive chart review carries downstream consequences for study design. This preprint suggests that an LLM-driven EHR pipeline can do that work with high agreement against analyst-identified cohorts, but the next step will require broader validation, additional sites, and careful testing across query types that are less forgiving than pediatric orthopedics.