Natural-language cohort queries worked best when the question was fully pinned down
A new hospital evaluation suggests that natural-language analytics can make clinical data access more approachable, but only when the system’s answers are tightly anchored to explicit definitions and a visible audit trail. In this proof-of-concept at a university hospital, investigators connected an on-premises platform, AIRE from Briya, to an institutional health data environment spanning three tertiary-care hospitals and tested it against a real-world adult ICD-10 C71-coded brain-tumor cohort of 1,594 patients.
The central tension is familiar to anyone who has tried to operationalize hospital data at scale. Clinicians and researchers want to ask questions in plain language, yet research-grade cohort work depends on rigid logic, code transparency, and reproducible filters that can withstand scrutiny months later. In this evaluation, those two worlds met inside a governed institutional setup that required regulatory review, pseudonymization, and substantial local IT work before the interface was usable.
The platform did not fail because the idea was wrong. It failed when language outpaced definition.
A real-world cohort test exposed where the interface held up and where it broke
The study used 25 natural-language prompts designed to identify and characterize the brain-tumor cohort. Each response was compared with manually written reference SQL, and investigators also checked whether a corresponding executed query appeared in the exported log. Responses were sorted into five categories, allowing the team to separate fully exact matches from answers that were technically traceable, plausible but wrong, or not answered at all.
Fifteen of the 25 responses landed in the usable range. Eleven were exact, one became concordant after definition alignment, and three were incorrect but still traceable to the logged SQL. Six responses were plausible yet incorrect and had no logged query, while four prompts remained unanswered. When the unanswered or incorrect items were reposed in a new thread, no incorrect unlogged result recurred, a finding that points to thread-level variability rather than stable but hidden system behavior.
Across the initial and repeat evaluations, 22 of 23 logged responses were either correct or traceable. By contrast, all eight unlogged responses were incorrect. That separation is the clearest signal in the entire dataset: provenance was not a decorative feature, but the difference between a query that could be audited and a result that could not be trusted.
Query provenance became the safety mechanism for a conversational data tool
The platform’s performance depended heavily on whether the request could be translated into an executable and visible SQL pathway. When definitions were explicit, the natural-language interface reproduced SQL-derived results. When the prompt left room for interpretation, the model could generate something that sounded reasonable while diverging from the reference logic.
In a hospital data setting, this matters because a cohort query is not just a search box exercise. A small change in inclusion criteria, diagnosis coding, temporal sequence, or variable definition can reshape the population under study and alter downstream inferences about treatment patterns, imaging findings, or outcomes. For brain tumor research, where code-based identification often serves as the first gate before manual chart review, even modest drift can cascade through an entire project.
The evaluation therefore reads less like a product demo and more like a warning label for adoption. Natural-language access may reduce technical friction for clinicians and researchers, but it does not eliminate the need for governance, validation, or a disciplined reconciliation step against SQL.
The implementation burden was substantial before a single prompt was tested
The platform was not simply switched on. Implementation took two years and included governance approval, regulatory work, and IT deployment across the institutional environment. That timeline is important because it shows that conversational querying may sit on top of a highly structured data ecosystem rather than replace one.
This kind of setup is consistent with the way clinical data infrastructures are usually built in large hospitals. The promise of natural-language access can be compelling precisely because the underlying systems are fragmented, heterogeneous, and technically demanding to navigate. Yet the cost of harmonizing those systems is borne before the interface ever reaches a user. A conversational layer can make retrieval easier, but it cannot create clean data, consistent terminology, or institutional trust from scratch.
The authors’ real-world testing also emphasized traceability as a design requirement rather than an optional enhancement. If a query cannot be tied to an executed log entry, it becomes difficult to know whether the answer is reproducible, whether it reflects a hidden mapping problem, or whether it merely mimics the structure of a valid response.
Clinical research teams will still need SQL literacy at the point of verification
A natural-language system may help researchers reach first-pass answers faster, but this evaluation shows that final verification still depends on conventional methods. The reference standard remained manually written SQL, and the platform’s best performances occurred when prompt wording and cohort definitions were aligned tightly with that standard. The fact that all unlogged responses were incorrect raises a practical concern: an interface can appear fluent while producing output that should not be used without checking.
For hospital research teams, the bedside implication is indirect but real. Data tools increasingly shape which patients are counted, which cohorts are analyzed, and which hypotheses are even considered. If a conversational interface lowers the barrier to querying but weakens the link to provenance, it can accelerate error as efficiently as it accelerates discovery. The safest path is not to replace structured querying, but to wrap it in a system that exposes exactly how the answer was generated.
A single-sentence takeaway fits the data: logged queries were usually reliable, unlogged ones were not.
The main limitation is not the platform concept but the narrow proof-of-concept design
This was a small, single-center proof-of-concept centered on one adult brain-tumor cohort and 25 prompts. The sample is far too limited to establish general performance across disease areas, hospitals, or query complexity. The data also reflect a controlled institutional deployment with local mapping support from the vendor, which may not resemble a less supported installation elsewhere.
Still, the study offers a concrete framework for future evaluations. The question is not whether natural-language cohort querying sounds useful; it is whether a system can preserve auditability when language becomes the front end of clinical data retrieval. Until then, the most clinically relevant feature may be the simplest one: can every displayed answer be traced back to an executable query, and does that query match the intended cohort logic?