Querying a hospital data platform in natural language: at Charité, the commercial tool AIRE is right when it runs a traceable SQL query and wrong every time it does not, across 25 questions

A neuroradiology team at Charité in Berlin connected a commercial natural-language query tool, AIRE from the Israeli company Briya, to the data platform of its three hospitals and asked it 25 questions about a cohort of 1,594 patients with brain tumours. The numbers were almost always correct or verifiable when the tool ran a query recorded in its log, and consistently wrong when it answered without running one. The study is small, single-site and partly funded by the vendor, but its main lesson is solid and transferable: a number produced by an AI on health data is only worth something if you can see the query that produced it.

The context

Building a cohort, that is, finding every patient in a hospital's records who meets a set of criteria (a diagnosis, an age band, a surgery, a time window), is one of the first steps of almost every retrospective clinical study and every trial feasibility check. In most hospitals this is done with SQL queries (SQL is the standard language for querying databases) written by informaticians or data managers, which creates a queue and makes clinicians heavily dependent on a few people.

Large language models (LLMs, models trained on huge text corpora that can generate text or code) promise to remove this barrier: you ask the question in plain language, the system translates it into SQL (this is called text-to-SQL), runs the query and returns the result. Academic work on the topic is plentiful, with benchmarks such as EHRSQL built on public intensive-care databases. But, the authors point out, nearly all of these evaluations use research databases or prototypes, and rarely a commercial product installed in a hospital's real infrastructure, with its local coding, legacy tables and governance rules. That is the gap this preprint, posted on medRxiv on 10 November 2026 and not yet peer reviewed, sets out to fill.

The method

The setting is the Health Data Platform (HDP) of Charité – Universitätsmedizin Berlin: a pseudonymised mirror (identities are replaced by codes) of the clinical systems of three university hospitals, holding administrative data, encounters, diagnoses and procedures. The tool under evaluation, AIRE, was installed on a virtual server inside the hospital network: the language model and query components run on site and no patient data leaves. The underlying LLM is not named in the preprint. The vendor itself built a FHIR layer (FHIR is an international standard for exchanging health data) from the platform's tables; AIRE generates SQL queries that it runs through the Trino engine, can ask clarifying questions, and keeps an exportable log of every query (timestamp, SQL code, row count, status).

The test cohort includes adults with an ICD-10 C71 code (malignant neoplasm of the brain), used as a proxy for glioblastoma: 1,594 patients according to the reference query. The authors designed a battery of 25 questions in three blocks: stepwise cohort construction (adults, first diagnosis between 2018 and 2023, then surgical resection), characterisation (mean age, for example), and 17 more varied questions (re-resection rate, top ten diagnoses, sex × age cross-tabulation, radiotherapy exposure, time from diagnosis to death). For each question, a reference SQL query had been written by hand before the tool was asked. Permitted rephrasings only clarified definitions (fields, dates, code prefixes), never the expected value.

Each answer was put into one of five categories: exact; concordant after definition alignment; wrong but traceable (a query exists in the log and explains the error); wrong and unlogged (a plausible number with no query behind it); not answered. Unanswered or wrong questions were asked once more in a new conversation thread. Two raters independently classified the 37 answers; their agreement, measured by Cohen's kappa (a chance-corrected agreement index, 1 = perfect agreement), was 0.74.

The results

Of the 25 questions in the first round, 11 answers were exact, 1 concordant after alignment, 3 wrong but traceable, 6 wrong with no logged query and 4 unanswered. At the very first attempt, without rephrasing, only 6 of 25 answers were strictly exact, and the first number displayed was right in 9 of 25 cases. An example of a traceable drift: at the resection step, the tool found 1,248 patients instead of the expected 1,275, because its explicit list of procedure codes (the German OPS classification) missed deeper sub-codes.

The central finding emerges when both rounds are pooled (37 answers). Of the 23 answers backed by a logged query, 22 were correct or at least verifiable; the exception was a case where the display (84 patients) did not match the result of its own query (101). Conversely, all 8 numbers produced without a query were wrong, sometimes by a wide margin: 117 biopsy-only patients instead of 258, or 512 patients on chemotherapy (32.1%) instead of 142 (8.9%). The mean age of the same cohort was given as 64.9 years (no query), then 55.1, 53.2 and 52.3 years depending on the thread.

A simple clue gives these invented answers away: time. Answers with a query took a median of 64 seconds (interquartile range 57 to 81 seconds); those without took 3 to 6 seconds. No wrong unlogged answer reappeared identically in a new thread, which suggests, according to the authors, that the tool was reusing values accumulated in the conversation instead of running a new query.

In concrete terms: a researcher who asked these 25 questions and copied the first number shown would have obtained 9 correct values, and 6 wrong but credible values that left no trace. Applying a strict rule (accept no number without a visible query) would have filtered out every invented value, but the researcher would still have had to read the SQL of the traceable answers to catch the 3 definition errors in the first round. The tool saves writing time, not checking time.

What is good

First, a real and rarely documented setting. The tool runs inside the production infrastructure of a large hospital, with its coding and legacy tables, not on a demo database. The preprint also describes what press coverage leaves out: nearly two years passed between the project start in October 2024 and the first successful query in June 2026, including about nine months of regulatory work (a data protection impact assessment and a two-stage IT security review), whereas the vendor's installation took only two weeks once the server was available.

Second, an evaluation scheme that separates verifiable errors from invisible ones. Rather than reporting a single accuracy rate, the authors distinguish wrong-but-traceable answers from wrong answers with no trace. This is the distinction that matters in practice, and it leads to operational advice: short threads separated by topic, definitions restated in every question, refusal of any number without a displayed query, and re-running critical queries in a fresh thread.

Third, unusual transparency about failures. The authors report a default filter on diagnosis verification status that the instruction "regardless of status" did not switch off, a display layer that deduplicated rows and lost patients, and structurally identical questions that succeeded or failed depending on the thread. The questions, the generated and reference queries and the matching log entries are provided as supplementary material.

What is less good

First, very small numbers and a possible misleading metric. 25 questions, one phrasing per question, a single retry, no confidence intervals: "8 out of 8" and "22 out of 23" are striking observations, not estimated rates. Sensitivity to rephrasing, which is decisive for a free-text tool, was not tested, and the battery was built over six weeks without preregistration.

Second, a reference that is not clinical truth, a broad form of biased comparator. The gold standard is the SQL query written by the team, not a review of patient charts. If the platform is incomplete, the tool and the reference are wrong together: the authors themselves note that an in-hospital mortality of 6.3% is implausibly low for glioblastoma, a sign that deaths outside the hospital are missing from the data. The C71 code also covers more than glioblastoma. Finally, neither rater was blinded to the reference values, and the first rater had run the battery himself.

Third, uncertain generalisation and heavy conflicts of interest. One site, one vendor, one product version, one cohort: a typical case of population bias applied to data, since another data platform with different coding and tables could give very different results. Briya partly funded the study, supplied the platform free of charge, did the data mapping and reviewed the manuscript for factual accuracy; the first author received partial salary support from Briya, the last author reports research collaboration agreements with the company, and the second rater is a former Briya employee. The authors declare all of this clearly, but readers should keep it in mind. Because the underlying language model is not named, the experiment cannot be reproduced outside this product.

What it changes

For the research community, this preprint shifts the question. The average accuracy of a text-to-SQL system matters less than its ability to expose what it actually executed. It argues for evaluations that measure provenance (the verifiable link between a number and the query that produced it) and for tools that explicitly flag a missing query as a reliability warning, as the authors suggest to vendors.

For clinicians and hospital data teams, the lesson is practical and immediate: a conversational interface can speed up a feasibility study, but it replaces neither reading the SQL nor knowing the pitfalls of the data platform (missing codes, implicit filters, uncaptured deaths). An instant answer with no displayed query should be treated as an invention, not a result. And integration takes time: here, two years of governance and IT work for two weeks of installation.

For patients and the public, the stakes are indirect but real: cohorts built with these tools are used to decide which studies to run and which patients to approach for trials. Plausible but wrong numbers that leave no trace can steer those decisions without anyone noticing. The study shows that a simple traceability rule is enough to filter them out in this setting, provided it is applied.

Further reading

The preprint: Desser D, Brunecker P, Dell'Orco A, Galperin G, Wattjes MP, Nawabi J. "Natural-language cohort queries on a hospital data platform: accuracy and provenance in a real-world evaluation", medRxiv, 2026, DOI 10.64898/2026.11.09.26367645. Questions, queries and log excerpts are in Supplemental Material S1; full transcripts are available from the corresponding author on request, and patient-level data cannot be shared. For context: the EHRSQL benchmark (Lee et al.) and its 2024 shared task on reliable text-to-SQL, and Criteria2Query for turning trial eligibility criteria into queries.