Pediatric emergencies: the Dr CaBot agent matches four emergency physicians on 100 admitted children, but on an endpoint near its ceiling
A team from Boston Children's Hospital and Harvard gave the records of 100 children who came through the emergency department and were then admitted to a large language model agent, Dr CaBot, and to four pediatric emergency physicians, revealing the information step by step. At the initial physician evaluation, both the agent and the physicians produce a differential containing the correct diagnosis, or a very close one, in 95% of cases, and the first-listed diagnosis is right in about six cases out of ten. The result is serious because it uses real records, but the endpoint sits near its ceiling, the notes were written by specialists and the children were already admitted: we do not yet know whether the agent would help where diagnostic errors actually happen.
The context
In the emergency department, a child who is misdiagnosed or diagnosed late can come back sicker or suffer an avoidable complication, especially where there is no pediatric emergency physician, which is the case in most general emergency departments. Hence the interest in large language models (LLMs, neural networks trained on huge text corpora that can produce written reasoning) as an aid to differential diagnosis, the ranked list of conditions compatible with what is observed.
Most evaluations have relied on published vignettes, often present in the models' training data. Real emergency records had been tested in adults, not in children, whose presentations depend heavily on age and on what parents report.
The tool tested, Dr CaBot, is an agent (an LLM that chains several steps on its own, here literature searches followed by reasoning) introduced in 2025 by part of the same Harvard team and designed to reason in the style of clinicopathological conferences.
The method
The study is retrospective and single-centre: it covers emergency visits at Boston Children's Hospital between 1 July 2024 and 30 June 2025 that were followed within 24 hours by an inpatient admission or an operation. Children cared for by one of the authors were excluded. The researchers randomly drew 25 visits from each of four age bands (under 1 year, 1 to 4 years, 5 to 11 years, 12 to 17 years), for 100 children: median age 4 years, 45% girls, 41% with a complex chronic condition, including 10 transferred from another hospital.
Information was extracted from the electronic health record and revealed in four stages: (1) triage, with the nursing note and vital signs; (2) the initial physician evaluation, with history of present illness, past history and physical examination, defined in advance as the primary endpoint; (3) the emergency department stay, with the physician's reasoning, labs, imaging and treatments; (4) the admission note written by the inpatient or surgical team. At each stage, each "diagnostician" proposed up to five ranked diagnoses.
Dr CaBot ran on OpenAI's GPT-5.4 (reasoning effort set to "high"), through an Azure endpoint compliant with US health data law (HIPAA), with a prompt (the initial instruction) casting it as a physician in the hospital's pediatric emergency department. Two configurations ran independently: one could query 1.6 million clinical abstracts, up to 25 searches per case; the other had no literature access. On the human side, four attending pediatric emergency physicians read the same data, in the same order, with no time limit and without seeing the agent's output.
The reference (the "true" diagnosis) was set by two other pediatric emergency physicians from the full record, including the discharge summary. They then scored each list with the Bond score, a 0 to 5 scale: 5 if the exact diagnosis is in the list, 4 for something very close, 3 for something closely related that might have helped, 2 for something related but unlikely to help, 0 if nothing comes close. A score of 4 or more counts as a "high-scoring" list. To mask their origin, all lists were rewritten by GPT-5.4 into spelled-out terms and presented in random order.
For each case, the authors compare the median score of the four physicians with that of the two agent configurations, using McNemar's test (suited to paired comparisons). The power calculation, based on an earlier adult study, targeted a 10-point difference and required at least 77 visits.
The results
At the initial physician evaluation, the primary endpoint, the physicians and Dr CaBot each produce a high-scoring list (Bond score of at least 4) in 95 cases out of 100. The difference is zero, with a 95% confidence interval from −5.8 to +5.8 points. At the other stages the figures remain very close: 91 versus 91 at triage, 96 versus 98 after the emergency department stay, 98 versus 99 at admission (physicians then agent), with no significant difference.
A stricter criterion: is the first-ranked diagnosis correct? Physicians achieve this in 53, 61, 68 and 73 cases out of 100 at the four stages, the agent in 50, 62, 74 and 76 cases, again with no significant difference. Giving the agent access to the 1.6 million abstracts did not detectably improve its performance.
Agreement between reviewers is imperfect (weighted kappa of 0.64, a chance-corrected agreement measure where 1 is perfect). And on 34 cases, reviewers guessed the origin of the lists 91.6% of the time: the agent's lists were longer and more esoteric.
Clinical translation, from the study's own figures: out of 100 children admitted after an emergency visit, at the point where the physician has just examined the child, about 5 have neither the correct diagnosis nor a very close one on the list, whether the list comes from the agent or the physicians; and for 38 to 39 children, the first-listed diagnosis is wrong. The study does not say whether the same children trip up both sides, nor whether combining the two would do better.
What is good
First, real data revealed in the real order. Hospital records from 2024-2025, never published, reduce the risk that the model saw the cases during training. The four stages, with a primary endpoint fixed in advance, show how performance evolves as information arrives.
Second, a declared methodology. Age-stratified random sampling, a justified sample size, a reference set by experts outside both arms, lists rewritten to mask their origin, reporting according to the STARD-AI guideline. The authors even publish the result that works against them: masking did not work.
Third, a set-up close to a possible deployment: a HIPAA-compliant endpoint, and a comparison with and without literature search showing that the simpler tool is enough here. Funding is public (NIH, grants T32HD040128 and R01DK137993), philanthropic (Dunleavy Foundation) and institutional, and competing interests are declared.
What is less good
First, a population bias combined with a form of information leakage. Only admitted children, in a large academic pediatric centre, were included. Yet many diagnostic errors and delays involve children sent home, and most children are seen in general emergency departments. Above all, the agent reads notes from expert pediatric emergency physicians, sometimes written after the work-up had begun: the physician's reasoning is already in them, as the authors acknowledge. Both the machine and the physicians being compared benefit from diagnostic work already under way, which inflates everyone's performance.
Second, a misleading metric through a ceiling effect. When 95% of lists are rated high-scoring on both sides, there is very little room to see a difference. The study was powered to detect a 10-point gap; it tests for a difference, not non-inferiority (showing that one tool is not worse than another beyond a margin fixed in advance). An interval of −5.8 to +5.8 points therefore does not establish equivalence. The Bond scale does not weight severity either: missing meningitis is not the same as missing gastroenteritis, and the study reports no error analysis, no serious missed diagnoses and no results by age.
Third, an unusual comparator and failed masking. The human arm is the median of four physicians, the machine arm that of two configurations: two aggregates are being compared, not one emergency physician against a tool. Physicians read notes, without seeing the child or facing time pressure, far from a real shift. And since reviewers recognised the origin of the lists more than nine times out of ten, their scores may have been influenced, in either direction. The agent's longer lists could also prompt unnecessary tests, a cost that was not measured. Two authors declare ties to Google (one employee of Google LLC, one visiting researcher at Google DeepMind), while the model tested comes from OpenAI.
What it changes
For the research community, this preprint offers a reusable protocol (real records, information revealed in stages, analysis planned in advance) and shows the limits of the Bond score when both arms are at the ceiling. Logical next steps: include children sent home, test notes from general emergency departments, measure serious errors and induced tests, then prospectively evaluate a physician assisted by the tool.
For clinicians, nothing changes today. Dr CaBot is a prototype with no regulatory status, and the study does not test a physician aided by the tool. It does indicate that, on well-documented records, a general-purpose LLM proposes differentials comparable to those of specialists: a serious candidate for a "second opinion" role, provided it does not multiply false leads.
For patients and the public, the promise is to bring pediatric-level reasoning to emergency departments without a pediatrician. The study shows that the tool does not lag behind specialists under the most favourable conditions, not yet that it reduces errors where they occur.
Further reading
The preprint: Agentic AI versus pediatric emergency physicians for differential diagnosis of acutely ill and injured children, Georgette, Buckley, Fine et al., Boston Children's Hospital and Harvard Medical School, medRxiv, posted 18 November 2026, DOI 10.64898/2026.11.16.26367440. Source data cannot be shared; anonymised scores are available on request.
The original agent: Advancing Medical Artificial Intelligence Using a Century of Cases, Buckley et al., arXiv, 2025.
On Tatakoto: an LLM judge calibrated on a physician panel to evaluate clinical reasoning and ChatGPT Health triage in multi-turn conversations.
Editorial transparency note: this article was produced by Tatakoto's automated editorial pipeline from version 1 of the preprint (medRxiv PDF, main text; supplementary tables not consulted). The "out of 100 children" conversion uses the published proportions directly. The English, Spanish and Chinese versions are AI-generated. Spotted an error? Contact us.