Transfusion reactions: three open-weight LLMs extract NHSN protocol signs from clinical notes with few false alarms, but miss one in four at best, on 50 notes and without an independent reference
A team from North America and India built a pipeline of open-weight large language models (LLMs) that reads clinical notes and extracts, item by item, the signs that define a transfusion reaction under the US NHSN hemovigilance protocol. On 50 notes reviewed by two physicians, the three models tested are very rarely wrong when they flag something (precision above 0.98), but the best one misses about one item in four and the other two more than one in three. It is an honest, reproducible proof of feasibility, not yet a surveillance tool: the sample is small, drawn from suspected cases, and the reference was built by reviewing the models' own outputs.
The context
A transfusion reaction is an adverse event linked to the transfusion of a blood product: fever, allergic reaction, hemolysis, circulatory overload (TACO, transfusion-associated circulatory overload) or acute lung injury (TRALI, transfusion-related acute lung injury). In the United States, the Hemovigilance Module of the NHSN (National Healthcare Safety Network, run by the CDC) provides standardized case definitions for twelve reaction types, based on signs, symptoms and laboratory results.
The problem is well known: most reactions are never reported. The authors cite an electronic surveillance study in which only 11 of 78 TRALI cases and 5 of 45 TACO cases detected by an algorithm had been reported to the blood bank. The clues often exist, but as free text in clinicians' notes, which neither structured-data queries nor keyword searches exploit well. Hence the idea of using LLMs, models trained on large amounts of text and able to extract information from it on instruction, to turn these notes into usable variables. Tatakoto has already covered related attempts, such as extracting radiotherapy side effects from recorded consultations or identifying non-epileptic seizures in clinical notes.
The method
The authors first translated the NHSN protocol (version 3.0) into a vocabulary of 181 items: 93 clinical or laboratory signs drawn from the definitions of the twelve reactions (dyspnea, pruritus, elevated LDH, positive blood culture, bilateral infiltrates, etc.) and 88 "pertinent negatives", meaning an explicit statement that a sign is absent, which helps rule a reaction out.
The pipeline is called "multi-agent", but every stage uses the same model with a different instruction (prompt). Stage 1 identifies items as present or absent, with the supporting quote from the text; answers are matched to the vocabulary deterministically and any off-list term is discarded. Stage 2 places each item in time relative to the transfusion, or marks it "uncertain". Stage 3 fills in the detailed NHSN schema variables, accepting only allowed values. Two exploratory stages (treatments, medical history) were not evaluated.
Three open-weight models (whose parameters can be downloaded) were compared: Qwen 3.6-27B, GPT-OSS-120B and Kimi K3. The first two ran at temperature 0 (near-deterministic answers), Kimi K3 at temperature 1.0, a value fixed by the provider. They were not run locally but through the online services Groq and OpenRouter, configured, according to the authors, for zero data retention.
The evaluation covers 50 notes: 10 expert-written scenarios from a validation study of the NHSN module published in Transfusion (Fung et al.), 20 published case reports from PMC-Patients, and 20 hospital discharge summaries from MIMIC-IV, a database from Beth Israel Deaconess Medical Center in Boston. Real cases were pre-selected by keywords, then triaged by an LLM and by the senior author; 18 development cases were excluded.
There is no independently established reference annotation. Two physicians reviewed every output of the three models in a blinded interface (anonymized models, random order): they accepted or rejected each item and could add those the models had missed. The primary analysis keeps only items on which both physicians agreed. The metrics are sensitivity (share of items judged present that the model found), precision (share of extracted items that are correct) and a "specificity" defined here as the share of absences stated by the model that were confirmed.
The results
The two physicians agreed on 1,201 of the 1,343 items they both rated (89.4%), with a Krippendorff's alpha of 0.73 (a chance-corrected agreement index, 1 = perfect agreement). On these concordant items, precision exceeds 0.98 for all three models. Sensitivity is 0.774 (95% confidence interval: 0.745-0.800) for Kimi K3, 0.636 (0.604-0.667) for GPT-OSS-120B and 0.624 (0.593-0.655) for Qwen 3.6-27B, giving F1 scores (the harmonic mean of precision and sensitivity) of 0.87, 0.78 and 0.77. "Specificity" ranges from 0.95 to 0.99. Adding back the discordant items barely changes the figures (0.78, 0.65 and 0.62) and not the ranking.
Errors are almost all omissions: 196 false negatives for Kimi K3, 317 for GPT-OSS-120B and 354 for Qwen, of which 95 to 100% are pure omissions. False positives are rare: 2 for Qwen, 4 for GPT-OSS and 12 for Kimi K3 in the main text, but 6 in a supplementary table, an inconsistency the preprint does not resolve. Misses mostly involve laboratory criteria and complex definitional criteria ("unexplained clinical illness consistent with infection"), and fine distinctions such as hematuria versus hemoglobinuria. Kimi K3's sensitivity ranges from 0.68 on expert scenarios to 0.82 on published cases. Timing is rated correct or reasonable in more than 92% of cases, clinical details in more than 97%.
In concrete terms, by our calculation from the published false negatives: on a typical note, Kimi K3 leaves out about four protocol items, the other two models six to seven. Out of 100 items judged present by the physicians, Kimi K3 finds about 77, the others about 63. Conversely, out of 100 flagged items, fewer than 2 are wrong. The tool is therefore reliable when it asserts and incomplete when it stays silent: for active surveillance, omissions are what cost, since one missing criterion can push a case outside an NHSN definition.
What is good
First, a clinically well-anchored task. Rather than asking the model "was there a reaction?", the authors have it work on the building blocks of the official NHSN definitions, with a supporting quote and timing. Every output becomes checkable by a human.
Second, a careful human evaluation that is transparent about its numbers. Two physicians blinded to model identity, inter-rater agreement reported with its confidence interval, a sensitivity analysis that adds back disagreements, results by source and an analysis of error types.
Third, genuine reproducibility. All three models are open-weight; the code, prompts, the 181-item vocabulary and the MIMIC-IV note identifiers are published on GitHub, and the other two sources are public.
What is less good
First, a misleading metric built into the reference itself. It is not an independent annotation but a review of the models' outputs: physicians could add misses, but an item that no model saw and the reviewers did not catch never enters the denominator, which tends to overestimate sensitivity. The interface also propagated judgments from one model tab to another, reducing the independence of the ratings. The primary analysis drops the 10.6% of discordant items, often the most ambiguous. Finally, the reported "specificity" measures the reliability of stated absences, not the ability to flag nothing when nothing is present: comparing it with a classic diagnostic specificity would be a mistake, as the authors acknowledge.
Second, selection and population bias. Fifty notes, pre-selected as likely reactions, with a single reaction assumed per note: we do not know how many actually describe a reaction, and the real task (finding a few reactions among thousands of notes from transfused patients) is not tested. The MIMIC-IV notes come from a single US academic hospital, and published case reports are by nature more complete and better written than routine notes. They are also public, and so may have been in the models' training data, a possible form of data leakage the study does not examine; it is on these cases that sensitivity is highest.
Third, an absent comparator. No external baseline is tested: no keyword rules, no single prompt without stages, no manual abstraction by a hemovigilance professional. So we do not know what the multi-stage pipeline adds, or whether 77% sensitivity beats current practice. Conditions are not identical across models either (temperature 1.0 for Kimi K3 versus 0 for the others), each note was processed only once, and run-to-run variability is not measured. On data handling, MIMIC-IV notes were sent to online services; the authors cite PhysioNet's rules and a zero-retention configuration without saying how it was verified. One author also declares having worked as a contractor contributing to OpenAI projects on the safety of health answers; no funding is declared.
What it changes
For the research community: this work provides a structured vocabulary, code and an evaluation protocol that can be reused for LLM-based hemovigilance. The logical next step is an evaluation on an unselected stream of notes from transfused patients, with an independent reference annotation established before seeing model outputs, and a comparison with current surveillance methods.
For clinicians and hemovigilance teams: nothing changes in practice today; the authors themselves state that the pipeline will need tuning before being connected to a reporting database. The error profile, very few false alerts but many misses, points to use as a review aid, where the tool pre-fills a record that a human completes, rather than as an autonomous filter.
For patients and the public: under-reporting of transfusion reactions is a real blind spot in patient safety, and this kind of tool could eventually help count them better. But a model that misses one sign in four at best cannot yet serve as a safety net.
Further reading
The preprint: Multi-Agent Generative Artificial Intelligence for Transfusion Reaction Feature Extraction, Kumari et al., medRxiv, posted 9 November 2026, DOI 10.64898/2026.11.06.26367389. Not peer reviewed.
Code, prompts and vocabulary: the authors' GitHub repository (license not stated in the preprint). Case definitions: NHSN Hemovigilance Module, CDC, version 3.0. Data: MIMIC-IV-Note (PhysioNet, credentialed access) and PMC-Patients.
Editorial transparency note: this article was produced by Tatakoto's automated editorial pipeline from version 1 of the preprint; the supplementary tables could not be consulted. The numbers of missed items per note and per 100 items are our calculations from the published figures. The English, Spanish and Chinese versions are AI-generated. Spotted an error? Contact us.