médical IA

Predicting ICU delirium from one hospital to another: what transfers, and what does not

Published on August 3, 2026 · 9 min read

Shengliang Ni and Kengo Sato (Institute of Science Tokyo) train five model families to predict ICU delirium on two large critical-care record databases — eICU, aggregated across 208 US hospitals, and MIMIC-IV, from a single Boston center — then move them between databases to see what survives the change of hospital. Risk ranking transfers reasonably well (external AUROC up to 0.88–0.95 in one direction), but probability calibration and the alert threshold do not. The lesson is not a number: it is a distinction — a clinical model's "transportability" breaks into layers that do not travel together, and mistaking good ranking for a deployable tool is the trap this work dismantles.

The context

Delirium is an acute, fluctuating disturbance of attention and awareness, very common in intensive care, often underdiagnosed, and tied to worse outcomes. It cannot be read off a lab value: it is detected by repeated bedside instruments, the CAM-ICU (Confusion Assessment Method for the ICU) and the ICDSC (Intensive Care Delirium Screening Checklist). A predictive model is appealing for one reason: flag at-risk patients early so surveillance and prevention can be reinforced.

The recurring problem with such models is distribution shift: a model trained in one hospital performs worse elsewhere because populations, documentation practices, and disease rates differ. Most published work stops at internal validation (same hospital, held-out data) and reports a flattering AUC. Ni and Sato ask a sharper question: when a model actually changes database, what degrades, and does everything degrade the same way? Their answer is that transportability is not all-or-nothing but a stack of layers — target definition, risk ranking, probability calibration, alert policy — that do not transfer at the same rate.

The method

The authors use two public critical-care databases. eICU covers 28,321 stays (2014–2015, 335 units across 208 US hospitals), i.e. 2,496,432 time windows; MIMIC-IV covers 49,828 stays from a single Boston center (2008–2022), i.e. 5,881,343 windows. A "window" is a moment at which the model, given the patient's recent history, must predict the near future. The primary target is persistent or recurrent delirium within the next 4 hours, a positive point counted as persistent/recurrent if the observed episode exceeds 48 hours or recurs after a gap of at least 48 hours.

A defining fact: delirium assessments are sparse and uneven. An assessed state is available in only 15.3% of eICU windows versus 7.8% of MIMIC-IV windows, and among explicit assessments the positive share is 9.5% in eICU versus 29.6% in MIMIC-IV — nearly a threefold difference in frequency between the two hospitals. Five model families are compared over 137 features (demographics, vital signs, labs, neurological scores, medications, respiratory support, observation-process variables): a LASSO logistic regression (sparse linear), XGBoost (boosted decision trees, nonlinear), and three sequence models — GRU, LSTM and Transformer — that read the chronological order of measurements. Validation is bidirectional: train in one database, evaluate in the other with no retraining (eICU→MIMIC-IV and MIMIC-IV→eICU), with a 70% / 15% / 15% split keeping all windows of a stay together. Reference comparators are descriptive curves based on the Glasgow Coma Scale (GCS) and the RASS sedation scale.

The results

In internal validation, discrimination — the ability to rank a future case above a future non-case, measured by AUROC (area under the ROC curve; 0.5 = chance, 1.0 = perfect) — is high: 0.87–0.92. As soon as the database changes, it drops: on the coarse label, external AUROC falls to between 0.66 and 0.83 depending on direction and model. On the refined persistent/recurrent target, scored at the stay level, ranking behaves better: external AUROC of 0.88–0.95 in the MIMIC-IV→eICU direction, but only 0.76–0.89 the other way — transportability is asymmetric. Crucially, on the new-onset target (strictly de novo cases), external AUROC collapses to 0.50–0.58, i.e. chance level. Risk sorting still retains utility: the top-risk decile concentrates future cases with an enrichment of 2.4 to 6.9-fold over the base rate.

Where does this discrimination come from? Removing all history of prior assessments cuts external AUROC by 0.16 to 0.32 points: the model leans heavily on the documented trace of past delirium, not on raw physiology. Calibration does not transport: there is no common mapping between predicted probabilities and observed frequencies across databases (in one direction the model overestimates in the top band, in the other it underestimates), and calibration slopes are too unstable to quantify for lack of events. Finally, the authors freeze a two-cutoff review policy (rule-out / review / alert) and apply it blind to the other database. The rule-out tier is very safe — negative predictive value of 99.8% and 98.5% by direction — but the alert zone captures only 9.2% and 11.0% of windows actually followed by delirium. Clinically: of ten windows heading toward delirium, the frozen alert layer misses about nine. The tool reliably reassures low-risk patients, but makes a mediocre early alarm when the threshold chosen elsewhere is transported as is.

What is good

Bidirectional cross-validation on two large real databases. Training then evaluating in both directions (eICU↔MIMIC-IV), over millions of windows and five model families, goes well beyond the usual internal validation. It is precisely this design that exposes the asymmetry (0.88–0.95 one way, 0.76–0.89 the other) that a one-way evaluation would have hidden.

The decomposition into layers, honest and operational. Explicitly separating target definition, ranking, calibration and alert policy is a useful conceptual contribution: it shows a model can "work" on one criterion (ranking) while being unusable on another (calibration or threshold). It is a reproducible framework, not a slogan.

An ablation analysis that reveals the mechanism. By quantifying the AUROC loss when assessment history is removed (0.16–0.32), the authors show where the performance comes from: the model follows a documented trajectory rather than predicting a biological onset. Naming this mechanism, instead of hiding it behind a high AUC, is exactly what a serious study should do.

What is less good

The misleading metric. An AUROC of 0.90 suggests a ready tool, whereas calibration does not transport and the frozen alert layer misses roughly nine future cases out of ten. Worse, on strictly new delirium, external AUROC (0.50–0.58) is at chance level: the fine overall discrimination rests mostly on already-documented cases, not on predicting a future event. Discrimination alone says nothing about bedside usefulness.

An observation-conditioned estimand, close to shortcut learning. Because delirium assessments are rare (7.8–15.3% of windows) and the model relies on their history, it partly learns surveillance intensity, clinician suspicion and documentation habits — not the disease alone. The authors acknowledge it: the estimand is conditioned on the observation process and must not be extrapolated to unassessed times or to true delirium incidence. This is a classic failure mode — the model learns a shortcut (the surveillance trace) correlated with the target.

Population bias, weak comparator, no prospective test. Positive rates differ threefold between databases (9.5% versus 29.6%), MIMIC-IV is single-center, and the study rests on only two retrospective US databases. The comparators (GCS and RASS curves) are descriptive, not strong references. The labels are operational definitions, not adjudicated diagnoses. Above all, no prospective evaluation, clinical benefit or workload gain is demonstrated — a point the authors themselves stress.

What it changes

For the research community, the paper offers a "layered" validation template that should become a minimum: reporting external AUROC alone is no longer enough; calibration and decision policy must also be tested in the target database. The distinction between what transfers (ranking) and what does not (probabilities, thresholds) explains why so many "validated" models disappoint once deployed.

For clinicians and integration teams, the message is clear: a model showing a good AUC in another hospital is not plug-and-play. It must be recalibrated locally and its thresholds re-chosen on local data. The rule-out tier (negative predictive value ~99%) can help triage and reassurance, but the alert layer transported as is misses most cases — a false sense of safety if used as an alarm.

For patients and the public, the stakes are invisible but real. ICU delirium is common and partly preventable through reinforced monitoring. A poorly calibrated tool would trigger too many or too few alerts depending on the hospital; a model's formal reliability does not guarantee its local accuracy. The useful lesson: demand that such tools be validated on the data of the very institution where they will be used, calibration and thresholds included.