médical IA

ICU mortality: predicting death from the first 24 hours on MIMIC-IV, and why calibration matters as much as discrimination

Published on June 24, 2026 · 10 min read

Three researchers train five machine-learning models to predict the in-hospital death of an ICU patient from the first 24 hours of laboratory data alone, on 53,866 stays from the MIMIC-IV database. Their best model, a calibrated ensemble, reaches an AUROC of 0.856 — but an AUPRC of only 0.449 on a mortality rate of 10.7%, and the study rests on a single hospital. The value of the work is not the unremarkable discrimination score: it is showing that calibrating probabilities and measuring decision benefit matter as much as beating a leaderboard.

The context

Predicting the risk of death for a patient admitted to intensive care is one of the oldest exercises in quantitative medicine. Since the 1980s, scores such as APACHE, SAPS or SOFA have aggregated a handful of physiological variables into a mortality probability, used to compare units, adjust studies and inform prognostic discussion. These scores are simple, transparent and calibrated on large populations — but they freeze a few variables and date from an era without the full electronic record.

The arrival of massive databases reopened the question. MIMIC-IV is the main one: a fully de-identified set of intensive-care records spanning more than a decade of admissions at a single US academic center (Beth Israel Deaconess Medical Center, in Boston), distributed publicly through the PhysioNet platform. On this playground, hundreds of teams have trained machine-learning models to predict mortality. The question is therefore no longer "can we predict?" — we can — but "is the prediction reliable and useful at the bedside?". That is precisely the angle claimed by this preprint by Abdallah Alsammani, Merasia Johnson and Jessica Elrefaei.

The method

The authors start from 53,866 adult MIMIC-IV (version 2.2) stays, of which 5,787 ended in in-hospital death — a mortality of 10.7%. This imbalance — a minority of positive cases drowned in a majority of survivors — is central to understanding the results. From the first 24 hours alone, they build 88 laboratory features. The methodological trick is not just to take the minimum, maximum and mean of each blood test, as most work does, but to add trajectory descriptors (is the variable rising or falling over the day?) and measurement frequency (how many times was the test ordered?). The idea: a closely monitored patient is not in the same state as one tested once.

Five models are compared. A regularized logistic regression (the classic statistical model, here constrained to avoid overfitting) serves as a simple baseline. A random forest and two gradient boosting models (XGBoost and LightGBM, ensembles of decision trees built sequentially, the state of the art on tabular data) represent modern machine learning. Finally, a calibrated weighted-voting ensemble combines the above. The data are split, in stratified fashion, into 64% training, 8% validation, 8% calibration and 20% test — the latter, never seen during learning, comprising 10,774 stays.

Four families of measures judge the models, and this is where the paper stands out. Discrimination first, via AUROC (area under the ROC curve): the ability to rank a future death ahead of a future survivor; 1.0 is perfect, 0.5 is chance. But AUROC says nothing about the probabilities themselves. Hence three complements. The AUPRC (area under the precision-recall curve), far more demanding when positive cases are rare, because its "chance" value equals the prevalence (here 0.107) rather than 0.5. The Brier score, which measures the gap between the stated probability and reality (the lower the better), and the calibration analysis, which checks that a stated risk of 30% really corresponds to 30% of observed deaths. Finally, decision curve analysis (DCA), which translates the model into net benefit by chosen alert threshold, and interpretation via SHAP, a method that attributes to each variable its contribution to a given patient's prediction.

The results

The calibrated ensemble achieves the best overall performance: AUROC of 0.856 (95% confidence interval: 0.846–0.867), AUPRC of 0.449 (0.418–0.480) and a Brier score of 0.078. XGBoost (AUROC 0.856; AUPRC 0.435) and LightGBM (AUROC 0.854; AUPRC 0.436) are virtually level. All three significantly outperform logistic regression (AUROC 0.823; AUPRC 0.376), but the AUROC gap is only 0.031 to 0.033 — about three hundredths. In other words, the modern machinery wins only a thin sliver of discrimination over the good old linear model.

The clearest gain is elsewhere: in calibration. Before treatment, the probabilities of XGBoost and LightGBM were poorly tuned (Brier scores of 0.134 and 0.151). After calibration, they fall to 0.078 and 0.076 — an improvement of 42% and 50%. This matters because a physician does not need a ranking but a number to trust: "30% risk" must mean 30%. Decision curve analysis confirms a consistent net benefit across a threshold range of 5% to 20%, that is, the zone where a unit might reasonably trigger enhanced monitoring.

On the variable side, the dominant predictors are age, blood urea nitrogen (a marker of renal failure and general severity), ICU subtype, measurement frequency and lactate-related variables (a sign of hypoperfusion). By subtype, AUROC stays above 0.79 everywhere, peaking at about 0.92 in cardiac ICU (where post-operative patients offer a clear prognostic signal) and dropping to about 0.79 in medical ICU (more heterogeneous cases). Of note: the random forest falls below 0.70 in four subtypes out of five — a structural weakness of that model, not of the problem.

The honest clinical translation fits in one sentence: an AUROC of 0.856 indicates good ranking, but an AUPRC of 0.449 on a mortality of 10.7% means that at a given alert threshold, a substantial share of patients flagged "at risk" will survive — precision on the rare class remains modest. What calibration brings, by contrast, is tangible: the displayed probability becomes a risk interpretable as is, without having to mentally re-translate it.

What is good

An evaluation that looks beyond AUROC. Most clinical-prediction papers stop at discrimination. Here, the authors add calibration, Brier score, decision curve and SHAP — exactly the battery that matters for real deployment, where one acts at a threshold and must be able to trust the number. This is methodological hygiene that remains too rare.

Time-aware feature engineering. Rather than crushing each test into an average, the pipeline captures 24-hour trends and monitoring intensity. This is closer to an intensivist's reasoning, who reads a trajectory (is the lactate falling?) as much as an isolated value.

A refreshing honesty about the small gain of the "modern". The paper does not oversell gradient boosting: it shows in black and white that the AUROC advantage over logistic regression is only three hundredths, and that the real progress comes from calibration. This refusal of sensationalism is rare and precious in a literature that readily headlines AUROCs of 0.9.

What is less good

A single center, no external validation: the population-bias failure mode. All the data come from a single Boston hospital. Yet a model trained on one population, one information system and one set of practices sees its performance — and above all its calibration — degrade elsewhere (domain shift). The authors explicitly admit it and call for multicenter validation. As it stands, nothing guarantees that "30% risk" in Boston will be worth 30% in Lyon.

A modest AUPRC masked by a flattering AUROC: the misleading-metric failure mode. On a task where only 10.7% of stays are deaths, the AUROC of 0.856 sounds good but the AUPRC of 0.449 reveals reality: precision on the rare class remains limited. This is the classic trap of imbalanced data, where the reassuring score is not the one that matters for use. To their credit, the authors report the AUPRC; the reader still has to know how to read it.

Measurement frequency as a predictor: the shortcut failure mode. That the number of ordered tests ranks among the best predictors cuts both ways. The model may learn the care process — a patient tested ten times is one judged severe — rather than physiology. This is a case of shortcut learning coupled with informative missingness: useful on the source data, but fragile across hospitals whose ordering habits differ. Add the acknowledged limits: a retrospective study, variables reduced to laboratory and demographic data (no vital signs, no treatments), fairness across subgroups not evaluated, recognized calibration drift, and code announced only "upon acceptance" (MIT license to come, hence not verifiable today). Finally, note that the three authors are affiliated with business schools (University of South Florida, Delaware State University), with no listed clinician co-author — no declared conflict of interest, but useful context to know.

What it changes

For the research community, the message is methodological before it is clinical: on MIMIC-IV, chasing an ever-higher AUROC no longer teaches much. The useful differentiator is calibration, net benefit and interpretability — the kit this work assembles cleanly and that others should adopt by default. It is also a reminder that, on tabular data, logistic regression remains a formidable comparator that must be beaten clearly, not by three hundredths.

For clinicians, nothing is deployable as it stands: a single-center model, not externally validated and without vital signs, will not pass the door of a unit. But the way of evaluating is exemplary — a mortality score that was not calibrated and whose decision benefit had not been measured would not deserve to be trusted. For patients and the public, the lesson is broader: a "risk" displayed by an AI is only worth something if it is calibrated and interpreted in context. A risk figure is not an individual prognosis nor a decision — it is the care team that integrates it into the full clinical picture.