médical IA

ICU sepsis: a reinforcement learning agent that lowers treatment as lactate rises, and that standard metrics score top of the class

Published on August 17, 2026 · 11 min read

Two engineers, one at Vanderbilt and one at Duke, took two reinforcement learning agents trained to dose fluids and vasopressors in septic patients from the MIMIC-III database, and subjected them to a test the literature almost never runs: artificially vary the patient's lactate and watch which way the recommended treatment moves. The model that imitates clinicians best — mean squared error of 0.0098, seven times lower than its competitor's — recommends lowering treatment intensity by about 15% as lactate rises, the exact reverse of what the Surviving Sepsis Campaign prescribes; the other model, a worse imitator, responds in the right direction across all fifteen levels tested. The diagnosis offered is right and important — imitation metrics do not measure clinical safety — but the demonstration rests on a single random seed, with no confidence intervals at all, in a six-page preprint that never states how many patients it used.

The context

The idea of learning a sepsis treatment policy from ICU records has a precise starting point: the AI Clinician of Komorowski and colleagues, published in Nature Medicine in 2018. The principle is offline reinforcement learning: rather than experimenting on patients, an agent is trained on already-recorded trajectories, learning to choose at each timestep the action (here, a fluid dose and a vasopressor dose) that maximises a reward meant to reflect survival. The structural problem is immediately visible: the learned policy can never be tested on the patients you have. You only have the decisions clinicians actually made, and you must estimate what a different decision would have produced.

Hence the usual evaluation toolkit, and its blind spot. Two metrics dominate. MSE (mean squared error) measures the gap between the action the agent predicts and the action the clinician actually took: it is a measure of imitation, not of quality. FQE (Fitted Q-Evaluation) estimates the expected value of the learned policy by fitting a value function to the observed data: it is a statistical extrapolation whose reliability collapses as soon as the evaluated policy departs from the one that generated the data. Gottesman and colleagues stated it plainly in Nature Medicine in 2019: these observational evaluations are largely insensitive to whether the agent learned causal logic or a correlation.

This paper adds a particularly uncomfortable variant of that problem, which it names "toxic mimicry". ICU records contain episodes of treatment withdrawal: when a patient transitions to comfort care, vasopressors are stopped, and the patient dies. Statistically, those trajectories associate biological deterioration with a reduction in treatment. An agent trained to imitate clinicians can therefore quite happily learn the rule "when the patient deteriorates, reduce" — descriptively accurate and clinically catastrophic if applied to a patient one is trying to save. The authors draw on the end-of-life ICU literature to show that these practices are widespread and heterogeneous: Mark and colleagues (2015) report withdrawal rates ranging from 0% to 84% depending on setting. Conceptually, the phenomenon is an instance of the causal confusion described by de Haan, Jayaraman and Levine in 2019: a model that learns to copy its own past actions rather than reason about the patient's state.

The method

The cohort. The authors extract from MIMIC-III v1.4 adult patients meeting Sepsis-3 criteria, explicitly following the Komorowski pipeline. The state space has 44 variables (vital signs, laboratory results, demographics), the action space is continuous and two-dimensional — intravenous fluid dose and vasopressor dose — which distinguishes it from the 25 discrete bins of the original AI Clinician. The sliding window is six four-hour timesteps, that is 24 hours of context, and the split is 70/15/15% at the patient level. This must be said immediately, because it conditions the reading of everything else: the number of patients, stays and transitions is stated nowhere, nor is the cohort's mortality or its demographic characteristics. There is no descriptive table.

The two agents. The first, MedDT (Medical Decision Transformer), adapts the Decision Transformer of Chen and colleagues (2021): a causal transformer that receives as input the sequence of returns-to-go, states and past actions, and predicts the next action. It is trained by pure imitation — an MSE on clinician actions, with no value function. The second, HCT-RL (Historical Causal Transformer), combines three elements. The first, called Causal Action Shielding, consists precisely of removing past actions from the encoder input — the name promises a runtime safety device, the mechanism is an architectural ablation. The second is a propensity-score weighting that reduces the weight of actions that are frequent in the data. The third is the Conservative Q-Learning of Kumar and colleagues, which penalises actions absent from the data. Both models share the same size (dimension 128, three layers, four heads) and are trained across five random seeds.

The audit. This is the paper's contribution. The Counterfactual Clinical Audit (CCA) does not measure a performance, it tests a response: a physiological variable in the input state is artificially modified and the movement of the policy is observed. Three probes. Audit I: inject Gaussian noise (standard deviation 0.5 in z-score units) into creatinine alone, a haemodynamically non-critical variable, and measure how far the recommended action shifts — any response is spurious by construction. Audit II: impose a constant lactate across all six timesteps, swept over fifteen values between −2 and +5 z-scores, and measure mean treatment intensity at each level; the summary is a Spearman rank correlation, positive if the response is guideline-concordant, negative if inverted. Audit III, the "scissor" probe: drop mean arterial pressure (from +1 to −2 z-scores) first alone, then simultaneously with rising heart rate and lactate, and check that the agent responds more strongly in the second case — a clinician facing hypotension accompanied by hyperlactataemia escalates faster.

The results

On standard metrics, MedDT wins. Its MSE is 0.0098 against 0.0742 for HCT-RL, roughly seven times more faithful imitation. Its FQE-estimated value is 65.29 against 63.43. Two important reservations accompany those figures: the reward function is never defined in the paper, which makes the FQE scale literally uninterpretable, and no value is reported for the clinicians' own policy. There is therefore no reference comparator.

On the audit, the ranking flips. In Audit II, HCT-RL raises its treatment intensity monotonically with lactate — Spearman correlation of +1.00 across the fifteen levels, for a rise of about 10% above baseline. MedDT does the reverse: its intensity falls about 15% below baseline at moderate-to-high lactate values, with a partial rebound at extreme values, for a ρ of −0.67. This is the paper's central result. In Audit I, sensitivity to creatinine noise is 0.060 ± 0.006 for MedDT against 0.032 ± 0.008 for HCT-RL on MIMIC-III. In Audit III on MIMIC-III, HCT-RL shows a positive scissor divergence while MedDT is roughly flat.

External validation is on MIMIC-IV v3.1, without retraining. In Audit III, HCT-RL retains a positive divergence of 0.04 to 0.12, while MedDT goes negative, from −0.12 to 0: under multi-organ stress it recommends lower vasopressor doses than when pressure alone falls. Audit II, the most important one, was not rerun on MIMIC-IV.

Clinical translation: it is absent, and that has to be said. Everything is expressed in z-scores and in norms of a normalised action vector. The paper never gives the mapping between its lactate sweep and millimoles per litre, nor between its "treatment intensity" and micrograms per kilogram per minute. One therefore cannot write "the dose falls by X µg/kg/min per mmol/L of lactate" — that figure does not exist in this work. What does exist, and what suffices for the argument, is a sign: in a condition where resuscitation demands escalation as lactate rises, the model with the best imitation metrics de-escalates.

What is good

The test is simple, reproducible and well aimed. Varying one variable and looking at the derivative of the policy costs almost nothing in compute, requires no additional data, and interrogates exactly what MSE and FQE cannot see. Choosing creatinine as the control variable in Audit I is shrewd: it is a negative control built on a physiological justification, not on intuition. And Audit III is the only one of the three to test an integration property — responding more strongly to contextualised hypotension than to isolated hypotension — rather than simple monotonicity.

The demonstration does not stop at one database. Audits I and III are rerun on MIMIC-IV without retraining, and Audit I is reported as mean ± standard deviation across five seeds. The authors cite Nauka and colleagues (npj Digital Medicine, 2025) on transportability failures of reinforcement learning policies in sepsis, which shows they know the problem rather than ignore it.

The limitations section is unusually candid. The authors write that CCA "audits behavioral safety but cannot directly certify outcome improvement", and describe it as "a necessary safety screen rather than a sufficient criterion". They explicitly acknowledge that both MIMIC databases come from a single academic medical centre in Boston, that the clipping bounds of their weighting were not tested, and — this is rare — that "cross-seed variability of Audit II and III metrics warrants further investigation". In other words, they say themselves that their headline result is not quantified in uncertainty.

What is less good

The headline result is single-seed, and the narrative does not match the table. The authors state that Audits II and III are reported for "the seed with lowest validation MSE". The ρ of −0.67 and the 15% therefore have no confidence interval, no p-value, and no cross-seed standard deviation — over fifteen measurement points. That fragility is visible elsewhere: on Audit I in external validation, the text announces a MedDT sensitivity of 0.10 against 0.01 for HCT-RL, a "ten-fold" improvement, while the same paper's multi-seed table gives 0.062 ± 0.028 against 0.003 ± 0.003. The factor of ten comes from the chosen seed; HCT-RL's standard deviation there equals its mean. It is a clear case of a misleading metric, in a paper whose subject is precisely the deceptiveness of metrics.

The palliative confounder is never demonstrated in the data. The whole thesis rests on the idea that end-of-life withdrawal episodes contaminate learning. Yet no treatment-limitation, advance-directive or comfort-care label is extracted from MIMIC-III to verify it. The mechanism is argued from the literature and inferred from model behaviour, never measured. To this is added an absence of ablation: HCT-RL's three components — removal of past actions, propensity weighting, CQL — are never tested separately, so the paper cannot say which one corrects the mimicry, even as it claims that the "causal shielding" and the weighting are responsible. Technically, the only mechanism that differs structurally is the removal of past actions from the input — that is, literally, the remedy known since 2019 against causal confusion.

The evidentiary base is thin, and the population narrow. No cohort counts, no written reward function, no clinician comparator, no published code, no funding, conflict-of-interest or ethics-approval statement. Both MIMIC databases come from the same Boston hospital, which makes the MIMIC-IV "external validation" a test of temporal drift rather than a test of population bias: nothing here says how these models behave in a different health system. Finally, Audit II — the only one producing the striking result — assumes lactate can be varied independently of the other 43 variables. The authors acknowledge this. It is a strong assumption: a lactate at +5 z-scores with unchanged blood pressure and heart rate is not a patient who exists.

What it changes

For the research community, the net contribution is not the result but the protocol. The idea that a learned policy must be subjected to a derivative test anchored in a clinical guideline — when this variable rises, this treatment must rise — is cheap, general, and ought to accompany every reinforcement learning publication in health alongside FQE. It operationalises Gottesman's 2019 warning, which had remained at the level of principle. That the model scored best by standard metrics is the one that fails the audit is, in itself, sufficient argument to change publication practice. One note, however: the two best-known technical critiques of the AI Clinician, by Jeter and colleagues and by Oberst and Sontag, are not cited.

For clinicians, nothing changes in practice, and this should be clear: no reinforcement learning agent is used today to drive fluids or vasopressors in intensive care, and this paper does not bring that prospect closer — it pushes it further away, by documenting one more failure mode. The transferable lesson is methodological: faced with a therapeutic decision-support tool, the question to ask is not "what is its performance?" but "what does it do when the patient deteriorates?". That is a question a clinician can put to a vendor without being an engineer.

For patients and the public, the important point is counter-intuitive and worth holding onto. A model can faithfully reproduce what doctors do and be dangerous precisely for that reason, because medical records also register end-of-life withdrawal decisions. The authors go further in their discussion, noting that these decisions vary by ethnicity and socioeconomic status, and that such a bias could therefore fall unevenly across groups. They write it as a hypothesis; no subgroup analysis is conducted in this paper to support it.