médical IA

Rare cases in medicine: training an LLM to choose the next action, not just the final diagnosis

Published on August 5, 2026 · 9 min read

Ofir Ben Shoham, Oriel Perets, Nir Grinberg and Nadav Rappoport (Ben-Gurion University of the Negev) start from a simple observation: most benchmarks that grade language models in medicine score only the final diagnosis, whereas real medicine plays out as a sequence of intermediate decisions — which test to order next, which imaging to obtain, which specialist to call. They build MedUPSQA, a set of 21,874 decision points extracted from 5,535 real rare clinical cases, then train models to predict the next action as information arrives; this reframing lifts accuracy from 55.2 to 66.7% on their best model, and — the more telling result — a small aligned model beats larger frontier models. The lesson is not the score: it is that learning to reason mid-stream matters more than model size — as long as you look closely at who decides the right answer.

The context

For three years, large language models (LLMs — systems trained to produce text from a prompt) have been measured in medicine by batteries of multiple-choice questions: USMLE, board questions, standardized vignettes. These tests share a structural flaw: they present the whole case at once and grade a single thing, the final diagnosis. But a clinician almost never meets a patient "all at once." They receive a complaint, ask a question, obtain a result, make a decision, and start again. Most of medicine lives in that chain of intermediate actions — the next lab, the next scan, the right specialist — long before a diagnosis is reached.

The problem worsens on rare and off-guideline cases: atypical presentations, uncommon diseases, unexpected treatment responses. That is exactly where clinical guidelines stop applying and step-by-step reasoning becomes decisive. The same team had already explored this ground with CUPCase (AAAI 2025), a dataset of rare clinical cases drawn from the literature to test LLMs on something other than exam questions. MedUPS extends that line but shifts the target: no longer "what is the diagnosis?" but "what should be done now, with the information available at this moment?".

The method

The authors start from 5,535 real clinical case reports and cut them into accumulating chronological "chunks": the first segment contains only the opening elements (complaint, history), then each following segment adds one piece of information (a test, a result, a change in course). At each boundary between two segments they define a decision point: given everything so far, what is the next relevant action? This segmentation yields MedUPSQA, 21,874 mid-stream decision points. The target is thus not a single diagnostic label but the expected action at that stage — the test to order, the imaging to obtain, the specialist to involve, the differential to pursue.

To align models to this task, the team uses reinforcement learning, specifically GRPO (Group Relative Policy Optimization): rather than imposing a reference answer, the model produces several candidate answers to the same situation, they are ranked, and the model is pushed toward the higher-ranked ones. The ranking comes from a "judge" — another language model (LLM-as-a-Judge) that compares the proposed answer to what actually happened in the case and assigns a reward. In parallel, the authors train classic supervised fine-tuning (SFT) versions, which learn directly to reproduce the documented next action, to check that the gain does not come solely from the reinforcement machinery. Three backbones are tested: Qwen3.6-27B, Qwen3.5-9B and HuatuoGPT-3-8B, a medical-specialized model. The main metric is next-step accuracy — the share of decision points where the proposed action is judged correct — with 95% confidence intervals.

The results

Mid-stream alignment clearly improves all three models. Next-step accuracy rises from 55.2 to 66.7% for Qwen3.6-27B, from 47.2 to 57.8% for Qwen3.5-9B, and from 37.8 to 44.4% for HuatuoGPT-3-8B. The gain is around ten points for the first two, more modest for the smallest medical model. The SFT versions also improve over the base, which, the authors argue, indicates that the signal comes from the task itself — learning the next decision — and not only from the reinforcement optimizer.

The most interesting result is not the absolute score but a comparison: across several model scales, the alignment objective improves accuracy more than scale does. In other words, a smaller but aligned model surpasses larger frontier models evaluated as-is. For a field dominated by the race for size, that is a useful signal: how you train can matter more than parameter count.

In clinical terms, keep a cool head. Even the best aligned model is still wrong about one time in three: on 100 intermediate decisions, it misses roughly 33. And these errors occur on rare cases, precisely the terrain where a bad "next action" — a useless test, a delaying scan, a wrong referral — costs the most in diagnostic time. A figure of 66.7% accuracy is therefore not a green light: it is measurable progress on a still largely open problem.

What is good

A well-judged reframing of the task, operationalized at scale. Moving from final diagnosis to the next action fits the reality of care much better, where information arrives piecemeal. The authors do not merely assert it: they turn 5,535 cases into 21,874 chronological decision points, yielding a substantial dataset to train and measure this sequential reasoning, where classic benchmarks grade only an endpoint.

Alignment beats scale — and the ablation confirms it. Showing that a small aligned model surpasses larger frontier models is a specific, economically meaningful result. Above all, adding an SFT variant is good experimental hygiene: since supervised learning also improves all three models, the gain is not an artifact of reinforcement — it comes from the training target itself. That is the kind of control that separates a demonstration from a showcase effect.

An open, reproducible release. The dataset, code and aligned checkpoints are released under CC BY 4.0, the most permissive license. In a field where many "clinical models" stay closed or under non-commercial terms, this openness enables replication, criticism and reuse — and transparently extends their earlier work (CUPCase).

What is less good

A judge that grades and rewards: the misleading-metric trap looms. The same kind of tool — an LLM-as-a-Judge — serves both as the reward signal during training and, in effect, as the definition of the "right answer." Two classic risks combine. First, reward hacking: a model trained to please an automatic judge can learn to produce answers that satisfy the judge without being clinically better. Second, circularity: an "accuracy" defined by a language model is not an accuracy validated by clinicians. Until the next action is adjudicated by independent physicians, the 66.7% figure mainly measures agreement with an algorithmic judge.

A fragile ground truth, and a data-leakage risk. The "correct" next action is the one actually documented in the case report. But a published case tells a single path, often chosen retrospectively because it led to the right diagnosis; other reasonable actions are then counted as wrong. Add a health-AI failure mode rarely discussed here: data leakage. Case reports from the literature are quite likely already present in the backbones' training data (Qwen, HuatuoGPT); without an explicit decontamination procedure, part of the performance could reflect memorization rather than reasoning.

A curated corpus, no real clinical noise, and zero prospective validation. Published rare cases are heavily selected: publication bias (the instructive, the solved, the spectacular get published), English dominance, absence of the mess of a real record (missing, contradictory, mis-entered information). A model may also learn the style of case reports — how authors telegraph what comes next — rather than the clinical content itself (shortcut learning). Finally, everything happens on retrospective text: no prospective evaluation, no clinician in the loop, no real deployment. Real-world performance remains unknown.

What it changes

For the research community, the methodological message is clear: grade the process, not just the endpoint. MedUPSQA provides a reusable template to evaluate sequential reasoning, and the SFT ablation shows the gain comes from the task. The obvious next steps are adjudication by independent clinicians (rather than a model judge), explicit decontamination against pre-training data, and a move to real records rather than literature cases.

For clinicians, nothing changes in practice today: this is a benchmark and alignment study on case reports, not a deployed tool. But it sketches a direction — assistants that suggest the "next action" rather than a diagnosis — and a warning: a model that confidently recommends the next test can be wrong one time in three on rare cases, the very ones where you would be tempted to trust it.

For patients and the public, the reading is sober. Answering well on carefully written rare cases is not the same as helping a real patient through a real care journey. The "next decision" framing is promising because it matches how medicine is practiced, but it remains far from the bedside. And if such systems ended up steering management decisions, they would fall under regulatory oversight as software as a medical device — not a mere conversational assistant.