Stroke prognosis: six neurologists, a classical model and a deep-learning model compared on the MR CLEAN trial

A team from the university hospitals of Zurich, Amsterdam and Maastricht staged a rare contest: on data from the randomised MR CLEAN trial, six neurologists predicted the three-month disability of patients who had suffered a large-vessel-occlusion stroke, and their forecasts were compared with those of two models — a classical statistical model and a deep-learning model that reads the imaging directly. On the full disability scale, both models clearly outperform the physicians, whose predictions suffer from systematic optimism and wide variability. The takeaway is not "the machine replaces the neurologist": it is that human error stems mainly from imprecise reading of the imaging, and that an end-to-end model, by removing that manual step, could serve as a reliable second opinion — without ever becoming the gatekeeper that decides on treatment alone.

The context

A large-vessel-occlusion (LVO) stroke is an emergency where every minute counts: a clot blocks a major cerebral artery, and the outcome depends partly on how fast treatment is delivered. Mechanical thrombectomy (endovascular treatment, EVT — removing the clot via catheter) has transformed care, but trajectories remain highly heterogeneous: at three months, only about half of treated patients regain their independence.

To anticipate the course, the standard yardstick is the modified Rankin Scale (mRS), graded from 0 (no symptoms) to 6 (death). Predicting this score at three months helps with decision-making and with informing the patient and family. Yet studies have shown for years that even experienced clinicians struggle to do so accurately. Statistical scores (THRIVE, DRAGON, MR PREDICTS) have been proposed and often beat physicians; more recently, deep-learning models ingest raw imaging directly. But one question remained underexplored: how do physicians really fare against these tools, how does model assistance change their judgement, and where exactly does human error come from? That is what this preprint, by Lisa Herzog, Nelly Blindenbacher and colleagues, sets out to take apart piece by piece.

The method

The authors use data from the randomised MR CLEAN trial (500 patients with large-vessel occlusion, treated within six hours, randomised between thrombectomy and no thrombectomy), a landmark thrombectomy trial. The task: predict the three-month mRS, both on the full ordinal scale (0–6) and in binary form (0–2, "favourable" independence, versus 3–6, "unfavourable").

Six neurologists from the University Hospital Zurich (4 to 20 years of experience, median 10.5) predicted the outcome of 40 cases, selected by propensity-score matching to balance treated and untreated patients. Each worked in two rounds: first from clinical and imaging data alone (non-contrast CT and CT angiography), then a second time with the help of MR PREDICTS, this time having to estimate the imaging parameters the tool requires themselves.

Two models serve as comparators. MR PREDICTS is a validated, public decision-support tool: an ordinal regression that combines clinical variables with manually extracted imaging features — the ASPECTS score (which quantifies the extent of early lesions on CT), the occlusion location and the collateral score (the quality of backup blood-supply routes). A crucial detail for interpretation: MR PREDICTS was originally developed on MR CLEAN, so it is effectively playing at home here. The second model is a multimodal deep-learning network that reads raw CT angiography directly via a transformer-type architecture (a network specialised in analysing data structured by relationships, here the image), combined with clinical variables, within a so-called interpretable ordinal-regression framework (the effects of clinical variables are expressed as readable odds ratios). It was trained on 449 patients using five-fold cross-validation. Performance is measured with the quadratic weighted kappa (QWK, agreement with the true score, penalising larger discrepancies more heavily), accuracy within one point, AUC (area under the ROC curve, the ability to separate favourable from unfavourable) and the Brier score (quality of probability calibration). The study follows the TRIPOD and CONSORT-AI guidelines.

The results

On the full ordinal scale, the models win clearly. In the comparison subset (40 cases), MR PREDICTS reaches a QWK of 0.51 and the deep model 0.49, versus only 0.27 for unaided neurologists — a marked gap. Above all, the physicians diverge enormously among themselves: inter-rater agreement (Fleiss' κ) is just 0.11, the sign of a very unstable prognosis from one practitioner to another. On the larger cohort (404 complete cases), MR PREDICTS obtains a QWK of 0.48 and the deep model 0.41; for binary prediction, the deep model reaches 71.3% accuracy versus 66.8% for MR PREDICTS, with an AUC of 0.74 (versus 0.78) but better calibration (Brier 0.17 versus 0.20).

The most illuminating point is not the score but the dissection of human error. Neurologists are systematically optimistic: they predict milder disability than actually observed, a bias present in every rater, with or without assistance, whereas the models show symmetric error. And when they have to extract the imaging parameters themselves, they often fail: occlusion location is correct in only 65.8% of cases, the collateral score in 44.6%, and their ASPECTS estimate deviates on average by 3.4 points from the confirmed value. Giving access to MR PREDICTS modestly improves and above all harmonises predictions (binary inter-rater agreement rises from 0.25 to 0.40, binary accuracy to 68.8%), without lifting physicians to the level of the standalone models on the ordinal scale.

The honest clinical translation: for the binary "will the patient be independent or not" triage, humans and machines are roughly equal (around 64 to 71% accuracy). It is on the nuance — predicting the exact degree of disability — that the models pull ahead, precisely where decisions and difficult conversations with families are made. An AUC around 0.77 remains modest, however: the authors note there is probably a biological ceiling, since events occurring after the acute phase (recurrence, complications) are not written into the baseline data.

What is good

A human comparator, and not just any. Most AI-health papers compare against a score or against nothing. Here, six practising neurologists predict under controlled conditions, and human-model interaction is measured too. That is the right clinical question: not "is the model good?" but "what does it change for the physician who decides?"

A dissection of the sources of error. Rather than proclaiming superiority, the study identifies why humans go wrong: systematic optimism and, above all, an inability to reliably extract the imaging parameters that classical models require. That is a strong, nuanced argument for end-to-end models that remove this manual step.

Transparency and partial reproducibility. The code is public on GitHub, the study follows the TRIPOD and CONSORT-AI reporting checklists, the deep model is strictly evaluated on unseen test data, and the authors declare no funding and no conflicts of interest. All signals of methodological seriousness.

What is less good

A comparator playing at home: the biased-comparator failure mode. MR PREDICTS was developed on the MR CLEAN trial; evaluating it on that same dataset likely inflates its performance through overfitting, as the authors acknowledge. The comparison with the deep model, which is tested on unseen data, is therefore not on an even footing: the classical model's score is probably optimistic.

A single trial, an old cohort: the population-bias failure mode. All the data come from MR CLEAN, collected between 2010 and 2014, before today's thrombectomy devices and protocols. The population is essentially Dutch. Nothing guarantees that the deep model, which still has no external validation, would hold up on today's patients, other countries or other scanners — the authors say so plainly.

A tiny comparison sample and an artificial setup. The experiment rests on 40 cases and 6 physicians: the confidence intervals are wide (the deep model's AUC on these 40 cases runs from 0.42 to 0.91). And the exercise — predicting from files, without real-time pressure or direct patient contact — does not reproduce on-call conditions, where clinical intuition draws on elements absent from the data. The "prognosis" measured here also has an intrinsic limit: part of the outcome depends on later events that no baseline data can contain.

What it changes

For the research community, the study shifts the debate. It confirms that models beat clinicians on fine-grained stroke prognosis, but above all it shows that the human bottleneck is imaging interpretation — which argues for end-to-end models, eliminating manual feature extraction, rather than scores that depend on it. It also calls for measuring not the model's performance alone but human-model interaction, since a tool's superiority does not automatically translate into a better human decision.

For clinicians, nothing is deployable as is: a single-trial, retrospective model with no prospective or external validation. But the direction is clear — an end-to-end model could offer an objective, fast "second opinion", useful to counter spontaneous optimism and to harmonise prognoses that today vary widely from one physician to another. The authors stress the ethical framing: such a tool must support the decision and the dialogue with families, never act as a gatekeeper that would withhold treatment. For patients and the public, the message is measured: the machine does not "see" the future, it estimates probabilities from day-one data; it can help the physician be more accurate and more consistent, but the prognosis of a stroke remains uncertain by nature, and the decision stays human.

To go further

The preprint is available on medRxiv (DOI 10.64898/2026.06.12.26355559), and the analysis code on the authors' GitHub repository. On AI-based prognosis after stroke from other angles, see our decryptages on the heterogeneous graph for post-stroke mortality and on cognitive prognosis from neuroimaging.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. English, Spanish and Chinese translations produced with AI assistance and reviewed.