Predicting 10-year ischemic stroke risk: an XGBoost that beats classic scores but whose absolute risk collapses from one hospital to the next

A team from the University of Alabama at Birmingham and the Icahn School of Medicine at Mount Sinai builds an XGBoost model that predicts 10-year ischemic stroke risk by combining clinical data, laboratory trajectories and twenty polygenic risk scores. On the development cohort it clearly outperforms reference scores (revised Framingham, Pooled Cohort Equations), and its ability to rank patients transfers to an external cohort from another hospital. But its estimate of absolute risk collapses when moving from one site to another — it predicts about eight times too many strokes — the genomic contribution is marginal except in one subgroup, and "self-reported race" adds almost nothing once the polygenic signal is accounted for.

The context

Predicting who will have a stroke within ten years serves to decide who to treat in primary prevention — statins, antihypertensives, antiplatelets — before the event. The historical tools, the revised Framingham Stroke Risk Profile (rFSRP) and the American Pooled Cohort Equations (PCE), rest on a handful of hand-entered factors (age, blood pressure, smoking, diabetes, cholesterol) combined by a regression. Two limitations are well known: they were calibrated mostly on white populations, and they ignore the vast information held in the electronic health record (EHR) and in the genome.

Two recent ingredients promise to do better. First, laboratory trajectories: instead of an isolated glucose or creatinine measurement, one summarises the repeated evolution of a marker (its mean, variability, slope, recency). Second, polygenic risk scores (PRS: a numerical summary of thousands of genetic variants associated with a disease, computed from an individual's DNA). The open question, and the point of this preprint: do these ingredients truly add anything beyond classic scores, and above all does their benefit hold when the hospital and the ethnic makeup change? The question is not rhetorical: PRS are mostly built from genetic studies conducted on people of European ancestry, and their transferability to other ancestries is the field's known weak spot.

The method

A retrospective cohort study. The model is developed in All of Us (AoU), a large American research program: 34,987 adults aged at least 45, with at least one year of medical records before inclusion, anchored to a baseline date (January 2010) then followed for ten years; 1,920 had an ischemic stroke (5.5%). External validation uses the Mount Sinai BioMe Biobank in New York: 10,693 participants, 107 strokes (1.0%), with a markedly different population — more Hispanic (35% vs 12%) and Black (22% vs 19%) participants. The measured event is the first inpatient ischemic stroke, defined by diagnostic codes (ICD-9 and ICD-10); non-ischemic strokes and outpatient-only cases are excluded, and controls must have at least ten years of follow-up.

The model is an XGBoost — a forest of decision trees built by gradient boosting, where each new tree corrects the errors of the previous ones; its useful feature here is that it handles missing values without artificially filling them in. The authors stack three nested tiers: M1 (clinical data: demographics, comorbidities, vitals, medications), M2 (M1 + laboratory trajectories), and M3 (M2 + twenty PRS from the public PGS Catalog). They train two families in parallel: a "full-feature" model (specific to AoU, including self-reported race) and a "harmonized" model (limited to features common to both cohorts, so that a single model trained on AoU runs as-is on BioMe). Each specification is tested in a "race-blind" and a "race-aware" version, the contrast isolating the specific contribution of declared race.

Two qualities of the model are measured separately, and this is the heart of the study. Discrimination — its ability to rank a future case above a non-case — via the AUROC (area under the ROC curve: 0.5 = chance, 1 = perfect). Calibration — the accuracy of the absolute risk announced — via the observed-to-expected ratio (O/E: ideally 1; below it, the model overestimates risk). Calibration is adjusted by Platt scaling on a dedicated AoU partition, then applied as-is to BioMe. Hyperparameters are tuned by cross-validation (via Optuna), confidence intervals obtained by bootstrap, AUROCs compared by DeLong's test, and feature importance decomposed by SHAP. Reporting follows the TRIPOD+AI guideline. M3 is finally compared with rFSRP and the PCE on the subgroups eligible for those scores.

The results

On the AoU test partition (6,998 participants, 384 cases), M3 reaches an AUROC of 0.813 (95% CI 0.788–0.837). The gap with classic scores is striking: +0.164 AUROC over the revised Framingham, +0.181 over the PCE (P < 0.001 in both cases). But the decomposition is instructive: adding laboratory trajectories (M2 vs M1) gains significantly (ΔAUROC 0.020; P = 0.016), whereas adding the twenty PRS (M3 vs M2) does not gain significantly (ΔAUROC 0.004; P = 0.253). In other words, most of the progress comes from the rich medical record, not from the genome.

External transfer tells two opposing stories. Discrimination transfers, attenuated: on BioMe, M3's AUROC falls to 0.745. Calibration, by contrast, collapses: the aggregate observed-to-expected ratio goes from 1.00 on AoU to 0.12 on BioMe. In plain terms, used as-is, the model predicts about eight times more strokes than actually occur in the external cohort. Recalibrating the intercept on BioMe repairs calibration in African American (O/E 1.05) and Hispanic (0.86) participants, but not European American ones (0.43, residual over-correction).

Clinical translation. At the threshold set for 90% specificity, M3 flags 54% of future strokes versus 43% for the clinical-only model, with a positive predictive value of 25% versus 20% — that is, under the same screening policy, a quarter more true positives and 10% fewer false positives. That is a real triage gain. But a model whose absolute risk is off by a factor of eight cannot, as it stands, be used to set a treatment threshold in a new hospital: it would rank patients well while massively overtreating. On the genome, the PRS contribution is significant in only one of the six cohort × ancestry combinations — Hispanic participants in BioMe (ΔAUROC +0.042; P = 0.003) — and null elsewhere (estimates from −0.019 to +0.012). As for self-reported race, it adds a small gain only when combined with the PRS (BioMe +0.022, P = 0.035; AoU +0.006, P = 0.053), and nothing without them.

What is good

A genuine external validation on an ancestrally distinct cohort. This is the exception rather than the rule: the authors themselves note that stroke risk models are rarely validated on cohorts of different ancestries. Here, the model trained on AoU is confronted with another hospital, another ethnic distribution (35% Hispanic vs 12%), and a stroke prevalence five times lower. It is exactly the test that 90% of published models dodge.

The discrimination/calibration distinction handled honestly. Many papers stop at a flattering AUROC. Here, the authors show in black and white that ranking ability transfers (AUROC 0.745) but that absolute risk collapses (O/E 0.12), and they say so. The dual display is exactly what makes it possible to see that a good ranking score can produce wrong absolute risks — the most useful lesson of the study.

Careful methodological rigour. Reporting following TRIPOD+AI, no post-inclusion information fed into the model (no temporal leakage), hyperparameter tuning by cross-validation, confidence intervals by bootstrap, AUROC comparison by DeLong's test, feature importance by SHAP decomposed into clinical domains. Funding is public (National Heart, Lung, and Blood Institute; Alabama Genomic Health Initiative) and the authors declare no commercial conflict of interest.

What is less good

A population bias and a single external validation site. This is the central failure mode. BioMe is a single New York hospital: calibration only holds if redone site by site, and even recalibrated it misses European Americans (O/E 0.43). Above all, the PRS, built mostly on European-ancestry studies, do not transfer: their contribution is null in five ancestry strata out of six. A model presented as fit for "diverse populations" whose genomic benefit materialises only in Hispanic participants of a single cohort remains, on that point, fragile.

A weak comparator and a flattering metric put forward. Two compounded failure modes — the biased comparator and the misleading metric. The 0.16–0.18 AUROC gap over Framingham and the PCE is real but pits an XGBoost fed the full medical record against scores designed one to three decades ago for a handful of hand-entered variables: the comparison is unbalanced by construction. The honest internal test (M3 vs M2) shows, moreover, that the twenty headlined PRS add almost nothing — the gain comes from the EHR, not from the advertised genomic innovation. And an AUROC of 0.745 alongside an O/E of 0.12 is a reminder that good ranking, in isolation, says nothing about the accuracy of the risk announced.

Limited power within strata and closed reproducibility. BioMe has only 107 strokes, split by ancestry into tiny subgroups: some per-ancestry AUROCs have very wide confidence intervals (up to 0.673–0.925), and the per-group conclusions rest on few events. Broadening the stroke definition drops the external AUROC to 0.646–0.674, a sign of strong sensitivity to the chosen phenotype. Finally, the data are under restricted access (AoU and BioMe), no code repository is published, and this is a preprint not yet peer-reviewed, released under a non-commercial, no-derivatives licence (CC-BY-NC-ND) — independent reproducibility is therefore limited.

What it changes

For the research community, the study delivers two messages. First, in 2026, progress on stroke prediction comes chiefly from exploiting the longitudinal medical record — laboratory trajectories — far more than from polygenic scores, whose cross-ancestry transferability remains the unresolved bottleneck. Second, a directly reusable methodological lesson: validate discrimination and calibration separately on an external cohort, and plan a site-specific recalibration before any deployment.

For clinicians, nothing is deployable today: the model is not calibrated outside its home site, is not approved, and rests on a preprint. The idea of a primary-prevention score fed by the medical record is credible in the medium term, but strictly on condition of local recalibration — without it, the announced risks would be unusable. For patients and the general public, the study illustrates two reading rules. "An AI predicts stroke better than Framingham" hides that the gain comes mostly from the medical record, not the genome, and that a model that ranks well can be off by a factor of eight on real risk in another hospital. And on the sensitive question of race: the study suggests that genetic ancestry (via the PRS) captures most of the signal that "self-reported race" contributed — an argument for calibrating by ancestry rather than writing race into the model as a variable.

To go further

The preprint is available on medRxiv (10.64898/2026.06.22.26356280). On the distinction between ranking well and calibrating well, at the core of this paper, see our decryptage of an ICU mortality model attentive to calibration and clinical utility; on AI applied to stroke from other angles, those of a deep-learning model set against neurologists' prognosis and a graph network for post-stroke mortality.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. English, Spanish and Chinese translations produced with AI assistance and reviewed.