BrainGFM: a generative foundation model for brain imaging, validated on 4,796 stroke patients — clear gains, a reader study to read with caution
A team from Southeast University, Case Western Reserve University and three Nanjing hospitals presents BrainGFM, a generative model pretrained on about 1.3 million brain MRI and CT slices, then adapted to 48 tasks and to three stroke tasks in six centres. The model clearly outperforms its comparators on lesion segmentation, subtyping and image synthesis, including on external centres. The results are solid for a preprint, but the reader study shows a benefit that varies greatly between radiologists, and the paper states neither code availability nor funding.
The context
In brain imaging, tasks keep multiplying (segmenting a tumour, spotting a haemorrhage, recognising an MRI sequence, enhancing a degraded image) while annotated data remain scarce and heterogeneous. Foundation models (large models pretrained without task-specific labels, then reused for many tasks) have already been proposed for the brain, such as MedSAM (general-purpose segmentation) or BrainIAC. Their limit, according to the authors: the knowledge stays locked in the weights and architecture of one model and does not transfer easily to other networks.
BrainGFM takes another route: a generative model that acts as a reusable "image prior". It can produce synthetic images to train another model when labels are missing, and can be fine-tuned to reconstruct images (for instance turning a low-dose CT into a better image).
The method
Pretraining has two stages. First, a tokenizer (a module that turns an image into a sequence of discrete tokens, like text) of the multi-scale vector-quantized type: a 4,096-entry codebook, 32-dimensional latents, ten scales from 1×1 to 16×16, shared between images and segmentation masks. Second, a "next-scale" autoregressive Transformer that predicts tokens from coarse to fine, conditioned on a category (T1, T2, contrast-enhanced T1, FLAIR, diffusion DWI, CT, unknown sequence, or mask). Both stages are initialised from weights trained on natural images, then adapted to the brain.
The data, named BrainGFM-1.3M, combine 11 resources (ADNI, FOMO-60K, ABIDE II, TCGA, TCIA, IBSR18, MSD Brain, BraTS2021, CQ500, RSNA Intracranial Hemorrhage, TopBrain): 64,754 3D volumes (63,745 MRI, 1,009 CT), i.e. 1,295,080 axial slices, 20 slices per volume. Images are skull-stripped, normalised and padded to 256 × 256. A portion is held out of pretraining for evaluation.
To adapt to a task, the base weights are frozen and a LoRA (low-rank adaptation: a small number of extra parameters, here rank 8) is added. For segmentation and classification, the model first generates synthetic images with their masks or classes, then a conventional network is trained on them. For image enhancement, it is post-trained using the source image as the condition.
Clinical validation covers 4,796 stroke patients: two internal centres (Jinling Hospital and Nanjing First Hospital) and four external centres (Nanjing Jiangning, Wuxi People's, Northern Jiangsu People's, Changzhou Second People's). Three tasks: lesion segmentation; five-class subtyping of ischaemic stroke (TOAST classification, by mechanism: large-artery, cardioembolic, small-vessel, etc.) from diffusion, ADC and the predicted mask; and FLAIR image synthesis from diffusion. Training uses 200 internal cases per task, with a 60-case regime also tested for segmentation and subtyping.
The results
Across the 48 tasks, mean segmentation Dice (the Dice score measures overlap between predicted and reference lesion: 100% = identical) is 72.4%, versus 54.4% for a UNet trained on little data, 70.0% for MedSAM and 71.4% for a UNet trained on all data. For classification, mean AUC (area under the ROC curve: 0.5 = chance, 1 = perfect) is 94.0% (few-data ResNet50: 89.4%; full-data ResNet50: 94.5%). For image enhancement, mean PSNR (reconstruction signal-to-noise ratio in dB, higher is better) is 37.8 dB over 20 tasks.
On stroke, ML metrics: lesion segmentation at 78.1% Dice internally and 76.9% externally (MedSAM: 72.0 and 70.3; full-data UNet: 69.3 and 65.1). TOAST subtyping at 87.4% AUC internally and 84.5% externally (best comparator, MAE: 82.5 and 79.6). DWI-to-FLAIR synthesis at 26.63 dB PSNR internally and 26.90 dB externally, versus 25.34 and 24.52 for MAE. That external figures stay close to internal ones across four centres is the most convincing result.
Reader study on the synthetic images: three radiologists scored 60 cases on a five-point Likert scale. Overall diagnostic quality: 3.51 for BrainGFM-generated FLAIR, 3.57 for truly acquired FLAIR, 2.87 and 2.76 for the two generative comparators. But at the strict threshold of 4 or more on all five criteria, 15.6% of BrainGFM cases pass versus 51.1% of acquired cases; at 3 or more, 81.7% versus 72.2%, which is surprising and calls for an explanation the text does not give.
Assistance study: in 300 patients, three radiologists classify the subtype (large-artery, cardioembolic, small-vessel) from diffusion and clinical data, then re-read the cases more than a day later with BrainGFM's prediction. The model alone reaches 79.7% accuracy. The radiologists' mean goes from 55.6% to 63.4% (+14.0% relative, i.e. +7.8 points). Clinical translation, our own calculation from these percentages: per 1,000 patients, about 444 wrong subtypings unaided versus 366 aided, i.e. 78 fewer errors; the model alone would make about 203. The effect is very uneven: radiologist 2 goes from 53.0% to 75.0%, radiologist 1 from 57.3% to 56.0% and radiologist 3 from 56.4% to 59.2%. Radiologist 2 corrected 80 of 102 initial errors, but also turned 15 of 22 correct answers into errors.
What is good
First, validation goes beyond the development centre: four external centres, 4,796 patients in total, and external performance close to internal performance for segmentation (76.9% versus 78.1% Dice). This is exactly what many imaging models lack.
Second, the comparators are varied and include recent models: MedSAM, BrainIAC, a MAE, CAR, generative models (Flow Matching, CycleGAN, RegGAN, diffusion) and networks trained on all data. The model adapted with few cases even beats a full-data UNet (69.3% internal Dice).
Third, the authors measure the effect of assistance on humans, with a re-read delay of more than a day between the two readings, and they report the downside: the rate of accepting wrong predictions, from 41% to 82% depending on the reader. Few imaging-AI studies publish this.
What is less good
First, a risk of population and site bias. All six validation centres are in Jiangsu province, China, while the pretraining data come from public datasets that are mostly Western. The paper does not say how the model behaves on other populations, other scanners or other countries. Nothing is reported by sex or age in the text consulted.
Second, the metric and reading protocol invite confusion. Image quality judged close to acquired is not diagnostic proof: on 60 cases and 3 readers, at the strict threshold, generated images remain clearly behind (15.6% versus 51.1%). A synthetic FLAIR can be plausible and yet invent or erase a lesion (hallucination), a failure mode specific to generative models. The authors acknowledge that interchangeability requires prospective validation.
Third, the assistance study relies on only three readers, with opposite responses: one gains 22 points, another loses 1.3. A reader who accepts 90% of suggestions (radiologist 2) is also the one who turns correct answers into errors: this is automation bias, not a proven benefit. The authors themselves call the confidence-stratified analysis exploratory. Acknowledged limits add to this: mostly 2D axial modelling, and gains that plateau between 1,000 and 3,000 synthetic images. The arXiv page consulted states no code repository, weights, funding or conflicts of interest; the paper's licence is CC BY-NC-SA 4.0. This is not a peer-reviewed article, and no prospective validation is reported.
What it changes
For the research community, the idea of a reusable generative prior that produces data to train other models is an interesting avenue when annotations are lacking. Other teams will need to test it, which requires code and weights to be released. The plateau seen between 1,000 and 3,000 synthetic images also suggests this approach does not replace the diversity of real data.
For clinicians, nothing changes today: this is a research prototype, with no CE marking or FDA clearance mentioned. In the medium term, assistance with ischaemic stroke subtyping is the most telling result, but its effect depends on the reader, and a prospective trial would be needed to know whether the gain survives real practice and the tendency to follow the AI when it is wrong.
For patients and the public, the impact is indirect: better models trained with fewer annotations could widen access to imaging tools, but the paper demonstrates no benefit for patient care.
Going further
Preprint: He Y., Wen X., Lan H., Liu C., Liu Y., Xi Y., Yu W., Kong J., Zhang Z., Jiang L., Ge R., Chen Y. and the Multi-Center Cooperative Clinical Study Group for "Lesion-Phenotype Brain Mapping in Acute Stroke", "A generative foundation model for computational brain imaging", arXiv:2611.00265, arxiv.org/abs/2611.00265. Pretraining datasets: ADNI, BraTS2021, CQ500, RSNA Intracranial Hemorrhage, among others. Code and weights: not stated on the arXiv page consulted.
This analysis was produced with AI assistance from the preprint text; calculations not found in the paper are flagged as such, and the English, Spanish and Chinese versions are AI-generated.