RadGrounder: a vision-language model for radiology that shows where it is looking, trained on 1.2 million slices without manual annotation

A team from the universities of Freiburg and Aarhus presents RadGrounder, a vision-language model that writes a radiology report, answers questions about an image and — above all — points, on the CT or MRI slice, to the structure it is talking about. Its real contribution is not a record score but a data-manufacturing method: 1.2 million image-text pairs and 9.6 million question-answer pairs were generated automatically, without manual annotation, by putting other language models to work on a single hospital's reports. The result rivals the best existing medical models on two public question sets — but the spatial grounding only points to organs, never to the lesions, and the entire training chain rests on labels produced by AIs that no radiologist ever checked.

The context

A vision-language model (VLM) is a system that takes an image as input and produces text as output: describing a scan, answering a question posed in natural language. In radiology, these models promise to help draft reports and query an image. Their most documented flaw is hallucination: the model confidently asserts something that is not in the image — a non-existent lesion, an invented location.

One way to discipline these models is spatial grounding: forcing the system to designate, with a bounding box or a mask, the precise region of the image its assertion refers to. A "grounded" output is verifiable — one can check whether the model points to the right place. The practical problem is cost: training a model for grounding normally requires radiologists to hand-draw thousands of boxes on images, slow and expensive work. It is this bottleneck the preprint tries to remove, by manufacturing the annotations automatically rather than by hand.

The method

The heart of the work is a dataset, RefRad2D, built from the clinical archives of a university medical centre (Freiburg, Germany) spanning a decade of practice. It gathers 1.2 million image-text pairs drawn from 945,000 CT slices and 321,000 MRI slices. The models here work on individual two-dimensional slices, not on whole 3D volumes.

The manufacturing of labels is almost entirely automated, and that is the decisive point. The reports, written in German, are translated into English by the Gemma 3 model (27 billion parameters), the translation quality refined using another model, GPT-OSS (120 billion parameters), as a judge. That same GPT-OSS then rewrites the captions to strip out temporal references ("the lesion has grown since the last exam") that make no sense on a single image and push the model to hallucinate. A question-answer subset — about 9.6 million pairs — is generated by Gemma 3, at several questions per image (open, yes/no, multiple choice).

For grounding, the authors use no radiologist: they pass the volumes through TotalSegmentator, an automatic tool that segments organs, producing masks for 117 anatomical categories on CT and 50 on MRI, harmonised into 121 classes. GPT-OSS spots the anatomical mentions in each caption and links them to the corresponding masks; an image is kept only if text and mask really refer to the same structure. This filtering yields 236,157 spatially annotated slices.

The model itself, named RadGrounder, is built on PaliGemma 2 (3 billion parameters): a SigLIP visual encoder (adapted to the medical domain then frozen, i.e. held fixed during final training) turns the image into tokens, which a Gemma-2B language decoder reads alongside the text. The grounding trick is to treat detection as text generation: a box's coordinates are cut into 512 discrete values that the model "writes" like words. Full training fits on a single H100 GPU in about two and a half days.

Evaluation combines an internal test set (from RefRad2D) and, crucially, two external public sets, Slake and VQA-RAD, the reference benchmarks for medical-imaging question answering. The metrics: CIDEr (description quality), F1 (harmonic mean of precision and recall), accuracy on closed questions, and G-IoU, a measure of overlap between the predicted box and the true region. An automatic quality score, the LLMScore (Gemma 3 acting as judge), was validated against three radiologists on 200 questions.

The results

On the external benchmarks, RadGrounder holds its own against the best medical models. On Slake it reaches an F1 of 87.7 and an accuracy of 90.3% on closed questions — neck and neck with BiomedGPT-B (85.2 and 89.9%) and LLaVA-Med (86.8% accuracy), and above Med-Gemini (75.8 F1) and MedGemma (72.3). On VQA-RAD, harder, it obtains the best open-question F1 among the compared models (50.7) but a more modest closed-question accuracy (64.7%). For grounding, the text-generation variant beats a dedicated segmentation-head variant: G-IoU of 43.6 against 36.9.

Clinical translation calls for restraint. A G-IoU of 43.6 means the predicted box on average overlaps less than half the target region: fine for a proof of concept, far from reliable localisation. And this grounding concerns whole organs, not abnormalities. The model can point to "the liver", not "the liver tumour". On the metrology side, the LLMScore validation is careful — inter-radiologist agreement of 0.958 (Krippendorff), correlation of 0.977 between the automatic score and the human mean — but it was measured on a small external sample, not on the distribution of questions manufactured by the model itself.

What is good

A data engine that removes manual annotation. Producing 1.2 million image-text pairs, 9.6 million question-answer pairs and 236,000 grounded slices without a human drawing a single box, on one GPU in two and a half days, is a real and reproducible methodological contribution. The detail of automatically removing temporal references from captions is a clever counter to a known cause of hallucination.

Spatial grounding as anti-hallucination discipline. Forcing a model to designate the region that justifies its assertion is the right direction to make its outputs verifiable. The choice to treat detection as text generation, simpler than a dedicated segmentation head, also turns out to perform better here (43.6 against 36.9 G-IoU), which streamlines the architecture.

A methodologically clean evaluation. Bootstrap confidence intervals over 10,000 resamples, validation of the automatic score against three radiologists, and above all correction of a known trap: VQA-RAD contains images shared between training and test, a data leakage (contamination of the test set by training, which inflates scores); the authors adopt a cleaned split to avoid it. That is the kind of rigour often missing from the literature.

What is less good

Labels manufactured by AIs, never checked by a radiologist: the absent-comparator and absent-ground-truth failure modes. Translations, caption rewrites, 9.6 million question-answer pairs, organ masks, text-image matching — all produced by Gemma 3, GPT-OSS and TotalSegmentator. No human ground truth validates these triplets. Any systematic error of these teacher models is learned and then embedded by the student, and RadGrounder's quality ceiling is that of its generators, not that of a radiologist.

The grounding points only to anatomy, not pathology: a shortcut that weakens the promise. TotalSegmentator segments organs, so "showing where it looks" designates normal structures, not lesions — precisely what a clinician would want localised. The advertised verifiability is therefore far weaker than it seems, and a G-IoU of 43.6 remains modest even on these easy anatomical targets.

A partly circular metric and a population bias. The LLMScore uses Gemma 3, the same model family that generated the training questions, to judge the answers — a risk of a misleading metric through complacency, which validation on 200 external items does not fully rule out. Add a population bias (a single German hospital, machine-translated rather than native English), an evaluation limited to benchmarks without a prospective clinical study, and an absence of release: code promised "upon publication" but not published to date, data not shared, licence unspecified.

What it changes

For the research community, the real contribution is the data engine: a reproducible template for building grounded radiology training sets without an annotation budget, and an elegant way to do detection through text generation. It is a tool other teams will reuse — provided they are aware they also inherit the biases of the generating models.

For clinicians, nothing is deployable today: the grounding concerns organs and not abnormalities, evaluation stays confined to question sets, the data come from a single centre, and the weights are not available. For patients and the general public, the study illustrates two reading rules. First, "the AI shows where it looks" suggests it points to your tumour, whereas here it points to the organ. Second, "trained on 1.2 million images" hides that the labels were written by other AIs, not by doctors — a distinction that changes everything when judging reliability.

To go further

The preprint is available on arXiv (2606.20477), with a dedicated project page (radgrounder.github.io); the automatic segmentation tool used for grounding is TotalSegmentator. On clinical vision-language models and their pitfalls, see our decryptage of a model where text dominates the image; on adapting radiology models with little data, the one on PromptRad; and on large language models in radiology, the one on GPT-4's explanation format.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. English, Spanish and Chinese translations produced with AI assistance and reviewed.