Are medical imaging foundation models interchangeable? What a controlled dissection of 25 encoders reveals about representational "convergence"

A German academic team tested an idea that has become received wisdom: as they grow, medical image encoders supposedly converge toward a shared representation, making them interchangeable. Dissecting 25 open-weight encoders (18 image, 7 text), from 7 million to 27 billion parameters, across five modalities and 650,982 chest radiographs, the authors show that convergence is real but modest, that it is driven by the self-supervised training objective rather than by scale or clinical supervision, and that it does not reproduce how a radiologist judges the similarity between two cases.

The context

A medical imaging foundation model is not a diagnostician: it is first an encoder, a pretrained neural network that turns an image — a radiograph, a CT slice, a slide — into a vector of numbers, a compact "fingerprint" meant to summarize its content. A small classifier is then attached for a specific task (detect pneumonia, a nodule, a fracture). The appeal: a single well-trained encoder serves dozens of tasks with little labeled data.

For two years, hospitals and vendors have swapped these encoders like interchangeable bricks — replacing one with another on the assumption that they "see" the same thing. That intuition rests on a fashionable machine-learning hypothesis, sometimes called the Platonic Representation Hypothesis: as models grow and train on more data, their internal representations supposedly converge onto a single shared geometric structure. If true in medicine, this would justify treating two different encoders as equivalent. The problem, the authors stress, is twofold: no one has properly checked that this convergence is real in medical imaging, and the tools that measure it — representational similarity measures — are notoriously fragile, meaning the same pair of models can look very close or very far apart depending on the chosen metric.

The method

The work is a controlled dissection, not a performance benchmark. The authors assemble 25 open-weight encoders (public weights, run locally, hence reproducible): 18 image encoders and 7 text encoders, from 7 million to 27 billion parameters, across five imaging modalities. The empirical base is massive on chest radiography: 650,982 images from six distinct datasets.

Three families of training objectives are compared. Self-supervision (self-supervised learning): the model learns from images alone, without labels, by solving internal tasks (reconstruct a masked patch, bring two views of the same image together). Label supervision: the model is trained to predict annotated diagnoses. And image-text learning: the model aligns images and reports, CLIP-style. To avoid confusing correlation with cause, the authors go beyond simply comparing existing models: they train their own encoders differing only in objective, with identical data, architecture and size, then reproduce the effect in a synthetic toy model. This is the decisive part of the protocol: it isolates what causes convergence.

Convergence is quantified by a representational alignment measure (how far two encoders organize the same images into similar neighborhoods), compared with a "random floor" — the alignment you would get by pure chance. Finally, a clinical-usability test: train a linear classifier (the simplest possible: a weighted combination of the fingerprint's dimensions) on one encoder's representations, then measure how much performance it retains when transferred to another encoder and to five hospitals never seen in training.

The results

Convergence exists, but it is modest: above the random floor, no more. Above all, it is not produced by what people believed. It is the matched self-supervised encoders that align the most — 40.4% on chest radiography — far ahead of label-supervised models (21.1%) and image-text models (3.3%). In other words, the training objective decides almost everything, and clinical supervision (diagnostic labels) aligns less, not more.

A second surprise: convergence does not grow with size. The correlation between parameter count and alignment is not significant (Spearman 0.302, p = 0.223). A bigger model is therefore not mechanically more "convergent" than a small one — the "scale eventually unifies everything" argument does not hold here. Convergence also stays within-modality: it does not link images to clinical language, and — crucially — it does not reproduce how radiologists judge that two cases resemble each other. The models' shared geometry and expert similarity judgment do not coincide.

Yet one result cuts the other way and qualifies the picture: a linear classifier trained on one encoder transfers to other encoders and to five unseen hospitals while retaining about 85% of the "home" performance. In operational terms: porting a decision head from one encoder or site to another costs on the order of 15% of relative performance. For a screening tool this is not trivial — if "home" sensitivity were 90%, a 15% relative drop would bring it to about 76%, i.e. one in seven flagged positives slipping into false negatives. Portability exists, but it is neither free nor guaranteed, and it requires local revalidation.

What is good

A causal protocol, not merely descriptive. Most work on "convergence" simply measures the resemblance of existing models — which mixes the effects of objective, data, architecture and size. Here the authors train encoders varying only in objective, with everything else held constant, and replay the effect in a controlled synthetic model. This experimental discipline supports a causal claim — "the objective produces the alignment" — that correlational studies cannot make.

Scale and transparency. Twenty-five open-weight encoders run locally, from 7M to 27B parameters, five modalities, 650,982 chest radiographs from six sources: the base is broad and reproducible, and the choice of public weights avoids the black box of proprietary APIs. The preprint is under the open CC BY 4.0 license.

Honesty about the fragility of the measure. The authors' starting point is that representational similarity measures are fragile — and rather than sweep the problem aside, they make it the core of the work: they test the clinical usability of convergence (classifier transfer, comparison to radiologists' judgment) instead of stopping at an abstract alignment number. Confronting your own metric with reality rather than presenting it as truth is exactly the stance one hopes for.

What is less good

A metric whose absolute meaning remains misleading. "40.4% alignment" has no direct clinical interpretation: it is a position on a similarity scale that depends on the chosen estimator, and we do not know which threshold would make two encoders "clinically equivalent". Even controlled, this misleading metric does not say whether a patient would be better or worse classified — it only says the geometries partly resemble each other. The risk, symmetric to the one it denounces, is that a hurried reader reads "convergence" as "proven interchangeability", when the paper says the opposite.

The point that matters most clinically is asserted but lightly instrumented. The authors conclude that one must validate "where the shared geometry is weakest, across patient subgroups". That is the right recommendation, but population bias — performance collapsing on subpopulations (age, sex, ethnicity, scanner, hospital) — is not mapped in detail in the body of the work, which is heavily concentrated on chest radiography and on predominantly Western public datasets. It measures a geometry and an average transfer, never a clinical outcome by subgroup, and without prospective validation.

From representation to patient, the road remains long. The "85% retained" transfer is relative to within-encoder performance, not an absolute sensitivity or specificity on a deployed task. The study evaluates a property of the fingerprints and a linear probe, not a device under real conditions; the interoperability promise ("it works with any encoder") is thus demonstrated neither clinically nor prospectively. Finally, this is a preprint not yet peer-reviewed.

What it changes

For the research community, the study buries a comfortable shortcut: no, scale and clinical supervision do not automatically make medical encoders converge onto a common representation. What governs alignment is the pretraining objective, and self-supervision contributes more than labels. Practical consequence: interoperability between models is not a gift of scale, it is a property to design for — via the training objective — then to validate precisely where the shared geometry is weakest. In passing, the work is a reminder of the hygiene needed around similarity measures, too often cited as truths.

For clinicians, nothing to deploy, but a useful warning: swapping one foundation model for another "under the hood" is not neutral. A vendor claiming its tool "works with any encoder" or "transfers to any hospital" is making a promise this work invites you to check site by site and subgroup by subgroup, not to take on faith. The right question is not "do these models converge?" but "do they preserve my sensitivity on my population?".

For patients and the public, the study tempers the image of a single great "medical brain" toward which all AIs would converge. Models from different labs are not secretly the same, and their internal resemblance does not reproduce a radiologist's judgment. Concretely, the safety of an imaging tool cannot be inferred from its kinship with another: it must be demonstrated on the ground, on the populations actually concerned.

To go further

The preprint is available on arXiv (2607.20274), posted on 22 July 2026 by Soroosh Tayebi Arasteh, Sebastian Ziegelmayer, Mahshad Lotfinia, Lisa Adams, Sven Nebelung, Jakob Nikolas Kather and Daniel Truhn (German university teams in radiology and medical informatics); it is under the CC BY 4.0 license and has not yet been peer-reviewed. On imaging foundation models and their blind spots, see our decryptages on the robustness and generalization of histopathology foundation models, on multimodal probing and confidence of cancer foundation models, and on hidden subgroup biases in medical imaging.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. English, Spanish and Chinese translations produced with AI assistance and reviewed.