Robustness of pathology foundation models: what a "perturbations plus distribution shift" benchmark reveals about their brittleness

Three researchers at the University of Warwick put twelve digital-pathology foundation models, and two reference convolutional networks, through eleven realistic image perturbations and a validation protocol built to break the resemblance between training and test data. The large models resist better than the older networks, but their robustness plateaus well before the billion-parameter mark, and all lose accuracy the moment they face genuinely different distributions. A useful reminder: clinical reliability will rest on the quality and diversity of training data, not on model size.

The context

Digital pathology — the analysis of high-resolution scanned tissue slides — has become one of the most active arenas of medical AI. For two years a new generation of tools has dominated it: foundation models, very large networks pre-trained by self-supervised learning (the model learns to represent millions of tissue images without labels, by setting itself prediction tasks) on massive corpora, then reusable across many diagnostic tasks. GigaPath, UNI, Virchow, Phikon: these models have pushed scores up on nearly every classification, segmentation and prognosis benchmark.

But a high score on a clean test set says nothing about reliability under real conditions. In pathology, two slides of the same cancer can differ in staining, scanner, laboratory protocol, patient population. A solid model must withstand these variations — the question of robustness — and stay accurate when deployment data differ from training data — that of out-of-distribution generalisation. This preprint, from the Tissue Image Analytics Centre team at the University of Warwick, asks three head-on questions: are these models resilient to clinically plausible image alterations? do they hold up under different distributions? and does the race to size still improve anything, or has it hit its plateau?

The method

The authors evaluate twelve pathology foundation models, from the mid-sized transformer encoder (Hibou, Phikon-v1/v2, EXAONEPath, UNI, between 80 and 300 million parameters) to the billion-plus-parameter model (GigaPath, H-Optimus-0/1), via the intermediate range (Virchow, Virchow2, UNI2, 600 to 700 million). Two classic convolutional networks (ResNet18 and ResNet50) serve as the "old-generation" comparison. All encoders are frozen: the model is not retrained, only a single linear classification layer is added on top and trained for each task (the linear probing technique, which tests the quality of the learned representations without altering them).

Four public datasets provide the ground, all cut into small image patches: NCT (colorectal tissue, reduced to a binary tumour / non-tumour task), PANDA (prostate biopsies graded by the Gleason score, reduced to cancer / benign), PanNuke (detection of tumour nuclei) and PatchCamelyon (about 330,000 lymph-node patches, presence or absence of metastasis). Baseline performance is measured in ROC-AUC (area under the ROC curve: the ability to correctly rank a positive case above a negative one, 0.5 = chance, 1.0 = perfect).

Two original devices structure the evaluation. First a battery of eleven realistic perturbations, produced by an in-house tool (REET), in three families: pixel-level (noise, brightness, JPEG compression), stain-level (hue variation, mixing in the histology colour space) and geometric (blur, rotation, crop, zoom). The authors summarise each model's resilience with an index, the PPI (Perturbation Performance Index), which integrates AUC as perturbation intensity rises: PPI of 1 = perfect robustness, lower values = degradation. Then a non-redundant validation protocol (NR-Kfold): instead of splitting patches at random between training and test, images that resemble each other in representation space are grouped, then whole groups are assigned to separate folds. This forces a dissimilarity between training and test, simulating the move from one hospital, scanner or population to another.

The results

On perturbations, the hierarchy is clear. Pixel-level alterations — noise, brightness, JPEG compression — are the most destabilising, with PPIs of roughly 0.4 to 0.6. Stain variations do moderate damage (PPI of roughly 0.6 to 0.75). Geometric transformations (blur, rotation, crop, zoom) are best tolerated (PPI of roughly 0.8 to 1.0), a sign that these models have learned spatial invariance well. Among the models, EXAONEPath, Virchow2, UNI2 and Phikon-v2 consistently lead; smaller or older foundation models (Hibou, Phikon-v1) and the ResNets fall behind, especially under pixel or stain stress.

The central result concerns size. Despite three orders of magnitude between the smallest and largest model, the correlation between parameter count and robustness stays weak (coefficient of 0.15 to 0.55 depending on the dataset). The improvement is real from small to mid-sized model, then clearly saturates beyond 300 to 600 million parameters: models above one billion parameters (H-Optimus, GigaPath) bring no further gain, and sometimes decline. In other words, what matters is the diversity and quality of pre-training, not sheer scale.

The generalisation test, for its part, exposes a hidden brittleness. As soon as the resemblance between training and test is broken with the NR-Kfold protocol, all models lose accuracy: the AUC drop ranges from −0.02 on NCT to −0.23 on PanNuke, that last figure betraying that this dataset's usual split shared patients and slides between training and test — a textbook case of data leakage. The large and mid-sized foundation models (Virchow2, UNI2, H-Optimus) take it best, with a mean loss of about −0.07 AUC and low fold-to-fold variability (standard deviation of 0.09 to 0.12). The older transformers and the ResNets drop more (up to −0.36 AUC) and turn unstable (standard deviation above 0.15). Here too size barely helps: the global correlation between parameters and out-of-distribution AUC is weak and non-significant (coefficient of 0.22, p = 0.096), and the smoothed trend even declines slightly beyond 600 million parameters.

What is good

A systematic, open benchmark. This is, to the authors' knowledge, the first evaluation that combines, on the same models, a robustness-to-perturbation analysis and an out-of-distribution generalisation analysis. Fourteen models, four datasets, eleven perturbations, two protocols, and a published toolbox (the pfm-robustness-toolkit repository): the work is reproducible and directly reusable by other teams — something sorely lacking in a literature where everyone reports their own figures on their own split.

A protocol that flushes out data leakage. Non-redundant validation is no gimmick: by forcing dissimilarity between training and test, it reveals that the usual random splits overstate performance. The −0.23 AUC on PanNuke is exactly the kind of drop that stays invisible in a standard evaluation, and surfacing it is a service to the field — a cheap way to detect, before any deployment, a model's optimism about its own data.

An argued case against sheer scale. Against the "bigger = better" reflex that dominates the foundation-model field, the authors show, with figures in hand, that robustness saturates around a few hundred million parameters. It is a useful, counter-intuitive conclusion that redirects effort toward the diversity of pre-training data rather than the race to size — and thus toward a more compute-frugal AI.

What is less good

An evaluation that stays at arm's length from the clinic. Everything plays out on isolated patches, with frozen encoders and a single linear layer on top. Yet a pathology diagnosis is made on a whole slide, often after fine-tuning the model. The benchmark thus measures the quality of the representations, not the performance of an actually deployed system. Above all, NR-Kfold is a synthetic proxy for distribution shift: grouping images that resemble each other in representation space is not the same as testing on a real second hospital, with its scanner and its population. Population bias, the main failure mode in health AI, is tested here only in a test tube.

Binary tasks that inflate the figures. Reducing NCT's nine tissues to "tumour / non-tumour", or PANDA's Gleason score to "cancer / benign", throws away the most useful clinical granularity — it is the distinction between grades that guides prostate-cancer treatment, not the mere presence of a tumour. These simplified tasks are easier and produce flattering AUCs: this borders on the misleading metric. The aggressive subsampling (10% of training data, 2.5 to 10% of the test on NCT) further reduces the statistical power of the comparisons.

No clinical translation of the measured gaps. The article stops at machine-learning metrics — ROC-AUC, PPI, AUC drop — without ever saying what a loss of 0.07 to 0.23 AUC means in misclassified patches, in false negatives on a metastasis, in slides to re-read. A sensitivity and a specificity at an operating threshold, the only figures meaningful to a clinician, are missing. Finally, while shortcut learning — the model latching onto a stain or scanner artifact rather than the tumour — is the implicit cause of these drops, no attribution map is offered to show what the models actually fixate on.

What it changes

For the research community, the contribution is methodological and immediate: robustness evaluation and the dissimilarity test should become a mandatory step before any foundation-model publication, on the same footing as AUC. NR-Kfold offers a cheap probe to detect data leakage and over-optimism. And the demonstration of the size plateau is a strong signal to reallocate compute: aim for source diversity, multimodality and domain alignment rather than the mere swelling of the model.

For clinicians and laboratories, nothing deployable here, but a concrete warning. A foundation model validated on one centre's slides can lose a significant share of its performance on another's scanner and staining — and that loss stays invisible until distribution shift is explicitly tested. The practical lesson: demand multi-centre validation, on your own slides, before granting the slightest trust to a published figure.

For patients and the general public, the message is the same as for any health-AI tool: an impressive score obtained in the lab does not guarantee reliability on your hospital's slides. The good news is that the field is equipping itself with tools to measure its own brittleness — the condition for a trustworthy digital pathology.

To go further

The preprint is available on arXiv (2607.04401), posted on 5 July 2026 by Dhyey Yajnik, Amina Asif and Fayyaz Minhas (Tissue Image Analytics Centre, University of Warwick, United Kingdom); it has not yet been peer-reviewed. The toolbox is published on GitHub (DhLYa/pfm-robustness-toolkit). On pathology foundation models and their limits, see our decryptages on GigaPath, one of the models evaluated here, on expert distillation for a pathology foundation model, and on the probing-based evaluation of multimodal foundation models in oncology.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. English, Spanish and Chinese translations produced with AI assistance and reviewed.