médical IA

CRS-Bench: scoring medical image encoders on four axes instead of AUROC flips 21 of 105 pairwise comparisons — but predicts cross-hospital transfer worse than AUROC does

Published on August 26, 2026 · 14 min read

Xingtao Lin, Hangqi Ren, Caiwan Sun and You Chen, of Vanderbilt University Medical Center, compared fifteen families of pretrained image encoders across three specialties — dermatology, ophthalmology, chest radiology — in 17,575 matched runs, scoring each on four axes rather than AUROC alone. Their composite score, the Clinical Reliability Score (CRS), reverses 21 of the 105 pairwise comparisons AUROC alone would produce, with a mean absolute displacement of 1.87 ranks. But when it comes to predicting performance at an outside hospital, that composite does worse (rho = 0.529) than the raw AUROC it is meant to supplement (rho = 0.864) — and it is the authors themselves who report this.

The context

An image encoder is the invisible half of nearly every medical imaging AI system: the network that turns a photograph of a skin lesion, a fundus image or a chest radiograph into a vector of a few hundred numbers. The classifier that follows — the part that says melanoma or not melanoma — is often trivial by comparison. Choosing the encoder, however, has become a modeling decision in its own right, because the supply has exploded into three families. Generalists trained on web images (ResNet-50, ViT-B/16, CLIP, SigLIP, DINOv2, MAE). Broad-medical encoders trained on biomedical article figures or medical image-text corpora (BiomedCLIP, LLaVA-Med, MedSigLIP, MedGemma). Specialists trained exclusively on skin (PanDerm), retina (RETFound) or chest (RAD-DINO, BioViL, CheXzero).

How is the choice made today? In the vast majority of papers, on a single table row: AUROC on a clean test set. AUROC (area under the ROC curve) is the probability that the model scores a randomly drawn positive case above a randomly drawn negative one — 1.0 is perfect, 0.5 is chance. It is a good measure of ranking, and a measure blind to three things that matter clinically. Is the stated confidence honest? How many expert annotations does it take to reach that level? What happens when the image is blurred, compressed, or comes from another hospital? The paper starts from that observation and tries to turn it into a measurable protocol.

The method

The data. Four public datasets. CheXpert (191,229 chest radiographs, 14 labels, multilabel) for radiology; ISIC 2019 (25,331 dermoscopy images, 8 classes) for dermatology; APTOS 2019 (3,662 fundus photographs, 5 ordinal diabetic retinopathy grades) for ophthalmology. MIMIC-CXR serves as an institutional shift target: probes trained on CheXpert are applied to it without retraining, the two databases sharing the same 14 labels but differing in institution, population, protocol and labeling.

The protocol. All encoders are frozen. Features are extracted in a single pass, standardized with training-split statistics, then an l2-regularized logistic regression is fit on top — this is linear probing, and the point is to measure representation quality, not optimizer quality. Two control probes (a two-layer perceptron, a k-nearest-neighbor with k = 5) verify that conclusions do not depend on the classifier. Five shared random seeds, matched splits, matched label subsamples, matched perturbation seeds, matched hyperparameter budget for everyone. Total: 17,575 runs and 3,515 seed-aggregated metric rows, over 43 of the 45 possible encoder-dataset cells (BioViL and CheXzero have no APTOS cell, their released interfaces being chest-specific; missing values are never imputed).

The four axes. Discrimination is macro-AUROC. Calibration is measured by ECE (expected calibration error, 15 bins): the average gap between stated confidence and actual accuracy — an ECE of 0.15 means that on average, when the model says 90 %, it is right about 75 % of the time. Label efficiency is the mean ratio of AUROC obtained with 1 %, 5 %, 10 % and 25 % of labels to AUROC at full labeling; it therefore measures preservation, not absolute attainment. Robustness combines the mean drop and the worst drop in AUROC under five perturbations (Gaussian blur, JPEG compression, contrast reduction, Rician noise, field-of-view crop) at three severities, plus, for CheXpert only, the real drop observed when moving to MIMIC-CXR.

The score. CRS aggregates these four axes, normalized by fixed anchors, through three components: a Pareto dominance term (how many panel peers the encoder dominates on all axes at once), a closeness to the ideal profile term, and a worst-axis performance term, weighted 0.60, 0.25 and 0.15 respectively. The authors state that these weights are explicit design parameters, not learned clinical utilities. Importantly, because the anchors are fixed, a new encoder arriving later does not change previously published scores.

The results

The frontier depends on the metric. In aggregate AUROC, MedGemma and MedSigLIP lead (0.909 and 0.908). In aggregate CRS, it is PanDerm (0.703) and MedSigLIP (0.701) — separated by 0.002, which is to say nothing. Across 10,000 paired-seed bootstrap replicates, PanDerm and MedSigLIP take rank 1 with probability 0.664 and 0.336, and the 95 % interval of their paired difference, [-0.097, 0.017], contains zero. The authors therefore decline to name a winner and speak of a stable leading tier: PanDerm, MedSigLIP, MedGemma, all three with probability 1.000 of being in the top three, against exactly zero for the other twelve.

The measured disagreement between AUROC and CRS. Across the fifteen encoders, Kendall tau = 0.600, Spearman rho = 0.811, 21 of 105 pairwise choices reverse, mean absolute displacement of 1.87 ranks, top-3 overlap of 0.667. Individual moves are telling: CheXzero goes from 11th to 5th, LLaVA-Med from 5th to 11th, PanDerm from 4th to 1st, DINOv2 from 6th to 9th.

Calibration is repairable, discrimination is not. Across six primary tests, post hoc recalibration (temperature, vector, isotonic or Dirichlet, fit on a held-out calibration split) reduces ECE by 54 to 77 % — mean absolute reduction 0.067, 95 % CI [0.024, 0.110] — without changing AUROC (mean delta about +0.001). Examples: DINOv2 on APTOS goes from 0.150 to 0.034; MedSigLIP on APTOS from 0.121 to 0.028. Practical translation: an encoder that ranks well but states its probabilities badly is not a bad encoder, it is an encoder nobody has recalibrated yet, and the operation costs a few hundred labeled cases.

Image degradation breaks confidence before it breaks ranking. Under perturbation, LLaVA-Med ECE on APTOS rises by 0.128 on average, peaking at 0.418; on ISIC, MedGemma and MedSigLIP see ECE rise by 0.113 and 0.094 while remaining good discriminators. Across all 43 cells, loss of discrimination and loss of calibration are only moderately associated (r = 0.63). In other words, a slightly blurred image can leave patient ranking nearly intact while making displayed probabilities plainly wrong — precisely the most dangerous failure mode for a decision-support tool that displays a percentage.

Institutional shift. Moving from CheXpert to MIMIC-CXR without retraining, medically pretrained encoders retain a macro-AUROC of 0.717 against 0.540 for generalists (Welch test, p = 0.0029). The 0.540 figure deserves to be read literally: on a binary ranking task, that is close to chance. And medical provenance is not sufficient — LLaVA-Med falls to 0.504.

Two useful secondary results. At 1 % of labels, MedGemma already reaches 0.884 AUROC on APTOS, and MedSigLIP, MedGemma and PanDerm reach 0.819, 0.815 and 0.806 on ISIC; on CheXpert, six medically pretrained encoders match ViT-B/16 at 1 % of labels while MAE and ResNet-50 never catch it. And intermediate layers beat the final layer in 5 of 8 depth sweeps: RAD-DINO gains 1.3 to 1.6 AUROC points around 50 % depth, DINOv2 prefers 75 %. The last block of a transformer is therefore not automatically the best clinical representation.

What is good

The matching is serious, and that is rare. Matched splits, matched label subsamples, perturbation seeds derived from a SHA-256 hash of the image identifier — so that the same degraded image reaches every encoder. Matched probe capacity, matched hyperparameter budget, matched wall-clock cap. Most published encoder comparisons match none of this, and their 0.01 AUROC gaps are indistinguishable from protocol noise.

The authors demolish one of their own candidate metrics. They show that the most common label-efficiency measure, area under the learning curve (AULC), is essentially a restatement of full-label AUROC: rho = 0.986 across encoders, 0.977 across the 43 cells. Their relative-retention measure drops these correlations to 0.339 and -0.342. That is quantified self-criticism, and it justifies the formula choice rather than postulating it.

Reference invariance is demonstrated, not asserted. Across 30 trials inserting three new encoders into a panel of twelve, cohort renormalization displaces already-published scores by 0.053 on average and up to 0.167; the fixed-anchor rule gives exactly zero displacement. That is the difference between a ranking you can cite in six months and one that moves every time a model is released.

What is less good

External validity of the composite is weak — and worse than AUROC. Removing MIMIC-CXR from the CRS computation and then trying to predict MIMIC AUROC, CRS gives rho = 0.529 against 0.864 for CheXpert AUROC alone. Under leave-one-dataset-out, rho is 0.550 / 0.709 / 0.271 for ISIC / APTOS / CheXpert, and the held-out winner is recovered in no fold. The authors are explicit: CRS is a benchmark-level comparative summary, not a predictor of AUROC on an unseen task. That is honest, but it heavily qualifies the use one would be tempted to make of it — choosing an encoder for a project that is none of the three tested.

The frozen ranking does not survive adaptation, yet nobody deploys a frozen encoder. With parameter-efficient LoRA fine-tuning (rank 8, alpha 16, eight epochs, three seeds), discrimination improves everywhere (+0.026 / +0.030 / +0.010 on ISIC / APTOS / CheXpert) but the order changes: Kendall tau of 0.829 / 0.667 / 0.448, and the in-domain leader changes in all three domains (MedGemma yields to MedSigLIP on ISIC and APTOS, MedSigLIP yields to RAD-DINO on CheXpert). Against frozen CRS, the four-axis comparison after adaptation gives tau = 0.562 and a mean shift of 2.0 ranks, with BiomedCLIP falling from 4th to 13th. The top three hold, the middle of the table reshuffles entirely. This is a biased comparator problem: the protocol measures a regime (frozen encoder) that is not the one used in practice.

Synthetic stresses model neither population nor pathology, and only one real shift is observed. The authors write it themselves: the perturbations model acquisition and image quality, not changes in anatomy, pathology or population. The benchmark's only genuine institutional shift is CheXpert to MIMIC-CXR, two American academic hospitals — which leaves population bias untouched, a particularly acute issue on ISIC 2019, whose underrepresentation of darker skin has been documented for years. No subgroup analysis, no operating-point selection, no human-AI interaction is in the protocol, as the authors acknowledge. Two further reservations they do not state: Rician noise is a noise model specific to magnitude MRI images, and applying it to dermoscopy, fundus photography and digital radiography is physically questionable; and the paper contains no code repository link, no funding statement and no conflict-of-interest declaration, while ranking models released by Google (MedSigLIP, MedGemma) and Microsoft (BiomedCLIP, RAD-DINO, BioViL).

What it changes

For the research community. The most reusable result is not the ranking, it is the quantified demonstration that disagreement between AUROC and a multi-axis profile is structural rather than marginal: 21 reversals out of 105, a tau of 0.600. Publishing a four-axis profile rather than a scalar becomes defensible. And the fixed-anchor idea — a ranking that does not renormalize with every new model — deserves to be picked up by other medical benchmarks, where adding a competitor today changes every predecessor's score.

For clinicians and deployment teams. Two immediate, low-cost consequences. First, if a tool displays probabilities, its calibration should be checked and, if needed, repaired on a local cohort: 54 to 77 % ECE reduction with no loss of ranking is the best effort-to-benefit ratio in the paper. Second, image quality degradation should be monitored in its own right, because it can leave ranking intact while making displayed percentages wrong — a silent failure mode no AUROC-based dashboard will catch.

For patients and the public. Nothing changes today: none of the fifteen models is evaluated here in clinical conditions, none has a regulatory status discussed in the paper, and the authors explicitly exclude prospective validation from their protocol. What the paper contributes to public debate is more modest and more durable: the sentence model X beats model Y is meaningless without saying on which axis, and the confidence percentage a medical tool displays is a quantity that can be measured, can be wrong, and can be corrected.