médical IA

OmniBreast: nine breast cancer tasks predicted from a mammogram and an H&E slide, for a 0.009 AUROC gain that does not survive multiple-testing correction

Published on August 25, 2026 · 14 min read

Ten authors from the First Affiliated Hospital of Bengbu Medical University, the Harbin Institute of Technology and Universiti Malaya trained, on 923 breast cancer patients, a system that predicts nine clinical variables — the three TNM components, clinical stage, histological grade, and ER, PR, HER2 and Ki-67 status — from a single mammogram and a single haematoxylin and eosin slide. Their central mechanism, task-conditioned expert routing, improves mean AUROC by 0.009 over fixed fusion (95% CI 0.005–0.013; P = 0.011), yet none of the nine per-task comparisons survives Benjamini-Hochberg correction. The most striking number in the abstract, an AUROC of 0.990 for M staging, rests on roughly seven metastatic patients in a 185-patient test set, and on images in which metastatic disease is, by construction, not visible.

The context

Managing breast cancer relies on three streams of information that arrive neither at the same moment nor in the same document. Imaging — mammography, ultrasound, MRI — detects the lesion and estimates its extent. Histopathology of the biopsy confirms type and grade. Immunohistochemistry then measures oestrogen (ER) and progesterone (PR) receptors, HER2 status and the Ki-67 proliferation index, four markers that determine most of systemic treatment. Each stream produces its own report, and the synthesis happens at the multidisciplinary tumour board.

AI applied to this pathway has followed the same fragmentation. Models were built to detect cancer on mammograms, others to predict histological grade on slides, others still to infer HER2 status. Each has its own training, its own endpoint, its own mandatory input modality. In clinical practice that rigidity creates a mundane but real problem: depending on where a patient is in the pathway, imaging may be available without the slide, or the slide without complete imaging.

The question this paper poses is therefore not only "how to fuse radiology and pathology" but "when is fusion worth anything". To answer it, the authors borrow a mechanism from the large language model vocabulary: the mixture of experts (MoE). The principle is simple — several specialised models coexist, and a small network called the gate decides, case by case, what weight to assign each of them. Here there are three experts per task (radiology only, pathology only, fusion of both), and the gate receives two extra pieces of information: which clinical question is being asked, and which modalities are actually available for this patient.

The method

The cohort. Retrospective, single centre. Of 1,428 patients treated at the First Affiliated Hospital of Bengbu Medical University (Anhui, China) between January 2016 and December 2023, 174 were excluded upfront (no paired radiology-pathology data n = 86, missing labels n = 48, duplicates n = 19, poor image quality n = 21), then a further 331 during curation (incomplete annotations n = 103, preprocessing requirements unmet n = 74, final curation criteria unmet n = 154). That leaves 923 patients, 64.6% of the initial screen, split at the patient level into 646 for training, 92 for validation and 185 for the held-out test set. A second, independent cohort comes from Hunan Cancer Hospital: 394 patients screened, 259 retained.

The inputs. On the imaging side, only pretreatment full-field digital mammograms, craniocaudal and mediolateral oblique views, across three vendor fleets (Hologic 46%, GE Healthcare 34%, Siemens Healthineers 20%), cropped to the breast, normalised between the 1st and 99th percentiles and resized to 512 × 512 pixels. No staging examination — no CT, no PET, no bone scintigraphy — is supplied to the model, and no free-text report either. On the pathology side, only whole-slide images stained with haematoxylin and eosin, digitised on Leica Aperio AT2 scanners at ×40 (about 0.25 µm per pixel) then analysed at ×20 equivalent; immunohistochemistry slides for ER, PR, HER2 and Ki-67 are not given to the model. Tissue is isolated by Otsu thresholding, cut into 224 × 224 pixel tiles with those containing less than 60% tissue discarded, and 128 tiles per patient are aggregated into a single representation by gated attention multiple-instance pooling — a learned weighting that decides which tiles count toward the overall verdict.

Tasks and models. Nine binary questions: T (Tis/T1 vs T2–T4), N (N0 vs N+), M (M0 vs M1), clinical stage (0–II vs III–IV), histological grade (low vs high), and ER, PR, HER2, Ki-67 (negative vs positive or high). For each task and each modality, five convolutional architectures are trained — ResNet, DenseNet, EfficientNet, RegNet, MobileNetV3 — and the best is retained on validation AUROC. That is ninety models adjudicated on 92 patients. Three random seeds (42, 3407, 2024) are used, and the checkpoint with the highest validation AUROC is kept. Fixed fusion is a late-fusion transformer consuming both modality embeddings. Learned MoE adds a multilayer perceptron that weights the three experts; task-conditioned MoE adds to it a vector representing the requested task and a binary modality-availability mask, with 0.30 modality dropout during training. Operating thresholds are set by Youden index on the validation set alone, then frozen. AUROC comparisons use paired bootstrap, with Benjamini-Hochberg correction across the nine per-task P values.

The results

The modality map. This is the paper's most interesting finding, and it is descriptive. Pathology clearly dominates for nodal involvement (AUROC 0.804 vs 0.606 for mammography) and for Ki-67 (0.748 vs 0.625). Radiology does better for HER2 (0.810 vs 0.795) and for clinical stage (0.735 vs 0.626). The two are equivalent for histological grade (0.949 and 0.946). In other words, the most informative modality depends on the question, not on a general hierarchy, and the apparent paradox — predicting nodal involvement from a primary tumour slide, predicting HER2 from a mammogram — is explained by indirect morphological correlations, not by direct visualisation.

The integration gains. Mean AUROC across the nine tasks: 0.813 for the best single-modality expert, 0.826 for fixed fusion, 0.830 for learned MoE, 0.834 for task-conditioned MoE. An oracle selector, non-deployable since it uses test labels, caps out at 0.854. The task-conditioned MoE gain over fixed fusion is 0.009 (95% CI 0.005–0.013; P = 0.011); fixed fusion's gain over the best single-modality expert is 0.012 (0.006–0.018; P = 0.008). Ablations confirm that both ingredients matter a little: removing the task embedding drops the mean from 0.834 to 0.828, removing the availability mask to 0.826.

The missing-modality stress test. This is where the architecture best justifies itself. Randomly masking one modality in 30%, 50% and then 70% of test patients, task-conditioned MoE reaches mean AUROCs of 0.804, 0.793 and 0.779, beating fixed fusion with zero imputation by 0.032, 0.051 and 0.067. The gap against simple rule-based availability routing is narrower: 0.015, 0.018 and 0.024.

The clinical comparator. The authors build reference models on five routine variables (age, menopausal status, breast density, view completeness, histological type), systematically excluding variables that define the target. Logistic regression: 0.698 mean AUROC. XGBoost: 0.724. Task-conditioned MoE beats them by 0.110 (95% CI 0.091–0.129; P < 0.001) and does better on all nine tasks. On the external cohort, two endpoint-extension tasks: mammography-based pathological complete response prediction reaches 0.758 AUROC (against 0.682 for the clinical baseline), and H&E-based triple-negative breast cancer prediction reaches 0.872 (against 0.746).

The clinical translation. Take HER2, the most telling example because it is the marker that gates access to anti-HER2 therapy. The best radiology model reports 76.7% sensitivity and 76.0% specificity. Projected onto 1,000 patients at a 20% HER2-positive prevalence, a usual order of magnitude, that would give roughly 153 HER2+ patients correctly identified, 47 missed, and 192 HER2-negative patients wrongly flagged. This calculation is an extrapolation from a 185-patient test set, not a measurement; it is nonetheless enough to show that none of these numbers replaces immunohistochemistry, and the authors do not claim otherwise.

What is good

The restraint of the reporting, starting with the title. The title says "internal validation", not "validation". The external analyses are explicitly labelled endpoint-extension tests, not external validation of the nine tasks. The conclusion states in plain terms that "the high internal M-stage performance was considered exploratory and requires independent validation because distant metastatic disease was not directly represented in the mammographic or H&E inputs". Above all, the authors apply Benjamini-Hochberg correction to their nine per-task comparisons, report that none remains significant after correction (all adjusted P ≥ 0.072), and themselves conclude that these analyses are exploratory. In a literature where p-hacking through endpoint multiplicity is the norm, that discipline deserves to be named.

The oracle as an upper bound. Adding a selector that cheats — it picks the best expert case by case by looking at the test label — improves nothing, but tells the reader a great deal: it shows that even a perfect routing policy would not exceed 0.854 mean AUROC, against the 0.834 achieved. All remaining headroom for this family of methods on this data therefore amounts to 0.020 AUROC. It is an honest and cheap way of telling readers where the ceiling lies, and it is rare.

The clinical comparator exists, and it is built properly. Many imaging papers stop at "our model reaches 0.83". Here the authors explicitly ask whether five routine variables, run through logistic regression and XGBoost, would do as well — the answer is no, by 0.110 with a narrow confidence interval. The care taken to exclude definitional variables (TNM components are not used to predict stage, immunohistochemistry results are not used to predict themselves) avoids the target leakage that would empty the comparison of meaning. The authors also state that they abandoned report-based models for exactly that reason: routine reports contain the information used to establish several reference labels.

What is less good

Seven metastatic patients carry the paper's most visible number. The abstract opens with "an AUROC of 0.990 observed for M staging". Yet the metrics reported for that task — accuracy 0.962, F1 0.667, sensitivity 1.000, specificity 0.961 on 185 patients — are arithmetically consistent only with roughly seven M1 patients and seven false positives. An AUROC over seven positive events is not a measurement, it is a very wide interval summarised by its midpoint. On top of that sits the substantive problem: distant metastatic disease is not in the image. A model reaching 0.99 on this task from a mammogram is necessarily learning a shortcut — a tumour size, a density, a care pattern correlated with M1 status in this centre. The authors acknowledge it, but in the conclusion, on page fifteen; the abstract remains unchanged, and it is the abstract that answer engines and hurried readers will see. This is a textbook case of a misleading metric on a rare class.

The paper's central mechanism is not demonstrated where it counts. Task-conditioned MoE gains 0.009 mean AUROC over fixed fusion. That 0.009 is an aggregate over nine mutually correlated tasks, estimated by paired bootstrap on the same 185 patients. Task by task, no difference resists multiple-testing correction. Decision curve analysis, the only one speaking in net benefit rather than discrimination, shows curves that overlap substantially and gains described as "small and task-dependent". What remains is an interesting methodological contribution and an empirical result whose magnitude, if one takes the authors' own statistical apparatus seriously, is indistinguishable from zero at the level of each individual clinical question.

Ninety models adjudicated on 92 patients, nothing reproducible, no human reader. Five architectures × two modalities × nine tasks, plus three seeds per configuration, with selection on a 92-patient validation set and evaluation on 185: the selection budget is large relative to the sample sizes. The availability statement points to the corresponding authors — no code, no weights, no data, no GitHub or Hugging Face repository. The efficiency measurements (latency, throughput, memory) are explicitly "simulated", not measured on hardware. Above all, there is no comparison to a radiologist or a pathologist: the authors state that standardised independent blinded readings were unavailable in this retrospective cohort, so no reader study could be performed. The only implicit human comparator is a five-variable clinical model. Finally, the cohort is single centre, Chinese, on three vendor fleets: distribution shift across scanners, staining protocols and populations remains entirely untested for the nine primary tasks.

What it changes

For the research community. Two lessons, one of which is not the one the title advertises. The first is the oracle gap: on this data, perfect case-by-case routing is worth 0.854 and learned routing 0.834. Investing further in routing architectures on cohorts of this size means arguing over two hundredths of AUROC; the headroom lies elsewhere, in data volume and diversity. The second is the per-task modality complementarity map — pathology for nodal status and proliferation, radiology for HER2 and stage — which is reusable independently of the architecture and deserves replication on a multicentre cohort. Conversely, the reporting protocol is copyable as is: multiple-testing correction announced and honoured even when it destroys the result, oracle published, external analyses named for what they are.

For clinicians. Nothing changes today, and the authors say so. OmniBreast is not packaged as a device, has no multicentre validation, no prospective evaluation, and has never been compared to a human reader. The most interesting number to retain is probably the external one: 0.872 AUROC for identifying triple-negative breast cancer from an H&E slide alone, without immunohistochemistry. That is far from what an IHC panel delivers, but enough to imagine a triage use in settings where immunohistochemistry is slow or unavailable — and that is exactly how the authors position it, as an auxiliary model rather than a replacement. The rest of the nine tasks remains, for now, feasibility demonstration.

For patients and the public. One idea, and one reading habit. The idea: a mammogram and a tissue slide contain, statistically, part of the information we otherwise go looking for with dedicated tests — but only part, and never with the reliability of the dedicated test. Predicting HER2 from an image is still wrong for roughly one patient in four; none of these models cancels a biopsy. The reading habit matters beyond this paper: when a reported performance concerns a rare event, the first question is not "what is the AUROC" but "how many cases were there". Here the answer was seven, and it changes entirely how the abstract's 0.990 should be read.