médical IA

Foundation models in multimodal oncology: what an audit of pathology and the transcriptome reveals

Published on June 20, 2026 · 8 min read

A team from the Alan Turing Institute, the Institute of Cancer Research and Genentech put five foundation models to the test on two modalities — digitized pathology slides and the transcriptome — in more than 7,600 patients with breast or lung cancer. The verdict is unsettling: on the omics data, a century-old principal component analysis beats the purpose-built foundation model, and fusing the two modalities only helps when neither already dominates the signal. This is not a breakthrough but an honest audit of what these models actually encode — and of how trustworthy they really are.

The context

For two years digital pathology has lived in the age of foundation models: large networks pretrained, without labels, on millions of histology slides, which turn an image into a vector of numbers — an embedding — meant to condense the useful information. A small classifier is then attached to those vectors to predict a tumor subtype, a biomarker, a prognosis. The names have multiplied — UNI, CONCH, Virchow, MUSK — each arriving with its own, often dazzling, performance figures. The same wave is reaching transcriptomics (the measurement of gene activity, or "omics"), with models like UCE (Universal Cell Embeddings) that aim to do for RNA what the others do for the image.

Three questions remain poorly settled, though. Do these representations hold up on out-of-distribution (OOD) data — that is, different from what was seen in training, and above all other than the public TCGA set on which almost everyone evaluates? Does fusing image and omics truly add anything? And can the predictions be trusted? It is these three questions — probing, fusion, trustworthiness — that this work tackles.

The method

The authors work by probing: the foundation models are frozen, never retrained; one merely learns a lightweight head on their fixed embeddings. This is the honest way to measure what the representations already contain. For the image, each case becomes a bag of encoded tiles aggregated by an attention head (a multiple instance learning, or MIL, method that learns to weight the fragments of a slide without pixel-level annotation); for omics, a multilayer perceptron on the gene vector.

The data do not come from TCGA but from a commercial database, the Flatiron Health–Caris CMDB, that is two US cohorts: 3,747 patients with breast cancer and 3,887 patients with non-small cell lung cancer (NSCLC), each with H&E slides and transcriptome. Eight classification tasks are evaluated: biopsy site, subtype (four classes for breast), the PR and PIK3CA biomarkers, loss of heterozygosity (LOH), tumor site and mutational burden (TMB) for lung. Classes that are too rare (under 8%) are dropped.

Three image–omics fusion strategies are compared: CONTACT (plain concatenation of the vectors, so-called early fusion), MCAT (intermediate fusion by cross-attention between modalities) and LateMIL (late fusion, at the decision level). Finally, trustworthiness is measured by conformal prediction: rather than a single label, the model returns a set of labels calibrated to contain the right answer with a guaranteed probability — here 90%. A small, well-covering set is reliable; a set that has to list half the classes to keep its guarantee is admitting its uncertainty.

The results

On the image side, the foundation models shine on morphologically obvious tasks: telling a metastatic breast biopsy from a primary one yields an AUC above 0.90 (Virchow 0.94, UNI 0.93). The AUC — area under the ROC curve — is 1.0 for a perfect sort and 0.5 for chance. But on the same type of task in lung, the AUC collapses to 0.63–0.65, and on biomarkers (PR, PIK3CA) it tops out around 0.79–0.82 at best: useful, far from a standalone test.

The real surprise comes from omics. The purpose-built foundation model, UCE, is beaten by both scVI and a plain principal component analysis (PCA), a dimensionality-reduction technique dating back to 1901. On breast subtype, PCA reaches an AUC of 0.8955, ahead of all image models and well ahead of UCE (0.76). PCA in fact posts the best AUCs on five breast tasks. The authors say it plainly: "building effective transcriptomic foundation models remains an open challenge."

Fusion does not systematically save the day. LateMIL is the most consistent, but multimodal beats the best unimodal only on some tasks — and sometimes does worse, as on lung mutational burden where omics alone (PCA) wins. The authors conclude that fusion helps "mainly when no single modality already dominates the signal."

On trustworthiness, the conformal sets keep their coverage guarantee (0.90 to 0.93 depending on the task), but at the cost of an average size of about two classes. Clinical translation: on breast subtype, a four-class task, the model returns on average two candidates rather than one — it honestly declines to commit half the time. The 0.90 coverage means that, out of 100 patients, the true label is missing for about 10 of them. And the 72.6% "rescue rate" indicates that, when the model's top choice is wrong, the right answer is still in the offered set in nearly three cases out of four.

What is good

A strong comparator, not a straw man. Including PCA and scVI against the foundation models is exactly the rigor the field lacks: it is this honest comparator that reveals the dedicated omics model adds nothing, and even degrades. Too many studies compare against a weak opponent to inflate the gap; here it is the opposite.

A genuinely out-of-distribution evaluation. By leaving TCGA for a US commercial database never seen in pretraining, the study tests transfer rather than memory. The risk of data leakage — same patients or same images in train and test, a classic plague of the field — is ruled out by construction, since the models are frozen and the cohort independent.

Trustworthiness treated as a metric, not a slogan. Conformal prediction offers a coverage guarantee with no assumption on the distribution, and set-valued predictions — a clinically meaningful format that says "here are the two plausible subtypes" rather than a misleading certainty. It is one of the rare evaluations to measure uncertainty as seriously as performance.

What is less good

An unquantified population bias. Both cohorts come from a single US commercial source (Flatiron–Caris) and only two cancers. The authors acknowledge "subgroup disparities" but publish no per-subgroup table (age, ancestry, center). Without those figures it is impossible to know whether reliability holds for under-represented populations — this is population bias left in the blind spot.

Metrics that can flatter. Accuracy is reported alongside AUC, but after dropping classes under 8%: on imbalanced tasks, removing rare classes mechanically inflates apparent performance. And the high AUC on biopsy site smells of shortcut learning — the model likely exploits a coarse morphological signature (metastatic vs primary tissue) rather than fine biology, which the gaping gap with biomarkers, far harder, confirms.

Limited reproducibility and independence. Neither code nor weights are released; the data are commercial and closed (access under Flatiron/Caris license). The image embeddings were generated in-house, on the industry side. Above all, one author is affiliated with Genentech (a Roche company) without any formal conflict-of-interest statement in the retrieved text — a gap on a topic where industry has a direct stake.

What it changes

For the research community, this paper is a useful reality check. It documents in black and white that today's transcriptomic foundation models have not yet surpassed a PCA, that multimodal fusion is no universal recipe, and that conformal prediction should become an evaluation standard. The bottleneck is not stacking modalities but aligning representations that genuinely complement one another.

For clinicians, nothing is deployable today: no task reaches standalone-tool performance, and everything rests on retrospective data from a single source. The usable message is subtler: a model that honestly answers "two plausible subtypes" via a conformal set is safer in practice than one that commits with unjustified confidence.

For patients and the public, AI in oncology is not a single magic model. To read a tumor's transcriptome, classical statistics remain competitive; to read a slide, large models help mostly on the easy questions. Reliability here is measured by the model's ability to recognize when it does not know — a more reassuring quality than a spectacular AUC.