Mixing mammography datasets does not improve screening AI: when the model learns the dataset's "signature" instead of the cancer

A team at Memorial University of Newfoundland (Canada) tested an assumption treated as obvious: adding biopsy-confirmed cancers from public databases to a genuine screening set should improve a mammography triage model. Training a single classifier on the NLBSD screening set (5,997 exams, 1.29% suspicious) and then adding the positive cases from CBIS-DDSM and CMMD, the authors find the opposite — performance drops with every addition, and a diagnostic test reveals why: the model separates images by their dataset of origin almost perfectly (AUC 0.9998), proof that it is learning the dataset's "signature" rather than the pathology.

The context

Mammography screening looks for rare cancers in a healthy population: among women in the screened age range, the prevalence of genuinely suspicious findings is around 1%. That is what makes the task so hard for a machine-learning model — the signals to detect are rare, subtle, and buried under an overwhelming majority of normal exams. To train a triage AI (separating "normal" from "suspicious"), you would want data that looks like that reality. Such data is scarce: most public mammography datasets — such as CBIS-DDSM or CMMD — are "abnormal-enriched" collections, built from already-biopsied cases, where cancers are numerous and often clearly visible, with almost no normal screening exams.

Hence an idea that sounds like common sense and is repeated everywhere: since positive cases are rare, why not top up your screening set with these confirmed cancers from elsewhere? More positives, better detector. The Memorial University of Newfoundland team puts that intuition to the test, using an unusual resource: the NLBSD (Newfoundland and Labrador Breast Screening Dataset), one of the few public sets that represents a true screening cohort, with a "normal / suspicious" labeling scheme and low prevalence. The precise question: does adding cancers from CBIS-DDSM and CMMD improve detection on NLBSD? To measure, the authors use the AUC (area under the ROC curve), a score between 0.5 and 1 that captures how well the model separates a suspicious case from a normal one: 0.5 is chance, 1 is perfection.

The method

The setup is deliberately simple and controlled — the goal is not to beat a record but to isolate a cause. As a feature extractor, the authors take an EfficientNet-B5 initialized with Mammo-CLIP weights (a model pretrained specifically on mammography) and freeze it: its parameters do not move. Only a small linear classification head is trained on top — a setup called a linear probe, which tests what the model's frozen representation already contains, without retuning it to the task. This is a strong methodological choice: if a signal appears, it is baked into the general representation, not manufactured by an opportunistic retuning.

Three datasets enter the stage. NLBSD supplies 5,997 screening exams (CC and MLO views of both breasts), with a suspicious-case prevalence of 1.29% (149 of 5,997) — the reflection of real life. CBIS-DDSM (1,644 cases from digitized film) and CMMD (3,712 images from 1,775 patients, natively digital) contribute only their biopsy-confirmed abnormal cases. Crucially: in every experiment, negative cases come exclusively from NLBSD; the external datasets serve only to add positives.

All datasets pass through exactly the same preprocessing: identical DICOM conversion, per-image intensity normalization to a common scale, breast cropping, common orientation. The idea is to neutralize "surface" differences so that any performance gap reflects a real domain difference, not a formatting artifact. The train/validation/test splits (70/10/20) are done at the patient level, so that no image of the same person appears in both training and test — the precaution that avoids data leakage, the bias where a model "cheats" by seeing test cases already seen in training. Performance is measured by image-level AUC, with bootstrap confidence intervals and DeLong-test comparisons, corrected for multiple comparisons (Holm). Finally, a diagnostic experiment repurposes the same setup: instead of predicting "normal / suspicious," the model is asked to guess which dataset each image comes from.

The results

The best model is the one that added nothing. Trained and evaluated on NLBSD alone, it reaches an AUC of 0.737 (95% CI 0.686–0.785). Adding external positives degrades performance every time, and increasingly so: 0.644 with CMMD cancers (−0.093), 0.640 with CBIS-DDSM (−0.097), and 0.620 combining both (−0.117). The decline is monotonic — the more sources added, the more is lost — and statistically significant (Holm-corrected p = 0.015, 0.005, and < 0.001). Evaluating on "mixed" test sets that include external positives reverses nothing: 0.653, 0.651, 0.648, never above the reference.

The clinical translation is the real warning. At a fixed decision threshold, the baseline model balanced sensitivity and specificity (0.66 each). The enriched models keep their specificity but see sensitivity fall to 0.53–0.55 — that is, they miss about 45 to 47% of suspicious cases, versus 34% for the reference. In screening, this is exactly the wrong direction: you want to miss as few cancers as possible, even at the cost of recalling a few more women. Naïvely adding data therefore both lowered discrimination and pushed the operating point toward the dangerous side.

Why? The diagnostic experiment shows it unambiguously. Asked to predict the dataset of origin rather than the pathology, the model separates the three sets almost perfectly: F1 above 0.98 for each dataset, mean AUC of 0.9998, confusion diagonal at 99.7 / 98.8 / 100.0%. Despite rigorously identical preprocessing, each dataset carries a "signature" the network reads with its eyes closed. This is the very definition of shortcut learning: rather than learning the hard concept — the subtle cancer of a screening population — the model latches onto an easy, spurious cue: where the image comes from. The authors recall the now-classic precedent of a chest-radiograph model able to recognize its source hospital, and generalizing poorly for that very reason.

What is good

A protocol that isolates a cause. Everything is held constant — architecture, optimization, preprocessing — and only the composition of the training data varies. The patient-level split cleanly rules out data leakage, and the frozen-encoder choice strengthens the conclusion: the near-perfect separability of the datasets is not manufactured by task-specific retuning, it is already present in a general mammography representation. One can therefore assert a cause — the domain of the added data — where a mere comparison of models would stay ambiguous.

An honest clinical anchor. The authors start from a true screening cohort (NLBSD) rather than a more convenient enriched set, and place their work fairly: without mammography pretraining, prior work topped out at an AUC of 0.6209 on NLBSD (versus 0.905 on the small curated INbreast set); starting from Mammo-CLIP, they obtain a firmer baseline (0.737), which they take as their starting point. The comparator is explicit and dated, not a straw man.

Turning failure into a demonstration. Rather than noting the underperformance and stopping there, the authors build the "guess the dataset" test that pinpoints the mechanism. This is good hygiene: naming the failure mode (shortcut learning), measuring it, and connecting it to a phenomenon documented elsewhere in medical imaging.

What is less good

A single model, a single way of training it. Everything rests on a frozen EfficientNet-B5 + Mammo-CLIP, with a linear probe and plain cross-entropy. The authors acknowledge it: full fine-tuning of the encoder, multi-view models, or objectives designed for class imbalance might behave differently. And the absolute performance stays modest — an AUC of 0.737 is not a deployable tool but a test bench. The result concerns "this setup," not all mammography AI.

Numbers to handle with care. The authors themselves note that the image-level test treats the four correlated views of a single exam as independent, making the p-values "somewhat optimistic" — the classic misleading metric, here honestly defused, but a reminder not to over-read significance thresholds (the monotonic ordering of the results, however, does not depend on this assumption). As for the 0.9998 AUC of the origin test, it measures dataset separability, not a clinical outcome: spectacular, but not to be read as diagnostic performance.

Generalization still to establish. NLBSD, valuable as it is, reflects a single population, a single equipment fleet, a single regional protocol (Newfoundland and Labrador). Nothing yet says the same degradation would appear on other true screening cohorts, nor how it distributes across subgroups (age, breast density, device) — this population bias is not mapped. There is no prospective validation, and this is a preprint not yet peer-reviewed, submitted to a journal but not published.

What it changes

For the research community, the message is clear: piling up heterogeneous datasets is not free, it is often counterproductive. Data quantity does not compensate for domain misalignment; explicit strategies are needed — intensity harmonization, domain adaptation, site-conditioned models, augmentation simulating acquisition variation — before a cancer from elsewhere can truly help. In passing, the study explains why the many mammography datasets remain islands: lacking a method to cross acquisition boundaries, they do not form a single resource.

For clinicians and decision-makers, it is an antidote to a common sales argument. A vendor announcing a model "trained on tens of thousands of mammograms from multiple datasets" guarantees nothing: if those datasets do not resemble the screened population, the addition can degrade sensitivity where it matters. The right question is not "how many cases?" but "do these cases resemble my patients, on my machines?" A screening tool must be validated on a screening distribution.

For patients and the public, the study dismantles the idea that "more data" equals "safer AI." A model may look like it is learning cancer while it is actually learning to recognize the device or the source of the image — an invisible shortcut that collapses in real conditions. The safety of a screening AI cannot be inferred from the size of its training set: it must be demonstrated on the population actually concerned.

To go further

The preprint "Dataset-Origin Signatures and Shortcut Learning in Screening Mammography AI: A Cross-Dataset Case Study" is available on arXiv (2607.15416), posted on 16 July 2026 by Parham Hajishafiezahramini, Matthew Hamilton, Oscar Meruvia-Pastor and Edward Kendall (Memorial University of Newfoundland, Canada) and submitted to Computerized Medical Imaging and Graphics; the work was funded by the university's Seed, Bridge, and Multidisciplinary Fund, with no declared conflicts of interest. The data are public (NLBSD via the Federated Research Data Repository; CBIS-DDSM and CMMD via The Cancer Imaging Archive). On mammography AI and the blind spots of generalization, see our decryptages on density-aware evaluation of mass detection, on hidden subgroup biases in medical imaging, and on the robustness and generalization of foundation models.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. English, Spanish and Chinese translations produced with AI assistance and reviewed.