When breast density skews the evaluation of screening AI: the Mass-Bench benchmark and the hidden degradation (Mathematics, 2026)

Published on 10 June 2026 in the journal Mathematics, this Mexican work builds Mass-Bench, a benchmark that unifies four public mammography datasets (32,930 images, 8,245 patients) to measure how AI models detect and classify breast masses — not globally, but stratified by breast density. The central result: performance degrades systematically as the breast grows dense, a weakness that the usual evaluations, run on imbalanced datasets, hide and turn into misleading optimism. The demonstration is useful and the problem real — but the paper itself reproduces some of the flaws it denounces, from a headline figure nowhere to be found in its own tables, to test cells of a single image, to no released code for something billed as a "benchmark."

The context

Breast cancer screening by mammography has an enemy radiologists know well: breast density. A "dense" breast contains a lot of fibroglandular tissue, which appears white on the mammogram — exactly the color of a suspicious mass. A tumor hides there like a snowflake in a snowstorm. The ACR (American College of Radiology) classification grades this density from A (fatty breast, easy to read) to D (extremely dense, hard), often coded 1 to 4. The higher you go, the more the sensitivity of mammography falls — for the human eye and, the paper hypothesizes, for algorithms too.

Yet almost all mass-detection AIs are evaluated on global figures: a single AUC, a mean sensitivity, on a dataset whose density composition is neither controlled nor representative of the real population. If that dataset holds mostly low-density, easy breasts, the model will look excellent — and will silently collapse on dense-breasted patients, precisely those for whom an aid would be most valuable. It is this evaluation bias the study sets out to make visible, by proposing a protocol that measures performance per density category rather than as a single aggregate number.

The method

The work (DOI 10.3390/math14122080), authored by Hector E. Zepeda-Reyes and Hayde Peregrina-Barreto (INAOE, Mexico) and Gabriela C. Lopez-Armas (CETI, Guadalajara), first builds Mass-Bench, an assembly of four public mammography datasets under a unified annotation schema: CBIS-DDSM (United States), INBREAST (Portugal), VinDr-Mammo (Vietnam) and DMID (India). In all, 8,245 patients, 32,930 images and 3,480 annotated masses, each carrying its ACR density category (1-4) and its BI-RADS score (the Breast Imaging Reporting and Data System, the standard radiological suspicion scale, from 2 "benign" to 5 "highly suspicious").

On this base, two tasks. First, mass detection, handled by three versions of the YOLO detector ("You Only Look Once," a family of networks that locate objects in an image in a single pass) — YOLOv5, v8 and v11. Then classification, of both density and BI-RADS, on lesion-centered patches. For this second task, images are turned into numerical vectors by four pretrained networks acting as feature extractors (VGG19, ResNet50, DenseNet121, EfficientNet-B3, not retrained), complemented by "handcrafted" descriptors (classical texture statistics), then classified by five standard algorithms (logistic regression, SVM, k-nearest neighbors, random forest, XGBoost).

Two methodological choices deserve immediate praise. The train/validation/test splits (70/20/10) are fixed at the patient level, before any preprocessing, so the same woman does not end up on both sides — the guard against data leakage, where the model "cheats" by seeing in the test cases it met in training. And the authors quantify each dataset's density imbalance via a Kullback-Leibler divergence (KL, a measure of the gap between two distributions) against a plausible clinical reference (10% A, 40% B, 40% C, 10% D). The verdict: VinDr-Mammo (0.57) and INBREAST (0.45) are far from that reality, whereas Mass-Bench (0.04) comes close.

The results

On detection, YOLOv11 leads on Mass-Bench with an mAP@50 of 0.663 (mAP, mean Average Precision, measures localization quality; @50 means a box counts as correct if it overlaps the true lesion by at least 50%) and an mAP@50-95 of 0.335, ahead of v8 (0.646) and v5 (0.598). But the abstract's headline figure — an AUC "up to 0.943" — comes from a single case: YOLOv8 on DMID. The other datasets are more modest (CBIS-DDSM 0.671, VinDr-Mammo 0.801).

On degradation by density, the heart of the argument, the signal is sharp on VinDr-Mammo: the detection F1-score falls from 0.750 for a low-density breast (ACR 2) to 0.571 for a very dense one (ACR 4). The trend — less success as density rises — recurs across datasets, which validates the original worry: a global figure smooths this drop and makes it invisible.

On classification, binary density (low- vs. high-density) classifies fairly well (0.90 accuracy on Mass-Bench, 0.95 on VinDr-Mammo), and binary BI-RADS (benign vs. suspicious) reaches 0.90 on Mass-Bench and 0.93 on DMID. By contrast, multi-class BI-RADS classification — finely distinguishing grades 2, 3, 4 and 5 — stays fragile, dropping to 0.53 accuracy on CBIS-DDSM. Clinical translation: coarsely sorting "worth watching" from "nothing alarming" is within reach; reproducing the nuanced judgment of a radiologist who assigns a precise grade is not — and it is that nuanced judgment that does or does not trigger a biopsy.

What is good

A protocol that makes a real bias visible. Measuring performance per density category rather than as a single aggregate figure is not a flourish: it is exactly the discipline most of the field's publications lack, and the demonstration that performance collapses on dense breasts is the kind of finding that helps anyone reading miraculous AUC announcements.

Honest care against data leakage. The patient-level split before any processing, and the demand for geometric consistency of annotations across datasets, are precautions often neglected. They make the figures more credible than the literature's average.

Explicit quantification of imbalance. Putting a KL-divergence number on how far each dataset strays from a plausible clinical distribution gives the reader a concrete tool to judge: a model trained on VinDr-Mammo, far from reality, should not be taken on faith outside its home turf.

What is less good

A headline figure that can't be found and single-image cells (failure mode: the misleading metric). The abstract announces an F1 "up to 0.976" in density classification, but the best F1 in the summary tables tops out at 0.951: the headline rests on a configuration peak absent from the tally. Worse, some "near-perfect" scores rest on tiny test cells — an F1 of 0.999 computed on a single image of category ACR 4. The authors half-acknowledge this, but the abstract keeps only the superlative.

Too few dense breasts, which weakens the thesis (failure mode: population bias). The paper means to warn about the under-representation of dense breasts — yet its own datasets suffer from it, and the ACR 4 categories hold a handful of examples in places. Demonstrating a degradation on such small numbers is measuring the problem with the very instrument that is its victim: the conclusion is plausible, but its statistical solidity stays limited.

A "benchmark" with no released code or weights (failure mode: reproducibility). Mass-Bench presents itself as a reusable benchmark, but no code repository, no curation script, no trained model is released — only the four source datasets are listed. Reproducing the assembly means redoing everything by hand. Add to this the absence of any funding statement and the authors' acknowledged use of ChatGPT (GPT-5) for "table generation" — a detail that, given the gap between abstract and tables, is not entirely innocent.

What it changes

For the research community, the methodological message is right and reusable, even if the tool is less so: stop publishing global AUCs and systematically stratify by density — and more broadly by subpopulation. It is an evaluation requirement, not a new model.

For clinicians, nothing changes in the reading room today. The work is retrospective, on public images, with no prospective validation and no comparison to radiologists: it does not say a tool is ready, it says how to judge the ones you will be shown. The useful reflex: ask for performance on dense breasts, not the average.

For patients and the public, the lesson is simple and durable: a screening AI "good on average" can be much less so on dense breasts, which are also where cancer is hardest to see. The promise of an automated second look stays credible, but its value depends entirely on the population it was actually measured on.

Going further

The full article is open access in Mathematics, 2026, 14(12), 2080 (DOI 10.3390/math14122080), under a CC BY 4.0 license, with ROC curves in the supplementary material. The four assembled datasets are public: CBIS-DDSM (TCIA), INBREAST (on request), VinDr-Mammo (PhysioNet) and DMID (Figshare). For the framework of breast density and BI-RADS, the American College of Radiology's BI-RADS atlas remains the reference.