médical IA

Report-derived labels versus image re-reading: for cardiomegaly in MIMIC-CXR, agreement is close to nil

Published on August 14, 2026 · 9 min read

Yesika Alexandra Agudelo-Londoño and colleagues had radiologists re-read 2,160 chest radiographs from MIMIC-CXR — the public repository underpinning much of chest imaging AI — and compared that reading with the labels the repository supplies, extracted automatically from radiology reports. For cardiomegaly, agreement is close to nil: a Cohen’s kappa of 0.011, with the label recovering only 1.3% of expert-confirmed cases, and the overwhelming majority of disagreements arising from plain omission in the report rather than explicit negation. The result does not indict the text extractors: it shows that a radiology report is not an inventory of what is visible on the image, and that assuming otherwise means training models on an ill-defined target.

The context

A handful of public repositories carry most chest radiograph AI research today: MIMIC-CXR, CheXpert, ChestX-ray14, PadChest. Together they supply millions of images paired with pathology labels, and serve both to train classifiers and to pretrain foundation models (large generic models reused through fine-tuning) and vision-language models.

Those labels almost never come from re-reading the images. They are extracted automatically from report text by natural-language tools — the CheXpert labeler and NegBio, which spot the mention of a finding and decide whether it is asserted, negated or uncertain. Without that procedure, no repository of this size would exist.

The problem sits upstream of the tool. A radiology report is not an exhaustive record: it is a clinical document answering a question posed by a referring physician. In acute inpatient care, the radiologist documents what prompted the examination and what has changed; a heart that is large but stable for ten years may go unmentioned altogether. The resulting label therefore encodes reporting practice as much as radiographic reality. The gap had already been flagged — by VisualCheXbert, by the literature on hidden stratification — without changing practice: the repository label is still treated as image-level ground truth.

The method

The authors propose a framework called Repository Supervision Auditing (RSA), which takes as its object not the model but the learning signal itself, before any training. Four stages.

An expert image-level reference. 2,160 unique studies (one patient, one encounter) from MIMIC-CXR, restricted to non-portable PA radiographs from inpatients without ICU transfer. Board-certified radiologists re-read each DICOM study using all available projections together with the report, annotating according to standardised terminologies (RadLex, SNOMED CT, Fleischner Society).

The audit proper. Repository labels (CheXpert and NegBio) were binarised with a deliberately generous rule: "uncertain" counts as positive, "not mentioned" as negative. Cohen’s kappa measures agreement between two raters after subtracting the agreement expected by chance: 0 means "no better than chance", 1 perfect agreement. PPA is the label’s sensitivity — of the cases the expert sees, what fraction the repository flags; NPA is its specificity, and PPV the label’s precision. Cardiomegaly was chosen as the index finding because it is defined primarily by the morphology of the cardiac silhouette on the image; pleural effusion and atelectasis serve as comparisons.

Characterising the disagreements. Every false negative was reclassified at the text level into three categories: explicit negation in the report, assignment of the repository "No Finding" label, or non-mention despite other documented abnormalities. This distinction is the heart of the paper: it separates a text-interpretation error from a reporting practice.

A rebuilt cohort, then a model. From 14,200 relabelled examinations, all expert-confirmed cases were matched to negatives on ten covariates. The largest imbalance, measured by the standardised mean difference, falls from 0.87 to 0.005. Quality control applied directly to DICOM files excluded non-frontal, annotated or poorly exposed studies. Final cohort: 2,072 studies, 1,020 positive and 1,052 negative, split without patient overlap into 1,453 / 311 / 308. A DenseNet121 pretrained on seven public datasets was fully fine-tuned on these expert labels. The decision threshold (0.436) was fixed on the validation cohort alone.

The results

The repository label does not recover visible cardiomegaly. Cohen’s kappa: 0.011 (95% CI 0.004–0.019), an agreement indistinguishable from chance. Of 1,080 studies where radiologists see cardiomegaly, the repository labels 14 — a PPA of 1.3%. CheXpert, NegBio and their union give exactly the same result. Conversely, PPV reaches 0.875: when the repository says "cardiomegaly", it is almost always right. The label is thus highly precise and nearly blind, the signature of a truncated rather than a noisy target. The failure is not uniform, though: pleural effusion obtains a kappa of 0.319 (PPA 21.7%) and atelectasis 0.120 (PPA 7.9%) — better, but far from acceptable agreement.

Disagreement comes from omission, not negation. Of 1,066 false negatives, only 55 (5.2%) correspond to explicit negation. The other 1,011 (94.8%) arise from non-mention, 979 of them under a "No Finding" label. Put differently: among the 1,998 studies the repository declares free of findings, experts see cardiomegaly in 979 — nearly one in two. The text extractors are doing their job correctly; it is the text that lacks the information.

On the audited cohort, a standard model works. The DenseNet121 reaches a ROC-AUC of 0.853 (95% CI 0.803–0.893) and a PR-AUC of 0.824 on the held-out test set, with sensitivity 0.862 and specificity 0.705. Calibration is moderate (Brier score 0.162, expected calibration error 0.099), with no significant difference by age or sex, and Grad-CAM maps concentrate attention on the cardiac silhouette rather than implanted devices.

Clinical translation. Out of 1,000 radiographs with the same proportion of cardiomegaly as this cohort — about one in two, far above any real workflow — the model would flag roughly 431 of 500 and miss 69, at the cost of 148 false positives among the 500 normal examinations. The comparison is brutal: on those same 500 cases, the repository label would flag six or seven.

What is good

Auditing supervision before modelling, and quantifying it. The notion that report-derived labels are imperfect has circulated for years as a general caveat. Here it becomes a measurement, with confidence intervals and an explicit protocol.

The breakdown of disagreements exonerates the extractors. Distinguishing explicit negation, "No Finding" and omission turns an observation into a diagnosis. The conclusion — 94.8% omissions — moves the target: improving natural-language processing on reports will fix nothing for cardiomegaly, because the information is missing at the source. That is the kind of precision that saves a decade of effort in the wrong place.

The dice were loaded in the repository’s favour, and it still fails. Treating "uncertain" as positive is the most favourable possible rule for the repository label.

What is less good

A narrow scope, not to be over-read. One repository, one source hospital, one index finding audited in depth, and a filtered population: non-portable PA radiographs, inpatients without ICU transfer. The expert reference covers 52 findings but only three are analysed. Nothing guarantees that the collapse observed for cardiomegaly recurs identically in CheXpert or PadChest, whose reporting practices differ — a population bias that here works against generalising the finding itself, and the authors say so plainly.

No comparator for the model. The 0.853 AUC floats without a reference point: no model trained on repository labels is set against it. The authors explain this honestly — with a PPA of 1.3%, there are almost no positive cases to train such a model on the same cohort — but the consequence stands: the practical benefit of the audit is postulated, not demonstrated. A missing comparator is a quieter failure mode than a weak one, and just as limiting.

A cohort that is too clean, and an audit that cannot currently be checked. The 2,072 studies were matched, balanced 50/50 and cleaned at DICOM level. That is sound for isolating the effect under study, but it produces a distribution that no longer resembles a clinical workflow, where a far lower real prevalence would crush positive predictive value. There is neither external nor prospective validation, calibration remains moderate, and nothing indicates that the expert annotations, the list of retained studies or the code will be released — which makes the audit itself hard for a third party to reproduce. On interests: the work was carried out within SURA’s imaging programme, several authors being employees or trainees there, one co-author is employed by IBM Research, compute was provided by Microsoft Azure, and no external funding is declared.

What it changes

For the research community, this is an invitation to re-read an entire literature. A model scoring 0.9x AUC on MIMIC-CXR cardiomegaly has not learned to measure a cardiothoracic ratio: it has learned to predict whether the radiologist would have mentioned the finding. The two tasks overlap only partially, and published scores measure the second. The implication bears especially on vision-language models pretrained by image-text pairing, whose signal is precisely that text — we have decrypted work showing that in such models the text largely dominates the image.

For clinicians, nothing changes today: this paper evaluates no deployed device. It does supply a question to put to any vendor of a radiograph-reading tool — what ground truth was the model trained on, reports or image re-reading? For findings defined by visible morphology, the answer is not a technical detail.

For patients and the general public, the lesson is broader: a medical AI’s performance is bounded by the quality of what it was taught to imitate. When a press release announces that a model "detects an abnormality with 95% accuracy", the real question is not the number but what it was measured against — as with the invisible biases in imaging datasets covered in our decryptions on underperforming subgroups and dataset signatures in mammography.