médical IA

Maxillary sinus on CBCT: when the text and the label come from the same LLM, AUC falls from 1.00 to 0.84

Published on August 19, 2026 · 13 min read

Three researchers from Queen Mary University of London and King Abdulaziz University compare, on the same 300 dental CT slices, eight vision networks, seven language models and five vision-language models on one binary task: is this maxillary sinus normal or diseased? The language models reach AUCs of 0.99 to 1.00 — until a radiologist re-reads the 297 cases, reclassifies 62 of them (21%), and sends those same models back down to 0.84–0.85, while the vision networks do not move at all. The paper is built to produce exactly this demonstration: when the training text and the label come from the same language model, performance measures a machine's consistency with itself, not diagnostic content.

The context

CBCT (cone-beam computed tomography) is the routine scanner of the dental practice: a reduced-dose 3D acquisition used to plan an implant, prepare a root canal treatment or oral and maxillofacial surgery. Its field of view almost always captures the maxillary sinuses, the air-filled cavities above the upper molars. Anomalies nobody was looking for therefore turn up regularly: mucosal thickening, retention pseudocyst, partial opacification, air-fluid level. These incidental findings are not trivial — they bear on whether to place a posterior implant or perform a sinus-floor elevation. But reporting them is entirely operator-dependent, and time-consuming.

Hence the drive to automate. So far, the dental CBCT literature has mostly evaluated single-modality convolutional networks. The authors — Seba Al-Hebshi, Hanadi Khalifa and Tuan D. Pham — report searching PubMed from January 2015 to July 2026 and finding no study that compared, on one dataset under matched cross-validation, all three families: image-only, text-only, and vision-language.

The real interest of this preprint lies elsewhere, though, and it reaches well beyond the maxillary sinus. A practice is spreading fast in health-AI research: having a large language model produce the textual descriptions — and sometimes the labels — needed to train other models, for want of enough human annotation. This paper builds a protocol to measure what that practice costs. The answer is quantified, and it is brutal.

The method

The data. The dataset is MMDental, published in Scientific Data in 2025 and freely available on Figshare under a CC-BY licence. The authors extract 300 mid-sagittal slices from it — one 2D image per case, no 3D volume. The source is a single hospital in China (ethics approval from the Medical Ethics Committee of Lishui University, no. 2022YR014). The number of patients is never given: cases are designated by image identifiers. The task is binary: "abnormal" covers mucosal thickening, retention pseudocyst, opacification and air-fluid level; "normal" denotes a well-aerated sinus with intact walls.

The label factory — the heart of the design. Each image is submitted three times independently to ChatGPT, with a detailed instruction to act as a consultant oral and maxillofacial radiologist and describe laterality, location, severity and relationship to adjacent teeth. The three readings are merged by majority vote into a representative description with a provisional label. That description and the image are then handed to a radiologist (one of the authors), who produces her own diagnosis, further checked by a prosthodontist on the team. Three cases are excluded as insufficient, leaving 297 cases, 146 normal and 151 abnormal.

The protocol then compares three conditions on the same folds: Raw (n = 300, ChatGPT labels), Pre (n = 297, ChatGPT's provisional labels) and Post (n = 297, radiologist-confirmed labels). The decisive point: between Pre and Post, only the label changes. The findings text stays rigorously identical. Any performance gap is therefore attributable to label provenance and to nothing else — not architecture, not image content, not a rewriting of the text.

The models. Eight ImageNet-pretrained vision networks fine-tuned for the task: ResNet-50, DenseNet-121, EfficientNet-B0, GoogLeNet, AlexNet, NASNet-Large, PNASNet-5-Large and ViT-B/16 (AdamW, learning rate 10⁻⁴, up to 50 epochs, batch size 16, seed 42). Seven language models trained on the findings field alone, stripped of the explicit classification line: BiLSTM, TextCNN, BERT-base, RoBERTa-base, DistilBERT, BioBERT and Bio_ClinicalBERT. Five vision-language models under linear probing (the encoder is frozen, a representation vector is extracted, and only a logistic regression is trained on top): CLIP ViT-B/16, BiomedCLIP, Qwen2.5-VL-7B, MedGemma-4B and LLaVA-Med v1.5, with image-only, text-only and image-plus-text probes wherever the architecture allows.

The evaluation. 10-fold stratified cross-validation, with the same fold assignment across every arm and both label conditions. Per-fold means ± standard deviation, 95% confidence intervals from the t-distribution, pooled confusion matrices. Grad-CAM for interpretability of the convolutional networks.

The results

Vision: honest and stable. Across the eight architectures, AUC — the area under the ROC curve, that is, the probability that the model scores a randomly drawn diseased sinus above a healthy one — ranges from 0.799 to 0.880. PNASNet-5-Large leads (0.880, 95% CI 0.850–0.910), followed by DenseNet-121 and EfficientNet-B0 (0.863 each). EfficientNet-B0 offers the most balanced profile: accuracy 0.800, sensitivity 0.787, specificity 0.813. ViT-B/16 lags clearly (accuracy 0.700, AUC 0.799) with a telling asymmetry: specificity 0.860 but sensitivity 0.540, meaning 69 diseased sinuses classified as normal. That is the expected signature of a data-hungry architecture trained on a few hundred images.

Language: the suspicious ceiling. The seven text models reach 0.977 to 0.993 accuracy and 0.992 to 1.000 AUC. Bio_ClinicalBERT attains a perfect AUC of 1.000; BERT-base, DistilBERT and BioBERT each reach 0.993 accuracy, two errors out of 300. A well-written radiological text can genuinely be easier to classify than an image — but a perfect score on a clinical task should always trigger a check.

The provenance test. This is where the paper earns its keep. The radiologist reclassified 62 of 297 cases, or 21%, almost evenly split: 32 under-calls (ChatGPT reads "normal" where the radiologist sees an anomaly, most often mild mucosal thickening) and 30 over-calls. Rerunning training with those corrected labels, input text unchanged:

The reading is unambiguous. The text produced by ChatGPT was not encoding diagnostic content: it was encoding the label ChatGPT had assigned itself. The language models were not learning to read a sinus, they were learning to recover another model's conclusion from that model's own phrasing. This is label leakage by circularity — a variant of classic data leakage, where information about the target escapes not through patients but through the annotation process itself. Correcting 21% of the labels is enough to cost 0.15 AUC points. The images, meanwhile, carried a real signal: their performance does not depend on who wrote the label, and even improves when the label gets more accurate.

Generative note drafting. An exploratory arm asks Qwen2.5-VL-7B, MedGemma-4B and LLaVA-Med v1.5 to produce a structured report. Qwen writes the longest notes (161 words on average against 78 and 88) and 66% of them stop without sentence-final punctuation, consistent with hitting the generation limit. More importantly, among abnormal cases whose input specified no side, Qwen invents a laterality in 17 of 150 cases (11%), MedGemma in 4 of 150 (3%), LLaVA-Med never. MedGemma also reproduces an unfilled clinical template, placeholder fields included, in 200 of 300 notes (67%).

Clinical translation. Out of 1,000 sinuses read by the best image model (EfficientNet-B0, sensitivity 0.787 / specificity 0.813, at an abnormality prevalence of 51%), roughly 109 diseased sinuses would be missed and 92 healthy ones flagged in error. At practice scale that remains a pre-reading tool, not a decision tool. And for automated report drafting, an invented laterality one time in nine is disqualifying: a wrong side in a sinus report is potentially a sinus lift on the wrong side.

What is good

1. The protocol isolates exactly the variable that matters. Same folds, same seed (42), same text, only the label changes between Pre and Post. This is a controlled experiment in the strict sense, and it produces a result that cannot be attributed to anything else. The health-AI literature is full of confounded effects; this one is not.

2. The paper documents the mechanism by which it could have fooled itself. The authors removed the "Classification: normal/abnormal" line from the input text — the correct precaution — and still got AUC 1.000. They then went looking for why, instead of publishing the number. Without the Post arm, this preprint would be one more paper announcing a perfect AUC in dental radiology. It is the opposite move to what the benchmark race rewards.

3. Material transparency is real. Public code on GitHub (sinus-cbct-benchmark, commit 32d5595 of 8 August 2026), public dataset under CC-BY, both the ChatGPT-generated text and the radiologist's text made available, preprint under CC BY 4.0, no funding, no declared conflicts of interest. Grad-CAM is used honestly: the authors show that AlexNet, on a correctly classified case, concentrates its activation on the dentition rather than the sinus cavity — a textbook case of shortcut learning, which they comment on by noting that "a correct prediction and a correct reason for it are not the same thing".

What is less good

1. The gold standard remains contaminated by what the paper denounces. The radiologist did not re-read the images blind: she received the image together with ChatGPT's description. Anchoring bias is therefore structurally possible — and the authors acknowledge it ("the reference standard was derived in part from LLM readings before radiologist confirmation, which may introduce systematic bias"). There is only one reader, no independent second radiologist, hence no kappa, no inter-observer agreement measure at all. The 21% reclassification rate is therefore probably a floor: blinded, it would plausibly be higher.

2. The experimental base is thin, and cannot be stratified. Three hundred 2D mid-sagittal slices from a single hospital in China, no external validation, no prospective validation, no held-out test set — everything rests on cross-validation. How the mid-sagittal slice was chosen (automatically? manually?) is not described, and the unit of analysis is the image, not the patient: since the patient count is never given, one cannot rule out that slices from the same patient land in different folds. The authors did not merge MMDental's metadata, so no subgroup analysis is possible: not by sex (the parent dataset reports 51.06% male), not by age, not by severity. Ethnicity is not recorded in the source. Finally the binary task flattens a clinical continuum: a 2 mm mucosal thickening and a complete opacification fall into the same box.

3. The statistical apparatus falls short of the argument's ambition. No significance testing — no DeLong, no paired test, no p-value anywhere — even though the paper's thesis rests entirely on gaps between conditions. No calibration analysis (no Brier score, no expected calibration error, no reliability diagram), even though ChatGPT emitted a confidence rating that is never used. The ChatGPT version is never specified — not the model, not the date, not the temperature, not API versus interface: the study's central result is therefore not exactly reproducible. The generative arm is qualitative and unscored, as the authors say. And the abstract reports image-only probes at "0.63 to 0.69" AUC while the results section reports up to 0.874: the abstract describes only the contrastive models and generalises wrongly.

What it changes

For the research community. This is the most direct point of application. LLM annotation is spreading because it is cheap and it scales; this paper supplies the minimal protocol for checking that it has not manufactured an empty result. The test fits on one line: retrain on human-corrected labels, input text unchanged, identical folds, and compare. If performance collapses, it was measuring the LLM's consistency with itself. That is cheap, and it should become a review requirement for any paper whose labels or descriptions are synthetic in origin. The corollary is also useful: here, it was the images that carried the signal, and their performance improved slightly with better labels. A modality robust to label noise is an indication — not a proof — that it is learning something real.

For clinicians. Nothing deployable in the short term. An AUC of 0.88 on a 2D slice from a single centre justifies no change in practice, and regulatory maturity is nil — this is neither a CE-marked device, nor an FDA-cleared SaMD, nor even a clinically validated prototype. The operational message is defensive instead: with any automated report-drafting tool, laterality is the first thing to check, because that is what models invent most readily when the information is missing — 11% of cases for the most verbose model.

For patients and the general public. The result comes down to something simple: when an AI is graded on a paper it wrote itself, it almost always scores full marks. That says nothing about its competence. Many impressive numbers published in medical AI rest on automatic annotations that nobody has checked against clinical reality. Here the check was done: a radiologist corrected one reading in five, and the perfect score vanished.

Further reading

Tatakoto does not provide individual clinical advice. This article describes and critiques a scientific publication; it recommends no tool and no diagnostic course of action.