médical IA

Thyroid ultrasound: an auditable "agentic" AI tested across 35 centres, and the three numbers the abstract leaves out

Published on August 16, 2026 · 10 min read

Fourteen authors, mainly from Sun Yat-sen University and Zhujiang Hospital in Guangzhou, present ThyroidXAgent: an "agentic" system that chains nodule segmentation, benign-malignant classification and thyroid ultrasound report writing, keeping every intermediate output as a piece of evidence the clinician can inspect and correct. Segmentation reaches a mean Dice of 87.21% across seven test sets, and on 4,999 cases from 35 independent Chinese centres the benign-malignant classification AUROC is 0.819 [0.808–0.831]. Three things are nonetheless missing: the paper never writes how benign versus malignant status was established on its classification test sets, the mean AUROC quoted in its own abstract — 0.9466 — cannot be recovered from its own supplementary table, and the word "overdiagnosis" does not appear once in a study devoted to thyroid cancer detection.

The context

The thyroid nodule is one of the most banal findings in imaging: depending on the series, a neck ultrasound detects one in 20 to 70% of adults, and the vast majority will never become disease. The radiologist's job is therefore less to find nodules than to decide which warrant fine-needle aspiration. That decision runs through standardised scoring grids — TI-RADS and its variants — combining composition, echogenicity, shape, margins and the presence of punctate echogenic foci. The known weakness of this chain is variability: two operators looking at the same image do not always score it the same way, and the final report is often free text, worded differently from one practitioner to the next.

This context has a second face, which the paper does not mention and which must be stated up front. The thyroid is the textbook case of overdiagnosis in oncology: in South Korea, reported thyroid cancer incidence rose roughly fifteenfold between the early 1990s and 2011 under the effect of ultrasound screening, with no movement in mortality from that cancer — the finding published by Ahn, Kim and Welch in the New England Journal of Medicine in 2014. In other words, for this particular disease, increasing detection sensitivity is not automatically progress. That is the reading frame this paper most conspicuously lacks.

On the AI side, the prior state is fragmented: one model to segment, another to classify, each returning an isolated and opaque verdict. The 2026 proposal is agentic: a language model acts as conductor, calls specialised tools and — this is the central claim of the work — keeps their outputs in a consultable, editable evidence record.

The method

The authors first assemble OpenThyroidDB, an aggregate of roughly 0.3 million ultrasound images and 24,000 paired reports. The subset used for segmentation and classification comprises 38,864 images from seven cohorts, six clinical centres across China, Vietnam and Colombia, and at least eight different ultrasound platforms. Three of those cohorts serve exclusively as external tests: DDTI (Bogotá, Colombia), RJH-7K (Ruijin Hospital, Shanghai) and ZJH-8K (Zhujiang Hospital, Guangzhou). To these is added a private cohort, NHC-MISD-TUS, drawn from the Chinese National Health Commission's reference imaging database: 8,721 cases from 35 centres for gland segmentation, 6,384 cases from 33 centres for nodule segmentation, 4,999 cases for classification.

The expert pool. Rather than a single model, the system trains a family of specialised models, each built on a shared DINOv3 backbone — an image encoder trained by self-supervision, that is, without human labels — topped with a lightweight task-specific head. These experts differ in input resolution (128, 224 or 448 pixels), in the composition of their training data, and in whether or not they start from DINOv3 pretrained weights. For segmentation, a U-Net-style decoder reconstructs the nodule mask.

The router and the evidence layer. This is the "agentic" component. A language model receives not the images but structured summaries — candidate masks, class probabilities, confidence, radiomic descriptors, acquisition metadata — and emits a decision in strict JSON: which expert to keep, or what weighting to apply to a subset. From the selected mask, a radiomics branch extracts shape, texture and intensity descriptors via PyRadiomics, then passes them to an AutoGluon tabular ensemble whose decisions are explained by SHAP, a method that assigns each descriptor a numerical contribution to the final score. If the clinician corrects the mask, those attributions are recomputed on the corrected mask: this is the loop the title claims.

Report generation. A planner language model produces a task graph, a ReAct-style executor calls the tools through an MCP server, then the text is assembled by template retrieval: BM25 selects a sentence template from a library of 7,147 quality-controlled reports and measured values are inserted into the designated slots. An important point, rarely stressed: there is no free decoding. The report is not "written" by an LLM, it is filled in — which largely explains the absence of hallucinations.

One methodological reservation is due here: neither the router, nor the planner, nor the "judge" used to validate the new metric is ever named or sized. We do not know which language model drives the agentic part of the system.

The results

Segmentation. Across seven test sets, the mean Dice — the overlap between predicted and reference mask, where 100% means perfect superposition — reaches 87.21%, against 85.66% for MedSAM2, the strongest baseline, and 79.66% for the UltraFedFM foundation model. The 95th-percentile Hausdorff distance, which measures the coarsest contour error, is 6.90 mm and is the lowest on all seven sets.

Benign-malignant classification. The abstract announces a mean AUROC of 0.9466 across five test sets. Supplementary Table S4 gives these five values: 0.8692 (TN3K), 0.9676 (ThyroidXL), 0.9472 (TN5K), 0.7991 (DDTI) and 0.9175 (ZJH-8K). Their arithmetic mean is 0.9001, not 0.9466 — and no sample-size weighting recovers the announced figure, which would require an AUROC above 1 on DDTI. The same divergence runs the other way on AUPRC: the abstract announces 0.8361, the table gives 0.8705. The segmentation means, by contrast, reconcile exactly, which rules out a simple rounding convention. We flag the discrepancy without being able to explain it; it is verifiable line by line in the preprint's supplementary material.

The multicentre external validation is the most informative result in the paper, and also the least flattering. Across the 35 centres of NHC-MISD-TUS, nodule segmentation holds up (Dice 82.31%), but thyroid gland segmentation collapses to 59.28% — and the system is best in only 17 of 35 centres on that task, with floor values of 3% to 37% in several of them. Benign-malignant classification falls to AUROC 0.819 [0.808–0.831], eight to fifteen points below the internal sets. By centre, AUROC ranges from 0.407 to 1.000; the largest centre (n = 269) scores 0.590 with a confidence interval running from 0.346 to 0.832.

Secondary tasks and clinician studies. Lymph-node metastasis: AUROC 0.8642 on 158 images, ahead of MedGemma (0.8403) and GPT-5 (0.8410), but with heavily overlapping confidence intervals and no significance test. Follicular versus papillary carcinoma: AUROC 0.8053 on 200 images, where GPT-5 scores 0.1604 — distinctly worse than chance, a result the authors publish without dressing it up. On 500 blinded images, two clinicians read with and without the system's support: the first goes from 79.2% to 84.6% accuracy, the second from 74.0% to 84.8%, the latter with sensitivity rising from 71.6% to 85.6% and specificity from 76.4% to 84.0%. Assistance cuts segmentation from 14.21 to 9.11 seconds per image (−35.9%), with an assisted Dice of 0.903 against 0.879 manual, and reporting from 2.5 to 1.8 minutes per case across 145 videos (−27.4%).

Clinical translation. The paper publishes no decision threshold, no sensitivity and no specificity for the model itself — only AUROCs. It is therefore impossible to convert the system's performance into false positives and false negatives, which is itself a gap. The second clinician's figures can be translated, however. On the 500-image set, from which a malignancy prevalence close to 50% can be inferred, per 1,000 nodules they were missing 142 cancers and wrongly flagging 118 benign nodules; with the system's support, 72 missed cancers and 80 false positives. But apply those same sensitivity and specificity figures to a real clinic population, where the proportion of malignant nodules is more like 10%, and 1,000 nodules yield 86 cancers detected, 14 missed — and 144 false positives, a positive predictive value of roughly 37%. This calculation is our own extrapolation, not a result of the paper, and it illustrates the problem exactly: a high AUROC on a cancer-enriched cohort says nothing about the volume of unnecessary fine-needle aspirations in practice.

What is good

The evidence layer is not decoration: it is measured. Many studies claim explainability by bolting a Grad-CAM onto the end of a pipeline. Here the clinician's correction of the mask triggers a recomputation of the SHAP attributions, and the cost of that correction is quantified: 9.11 seconds against 14.21 for manual segmentation, with Dice going up rather than down (0.903 against 0.879). That the report is assembled from templates rather than freely decoded is a conservative engineering decision and, on a medicolegal document, a defensible one.

The external validation is published centre by centre. This is rare and does the authors credit: the supplementary tables give the performance of all 35 centres separately, including the centres where the system scores 3% Dice on the gland. Prevailing practice is to publish only the pooled mean, which here would have concealed considerable heterogeneity. That granularity is what makes the work criticisable — and therefore useful.

The comparators are numerous and current. MedSAM2, MedSegX, UltraFedFM, TransUNet on the segmentation side; MedGemma, Qwen3-VL-8B, GPT-5, Gemini-2.5-Pro, BiomedCLIP, MedSigLIP on classification and reporting. The authors publish results unfavourable to their competitors as well as absurd ones — GPT-5 at an AUROC of 0.1604 on follicular versus papillary — without smoothing them over.

What is less good

The reference standard for the primary endpoint is never stated. This is the central weakness. The paper asserts histopathological confirmation in exactly two places: for the sub-labels of the ZJH-8K cohort (nodal status, classic papillary versus follicular variant) and, for the LymphUs nodal cohort, confirmation by fine-needle aspiration. For none of the five benign-malignant classification test sets — TN3K, TN5K, ThyroidXL, DDTI, ZJH-8K taken as a whole — nor for the private 35-centre cohort does the text indicate whether the labels come from surgery, cytology or an image reading. No proportion of histologically confirmed nodules is given anywhere. Worse, the reader study's ground truth is explicitly annotation-derived: "cases containing any malignant annotation were classified as malignant or suspicious". In other words, the endpoint may be measuring the model's agreement with the annotating radiologist, not with the disease — a manual instance of shortcut learning in which the learned target is not the one assumed.

The chosen metrics flatter the system, and the external comparator is empty. Three mechanisms compound. First, the ZJH-8K cohort, presented as an external test, contains 723 malignant cases out of 1,149, a 62.9% prevalence — a surgical-series profile, the opposite of a screening population; an AUROC there is structurally optimistic. Second, on the private 35-centre set the only comparators are zero-shot models — BiomedCLIP and MedSigLIP, which have never seen a labelled thyroid and run at 0.43–0.52, that is, chance. Setting 0.819 against 0.44 establishes nothing: no supervised classifier trained on the same data was put up against it. Third, no calibration is reported — the word appears only in the limitations, as a property the system depends on — no threshold is set, no comparison with TI-RADS is made, and no clinical decision curve is produced. For a disease where the whole question is "should this nodule be biopsied?", that is precisely the missing measurement.

Overdiagnosis is absent, the clinicians number two, and the code is "open" with conditions. The terms overdiagnosis, microcarcinoma, indolent and active surveillance appear not once in the paper or its bibliography: the objective function is always detection, never the avoidance of an unnecessary procedure — even though raising a reader's sensitivity from 71.6% to 85.6% on the thyroid is exactly the intervention whose limits Korean epidemiology exposed. The reader studies rest on two clinicians whose seniority and specialty are not specified, with no p-values and no confidence intervals; the reporting study is a crossover between readers rather than within reader, so that with two operators the tested condition is perfectly confounded with reader identity, and the consistency figure of 70.3% → 86.2% highlighted in the abstract appears only there, with no table and no test. The data, finally, are almost entirely Chinese — a direct population bias. As for the "publicly available" code: both Hugging Face repositories are gated behind manual approval, none of the three repositories (GitHub, weights, data) declares a licence, the website is promised "after acceptance", and the private 35-centre cohort has no access route at all.

What it changes

For the research community, the net contribution is not the score but the architectural pattern: a router arbitrating between several experts on the basis of structured signals, and a persistent evidence layer the clinician can correct with downstream recomputation. The most solid lesson is in fact the internal-versus-external gap — 0.90 against 0.819 in classification, 87% against 59% in gland segmentation — measured across 35 real centres and published centre by centre. It is this kind of table, more than pooled means, that ought to become the publication norm in medical imaging.

For clinicians, nothing changes today: the evaluation is retrospective, the authors say so, and no operating threshold is published. The most directly usable result is the least spectacular: assisted correction of a contour takes 9 seconds instead of 14 without degrading quality, and a structured, template-assembled report saves some forty seconds per case. The demonstrated gain is one of time and wording consistency — not a diagnostic gain.

For patients and the public, the caution bears on what "detecting better" means here. On the thyroid more than almost any other site, a system that increases sensitivity without our knowing how many false positives it produces at real prevalence can produce more fine-needle aspirations, more thyroidectomies and more lifelong hormone therapy without a single life being saved. This is not an indictment of this paper: it is the question to put to any tool of this kind, and it is the one this paper does not put.