AI-generated pathology reports: why BLEU scores overstate quality, and what the clinical metric CRQS fixes

A team from Stony Brook and the University of Utah built the first standardized benchmark for automatically generating pathology reports from digitized slides, re-evaluating four models and three image encoders across three datasets. Their finding: the usual language metrics (BLEU, ROUGE, METEOR) reward surface similarity and can score highly a report that flipped the diagnosis. To fix this they propose CRQS, a score that compares extracted clinical facts rather than words — but the judge metric itself relies on a language model and is measured against noisy references.

The context

Generating a pathology report from a slide means asking a model to read a whole-slide image (WSI) — the scan of a stained tissue slide, a "gigapixel" image of billions of pixels — and write the diagnostic report: tumor type, grade, stage, margins, lymph-node involvement. The task is recent and becoming realistic thanks to pathology foundation models, vision networks pretrained on huge slide collections (here CONCH, UNI, H-Optimus) that turn each image fragment into usable feature vectors.

Two problems hold the field back. First, comparability: each team uses its own data, encoder, preprocessing and protocol, so it is unclear whether a reported gain comes from the model or the experimental setup. Second, evaluation. Most work grades generated reports with metrics borrowed from machine translation — BLEU, ROUGE, METEOR — which measure lexical overlap, i.e. the share of common words or short sequences between the generated and reference text. The problem is clinical: a report can be word-for-word close to the reference while omitting a diagnosis, getting the grade wrong, or inventing lymph-node involvement. These errors are not linguistic, they are medical — and a surface metric does not see them.

The method

The work has two building blocks. The first is a standardized benchmark. The authors reimplement four representative methods in a single codebase: WSI-Caption (a.k.a. MI-Gen, a multiple instance learning generator that aggregates fragment features into a slide report), HistGen (hierarchical local-global encoding), BiGen (a retrieval-augmented model that fetches external textual knowledge), and SCOUT (a context-aware multimodal transformer). Each is paired with three pathology encoders — CONCHv1.5, UNI2-h and H-Optimus-1 — to isolate the effect of the visual representation. Everything runs through the same pipeline: feature extraction with the TRIDENT tool on fragments at 20× magnification, the same tokenizer, the same train/validation/test splits, training on H100/H200 GPUs with a batch size of 1. Three datasets with complementary profiles: TCGA (7,074 / 884 / 885 slides; long, free-text reports with high annotation noise), HistAI (9,980 / 1,248 / 1,248; multi-institutional, moderately structured), and REG 2025 (5,924 / 740 / 741; short, semi-structured, low noise).

The second block is the CRQS metric (Clinical Report Quality Score). Rather than comparing two texts, CRQS extracts from each report — generated and reference — a set of structured clinical fields (diagnosis, histologic type, grade, stage, margin status, lymph-node involvement, biomarkers), via a language-model pipeline, then normalizes the vocabulary. It then computes four sub-scores: fact coverage (share of reference fields correctly recovered), key information recall (the most decisive fields: diagnosis, grade, stage, margins, nodes), hallucination rate (share of facts asserted by the model that are absent from the reference), and clinical discordance (direct contradictions on a field present on both sides). The combination weights key-information recall most heavily and penalizes discordance more than hallucination — a contradictory value being more dangerous than a non-conflicting addition — then normalizes to between 0 and 1. The field schema was designed with a pathologist, prioritizing elements genuinely inferable from the slide.

The results

The contrast between datasets is striking. On REG 2025 (short, structured reports), lexical and clinical metrics agree: BLEU-1 around 0.83–0.86 and CRQS around 0.69–0.71. On TCGA (long free text), CRQS collapses to about 0.18–0.19 and the hallucination rate reaches about 0.5 — nearly half the clinical facts asserted by the model unsupported by the reference — while the lexical scores stay "presentable" (BLEU-1 ≈ 0.34–0.39). In other words, on realistic data the classical metrics clearly overstate clinical quality. HistAI sits in between (CRQS ≈ 0.27–0.29).

Above all, the best model changes with the metric. On REG 2025, SCOUT tops the language scores but BiGen achieves the best average CRQS: picking your system on BLEU would have crowned a different winner than the clinical criterion. Two examples make the dissociation tangible. Case 1: the model downgrades a cervical lesion from high-grade (HSIL / CIN 2) to low-grade (LSIL / CIN 1) — a clinically heavy error — yet METEOR gives it 0.4264 (a respectable mark) while CRQS crushes it to −0.0714. Case 2: the model writes "reactive lymph node hyperplasia" where the reference says "reactive lymph node, no metastatic carcinoma" — same clinical meaning, different words — and METEOR punishes it at 0.0114 (a catastrophic mark) when CRQS recognizes 0.9143. Encoder choice has a measurable effect but none dominates everywhere: the optimal visual representation depends on the dataset and the method.

The clinical translation of Case 1 captures the stakes: describing as "low-grade" a genuine high-grade precancerous cervical lesion risks that a patient who needs treatment or close surveillance is simply reassured. A metric that scores that error 0.43 out of 1 would let a dangerous model through; that is exactly what CRQS aims to prevent.

What is good

Real, reproducible, public standardization. Shared fixed splits, unified TRIDENT extraction, reimplementation of the four methods in a single PyTorch Lightning codebase: the setup directly attacks the field's comparability problem. Tellingly, several re-evaluated models beat their own originally reported numbers, suggesting part of the published "progress" was preprocessing rather than architecture. The framework and splits are open (PathBench repository on GitHub), hence replayable.

A metric that speaks clinical, not linguistic. By decomposing quality into interpretable failure modes — omission, missed key field, hallucination, contradiction — CRQS does exactly what BLEU does not, and in both directions: it punishes a grade flip that METEOR found acceptable, and rewards a correct rephrasing that METEOR judged worthless. The schema, built with a pathologist and centered on slide-visible elements, anchors evaluation in real synoptic practice.

Public funding, academic teams, no conflict of interest. The work is supported by NIH grants (including 1R01CA297843-01) and NSF (2442053), led by university labs (biomedical informatics at Stony Brook, pathology at the University of Utah), with no commercial sponsor or declared conflict. On an evaluation question, independence matters: whoever defines the metric decides which models "win".

What is less good

The misleading metric… fixed by a metric that judges with a language model. To extract the clinical facts, CRQS relies on an LLM pipeline. The clinical judge is therefore itself a language model, whose extraction error rate is not quantified at scale against pathologists — it is only "verified by experts" qualitatively. This risks shifting the very problem it claims to solve up one level: an LLM judge can hallucinate a field or wrongly normalize a synonym, reintroducing exactly the hallucination and the misleading metric it is meant to measure. It is the same pitfall as an LLM evaluator in the clinic, which we have described elsewhere.

It measures fidelity to a reference, itself noisy, never a patient outcome. CRQS compares the generated report to the reference report — yet TCGA reports are explicitly described as "high annotation noise". Discordance and hallucination are therefore computed against an imperfect ground truth: a "hallucinated fact" may be a correct observation absent from a sloppy reference, and discordance penalizes disagreement with a sometimes-wrong reference. The 0.5 hallucination rate on TCGA partly reflects reference quality, not only model failure. At no point does the metric touch a clinical outcome: no reader study with pathologists on the final reports, no prospective validation. Add an acknowledged population bias — only three datasets, predominantly Western (including the US TCGA): generalization to other labs and populations is untested.

Untested design choices. The CRQS weights (0.30 / 0.40 / 0.20 / 0.40) are set by hand, with no sensitivity analysis showing how rankings would shift under reweighting; and an individual report can receive a negative score (the CIN case at −0.07), so the 0-to-1 normalization only holds on average. Finally, the batch size of 1 and a fixed decoding strategy do not necessarily exploit each method's best configuration: the absolute numbers count as within-benchmark comparisons, not as the performance ceiling of any given model.

What it changes

For the research community, the message is clear: in pathology report generation, clinical correctness — not lexical overlap — should become the primary criterion, and evaluation should span several datasets and encoders to avoid single-setting conclusions. The reusable framework and splits lower the cost of an honest comparison. What remains is to "validate the validator": quantify CRQS's extraction error against pathologists, test sensitivity to the weights, and link evaluation to clinical outcomes.

For clinicians, nothing to deploy — this is a tool for grading research systems, not a device. But the lesson is directly useful: a vendor touting "BLEU/ROUGE state of the art" for an auto-report tool is saying almost nothing about clinical safety. The right questions are fact coverage, hallucination rate and discordance measured against pathologist-verified references.

For patients and the public, the study tempers headlines of the "an AI writes your pathology report" type. On realistic free-text data (TCGA), current systems hallucinate about half their clinical facts and score low on clinical fidelity, even when their reports "read well". The automated pathology report remains a research problem, not a clinical product — and real progress will be measured by the accuracy of the facts, not the fluency of the text.

To go further

The preprint is available on arXiv (2607.18448), posted in July 2026 by Suryakant Singh, Sejuti Majumder, Beatrice Knudsen, Joel Saltz and Prateek Prasanna (Stony Brook University; University of Utah); it has not yet been peer-reviewed. The evaluation framework and splits are released as PathBench on GitHub. On language-model-based evaluation and pathology foundation models, see our decryptages on the limits of an LLM evaluator for grading medical AI, on building a pathology foundation model by distilling experts, and on the robustness and generalization of histopathology foundation models.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. English, Spanish and Chinese translations produced with AI assistance and reviewed.