VOICE-AE: grading radiotherapy adverse events from visit audio, near-expert agreement but an "always grade 0" model already scores 76.9%
A Mayo Clinic team built VOICE-AE, a pipeline that transcribes the audio of a radiotherapy visit, identifies who is speaking, then asks a large language model (LLM) to detect and grade ten adverse effects on the CTCAE scale. On 390 assessments drawn from 39 visits, agreement with an expert reference is decent (mean weighted kappa 0.717), but 76.9% of assessments are "grade 0": a model that always answered "nothing" would already reach 76.9% exact accuracy. The signal is real but modest, grades 1 and 2 are still poorly recognised (F1 of 0.58 and 0.57), and the authors acknowledge that severe toxicity is unproven.
The context
In oncology, every follow-up visit requires the clinician to record the side effects of treatment. The standard is CTCAE (Common Terminology Criteria for Adverse Events): for each toxicity (urinary discomfort, diarrhoea, and so on), a grade from 0 (absent) to 5, grade 2 generally being where some management becomes necessary. This documentation takes time and varies between observers: two clinicians listening to the same patient do not always record the same thing.
The idea is therefore to let the visit run normally, record the conversation, and automatically produce a draft grading. Earlier work mostly extracted toxicities from text already written in the record; here the raw material is spontaneous speech between doctor and patient. That is closer to real practice, and harder: hesitations, rephrasing, symptoms mentioned in passing.
The method
The task. For each visit and each of the ten genitourinary and gastrointestinal toxicities followed in patients treated for prostate cancer, assign an ordinal CTCAE grade (0, 1, 2, and so on). It is an ordinal classification, where being one grade off is less serious than being three off.
The pipeline. VOICE-AE chains automatic speech recognition (ASR, which converts audio to text), speaker diarization (which assigns each sentence to a speaker, doctor or patient) and an LLM (a large language model, which reads the transcript and infers the toxicities and their grade). We could only read the abstract and metadata of the early-access version: the specific models used, the instructions given to the LLM and the recording set-up are not in what we read, and we do not invent them.
The data. 31 patients, 39 visits, i.e. 390 visit–toxicity pairs, at Mayo Clinic (Rochester, Minnesota). One centre, patients receiving radiotherapy for prostate cancer.
The reference. Six blinded raters graded the same transcripts. Hierarchical clustering (a statistical method that groups raters who resemble each other) isolated a subgroup of three "expert" raters whose consensus serves as the reference. Agreement among all six, measured by ordinal Krippendorff's alpha (1 = perfect agreement, 0 = chance), is 0.677.
The results
ML metrics. Against the reference, VOICE-AE obtains a mean weighted kappa of 0.717 (kappa measures chance-corrected agreement, the weighting penalising large errors more), a Krippendorff's alpha of 0.669, an exact-grade accuracy of 82.6% and a within-one-grade accuracy of 99.2%. The authors themselves give the trivial comparator that always answers "grade 0": 76.9% and 93.6%. On the grades that matter in practice, the F1 (harmonic mean of precision and recall) is 0.58 for grade 1 and 0.57 for grade 2. For binary detection (toxicity present or not): sensitivity 77.8%, specificity 89.0%, precision 68.0%.
Like-for-like comparison: the weighted kappa between VOICE-AE and the three experts is 0.666, against 0.681 among the experts themselves. The system therefore sits slightly below inter-expert agreement, in the same range.
Clinical translation. Of the 390 assessments, 90 have a non-zero grade (390 minus 300). A sensitivity of 77.8% means catching about 70 of those 90 and missing about 20; a specificity of 89.0% means about 33 false alerts among the 300 normal assessments (70 true positives out of 103 alerts, which does give 68% precision). Scaled to 1,000 assessments with the same proportion of toxicities, that would be about 180 toxicities caught, 50 missed and 85 false alerts. The accuracy gain over the trivial model amounts to about 22 more correct assessments out of 390. These orders of magnitude are our calculations from the abstract, not published values.
What is good
A trivial comparator published by the authors themselves. Giving 76.9% and 93.6% next to 82.6% and 99.2% is rare honesty: the reader can judge the real gain instead of being impressed by "99.2%".
A reference that admits human disagreement. Rather than pretending a single truth exists, the study measures agreement among six raters (alpha 0.677) and compares the system with agreement among experts (0.666 versus 0.681). That is the right question for a task on which clinicians themselves diverge.
Named limits and open access. The authors write that accuracy at grade 2 or above and detection of severe toxicity remain unproven, declare institutional funding (a Mayo cancer centre grant, with no role for the funders in the study) and no competing interests. The article is published under a CC BY 4.0 licence.
What is less good
A misleading metric on imbalanced classes. With 76.9% grade 0, raw accuracy says little; the gap over the trivial model is only 5.7 points on exact grade. And the F1 of 0.58 and 0.57 on grades 1 and 2, the ones that matter, indicates mediocre recognition of those intermediate levels. A precision of 68% means about one alert in three is a false alert: a draft to read closely, not a grading to sign.
A tiny, non-independent sample from a single centre. 31 patients for 39 visits: some patients contribute more than once, and the ten toxicities of a given visit are related to each other. The 390 assessments are therefore not 390 independent observations, and the confidence intervals, which we did not see, should reflect that. It is a classic population bias: one department, one disease, one patient population; nothing says how the pipeline behaves with different accents, another language or different room noise.
A reference that depends on those who define it. The expert subgroup is chosen by statistical clustering among the same six raters, so the reference is partly built from the data. The raters graded transcripts; we do not know whether these were the texts produced by the system's own speech recognition, in which case part of the listening errors would be absent from the comparison. This is a retrospective validation on 39 visits: no prospective evaluation in clinical conditions, and no measured effect on workload or quality of follow-up.
What it changes
For the research community. The paper offers a useful evaluation frame: a trivial comparator, inter-rater agreement and a like-for-like comparison. The natural next steps are larger cohorts with more grade 2 or higher toxicities, several centres, end-to-end evaluation on raw audio, and release of the code and prompts.
For clinicians. Nothing changes today: no clinical use is demonstrated and the pipeline has not been tested on severe toxicities, the ones where an omission costs the most. In time, a draft grading reviewed and corrected by the clinician could lighten documentation, provided the false-alert rate is measured in the department concerned.
For patients and the public. Recording a consultation raises confidentiality and consent questions that the abstract does not address. The hoped-for benefit is more consistent tracking of side effects from one visit to the next; that remains to be shown.
Going further
Paper published in npj Digital Medicine: doi.org/10.1038/s41746-026-03373-z (received 10 April 2026, accepted 26 September 2026, published online 10 October 2026, early-access version, subject to further edits). Corresponding author: Mark R. Waddle (Mayo Clinic). Funding: Mayo Clinic Comprehensive Cancer Center grant (P30 CA015083) and the Paul Calabresi Program (K12CA090628); the authors declare no competing interests. CC BY 4.0 licence. A supplementary file is linked to the article; code and data availability does not appear on the page we read.
Editorial transparency: this article was produced with the help of artificial intelligence tools from the early-access article page (abstract and metadata); the full Methods section could not be consulted and the journal version may differ. The English, Spanish and Chinese versions are machine translations.