Lung cancer screening: LungAI refines Lung-RADS triage on the NLST trial, but on a single cohort, with a closed model and a study run by its developer
A US start-up, Oatmeal Health, tested its deep learning model LungAI on 2,747 lung cancer screening CT scans from the NLST trial, to see whether it adds information that Lung-RADS, a grading system based on nodule size and appearance, does not capture. The model ranks malignant nodules better than size alone or the public Sybil model and, combined with Lung-RADS, would roughly halve the number of cancers sent back for a one-year follow-up scan. The signal is serious, but it rests on a single cohort from the 2000s, a proprietary model whose training data are not described, and a study funded, designed and partly annotated by the company itself.
The context
Lung cancer screening with low-dose CT (LDCT) in heavy smokers reduces mortality, but it finds a great many nodules, the vast majority benign. To decide what to do with each one, US radiologists use Lung-RADS, a system published by the American College of Radiology (version 2022). It assigns each exam a category, from 2 (scan again in a year) to 4B/4X (invasive work-up, often biopsy), mainly according to nodule size, appearance (solid, part-solid, ground-glass) and change over time.
This logic has a known flaw: small cancers wait twelve months, and large benign nodules lead to unnecessary biopsies. Several AI models already estimate risk directly from the image, including Google's (Ardila et al., Nature Medicine, 2019) and Sybil (MIT and Mass General, 2023), a public model that predicts cancer risk one to six years ahead. The question here: does AI add anything to Lung-RADS, and what would happen if it were used to move patients one step along the management ladder?
The method
LungAI is a locked computer-aided diagnosis (CADx) model, meaning its parameters no longer change during evaluation. It gives each nodule a score from 0 to 10, and each exam the maximum score of its nodules. Nodules are not detected by LungAI but by third-party software already cleared by the FDA, Coreline's aView LCS, which also measures them and matches them across exams. The paper describes neither the network architecture nor its training data: neither source nor size. The authors state only that the test partition was never used to develop LungAI or Sybil.
That partition is the NLST test set defined by Ardila et al., the same one on which Sybil was published. Of 4,872 exams with at least one detected nodule, the authors exclude 164 exams where the cancerous nodule was not visible, 257 exams with slices thicker than 2.5 mm, and 1,704 exams without a solid or part-solid nodule of 4 to 30 mm. That leaves 2,747 exams from 1,244 patients, including 89 pathologically confirmed cancers, a prevalence of 3.2%. The population is the NLST one: median age 61, 62% men, 93.2% white patients, scanners from 2002 to 2004.
Comparators are nodule diameter, nodule volume and Sybil (one-year risk, the horizon most favourable to it). Comparing with Lung-RADS required categories: a single radiologist, blinded to the LungAI score and to outcomes, assigned them on a cancer-enriched subset of 226 eligible exams (78 cancers, 148 benign). Because this subset contains 87.6% of cancers but only 5.6% of benign exams, each benign exam "weighs" about 18 exams of the full cohort. The main metrics are the AUROC (area under the ROC curve, the probability that a cancer scores higher than a benign nodule) and the AUPRC (area under the precision-recall curve, more demanding when cancers are rare: a model guessing at random would score about 0.03 here).
Finally, a reclassification algorithm moves each patient one step up or down the Lung-RADS management ladder (12-month scan, 6-month scan, 3-month scan or PET-CT, biopsy), according to score cut-offs set per category.
The results
In ML metrics, LungAI reaches an AUROC of 0.92 (95% confidence interval 0.88-0.95) and an AUPRC of 0.47 (0.35-0.61). Sybil reaches 0.85 and 0.25, diameter 0.78 and 0.13, volume 0.79 and 0.12; all differences from LungAI are significant. Against Lung-RADS, on the weighted subset, the picture is more nuanced: Lung-RADS has an AUROC of 0.89, a gap of only 0.03 (-0.04 to 0.10), not significant (p = 0.20); the AUPRC gap of 0.31, however, is significant (p = 0.003). Added to a diameter model, the score raises AUROC by 0.10: it does more than measure size. Performance is lower for stage II and higher cancers (AUROC 0.83, versus 0.95 for stage I). No calibration analysis is reported.
Clinical translation, as projected by the authors per 1,000 nodule-positive exams (about 32 cancers): with Lung-RADS alone, 3.1 cancers would be sent back for a one-year scan; with LungAI, 1.6. Cancers referred straight to tissue sampling would rise from 18.3 to 23.9, and biopsies in patients without cancer would fall from 32.7 to 26.2, 6.5 fewer unnecessary biopsies, with total biopsy volume almost unchanged (51.0 versus 50.1). In modelling, with perfect adherence, this would amount to a cumulative 25 months of diagnostic lead time and about $2,200 saved per 1,000 exams, barely more than $2 per exam.
What is good
First, a readable and comparable evaluation protocol. The model is locked, the test partition is the one already used for Sybil, and the authors follow the STARD 2015 reporting guideline. The main comparator, Sybil, is public and re-run on the same exams, at the horizon most favourable to it. Reporting AUPRC at 3.2% prevalence is a good practice that remains too rare.
Second, a clinical question asked in the right place. Rather than offering one more score alongside Lung-RADS, the authors insert it into the existing ladder, moving patients by only one step, and they express effects as cancers kept waiting and biopsies avoided per 1,000 exams. A sensitivity analysis that scales all cut-offs by a factor of 0.5 to 2 shows that the gain among cancers (net reclassification index between +0.16 and +0.27) does not depend on fine-tuning.
Third, genuine transparency on conflicts of interest and limitations: equity holdings, the paid advisory role of the annotating radiologist, LungAI's uncleared status (510(k) pending), the descriptive nature of the stage analyses.
What is less good
First, a population bias that is hard to get around. Everything rests on a single cohort, the NLST, with scanners more than twenty years old, 93.2% white patients and only four cancers in non-white patients, which makes the analysis by race uninterpretable, as the authors acknowledge. Ground-glass nodules and thick-slice exams are excluded, as are 164 exams where the cancer was not visible on the index scan.
Second, a fragile metric behind the most striking figures. The comparison with Lung-RADS and the whole reclassification rest on 148 annotated benign exams, each weighted by about 18. By our calculation, one benign exam therefore counts for about 6.5 exams per 1,000: the drop from 32.7 to 26.2 unnecessary biopsies, presented as a 20% reduction, corresponds roughly to the net move of a single annotated benign exam. In addition, the so-called "a priori" cut-offs were chosen to match Lung-RADS performance estimated on that same subset, drawn from the test partition. This weakens the independence of the evaluation. Finally, the AUROC gap with Lung-RADS is not significant.
Third, structural conflicts of interest and an unverifiable model. The study is funded by Oatmeal Health, whose employees designed the analysis, interpreted the results and decided to publish; seven authors hold equity in the company. The only radiologist who assigned the Lung-RADS categories, the reference comparator, is a paid advisor to the company. The model is proprietary, its training data are not described, and the analysis code is available only "upon reasonable request". Under these conditions, it is impossible to rule out indirect data leakage, for example other exams from the same NLST patients used in training, or to reproduce the results.
What it changes
For the research community: this work offers a useful framework, measuring a model's contribution not in raw AUROC but in how patients move along an existing management ladder. The logical next step is external validation on recent, diverse screening cohorts, with several independent Lung-RADS readers and model access for teams with no link to the company.
For clinicians: nothing changes today; the device is not cleared. The takeaway is rather that, within a single Lung-RADS category, actual risk varies widely: in category 4A, the share of cancers ranges from 1.0% below LungAI's low cut-off to 46.2% above its high cut-off. If this gradient holds elsewhere, such a score could help triage intermediate nodules.
For patients and the public: the stakes are concrete, but the announced gains are modest in absolute terms (about 1.5 fewer cancers kept waiting per 1,000 nodule-positive exams) and rest on projections. An AI score presented as a "biomarker" still has to prove itself outside the cohort in which its own developer evaluated it.
Further reading
The preprint: LungAI: An AI Biomarker that Refines Lung-RADS v2022 Triage of Screen-Detected Pulmonary Nodules, Suri et al., medRxiv, posted 10 November 2026, DOI 10.64898/2026.11.09.26367662. Not peer reviewed.
Sybil: Mikhael et al., Journal of Clinical Oncology, 2023 (public code). NLST test partition: Ardila et al., Nature Medicine, 2019. NLST data: National Cancer Institute Cancer Data Access System. Lung-RADS v2022: American College of Radiology.
Editorial transparency note: this article was produced by Tatakoto's automated editorial pipeline from version 1 of the preprint; the supplementary tables and figures could not be consulted. The weight of one benign exam per 1,000 exams is our own calculation from the published weights. The English, Spanish and Chinese versions are AI-generated. Spotted an error? Contact us.