Whether to operate on a pancreatic cancer: what a multimodal CT + clinical-data model really learns to triage

A team from the University of Basel, Kantonsspital Aarau and the Royal Free Hospital in London trains a deep-learning model that fuses contrast-enhanced 3D CT with 17 clinical variables to sort 159 pancreatic ductal adenocarcinoma patients into the three NCCN resectability categories. AUC reaches 0.86 in internal cross-validation and 0.86 on an independent external cohort of 52 patients from Aarau — a rare external validation in this field. But the ground truth is only a multidisciplinary tumor-board consensus, the cohort is small and entirely Swiss, and no study compares the model to radiologists in real conditions.

The context

Pancreatic ductal adenocarcinoma (PDAC) is one of the deadliest cancers: it accounts for over 90% of pancreatic cancers, with five-year survival below 10%. The only curative option is complete surgical resection (so-called R0), yet fewer than 20% of patients are eligible. Deciding who can be operated on — "resectability" — rests above all on the relationship between the tumor and the major peripancreatic vessels seen on contrast CT: the superior mesenteric artery (SMA), superior mesenteric vein (SMV), portal vein and celiac trunk. The US NCCN framework defines three categories: upfront resectable, borderline, and locally advanced. The stakes are high: a borderline patient generally receives neoadjuvant chemotherapy then surgery, a locally advanced one shifts toward non-surgical treatment.

The problem is that this judgment is poorly reproducible: inter-observer agreement on resectability often stays below 70%, even with standardized criteria. Hence the idea of automating the reading. Prior work was limited either to segmenting organs or to predicting from the image alone, without integrating the clinical variables that weigh at the tumor board (comorbidities, tumor markers such as CA 19-9). Many additionally required a segmentation mask at inference time, which complicates deployment. This paper proposes an end-to-end framework that combines image and clinical context and no longer needs a mask once trained.

The method

The core is a Swin-UNETR: an encoder-decoder network whose encoder is a "shifted-window" transformer — the architecture that processes data by weighting the relative importance of its elements — adapted to 3D and particularly good at delineating fine vascular structures. During training, the decoder learns an auxiliary task of segmenting 16 anatomical classes (pancreas, tumor, portal vein, SMV, SMA, celiac trunk, aorta, vena cava, hepatic arteries, dilated pancreatic duct, bile duct…). This supervision forces the encoder to produce "vessel-aware" representations. The encoder bottleneck outputs a 256-dimensional vector.

In parallel, 17 routine clinical variables (age, sex, BMI, Charlson Comorbidity Index, ASA score, diabetes, smoking, alcohol, baseline CA 19-9, its delta and its value after neoadjuvant therapy, CEA, blood glucose, HbA1c, bilirubin, histological grade, neoadjuvant status) pass through a small multilayer perceptron (MLP) that produces a 32-dimensional vector. The two vectors are concatenated into a 288-dimensional representation, then a classification head predicts the three NCCN categories. Training follows a dynamic multitask objective: the relative weight of segmentation and classification adjusts automatically according to the tumor Dice (the overlap rate between predicted and true segmentation) — anatomy is learned first, then the model shifts toward the decision. An important practical detail: masks are used only during training; at inference the model predicts resectability without any segmentation step. CT scans are resampled to 1 mm isotropic, cropped to 160×160×160 voxels, normalized, and the encoder is pretrained on the public abdominal BTCV set.

The main cohort comprises 159 patients with histologically confirmed PDAC from University Hospital Basel and St. Clara Hospital (imaging from three institutions): 85 upfront resectable (53.5%), 47 borderline (29.5%), 27 locally advanced (17%). The NCCN labels are those decided at the multidisciplinary tumor board, then reviewed by an independent physician. An external cohort of 52 patients from Kantonsspital Aarau (28/14/10), never used for development, serves as a generalization test. Internal evaluation uses stratified nested five-fold cross-validation: 20% of the data is held out as test at each outer fold, the rest serving for training and tuning. Performance is read as AUC (area under the ROC curve: the probability that the model correctly separates two classes), macro-F1 (a balanced average of precision and recall across the three classes) and accuracy.

The results

In internal cross-validation, the multimodal model reaches an AUC of 0.86, a macro-F1 of 0.79 and an accuracy of 0.85. On the external Aarau cohort it holds: AUC 0.86, macro-F1 0.81, accuracy 0.87 — an unusual stability across centers. The auxiliary segmentation reaches a Dice of 0.82 for the pancreas, 0.71 for the tumor and 0.67 for the major vessels: imperfect, but enough to guide the encoder. The model beats the two comparators re-implemented by the authors: a segmentation-plus-geometry approach (Viviers, AUC 0.79 internally) and a texture-aware transformer (TAT, 0.83). The ablations are telling: the image-only model gets 0.82, the clinical-only model just 0.74, and a "knockout" analysis (replacing one modality by its mean) shows that most of the signal comes from the image (0.83 without the clinical, 0.77 without the image). The image carries the diagnosis, the clinical data refines it.

The clinical translation is decisive here, because the three classes are neither equally easy nor equally serious to confuse. The per-class F1 is 0.83 for resectable, 0.78 for borderline, but drops to 0.70 for locally advanced — the rarest class (27 internal cases, 10 external). Yet this is precisely the most consequential boundary: wrongly classifying a locally advanced patient as resectable can send them to futile, morbid surgery; the reverse can deny a potentially curative operation. With an accuracy of 0.85, roughly one patient in seven is misclassified — a figure to set against a human agreement itself often below 70%, but one that forbids any autonomous use. The authors add a sanity check: from the predicted segmentations, they recover tumor-vessel contact angles consistent with the NCCN definitions (arterial contact ≥ 180° ↔ locally advanced), suggesting the model learned a plausible geometry rather than a shortcut.

What is good

An external validation, a rare thing. Most imaging AI models are never tested outside their home center — and that is where they collapse. Here, performance stays identical (AUC 0.86) on an Aarau cohort never seen in training, with a properly designed nested cross-validation (test held out from hyperparameter tuning). A notable detail: the external cohort had complete clinical data, with no imputation, indicating that the imputation applied to training introduced no detectable bias.

Mask-free inference, designed for the clinic. Segmentation serves as pedagogical scaffolding during training and then disappears: the final model predicts resectability directly from the CT, with no manual segmentation step. This is exactly the deployment obstacle that prior pipelines dragged along. The adaptive multitask objective (shifting from anatomy to decision as tumor Dice improves) is a simple, well-ablated idea.

Honest comparators and ablations. The authors do not merely display their best number: they re-implement two competing methods, test image-only, clinical-only, and the "knockout" of each modality. One sees precisely where the signal comes from (the image), and finds that simple concatenation fusion suffices — more complex cross-attention mechanisms added nothing and destabilized training at this cohort size. Public code, CC BY 4.0 licence.

What is less good

The ground truth is a human decision, not a surgical outcome. This is the deepest limitation. The NCCN labels are those set at the tumor board, not confirmed by surgery or pathology (actual R0 resection). The model therefore learns to reproduce what a tumor board would say — with its biases and, mechanically, part of the 30% inter-observer disagreement the authors themselves invoke to justify the study. A tool that mimics an imperfect consensus cannot, by construction, do better than that consensus; above all it can homogenize it. This is a biased-comparator failure mode: the reference is not a hard fact but a judgment.

A small, geographically narrow cohort. 159 + 52 patients, all Swiss, with a "locally advanced" class reduced to 27 and 10 cases — hence wide intervals and an F1 of 0.70 on this critical class. The Aarau "external" validation shares the same country, the same guidelines and similar equipment: the domain shift is small. Nothing says the model would hold on scanners from another continent, other injection protocols, or a different population. This is a misleading-metric failure mode coupled with a population bias: a high macro-AUC over three imbalanced classes smooths over the weakness on the minority that matters most.

No reader study, no prospective evaluation. The paper does not compare the model to radiologists or surgeons under standardized conditions, does not measure whether it actually reduces inter-observer disagreement, and remains retrospective. The segmentation masks are semi-automated (reviewed by up to five physicians), and the vascular Dice is only 0.67: the vaunted "vessel-aware" representations rest on an imperfect delineation of the vessels themselves. Explainability is deferred to future work. In other words, this is a solid proof of concept, not a tool validated for decision-making.

What it changes

For the research community, the paper is a clear demonstration that fusing a "vessel-aware" 3D image with structured clinical variables, with mask-free inference, is feasible and generalizes between two Swiss centers. The recipe (auxiliary anatomical supervision + adaptive multitask weighting + simple fusion) is reproducible and the code is open. The obligatory next step is twofold: a multi-continental validation on heterogeneous scanners, and above all an anchoring of the ground truth on real surgical and pathological outcomes, not just on a board's verdict.

For clinicians, nothing changes in practice today. At best, one glimpses a future aid to standardizing resectability reading — useful precisely because human agreement is weak — but it will first require a prospective reader study showing that it reduces errors without introducing new ones, particularly at the borderline / locally advanced boundary where operative decisions are made.

For patients and the public, the message is to temper enthusiasm: "AUC 0.86" does not mean "AI decides who gets operated on." It means that a model, on a limited Swiss population, reproduces reasonably well the classification of a panel of experts — a panel that itself errs in about a third of contentious cases. The decision to operate on a pancreatic cancer remains, and will for a long time, a collegial human decision.

To go further

The preprint "Multimodal Assessment of Pancreatic Cancer Resectability Using Deep Learning" (accepted at MIDL 2026) is available on arXiv (10.48550/arXiv.2607.13826), by Vincent Ochs, Christoph Kuemmerli, Florentin Bieder, Julia Wolleb, Joël L. Lavanchy, Julia Ruppel, Jan Liechti, Stephanie Taha-Mehlitz, Christian A. Nebiker, Beat Müller, Giuseppe K. Fusai, Joerg-Matthias Pollok, Anas Taha, Philippe C. Cattin and Sebastian Staubli (University of Basel, Department of Biomedical Engineering; Clarunis, University Digestive Health Centre, Basel; Kantonsspital Aarau; Royal Free Hospital, London). Funding: University of Basel Research Fund for Junior Researchers, Swiss National Science Foundation (P5R5PM 21766), Novartis Foundation for medical-biological Research (23C162), Vontobel Foundation; no conflict of interest declared. Public code under CC BY 4.0 (github.com/vincentochs/pancreas_resectability), data and weights on request. On segmentation and multimodal fusion in oncology AI, see our decryptages on UNETR segmentation of prostate cancer in PSMA PET/CT and survival prediction, on multimodal fusion of foundation models in oncology, and on deep radiomic signatures for tumor classification.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. The metrics (AUC 0.86 internal and external, macro-F1 0.79/0.81, F1 of 0.70 on the locally advanced class) are those reported by the authors on cohorts of 159 and 52 patients. English, Spanish and Chinese translations produced with AI assistance and reviewed.