Predicting the heaviest Mohs surgeries before the operation: what thirty machine-learning models mostly learn from tumour size

A Sydney team trains thirty machine-learning algorithms to predict, before surgery, whether Mohs micrographic surgery will need 13 or more tissue sections — a "heavy case" threshold copied from the Australian reimbursement schedule. On 408 procedures from a single centre, the best model, a "stacking" ensemble, reaches a test AUC of 0.884 and 81.7% accuracy, above the 0.70–0.78 of prior work. But the test set holds just 82 cases, no external validation is done, and the feature-importance analysis shows the model mostly replays an anatomical truism: the bigger the tumour, the more you have to cut.

The context

Mohs micrographic surgery (MMS) is the reference treatment for keratinocyte skin cancers — basal cell carcinoma (BCC) and squamous cell carcinoma (SCC) — in anatomically delicate sites such as the face, with cure rates above 98% for primary tumours. Its principle: remove the tumour layer by layer, examining each layer under the microscope immediately, and repeat until the margins are clear. The number of "sections" and stages varies enormously between patients — from a single section to more than forty in complex cases.

That number has very concrete consequences: heavy cases lengthen operating time, tie up more resources and demand more complex closures. Knowing in advance that a procedure will be long would help schedule the theatre, counsel the patient and plan the reconstruction. Prior work attempted this prediction with AUCs of 0.70 to 0.78 — the AUC (area under the ROC curve) measuring, between 0.5 and 1, a model's ability to separate two classes, here "heavy case" and "light case." The question here: from data available before the incision alone, can a machine-learning model predict a 13-section-or-more case?

The method

The task is a binary classification: 13 sections or more versus fewer than 13. The threshold of 13 is not biological — it matches the highest reimbursement tier in the Australian schedule (Medicare Benefits Scheme), taken to mark resource-hungry cases. The data come from a single institution, The Skin Hospital, across two Sydney sites, over 2012–2017: 408 consecutive procedures. Patients average 68.5 years (± 12.9), are 57.8% male; 93.4% of tumours sit on the head and neck, and 90.0% are basal cell carcinomas.

The model receives only pre-operative variables — the authors deliberately exclude data known only after surgery (defect size, closure type, margin status) to preserve upstream clinical utility. Sixteen variables are kept: age, sex, the two tumour diameters, tumour area (computed as the area of an ellipse from those two diameters), tumour type, prior recurrence, aggressive histology, body site and risk "zone," laterality, fine anatomical unit, surgeon experience, a "see-and-do" status, biopsy status and smoking.

Thirty algorithms are put in competition, across six families: gradient boosting (XGBoost, LightGBM, CatBoost — methods that add up small decision trees each correcting the previous ones' errors), ensemble methods (stacking combines several models' predictions through a meta-model, here a logistic regression; voting makes them vote), neural networks of 3 to 7 layers, support vector machines and classic classifiers. The data are split into training (80%, 326 cases) and test (20%, 82 cases) by stratified sampling, with 5-fold cross-validation on the training set and bootstrap confidence intervals (1,000 resamples). A welcome point: the outcome is balanced, 195 cases (47.8%) at 13 sections or more versus 213 (52.2%) below. Feature importance is read via SHAP values (a method that assigns each variable its average contribution to the prediction), and an uncertainty estimate is produced by retraining 20 random forests with different seeds. No external or temporal validation is done.

The results

The declared winner, the stacking ensemble, reaches an AUC of 0.891 in cross-validation (95% CI 0.849–0.934) and 0.884 on the test set, with 81.7% accuracy and an F1 of 0.81. The random forest matches the same cross-validation AUC (0.891) but falls to 0.851 on the test; CatBoost gets 0.885 in cross-validation, 0.881 in test and the best calibration (Brier score of 0.134, the lowest — so its predicted probabilities are the most faithful). All methods sit within a whisker: AUC from 0.867 to 0.891. Notably, a wide 3-layer neural network reaches a test AUC of 0.892 — above the model crowned best.

The uncertainty stratification is the most interesting result: 58 of the 82 test cases (70.7%) receive a "high-confidence" prediction (uncertainty below 15%), and on that subgroup accuracy climbs to 91.4%, versus 70.8% for less certain predictions. Learning curves plateau by ~260 examples, suggesting more data of the same kind would help little. On the variables, the SHAP analysis is unambiguous: tumour area dominates (mean importance 0.141), well ahead of the first diameter (0.086), the second diameter (0.068), aggressive histology (0.046), recurrence (0.035) and age (0.035); body zone (0.006) and surgeon experience (0.008) count for almost nothing. A threshold effect appears around 1.5 cm² of area, beyond which the heavy-case probability rises sharply. The univariate analysis confirms it: cases at 13 sections or more involve tumours of 6.32 cm² on average versus 0.75 cm² (p < 0.001, Cohen's effect size d = 0.98, large).

In clinical translation, two caveats. First, the 81.7% accuracy must be judged against a base rate of about 52% (the majority class): the model adds real value, but on 82 test cases it is still wrong about fifteen times — and the preprint body reports neither sensitivity nor specificity at the chosen threshold, both needed to judge use. Second, the bulk of the signal lies in a single measurement made with a ruler: the tumour's area.

What is good

A balanced outcome and honestly plural metrics. Unlike many AI-health studies where an imbalanced prevalence inflates accuracy, here the two classes are near parity (47.8% versus 52.2%): accuracy and F1 are therefore interpretable without a prevalence trap. The authors do not stop at AUC; they report calibration (Brier score) and discuss which model is best calibrated — rigour too rarely seen.

Coherent, plausible interpretability. The SHAP values are stable across models (Spearman correlation ρ > 0.85 between all pairs), the threshold effect around 1.5 cm² is clinically credible, and the variable ranking tells a legible story rather than a black box. One sees why the model decides.

An uncertainty stratification useful for deployment. Rather than a single score, the system separates reliable predictions (70.7% of cases, at 91.4% accuracy) from doubtful ones (70.8%): a clinician would know when to trust the tool and when to distrust it. This is exactly the safeguard most clinical calculators lack. The study is also ethics-approved (protocol HREC 2017/021) with patient consent.

What is less good

A tiny test set and a "validation" that is mostly internal. The title announces "validation," but everything rests on a single random split of one centre: 82 test cases, about 39 positive. At that size, differences between models are within the noise — the 3-layer neural network (test 0.892) actually beats the proclaimed winner (0.884), and CatBoost (0.881) ties it. The "best model" is chosen on cross-validation ties settled at the third decimal, making the ranking unstable. No external or temporal validation is done, though that is precisely the step that makes most models collapse when the hospital changes.

Thirty models, no simple comparator, a risk of over-optimism. Evaluating thirty algorithms on 408 cases and keeping the best invites the winner's curse: the best AUC among thirty is mechanically optimistic, with no correction for that multiplicity. Above all, the most obvious comparator is missing: a plain logistic regression on tumour area alone. Since SHAP shows that area dominates everything, nothing proves the artillery of thirty models beats a tape measure and a rule of three. This is the comparator biased by omission: they beat the old literature (0.70–0.78) without measuring against the trivial baseline.

A near-tautology dressed as prediction, on a narrow population. Tumour area is deterministically computed from the two diameters, which are themselves counted among the "sixteen variables": the top three SHAP variables are thus the same information repeated. The model replays a truism — a large tumour requires excising more tissue — which the authors indeed concede: "13 or more sections serves primarily as a surrogate for tumour size." Add a strong population bias: a single Australian centre, 90% basal cell carcinomas, 93% head and neck. Finally the threshold of 13 is a billing artifact (the Medicare tier), partly dependent on the surgeon's habits — the number of sections per stage goes from 3.0 to 8.6 by zone — rather than a biological boundary. Note too that no code is released, the data are available only on request, and an online calculator derived from this work displays outputs (operating time, billing codes, defect size) far beyond the actually validated binary model, and rests on a six-model ensemble different from the one presented as best.

What it changes

For the research community, the study offers a tidy template for pre-operative prediction — good use of SHAP, reported calibration, uncertainty stratification — but above all a lesson in evaluation. Comparing thirty models on 82 test cases, without external validation and without a trivial baseline, yields a fragile ranking and a likely optimistic headline AUC. The remedy is known: pre-register a simple comparator (here, tumour area alone), validate externally and over time, report sensitivity and specificity at the operating point, and correct for the multiplicity of models tested.

For clinicians, a Mohs surgeon already anticipates heavy cases by looking at tumour size and location; this work formalizes that intuition without yet showing it beats a centimetre. The potential is real for theatre scheduling and patient counselling — provided external validation. The online calculator's extra outputs (operating time, reimbursement codes) are, for their part, not supported by the presented data.

For patients and the public, two simple ideas. No, no app can reliably tell you today that your Mohs surgery will be long: the result comes from a single centre, on 82 test cases, and has not been validated elsewhere. And an "AUC of 0.89" impresses less once translated: large tumours in delicate places need more surgery — which we already knew.

To go further

The preprint "Development and Validation of Machine Learning Models for Predicting 13 or More Sections in Mohs Micrographic Surgery" is available on medRxiv (10.64898/2026.07.20.26358484), by Yagiz Alp Aksoy, Simon Lee and Gilberto Moreno-Bonilla (University of Sydney, Daffodil Centre and the Centenary Institute's Biomedical AI Centre; Royal North Shore Hospital; The Skin Hospital, Sydney). The study is approved by the University of Sydney ethics committee (HREC 2017/021), with no declared funding or conflict of interest; the licence is CC BY-NC-ND 4.0. Data are provided on request and no code repository accompanies the article. On AI in skin cancer and validation pitfalls, see our decryptages on cascade classification of dermoscopic lesions and its external clinical validation, on a deep-learning model collapsing at external validation, and on random forests versus deep learning when data are scarce.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. Sensitivity and specificity at the chosen threshold do not appear in the body of the preprint and are therefore not reported here. English, Spanish and Chinese translations produced with AI assistance and reviewed.