Severe acute pancreatitis: a random forest beats every deep-learning model on 722 patients

A single-author preprint (Netanel Stern, "PenuX Research Initiative") compares eleven machine-learning and deep-learning models to predict, at admission and from thirteen routine laboratory tests, which patients will develop severe acute pancreatitis — on a public Chinese cohort of 722 patients. The result is clear and instructive: classical models win, a random forest reaching an AUC of 0.877, while every deep-learning model does worse, some collapsing to the point of predicting "severe" for almost everyone. It is an honest, useful negative result — trees beat neural networks on small tabular data — but the cohort contains 81% severe cases, the inverse of clinical reality, and the study never compares itself to the clinical scores it claims to beat: it therefore proves nothing yet about its usefulness at the bedside.

The context

Acute pancreatitis is one of the most common digestive emergencies, with a global incidence estimated at about 34 cases per 100,000 people per year. In most cases it resolves on its own, but 10 to 20% of patients progress to a severe form (severe acute pancreatitis, SAP), defined by the 2012 Revised Atlanta Classification as the presence of persistent organ failure — lasting more than 48 hours. This severe form is formidable: mortality is on the order of 20 to 40%, and it places heavy demands on intensive care.

The whole challenge is anticipation. Spotting at-risk patients early allows escalation to intensive monitoring, vigorous fluid resuscitation and multidisciplinary management. Yet the existing severity scores — Ranson, APACHE II, BISAP, the Modified CT Severity Index — share one limitation: they require data accumulated over 24 to 48 hours, or a contrast CT not always appropriate in the acute phase. Hence the appeal of a model that would decide on arrival in the emergency department, from the first blood draw alone. That is the goal this preprint sets itself.

The method

The data come from an already-published Chinese cohort deposited publicly on GitHub: 722 patients with acute pancreatitis, of whom 585 (81.0%) classified as severe and 137 (19.0%) as mild or moderately severe. This proportion is essential to keep in mind: it does not reflect the real frequency of the severe form, but the recruitment of a referral centre, where serious cases are over-represented. Each patient is described by thirteen variables available at admission: white blood cell count, C-reactive protein, creatinine, urea, blood glucose, calcium, LDH, albumin, haematocrit, age, body mass index, heart rate and systolic blood pressure.

The author pits eleven models against each other in three families. Three classical models: a logistic regression, a random forest (an ensemble of hundreds of decision trees that vote, here 500 trees with class weighting) and a gradient boosting (trees added successively, each correcting the previous one's errors). Three feedforward neural networks: a multilayer perceptron (MLP, the basic architecture where information passes through successive layers), a residual-connection variant, and an attention-mechanism variant (which weights the relative importance of the variables). Finally five recurrent LSTM-type models — an architecture designed for temporal sequences. A decisive detail: the tabular data, which have no temporal dimension, are here presented to these networks as "sequences" of length 1, a use outside their design domain.

Evaluation rests on five-fold stratified cross-validation (the cohort is split into five parts, preserving the proportion of severe cases, each part serving in turn as the test set). A notable point of methodological hygiene: normalisation and imputation of missing values are computed only on the training folds, which avoids data leakage — the contamination of the test set by information from training, which artificially inflates performance. The decision threshold is chosen to maximise F1 (the harmonic mean of precision and sensitivity). The reported metrics are the AUC (area under the ROC curve, the ability to separate severe from non-severe), F1, sensitivity and positive predictive value. No external validation is performed.

The results

The classical models dominate. The random forest comes first with an AUC of 0.877, an F1 of 0.917, a sensitivity of 96.8% and a positive predictive value of 87.1%; at its optimised threshold of 0.535, it correctly identifies 96.8% of severe cases (about 566 true positives and 19 false negatives across the cohort). Gradient boosting follows very closely: AUC 0.874, F1 0.918, sensitivity 97.1%. Logistic regression reaches 0.817.

Deep learning systematically does worse. The best feedforward network, the plain MLP, tops out at 0.836 — respectable but below the trees. The recurrent models collapse: the best of them, a CNN-LSTM, reaches only 0.777, the standard LSTM 0.699, and the LSTM-with-attention variant brings up the rear at 0.684. The gap between the random forest and the best recurrent model is 0.100 AUC points, a wide margin. Above all, the LSTMs show sensitivities of 98.3 to 100% at the cost of near-zero specificity: they have tipped toward the majority class and predict "severe" for almost everyone. A sensitivity of 100% obtained by declaring everyone ill is worth nothing — a hands-on illustration of a misleading metric.

An honest clinical translation calls for a warning about these flattering figures. The 87.1% positive predictive value was measured in a population that was 81% severe. Yet a test's precision depends on prevalence: applied to a real emergency department where only 10 to 20% of patients will progress to the severe form, the same model would produce far more false alarms for the same sensitivity. The author notes that his sensitivity "compares favourably" with the published 60 to 80% values for BISAP and APACHE II within 24 hours, but without ever computing those scores on the same cohort — so the comparison remains indirect.

What is good

Methodological hygiene against data leakage. Normalisation and imputation computed only on the training folds, stratified cross-validation preserving the class imbalance, threshold set on the training portion: these precautions, often neglected in the AI-health literature, are correctly applied here and limit the risk of artificially inflated performance.

An owned negative result. Reporting that deep learning loses to a random forest, and that LSTMs collapse, runs against the publication bias that favours positive announcements. The choice of variables is also clinically sensible: only routine tests available from admission, without CT, hence early and inexpensive.

Transparency. The code is public on GitHub, the study builds on an open dataset, the author declares no funding and no conflict of interest, and an explicit warning states that the work is not intended for clinical use. So many signals of seriousness, rare for a preprint.

What is less good

A cohort 81% severe: the population-bias and misleading-metric failure modes. The study's prevalence is the inverse of epidemiological reality (10 to 20% severe forms). All the performance figures — and especially the positive predictive value — are mechanically embellished by it. The headline 96.8% sensitivity says almost nothing about what the model would do in an emergency department, on a population where serious cases are rare.

No clinical comparator: the absent-comparator failure mode. The paper never confronts its models with the BISAP, APACHE II or Ranson scores on this cohort. Yet the entire rationale — beating slow scores — rests on that comparison, which is not made. Claiming superiority against values drawn from other studies, on other populations, has no probative value.

No external validation, and a fragile provenance. Everything rests on a single Chinese cohort, reused from a public repository; nothing guarantees generalisation to other countries, other aetiologies or other practices. The author acknowledges that cross-validation may overestimate performance compared with a true independent test set. And the work is a non-peer-reviewed preprint, by a single author affiliated with a "research initiative" with no identifiable hospital or university affiliation: the maturity remains that of a prototype.

What it changes

For the research community, the study is a salutary reminder: on small tabular data, boosted trees and random forests remain the benchmark to beat, and applying LSTMs to data with no temporal dimension — folded into length-1 sequences — is a category error. The "medical problem, therefore deep learning" reflex is clearly caught out here.

For clinicians, nothing is deployable: no external validation, no comparison with the scores they already use, and an unrepresentative prevalence. The model has not shown that it adds anything beyond existing tools. For patients and the general public, the study above all illustrates a reading rule: a headline like "an AI predicts severe pancreatitis with 97% accuracy" hides that the cohort was already 81% severe — a figure that does not transfer as is to a real emergency room.

To go further

The preprint is available on medRxiv (DOI 10.64898/2026.06.20.26356146); the code is published on the author's GitHub repository, and the source cohort on a dedicated public repository. On the critical reading of clinical models — calibration, decision curves and prevalence — see our decryptage of mortality prediction in intensive care on MIMIC-IV; on classical learning and explainability, the one on thromboembolism risk in endometrial cancer.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. English, Spanish and Chinese translations produced with AI assistance and reviewed.