médical IA

Colorectal surgery: one model segments the operation into phases across 54 videos, and loses a third of its performance as soon as a centre leaves the training set

Published on August 22, 2026 · 14 min read

Thirteen authors from IHU Strasbourg, Gemelli and Sapienza in Rome, Policlinico di Milano and Virgen Macarena in Seville train a single model to automatically segment minimally invasive colorectal surgery videos into operative phases and steps, from 54 operations drawn from four centres and one public German dataset. The model reaches a macro F1 of 73.01% for phases and 39.82% for steps, and outperforms models trained centre-by-centre or procedure-by-procedure in most tested configurations. The most instructive result in the paper is the one that undercuts the previous sentence: when a centre is removed from training and the model is evaluated on it, phase F1 falls to 48.42% on average, and as low as 37.49% at the worst-off centre.

The context

Minimally invasive colorectal surgery — laparoscopic or robotic colectomies and rectal resections — is characterised by wide variability in practice and inconsistent outcomes across surgeons and centres. To make that variability measurable, the field has adopted a tool: video-based assessment (VBA), evaluation from the recorded video of the operation. The principle is simple — annotate the video to establish who did what, in what order, for how long — and it feeds training, credentialing, and research into the link between technique and complications.

That requires a shared vocabulary. Hence ColoWorkflow, an ontology published in 2026 in BJS Open by part of the same team, built through a modified Delphi process with more than forty international experts. It defines 9 phases that apply to every colorectal procedure (port placement and exploration, vascular dissection, colon mobilisation, transection, anastomosis, completion, preplanned additional procedures, unplanned procedures, extracorporeal steps) and 34 steps specific to each procedure type. The problem is cost: manually annotating a video according to ColoWorkflow takes, per the authors, roughly 60% of the video duration. At two and a half hours per operation, VBA does not scale.

Hence the temptation to automate it. Automatic surgical phase recognition is a mature field — on the Cholec80 laparoscopic cholecystectomy dataset, models exceed 90% accuracy. But cholecystectomy is short, highly standardised, and split into few phases. In colorectal surgery, existing work remains single-procedure: Nakajima et al. on sigmoid resection, Kolbinger et al. on robot-assisted rectal resection. No model had yet attempted to apply a single consensus ontology across the whole colorectal spectrum. That is the gap this paper targets.

The method

The data. Fifty-four operative videos, from five sources: four partner centres anonymised as "Centre 1" through "Centre 4" (located in Italy, Spain and France, with no mapping given between number and hospital), and the public HeiCo dataset from Heidelberg, which alone contributes 22 of the 54 videos. Five procedure types: left hemicolectomy (n = 9), right hemicolectomy (n = 13), rectal resection (n = 10), sigmoid resection (n = 12), total proctocolectomy (n = 10). Mean duration 141 ± 59 minutes, from 93 ± 32 minutes for right hemicolectomy to 210 ± 50 minutes for total proctocolectomy. Frames are extracted at one frame per second and resized to 224 × 224, for 462,000 frames in total. Local de-identification via the Endoshare application, annotation on the MOSaiC platform, ethical approval from Policlinico Gemelli (ID 6456).

The annotation. A single physician, trained in VBA and involved in developing ColoWorkflow, annotated the entire corpus. Of the ontology's 9 phases and 34 steps, 8 phases and 33 steps are actually evaluated: phase P7 (preplanned additional procedures) and step S25 (ileoanal pouch preparation) are absent from the test set.

The architecture. Two blocks. First a visual extractor: a DINOv3 vision transformer — an image model pre-trained in a self-supervised way, that is, without human labels, on very large generic image corpora — fine-tuned on the annotated frames to produce a vector representation of each frame. Then a temporal model: SAHC, a multi-stage temporal convolutional network stacking dilated causal convolutions, meaning they only look at the past — so the model can in principle run live during the operation, without waiting for the video to end. Each stage refines the previous stage's predictions, which reduces over-segmentation (the model changing its mind every two seconds). The distinctive feature: a shared temporal trunk with two output heads, phase and step, jointly optimised through a combined cross-entropy with equal weighting. The idea is that phase context regularises step prediction, and vice versa.

Splits and comparators. Case-level partition — 32 training videos, 11 validation, 11 test — which prevents the same operation appearing on both sides. The test set comes from the same five sources as training: the authors themselves call this configuration "the most optimistic scenario". Four comparators: a majority-class baseline, models trained on a single centre, models trained on a single procedure type, and a leave-one-centre-out (LOCO) cross-validation that withholds each centre in turn to evaluate the model on that unseen centre. A paired Wilcoxon test on the 11 test videos compares the model to the baseline. Reporting follows a modified version of the QUAIDE checklist.

The results

The headline number. On the test set, AI-ColoWorkflow reaches a macro F1 of 73.01% ± 10.27 for phases (balanced accuracy 73.43%, precision 73.19%, recall 73.43%) and 39.82% ± 7.06 for steps (balanced accuracy 38.65%, precision 48.36%). The majority-class baseline tops out at 6.71% and 2.29% (p = 0.001). Note what the standard deviation is: not across random seeds nor across cross-validation folds, but across the 11 test videos. The 73.01% comes from a single split.

Per phase. Extracorporeal steps (P9) are recognised at 94.73% F1 — unsurprisingly, the image changes radically when the scope leaves the abdomen. Then vascular dissection (P2, 76.58%) and anastomosis (P5, 75.27%). At the bottom: unplanned procedures (P8) at 41.48%, colorectal transection (P4) at 64.11%, completion of operation (P6) at 66.45% with 82.55% precision for 55.61% recall.

Per step. The table is published in full, and it is harsh. Three steps exceed 79%: enterotomy closure on the right (S31, 84.61%), distal transection on the right (S22, 81.53%), externalisation of the resected segment (S21, 79.96%). But two steps score 0.00% F1 — preparation for intracorporeal anastomosis on the right (S29) and additional suturing (S32) — because they are too rare in the test set. Step S23 shows 100% precision for 3.88% recall, an F1 of 7.46%: the model almost never predicts it, but when it does, it is right. The authors write plainly that some short steps had fewer training frames than test frames.

Global versus specialised. For phases, the global model beats centre-specific models in 4 out of 5 cases (Centre 2: +18.9 points; Centre 4: +14.3; Centre 1: +9.1; Centre 3: −1.0) and procedure-specific models in 4 out of 5 (total proctocolectomy +10.4; sigmoid +10.2; rectal +6.6; right hemicolectomy +1.8; left hemicolectomy −2.2). For steps, it beats centre-specific models in 4 out of 5 (up to +23.4 at Centre 4) but only 2 out of 5 procedure-specific ones: right hemicolectomy loses 10.2 points and rectal resection 5.7 points against a dedicated model.

The generalization test. This is where the paper becomes interesting. Under LOCO, mean phase F1 on the unseen centre is 48.42%, against 61.59% on the centres that stayed in training during the same folds, and 73.01% for the global model. The range runs from 37.49% (Centre 2) to 65.78% (Centre 4). Expressed as retention relative to the global model, that is 51.3% to 90.1%: Centre 4 keeps nine tenths of the performance without its own videos in training, Centre 2 keeps half. For steps the drop is sharper still: 21.94% on average on the withheld centre, against 40.61% on the seen ones.

The ablation. Training phase and step jointly rather than separately yields +2.8 points of phase F1 (70.2% → 73.0%) and +1.7 points of step F1 (38.1% → 39.8%).

Operational translation. Balanced accuracy of 73.43% for phases means that, averaged over the eight phases, slightly fewer than three frames in four receive the correct label. On a 141-minute operation, that is enough to propose a pre-segmentation for a human annotator to correct — which is precisely the intended use — but not enough to produce a usable phase duration without review. At 38.65% for steps, the fine-grained timeline is not usable as-is. The saving on the 60% of video duration that manual annotation costs is therefore not demonstrated: it is plausible at phase level, it is not at step level.

What is good

The LOCO experiment is published, and it contradicts the headline. Nothing required the authors to withhold one centre at a time and publish the collapse that follows. They do, they report all five folds, and they go as far as computing a retention rate per centre — 51.3% for Centre 2, 90.1% for Centre 4 — then draw the conclusion that works against them: "the global model's performance at these sites is heavily dependent on the presence of local training examples, a finding with direct implications for the adoption of AI-ColoWorkflow at new, data-poor institutions." That is exactly the sentence most papers of this format do not write.

The per-class table is complete, not summarised. All 8 phases and 33 steps are published with precision, recall and F1, standard deviations included. That is what makes it visible that the 39.82% step figure covers two classes at zero and one class at 100% precision for 3.88% recall. A paper publishing only the macro average would tell the same story while making it unverifiable. Three further choices are sound: case-level partition, macro F1 as the primary metric rather than an accuracy dominated by long phases, and a majority-class baseline with a statistical test attached.

The losses are reported as precisely as the wins. The global model loses to the procedure-specific model on right hemicolectomy (−10.2 points at step level) and rectal resection (−5.7), and to the centre-specific model at Centre 3 (−1.0 for phases) and HeiCo (−1.5 for steps). The authors draw a qualified recommendation — pooled training for phases, retained specialisation for steps in procedures with constrained ontologies — rather than a blanket "our model wins". The multi-task ablation is in the same register: quantified at +2.8 and +1.7 points, that is, modest, and presented as such.

What is less good

The step macro F1 is an average of noise. This is the misleading metric failure mode in a rarely discussed variant: aggregating 33 classes by unweighted mean when several of them are measured on a single test video. The standard deviations give it away — S8, S9, S31 and S32 show standard deviations of 0.00, meaning the class appears in only one video; S23 shows precision of "100.00 ± 32.69", a mean of 100% with a standard deviation of 32.69, which is not interpretable as stated. The authors acknowledge that some steps had fewer training frames than test frames, which makes, they write, "reliable model learning impossible". As a result, 39.82% is not a performance figure: it is the average of roughly twenty informative measurements and about ten that measure nothing.

Eleven test videos, and no confidence intervals on the comparisons that carry the conclusion. The Wilcoxon test is applied only against the majority-class baseline, that is, against an opponent scoring 6.71% — a win that teaches nothing. No test is reported between the global model and the specialised models, which is precisely where the paper's thesis lives. Several of the invoked gaps are tiny: +1.8 points on right hemicolectomy, −1.0 at Centre 3, −1.5 on HeiCo. With one or two test videos per centre, those differences are indistinguishable from sampling noise. Population bias is likewise uncharacterisable: no patient demographics, no surgeon count, no robotic-versus-laparoscopic breakdown, no collection years are reported — the authors themselves list robustness to camera, lighting and robotic platform shifts as future work.

Ground truth, reproducibility and interests. A single annotator, and that annotator co-developed the ontology being applied: the rule and its application are not independent. No inter-rater agreement is measured in this study — the authors point to the "moderate" agreement of the original ColoWorkflow paper, without a figure. On reproducibility, the retrievable text contains no code, weights or data availability statement, and the preprint specifies neither the DINOv3 variant used, nor the number of SAHC stages, nor the learning rate, nor the number of epochs; of the five sources, only HeiCo is public. Finally, two authors — Pietro Mascagni and Nicolas Padoy — declare being co-founders and shareholders of Scialytics, a company operating precisely in surgical video analysis; and the "Marginal Gains" work cited as immediate prior art is co-signed by three of this paper's authors. None of this is hidden, all of it is declared, but taken together it describes an ecosystem where the ontology's authors, the annotator, the evaluators and the commercial beneficiary substantially overlap.

What it changes

For the research community. The reusable number in this paper is not 73.01%: it is 48.42%. It is the first published measurement of what a colorectal workflow recognition model is worth on a centre it has never seen, under a consensus ontology and across five procedure types. Any subsequent method — federated learning, domain adaptation, scaling up — will have to compare against that number, and against its 37.49-to-65.78 spread. The second lesson is the dissociation between the two granularities: pooling data from all centres clearly helps at phase level, but not systematically at step level, where a model dedicated to a procedure with a constrained ontology retains the advantage. That argues for hybrid strategies rather than one more single model. The third is stated explicitly by the authors and holds beyond surgery: the long-tail distribution of clinically important but infrequent actions "is not solvable by architectural choices alone" and requires substantially larger annotated corpora.

For clinicians. Nothing today. No regulatory status is mentioned, no prospective validation has taken place, and the link between detected patterns and clinical outcomes — operative time, complication rates, learning curves — remains, by the authors' own admission, "to be investigated". The useful reading is this: at a centre that contributed training videos, a phase-level pre-segmentation reviewed by a human is plausibly within reach; at a centre that contributed none, the model loses 10 to 49% of its performance depending on the site, and the question of how many local videos are needed to close that gap is not addressed. That is exactly the question to put to any vendor offering this tool.

For patients and the general public. The underlying stake is not an algorithm but an idea: measuring surgical quality by something other than its complications, by looking at what actually happened during the operation. Operative video makes that measurement possible, and automating it would make it generalisable. The honest state of the file, at this stage, is a pre-clinical feasibility proof: the coarse segmentation of the operation is beginning to work, the fine-grained segmentation does not yet, and nothing tells us what happens in a hospital that did not take part in the training.