When a skin-cancer detector changes country: a cascade classifier of dermoscopic images falls from 0.96 to 0.80 AUC between the ISIC archive and a Russian clinic (arXiv, 2026)

Posted on 11 June 2026 on arXiv, this preprint by Elena Kozachok and colleagues compares four neural-network architectures and three ways of organizing the decision to classify dermoscopy images (skin examined at high magnification) of cutaneous lesions, then tests their ability to generalize from a large open international archive (ISIC) to two independent Russian clinical datasets. The central result is a warning: an almost perfect benign/malignant discrimination internally (area under the ROC curve of 0.952 to 0.966) collapses on new ground — on the Sechenov University data, the AUC falls to 0.797–0.893, sensitivity to 0.53–0.67, and calibration degrades sharply, the model erring in the dangerous direction by underestimating cancer risk. The methodological contribution, a cascade classification with a tunable triage threshold, allows explicit control of sensitivity, which a single-stage classifier cannot do. The paper's merit is to measure and publish this generalization gap that most work prefers to ignore; its limits lie in very small, imbalanced validation cohorts, with no comparison to a dermatologist and no prospective trial.

The context

Skin cancer is one of the oldest and most active arenas of medical AI. Since 2017, many models have reached, on "clean" test sets, performance comparable to dermatologists in distinguishing a benign from a malignant lesion on dermoscopy images — the technique of examining skin through a lit magnifier to reveal structures invisible to the naked eye. Most of these models are trained and evaluated on the same data source: the ISIC archive (International Skin Imaging Collaboration), a public repository aggregating hundreds of thousands of dermoscopic images, mostly from fair-skinned populations and Western centers.

The known but rarely quantified problem is domain shift: does a model that shines on ISIC images still behave correctly on images acquired elsewhere, with other devices, other protocols, other patients? In skin oncology, the stake is not abstract. Missing a melanoma (the deadliest skin cancer) or a carcinoma has direct consequences. This paper tackles the question head-on by confronting ISIC-trained models with the reality of Russian clinical practice, and comparing along the way several ways of architecting the diagnostic decision.

The method

The authors compare four modern vision architectures, all pretrained on ImageNet (a large general-purpose image dataset) then re-trained on dermoscopic data. Two are vision transformers — networks that cut the image into small patches and learn their relationships through a so-called attention mechanism: ViT-B/16 and Swin-S. Two are convolutional networks, the classic family specialized in detecting local patterns: ConvNeXt-S and EfficientNetV2-S. All use the same data-augmentation protocol (rotations, crops and variations meant to artificially diversify training), so the comparison is fair.

Above all, the authors pit three classification schemes against each other. The first is binary: benign versus malignant. The second is a single-stage four-class ranking: benign, melanoma (MEL), cutaneous squamous cell carcinoma (SCC) and basal cell carcinoma (BCC), the network directly picking the most probable class (the "argmax" rule). The third is a two-stage cascade: a first model performs a benign/malignant triage, then a second differentiates, among lesions deemed malignant, between MEL, SCC and BCC. The claimed advantage of this cascade is that the triage threshold can be tuned: lowering it raises sensitivity (fewer cancers missed) at the cost of more false alarms — a slider that the fixed argmax rule does not allow.

Training relies on aggregated open data from the ISIC archive. Evaluation occurs at three levels: an internal held-out sample (from the same distribution as ISIC), then two independent external clinical datasets — a subsample of the commercial mobile system Melanoscope AI (472 images: 450 benign, 22 malignant, of which 14 melanomas, 1 SCC and 7 BCC) and a dataset from Sechenov University (77 images: 20 benign, 57 malignant, of which 28 melanomas, 6 SCC and 23 BCC). Performance is measured by the ROC-AUC (ability to separate a malignant from a benign case), sensitivity (share of cancers correctly detected) and ECE (Expected Calibration Error: the gap between the confidence the model displays and its actual accuracy — a low ECE means an 80% score really corresponds to 80% correct answers).

The results

Internally, all is well: the binary benign/malignant stage reaches a ROC-AUC of 0.952 to 0.966 depending on architecture, the expected level for modern models on data from the same source as training. It is on external ground that the picture flips. On the Sechenov University set, the binary AUC drops to 0.797–0.893, and above all the sensitivity falls to 0.53–0.67: depending on architecture, between a third and nearly half of truly malignant lesions are no longer flagged as such. In the same move, the ECE jumps from 0.02 to 0.27–0.39: the model is no longer calibrated at all, and the drift goes in the most dangerous direction, underestimating malignancy — assigning reassuring confidence scores to lesions that are in fact cancers. The cascade, for its part, delivers on its methodological promise: by tuning the triage threshold, one regains explicit control of sensitivity, impossible with single-stage argmax, and the "triage then differentiation" chain better reproduces the logic of clinical differential diagnosis.

Clinical translation: a sensitivity of 0.53–0.67 on Sechenov's 57 malignant lesions means roughly 19 to 27 cancers out of 57 would be wrongly classed as benign. For a melanoma, such a false negative can cost months of diagnostic delay on a tumor whose prognosis hinges closely on early management. The degraded calibration worsens the picture: not only does the model err more often outside its training domain, it does so with confidence, which is precisely the error profile most likely to mislead a user who trusts the score. Conversely, the cascade's sensitivity slider shows the way out: for a screening tool, one would set the threshold toward high sensitivity, even at the cost of multiplying false alarms and therefore unnecessary exams and biopsies — a trade-off that must be made explicitly, not endured.

What is good

A real external validation, in another country, and a generalization gap published without spin. This is where the paper's value lies: instead of reporting a flattering AUC on an internal ISIC-derived test, the authors confront their models with two independent Russian clinical sets and publish the drop — AUC from 0.96 to 0.80, sensitivity as low as 0.53. This is exactly the test that the dermoscopy literature, massively trained and tested on ISIC, tends to dodge.

A cascade that makes sensitivity tunable and matches clinical logic. Separating the benign/malignant triage from the differentiation between cancer types, with an adjustable threshold, is not just technical elegance: it gives the designer back a sensitivity/specificity slider that the single-stage argmax decision locks away. For screening use, where missing a cancer is far more serious than a false alarm, it is the right lever in the right place.

Calibration measured, not just accuracy. By reporting ECE and not only AUC or accuracy, the authors document a failure mode that almost all papers pass over in silence: a model can stay "roughly correct" on average while becoming dangerously overconfident out of distribution. Showing that ECE explodes from 0.02 to nearly 0.4 is clinically more telling than many decimals of AUC.

What is less good

A domain shift and a likely population bias driving the collapse (failure modes: population bias and domain shift). The models are trained on ISIC, an archive dominated by fair-skinned populations and Western centers, then tested on Russian clinical practice with other devices and acquisition protocols. The drop in AUC and sensitivity is the classic signature of this shift. The paper observes it but does not correct it (no local re-training, no evaluated domain adaptation), so the message remains a diagnosis of the problem more than a solution.

Tiny, highly imbalanced external cohorts (failure modes: the misleading metric and selection bias). The Sechenov set has only 77 images (with just 6 SCC) and the Melanoscope set 472 images with barely 22 malignant lesions (including a single SCC). On such small numbers, every case weighs heavily: a sensitivity of 0.53 to 0.67 has a very wide confidence interval, and the SCC class is too sparsely represented to conclude anything. Moreover, Sechenov's benign/malignant balance (a majority of malignant) is inverted relative to a real screening context where benign lesions vastly dominate — which makes the raw figures hard to transpose as is.

No dermatologist comparator, a commercially sourced validation set, and mere preprint status (failure modes: biased comparator and potential conflicts of interest). The only comparison is internal to the paper (architectures and schemes against one another); there is no benchmarking against dermatologists' judgment, which remains the relevant clinical yardstick. One of the two external sets comes from a commercial mobile system (Melanoscope AI), raising a question about data provenance and representativeness to keep in mind. Finally, this is a preprint not yet peer-reviewed, retrospective, with no prospective validation or regulatory status — usual caveats, but decisive here given the stake (a missed cancer).

What it changes

For the research community, the paper provides a sober, useful benchmark on the size of the generalization gap in dermoscopy, and two reusable ideas: cascade classification with tunable sensitivity, and the systematic evaluation of calibration (ECE) under domain shift, not just AUC. By example, it argues for making external validation and calibration default requirements rather than bonuses.

For clinicians, nothing changes in practice today. A model that lets through a third to a half of cancers outside its training domain, and does so with miscalibrated confidence, cannot serve as a standalone tool. The practical message is rather negative and valuable: a classifier trained on ISIC should not be deployed on a new population without prior local validation and recalibration.

For patients and the public, it is a reminder of caution toward "skin-cancer screening" photo apps. A tool that shows excellent figures on international data can clearly underperform on another population, and a reassuring score is not a guarantee. The tunable-sensitivity track does, however, point toward safer screening tools — calibrated to prefer catching cancers, at the cost of extra exams — provided that trade-off is explicit and validated locally.

Going further

The full preprint is available on arXiv (DOI 10.48550/arXiv.2606.13135), posted 11 June 2026 by Elena Kozachok and colleagues, with clinical validation anchored notably at Sechenov University (Moscow). The training data come from the ISIC archive, a public repository of dermoscopic images. For the clinical framework of melanoma screening and the critical reading of AI tools in dermatology, the guidelines of dermatology societies and reviews devoted to external model validation offer reliable entry points.