médical IA

Skin lesions: 96% accuracy and 0.99 AUC — on the three quarters of images the model agreed to decide

Published on August 18, 2026 · 11 min read

Two authors, Rofiqul Islam and Lilatul Ferdouse, combine three neural networks with an uncertainty estimation mechanism to classify seven skin lesion types on the HAM10000 dataset, and report 96% accuracy with an AUC of 0.99. Those figures are computed only on the images the system deemed reliable enough: the results table covers 1,508 images, whereas a 20% test split of 10,015 images holds 2,003 — nearly a quarter of cases discarded, a rate the paper reports nowhere. The methodological instinct is right; the execution documents its own blind spot, because abstention is not uniform across classes, and it is highest on melanoma.

The context

Automatic classification of skin lesions from dermoscopy images — close-up photographs of a mole taken through polarised optics — is one of the oldest testbeds for AI in medicine. Since Esteva and colleagues published in Nature in 2017, the field has had a dominant benchmark: HAM10000 (Human Against Machine with 10,000 training images), released by Tschandl, Rosendahl and Kittler in Scientific Data in 2018. It gathers 10,015 annotated dermoscopic images across seven diagnostic categories, collected at the Medical University of Vienna and at a screening practice in Queensland, Australia.

Improving overall accuracy on that benchmark is no longer the field's problem: it has plateaued for years inside a narrow band where papers no longer meaningfully separate. The question that matters has become per-case reliability. A clinically useful model is not the one that is right most often, it is the one that can say "I don't know" and refer the case to a dermatologist rather than guessing.

That idea has a precise name in machine learning: selective prediction (classification with a reject option), formalised as early as 1970 by C. K. Chow and brought back to the centre of the field by Geifman and El-Yaniv in 2017. It comes with one non-negotiable reporting rule: never publish selective performance without the associated coverage — the fraction of cases on which the system agrees to commit. 96% accuracy at 100% coverage and 96% accuracy at 50% coverage describe unrelated objects. That is precisely where this preprint fails.

The method

The data. HAM10000, 10,015 RGB images, seven classes: melanocytic nevus (nv), melanoma (mel), benign keratosis-like lesions (bkl), basal cell carcinoma (bcc), actinic keratosis and intraepithelial carcinoma (akiec), vascular lesion (vasc), dermatofibroma (df). The paper describes an 80% / 20% split by "stratified sampling", meaning class proportions are preserved. No separate validation set is mentioned. Images are resized to 224 × 224 with standard ImageNet normalisation, and augmentations are given with exact magnitudes (horizontal and vertical flips, rotation up to 25°, brightness, contrast and saturation jitter of 0.2, affine shear of 10°). Class imbalance — HAM10000 is dominated by benign nevi — is handled by weighted sampling during training and by a focal loss, a loss function that puts more weight on hard examples and rare classes; its two parameters, α and γ, are never given.

The models. An ensemble of three ImageNet-pretrained architectures fine-tuned on the task: MaxViT-Tiny, a hybrid vision transformer alternating block-local attention and sparse global attention; ConvNeXt-Tiny and EfficientNetV2-B0, two modern convolutional networks. The choice is defensible: mixing different inductive biases (local convolution versus attention) is the most reliable way to reduce an ensemble's error. The combination is a plain arithmetic average of probabilities across the three models, with no learned weighting. Training: AdamW optimiser, learning rate 10⁻⁴, weight decay 10⁻⁴, cosine annealing schedule, 15 epochs, batch size 16, automatic mixed precision, gradient clipping at 1.0. GPU model, parameter counts and training time are not stated.

The uncertainty. This is the paper's declared core. The mechanism is MC Dropout (Gal and Ghahramani, 2016): dropout layers — which randomly switch off a fraction of neurons — are kept active at inference, and prediction is repeated several times, the spread of outputs serving as an uncertainty proxy. Here, T = 20 stochastic forward passes per model, sixty in total. The dropout rate is never specified, nor the location of the layers involved. The uncertainty measure is the predictive entropy of the distribution averaged over the sixty passes — a measure of how spread out the probability mass is across the seven classes, maximal when the model splits it evenly. A prediction is declared reliable if entropy is below 1.0 and the maximum probability reaches 0.7. Neither threshold is justified, swept, or tuned on a validation set.

Worth noting: because entropy is computed on the averaged distribution, the spread across the sixty passes is never used as such. MC Dropout therefore only smooths the probabilities here; it does not separate irreducible data uncertainty from model uncertainty — the very distinction the method is supposed to enable.

The interpretability. Grad-CAM++ produces heatmaps highlighting the image regions that weigh most on the decision. They are computed on the final layer of MaxViT-Tiny alone.

The results

The headline numbers. 96% accuracy, AUC of 0.99, macro-average precision, recall and F1 of 0.94, 0.95 and 0.95, and 0.96 weighted average. Per class: nv 0.99 / 0.97 / 0.98 on 1,044 images; bkl 0.96 / 0.94 / 0.95 on 141; mel 0.81 / 0.92 / 0.86 on 150; bcc 0.95 / 0.98 / 0.96 on 84; akiec 0.90 / 0.94 / 0.92 on 47; vasc 1.00 / 0.96 / 0.98 on 26; df 1.00 / 0.94 / 0.97 on 16.

The column that changes everything. The table states, explicitly, that it covers 1,508 images. A 20% test split of 10,015 images contains 2,003. The gap, about 495 images, corresponds to the cases discarded by the uncertainty filter: actual coverage is therefore around 75%. That calculation is ours. The paper never gives this figure, reports no full-coverage performance, and plots no risk-coverage curve. The word "coverage" appears only in two qualitative sentences of the discussion.

And abstention is not uniform. Comparing retained counts against those expected from a 20% stratified split, the rejection rate varies sharply by class: roughly 7% for vascular lesions, but around one third for melanoma (150 retained against about 223 expected) and for benign keratoses (141 against about 220). In other words, the system abstains most on the dangerous class and on its main source of confusion. Here too, the paper reports no per-class rejection rate.

Clinical translation. Out of 1,000 images fed to the system, roughly 250 would be referred to a dermatologist and 750 decided automatically. Among retained melanomas, a recall of 0.92 means about 8 melanomas in 100 would be labelled benign with no alert; the confusion matrix gives the raw shape — 10 melanomas classified as nevi, and 30 nevi classified as melanoma among confident cases only. But a third of the test melanomas were never decided at all: the effective sensitivity of the full pipeline depends entirely on what the dermatologist does downstream with the referred cases, which the paper neither models nor quantifies. That is the difference between evaluating a classifier and evaluating a triage system.

One internal inconsistency is worth flagging, because it errs on the side of apparent caution: the text states that "the framework still achieves a recall of 0.86" for melanoma, while the table gives 0.92 for recall and 0.86 for F1. The authors quote their own F1 believing they are quoting recall.

What is good

The instinct is right. Building a classifier that refuses to decide ambiguous cases and refers them to the clinician is exactly the direction dermatological AI should take. The setup is coherent end to end: a heterogeneous transformer-plus-convolution ensemble, MC Dropout at inference, a two-part rejection criterion (entropy and maximum confidence) rather than a bare softmax threshold, and an explicit reading of abstention as "refer to the dermatologist".

The training protocol is precisely described. AdamW at 10⁻⁴, weight decay 10⁻⁴, cosine annealing, 15 epochs, batch size 16, mixed precision, clipping at 1.0, and augmentations reported with exact magnitudes. For a five-page paper, that is more complete than the HAM10000 literature average, where results with no hyperparameters at all are common.

Results are given per class with sample counts. That is what makes the paper criticisable — and it is a merit. Many HAM10000 papers stop at global accuracy, from which nothing can be reconstructed. Here, the table's "Samples" column read against the stated test-set size is enough to recover the abstention rate the text omits. The authors supply the very data needed to measure what they do not measure.

What is less good

1. Misleading metric: selective performance without coverage is not performance. This is the central problem. Every reported metric is post-filtering, none is given at full coverage, no risk-coverage curve is produced, and the thresholds (entropy 1.0, confidence 0.7) are asserted without justification or sensitivity analysis. Direct consequence: these 96% are comparable to no published HAM10000 figure, all of which are at 100% coverage. The preprint never flags that non-comparability. Add that rejection is highest on melanoma: a triage aid that preferentially abstains on the class you are trying not to miss shifts the workload exactly where it is already heaviest.

2. Risk of lesion-level data leakage. HAM10000 contains several images of the same lesion — the dataset ships with a lesion identifier precisely so these duplicates can be grouped. The paper writes only "stratified sampling", with not a word about grouping by lesion or by patient. If the split is done at image level, near-identical views of the same lesion land on both sides of the train/test boundary, and the measured accuracy incorporates a share of memorisation. This is the best-documented failure mode of this dataset. It is neither addressed, nor mentioned, nor ruled out.

3. No comparator, no ablation, a single run. There is no comparison table against the literature. There is no result for each of the three models taken alone, so nothing demonstrates that the ensemble contributes anything. There is no ablation of MC Dropout, the focal loss or the weighted sampling, and no comparison against rejection by a plain softmax threshold — the obvious baseline. A single random seed, no confidence intervals, no statistical test, while the precisions of 1.00 reported for df and vasc rest on 16 and 26 images respectively. No external validation on a second cohort, which the authors acknowledge in two lines of conclusion. No calibration analysis — neither expected calibration error nor reliability diagram — even though the paper sells the quality of its uncertainty estimates. And on transparency: no code, no weights, no legible institutional affiliation, no funding statement, no conflict-of-interest declaration.

Two secondary reservations, no less structural. Population bias: HAM10000 comes from two sources, Vienna and Queensland, that is, overwhelmingly fair-skinned populations; nothing indicates how the model behaves on darker phototypes, where melanoma is rarer but diagnosed later. Interpretability as decoration: the Grad-CAM++ maps are evaluated by no localisation metric and no clinician reading study, and the four examples shown are all correctly classified cases — no failure is illustrated. They are moreover computed on MaxViT-Tiny alone, and therefore do not explain the predictor actually evaluated, which is the average of the three models.

What it changes

For the research community. This preprint is a useful case study, less for what it proposes than for what it illustrates. Selective prediction is fast becoming the default frame for diagnostic AI, and it arrives with a reporting requirement the imaging community has not yet absorbed: coverage is a metric, not an implementation detail. The minimum standard should be a risk-coverage curve, or at least a triplet (accuracy, coverage, per-class rejection rate) — cheap to produce, and enough to make numbers comparable again. A reviewer should refuse any selective performance table lacking that information.

For clinicians. Nothing, today. A model trained and tested on a single public dataset, with no external validation, no code, no weights, no calibration and no reported coverage is not an evaluable object in practice. The question to put to any vendor presenting such a device is simple and sufficient for triage: on what fraction of cases does your system agree to commit, and what is the performance on the remainder? A triage device is not judged on the cases it finds easy.

For patients and the public. An accuracy figure announced without its perimeter means nothing. "96% accuracy" can describe a model right 96 times out of 100 on all cases, or a model right 96 times out of 100 on the three quarters of cases it chose — leaving the rest to a human. In the second case, the number describes the selection as much as the performance. It is a question worth asking systematically, including of consumer mole-photography apps, which routinely build their marketing on metrics of this kind.