Segmenting the inner retina in retinitis pigmentosa: two AI models, including the SAM foundation model, trained on just 228 OCT slices

A team from the University of Göttingen presents two deep-learning models that automatically segment the inner retinal layers on OCT images of patients with retinitis pigmentosa, a blinding genetic disease. Its contribution is not a record score but a deliberate frugality: by combining pretraining on 1,700 public slices, an interactive annotation loop and anatomy-aware post-processing, the models reach excellent performance with only 228 annotated patient slices. The result is strong on the inner layers — those that decide eligibility for future optogenetic therapies — but it fails precisely on the layer used to grade disease severity, and it was validated on only about thirty patients from a single hospital.

The context

Retinitis pigmentosa (RP) is a group of genetic retinal diseases, the leading cause of blindness in people under 60, with a prevalence of about 1 in 4,000 and more than 3,000 identified mutations. It first destroys the rods, then the cones; the outer retinal layers degenerate first, followed by the inner layers. OCT (optical coherence tomography) is the reference examination for tracking this degeneration: it produces cross-sectional images of the retina in which the thickness of each layer can be measured.

Why does the inner retina, specifically, become the issue? Because a new family of treatments, optogenetics, targets it directly. Unlike gene therapy, which only works for one precise mutation, optogenetics is mutation-independent: it makes the neurons of the inner retina (ganglion cells and bipolar cells) light-sensitive to compensate for the destroyed photoreceptors. This approach requires an inner retina that is still largely intact — hence the need to finely measure the state of the inner layers, to select eligible patients and monitor treatment effect. Yet the automatic segmentation tools shipped by OCT manufacturers (such as Heidelberg Engineering) fail on degenerated retinas, for lack of the anatomical landmarks present in a healthy retina. And deep learning, so far, had focused on the outer layers in RP, never on the inner layers. It is this double gap — clinical and methodological — that this preprint sets out to fill.

The method

The authors train two complementary models. The first, OCT-SAM, derives from the Segment Anything Model (SAM), a foundation model (a large pretrained model, reusable for many tasks) designed by Meta to segment any object in an image from a single click or box. The authors start from MedicoSAM, an adaptation of SAM to medical imaging with a decoder able to segment automatically, without intervention. The second model, nnU-Net, is the established reference for semantic segmentation (each pixel is directly assigned the identity of its layer).

The data come from a retrospective cohort of about thirty RP patients (31) followed at the Göttingen ophthalmology department between 2019 and 2025, all imaged on the same device (Spectralis, Heidelberg). The ground truth — the reference tracings — was produced semi-automatically (an edge detector) then corrected by hand by a first grader, validated by a second, with a third arbitrating disagreements, across seven retinal layers. A clever point: rather than annotating everything by hand, the authors proceed in four cycles where OCT-SAM pre-segments the next slices, a human corrects, and the model is retrained — a loop that speeds up annotation. The final model is thus trained on 228 slices (called B-scans, a vertical cross-section of the OCT), after pretraining on 1,700 slices from two public datasets not specific to RP. A bespoke post-processing step then enforces anatomical consistency on the detected layers.

The authors integrate everything into open-source software (a napari plugin) that automatically measures layer thickness and central foveal thickness (CFT). Evaluation combines a 20-slice test set (overlap metrics: Dice and F1, which measure how well the predicted region covers the true one) and a validation of thickness measurements on 90 points compared with an expert's manual measurements, via Bland-Altman plots (a standard method for measuring agreement between two ways of measuring: it gives a mean bias and limits of agreement). Finally, a longitudinal demonstration follows three patients over at least three years.

The results

On the test set, nnU-Net dominates in automatic segmentation: precision, recall and F1 at 0.96, Dice at 0.88. OCT-SAM follows one notch below (precision 0.93, recall 0.80, F1 0.85, Dice 0.77) but brings the interactivity nnU-Net lacks. A striking result on frugality: performance jumps from just 5 to 10 specific images and plateaus around 100 — a large annotated dataset is therefore not necessary, which is decisive in a rare disease where data are scarce.

Clinical translation is more nuanced. Against an expert's manual measurements, OCT-SAM's automatic measurements are reliable for the inner layers that matter for optogenetics — relative differences of 4.4% to 11.2% for central foveal thickness, the nerve fibre layer (RNFL) and the ganglion cell layer (GCIPL). But they degrade markedly elsewhere: 21.8% difference for the inner nuclear layer, 22.5% for the outer nuclear layer, and above all 54.8% for the ellipsoid zone (EZ) and 41.1% for the pigment epithelium. The ellipsoid zone is, moreover, almost never detected: 6 times out of 90 by the model — but the human expert too spots it only 21 times out of 90, a sign that the target is objectively hard. In concrete terms, the tool measures well what optogenetics looks at (the inner retina), but fails on the layer used to establish the disease stage.

What is good

A useful and well-targeted first. This is, to the authors' knowledge, the first deep-learning work devoted to the inner retinal layers in RP, and the first application of SAM to retinal layer segmentation on OCT. It is not a gratuitous novelty: it answers a precise clinical need created by the arrival of optogenetic therapies, which require assessing the integrity of the inner retina.

Data frugality demonstrated, not merely asserted. Reaching excellent scores with 228 slices, and a plateau from around a hundred, thanks to pretraining on 1,700 public slices, the interactive annotation loop and anatomy-guided post-processing, is a concrete answer to the scarcity of imaging data in rare diseases. The curve showing the gain from just 5 to 10 images is the kind of result other teams can reuse directly.

Open science and careful metrology. The code, both models and their training routines are published in a public repository, which makes the work reproducible. Measurement validation relies on Bland-Altman plots compared against an expert rather than a single global score, and the funding (public, essentially the German Research Foundation) is declared with no commercial conflict of interest.

What is less good

A population bias and an absent external validation. Thirty-one patients, a single hospital, a single scanner model: nothing guarantees that performance holds on another device, another population, another country. The authors themselves acknowledge that not all genetic variants of RP, with their differing degeneration profiles, are represented in so small a sample. No external validation, on an independent centre, was carried out — the classic failure mode of the AI-in-health literature.

A fragile ground truth, precisely where it counts. The reference is a manual segmentation, yet even the human expert detects the ellipsoid zone in only 21 cases out of 90: the "gold standard" is itself uncertain on that layer. And that is exactly where the model fails (54.8% difference), whereas the ellipsoid zone is used to stage RP in the classification the study relies on. The model is therefore weak where the clinic would need it most — a mismatch between the measured task and the intended use, compounded by a small test set (20 slices, 90 measurement points).

Minimal clinical validation and partly misleading automation. The longitudinal demonstration covers only three patients, retrospectively, with no prospective protocol and no quantified comparison to the manufacturer's tool. Above all, for the difficult layers, segmentation still requires human clicks (the "interactive" mode): the "automatic" label fully holds only for the easy inner layers. Finally, this is a preprint not yet peer-reviewed, and the manuscript is released under a non-commercial, no-derivatives licence (CC-BY-NC-ND).

What it changes

For the research community, the main contribution is a frugal recipe — pretraining on public data, looped interactive annotation, anatomy-guided post-processing — directly transposable to rare diseases where annotated images are missing. Added to this are the first application of SAM to retinal layers and an open tool others can extend, notably towards the three-dimensional segmentation the authors announce.

For clinicians, nothing is deployable today: a single-centre cohort, no external validation, weakness on the outer layers needed for staging, a demonstration limited to three patients, and manual corrections still required. The direction remains promising against manufacturers' software, which fails on degenerated retinas. For patients and the general public, the study illustrates two reading rules. First, "an AI segments the retina" hides that it was validated on about thirty patients from a single hospital. Second, a tool can excel on part of the image while stumbling on the very layer used to measure disease severity: average performance says nothing about performance where it counts.

To go further

The preprint is available on medRxiv (10.64898/2026.06.16.26355668), and the code and models are published on GitHub. On foundation models in medical imaging, see our decryptage of GigaPath in pathology and our evaluation of foundation model representations for cancer; on automatic segmentation applied to another organ, the one on Fine-UNETR in prostate cancer.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. English, Spanish and Chinese translations produced with AI assistance and reviewed.