EEG seizure prediction: five pipelines, one near-chance result once evaluation truly separates patients (CHB-MIT)

A team at Auckland University of Technology kept five published EEG seizure-prediction pipelines fixed and changed only one thing: how they were evaluated on the pediatric CHB-MIT dataset. Once test patients are truly absent from training, all five pipelines fall to an AUC around 0.5, meaning chance, whatever their architecture. The message is not that AI will never work here, but that the high performances in the literature depend first on the evaluation protocol; read it as an exploratory, non-peer-reviewed preprint.

The context

About a third of people with epilepsy keep having seizures despite treatment. For them, a system that could warn a few minutes ahead would change daily life: stop driving, lie down, trigger stimulation or a rescue medication. Hence a large literature on seizure "prediction" from the electroencephalogram (EEG, a recording of the brain's electrical activity through scalp electrodes), with reported performances often above 90%.

The problem is old and methodological. An EEG is a highly autocorrelated time series: two neighbouring 20-second windows look very much alike. If a recording is cut into windows and these are randomly split between training and test, the model "recognises" at test time windows nearly identical to those it saw in training. This is data leakage: test information seeps into learning. This study measures, on existing pipelines, how much of the published performance disappears once these leaks are closed.

The method

The task is binary classification of 20-second EEG windows: "preictal" (5 to 30 minutes before seizure onset) versus "interictal" (far from any seizure). Seizures themselves, the last 5 minutes before onset and the 4 hours after seizure end are excluded. It is therefore not a continuous alarm system but a retrospective window-by-window classification test.

The main data come from CHB-MIT, a public PhysioNet dataset of recordings from children followed in Boston: 182 annotated seizures in 22 participants, sampled at 256 Hz, filtered 0.5 to 40 Hz, on 18 bipolar channels. Two identifiers belonging to the same person are grouped, patient chb24 is dropped for lack of interictal windows, and the final set has 17,424 windows; preictal windows come from only 69 of the 182 seizures. A second dataset, Siena Scalp EEG (13 participants, 5,657 windows), serves as an external control.

Five pipelines are compared, with no architecture search: PSD+LDA (90 band-power features followed by linear discriminant analysis, a classical linear classifier), EEGNet (7,002 parameters), a temporal convolutional network TCN (245,122), a 1D-CNN (520,322) and an EEG-Conformer, mixing convolutions and attention (852,746).

The core of the work is the evaluation design. In the "cold-start" scenario, 20 random partitions assign 14 subject groups to training, 4 to validation and 4 to test: no test patient was seen. Three controls are added: (1) in 11 patients, random window allocation versus chronological seizure blocks; (2) an audit of the earlier index-based split on 10 cases; (3) a threshold-transfer test, where the decision threshold is set on validation to flag only 1/60 of interictal windows, then applied unchanged to test subjects.

The results

ML metrics first. The AUC (area under the ROC curve) is 0.5 for a classifier no better than a coin toss and 1 for a perfect one. In cold-start, the mean pooled AUC over 20 partitions is 0.527 ± 0.070 for PSD+LDA, 0.534 for EEGNet, 0.553 for TCN, 0.541 for the 1D-CNN and 0.527 for EEG-Conformer. Weighted so each subject counts equally, it is between 0.502 and 0.523, with standard deviations of 0.09 to 0.13: the uncertainty is larger than the differences between architectures.

In the within-patient comparison with PSD+LDA, AUC goes from 0.815 ± 0.089 with random window allocation to 0.572 ± 0.171 with chronological blocks, a mean drop of 0.243 (0.024 to 0.601 depending on the patient). The index-split audit shows raw samples crossing the train-test boundary in all 10 eligible cases (16 overlapping window pairs in total), with differences of 0.139 to 0.373. On Siena, PSD+LDA applied without retraining reaches 0.480 ± 0.022.

Threshold transfer fails too: with a target of 1/60 (about 1.67%) false detections, observed rates range from 1.81% to 3.13%, and every pipeline exceeds the target in 6 to 10 of 20 partitions. Sensitivity at that threshold is only 2.33 to 5.56%.

Clinical translation. With the Youden threshold from validation (the one balancing sensitivity and specificity), EEGNet detects 40.7% of preictal windows and correctly rejects 64.9% of interictal ones. Per 1,000 preictal windows, about 593 are missed; per 1,000 interictal windows, about 351 are false alerts. If the 20-second windows follow each other without overlap, i.e. 180 per hour, a 1.67% false-positive rate means about three false alerts per hour, and 3.13% almost six, with under 6% of preictal windows detected. This calculation is ours, derived from the paper's figures and assuming contiguous windows; no wearable device would be tolerated with this profile.

What is good

First, the protocol isolates a single variable. Architectures, features and hyperparameters are fixed; only the evaluation scheme changes. That is what allows the claim that evaluation design weighs more than model choice, rather than comparing papers that differ on ten points at once.

Second, the audit is concrete. The authors do not merely say leakage is possible: they count 16 overlapping window pairs across the train-test boundary in the 10 verified cases and measure the induced AUC gap (0.139 to 0.373). They also handle the duplicated-patient trap (chb01 and chb21) by grouping both identifiers.

Third, transparency: code on GitHub audited at a specific commit (845e973), partition manifests and source tables supplied, public datasets on PhysioNet, university-only funding and no declared conflict. A control on a second dataset (Siena) is added, even if limited.

What is less good

First, the dataset is small and pediatric: 22 subject groups, and only 69 of 182 seizures provide usable preictal windows. No analysis by sex or age is done. This is the population-bias failure mode: we do not know whether the result holds for adults, other epilepsy types or other acquisition systems. The 20 repeated partitions share the same subjects, which limits inference to a wider population.

Second, the metric and task are not those of real use. The paper evaluates isolated windows, not continuous alarms with warning time, refractory period and false alarms per day, which are the quantities a patient would experience (a misleading metric by construction). It therefore cannot say whether a finer strategy, for instance patient-specific, would do better.

Third, the chronological and threshold analyses were added after the initial analyses and are called exploratory by the authors; the filtering is non-causal (it uses future samples, impossible in real time); the Siena control covers only one model, PSD+LDA, without the deep models. The five pipelines are reimplementations without architecture search: a negative result on unoptimised versions does not prove that no optimised version could succeed. Finally, this is a non-peer-reviewed preprint, with no prospective validation and no regulatory status.

What it changes

For the research community, the paper argues for evaluation standards: separate patients between training and test, group identifiers of the same person, respect the chronological order of seizures, set the threshold on separate data and report false alarms per hour. It also invites rereading published figures by first checking the partition scheme. It does not by itself show that every earlier paper is distorted, only that the risk is measurable and large.

For clinicians, nothing changes in practice today, and that is the useful information: no scalp-EEG seizure-prediction system advertised with spectacular performance should be taken seriously without validation on never-seen patients and in continuous time. For patients and the public, this does not mean prediction is impossible, but that the promise of a reliable warning remains, for now, undemonstrated with these methods.

Going further

The preprint: Song Z., Pedersen M., Henare D., "Same pipelines, different answers: how evaluation design shapes EEG seizure prediction", medRxiv, 29 October 2026, doi:10.64898/2026.10.28.26366763. Code: github.com/TsikonSong/EEG-Seizure-Prediction. Data: CHB-MIT and Siena Scalp EEG on PhysioNet.

This Decryption was produced with AI assistance from the preprint text and submitted to editorial review; derived clinical calculations are flagged as such.