Epilepsy or psychogenic seizures: what EpiScreen is worth, LLMs fine-tuned on clinical notes and tested at two hospitals, before prospective evidence

A University of Minnesota team fine-tuned four large language models (LLMs) with QLoRA to tell epilepsy from psychogenic non-epileptic seizures (PNES) using clinical notes, then tested them on two hospital record sets, MIMIC-IV and a private Minnesota cohort. AUCs (area under the ROC curve) are 0.88 to 0.89 in internal validation on MIMIC-IV and 0.97 to 0.99 on the Minnesota cohort, and stay high when trained at one site and tested at the other; accuracy, however, drops to 0.70–0.77 when trained on Minnesota and tested on MIMIC-IV, about what a model answering "epilepsy" every time would score. The signal looks real, but the reference label is a hospital discharge diagnosis rather than video-EEG, and there is no prospective validation.

The context

Epilepsy and psychogenic non-epileptic seizures (PNES, seizures of psychological origin with no abnormal electrical discharge in the brain) look alike to the naked eye: convulsive movements, loss of contact, falls. Yet management differs radically: the former call for antiseizure drugs, the latter for psychiatric and psychotherapeutic care. Getting it wrong one way means unnecessary treatment and side effects; the other way, untreated seizures.

The reference test is prolonged video-EEG (simultaneous video and electroencephalogram recording during a seizure), which is costly and mostly available in specialist centres. Hence the idea of an upstream triage tool that would use what the medical record already holds: free-text clinical notes. Earlier approaches relied on phenotyping rules or on smaller language models such as BERT and ClinicalBERT. The question here: does an LLM fine-tuned on a few thousand notes do better, and above all does it do better outside the hospital where it learned?

The method

The task. Binary classification, epilepsy versus PNES, from the text of a clinical note. The model returns a probability of epilepsy.

The models. Four open-source LLMs of 32 to 70 billion parameters (Qwen-2.5-32B, DeepSeek-R1-32B, Llama-3.3-70B and Med42-70B, the latter specialised in medicine) are fine-tuned (training of the model is continued on the target task). The technique, QLoRA, freezes the original model, compresses it to 4 bits and trains only small corrective matrices (rank 16), which makes it feasible on four A100 GPUs. The model receives the note and an instruction and outputs a diagnosis; the probability is read from the two output logits (raw scores).

The data. MIMIC-IV, a public database from Beth Israel Deaconess Medical Center in Boston: 3,451 patients (2,651 epilepsy, 800 PNES). A private University of Minnesota cohort: 10,182 patients (8,011 epilepsy, 2,171 PNES), drawn from a source cohort of 32,660 patients, with no explanation of the selection. Labels come from diagnosis codes (ICD) of the stay, checked by manual review of discharge summaries: only cases whose primary discharge diagnosis matches the code are kept. Patients with both diagnoses are excluded. Notes are truncated to dates before the diagnosis, following a procedure described in a supplementary note we could not read.

The validation. A random 70 / 10 / 20% split within each set (internal validation), then crossed external validation: train on one site, test on the other. Uncertainty is given as 95% bootstrap confidence intervals (resampling the test set, 1,000 draws).

The comparators. A phenotype-based method (FSLS), BERT and ClinicalBERT (smaller language models trained on the task), and the four LLMs prompted without training (zero-shot). In a separate experiment, four clinicians (two neurology residents, two senior neurologists with over ten years of experience) review 300 cases drawn from the MIMIC-IV test set.

The results

ML metrics. Reminder: AUC measures the ability to rank a positive case above a negative one (0.5 = chance, 1 = perfect); accuracy is the share of correct answers. In internal validation, the EpiScreen models reach an AUC of 0.882 to 0.891 on MIMIC-IV and 0.974 to 0.992 on Minnesota. The best trained comparators score 0.769 (ClinicalBERT) on MIMIC-IV and 0.960 on Minnesota; the best untrained LLM scores 0.813 and 0.900.

In external validation, the abstract's figures: trained on Minnesota and tested on MIMIC-IV, EpiScreen reaches AUCs of 0.829 to 0.875 (accuracy 0.70–0.77); trained on MIMIC-IV and tested on Minnesota, 0.966 to 0.980 (accuracy 0.89–0.92). ClinicalBERT falls to 0.749 and 0.788, FSLS to 0.498 and 0.485 (chance).

Clinician–AI study (300 cases). Accuracy: residents alone 0.793, seniors alone 0.817, EpiScreen alone 0.810 (no significant difference among the three). With the predictions and the most influential sentences, residents rise to 0.880 and seniors to 0.893.

Clinical translation. Out of 1,000 MIMIC-IV-type notes, an internal accuracy of 0.83 means about 165 errors; in transfer from Minnesota to MIMIC-IV, at 0.70 to 0.77, that is 230 to 300 errors. In the MIMIC-IV test set (about 690 notes, about 530 of them epilepsy by our estimate), the paper counts 49 false negatives for EpiScreen (Qwen-2.5-32B): roughly 9 in 100 epilepsies classed as PNES, against 93 false negatives for ClinicalBERT. False positives per site are not detailed. These orders of magnitude are our calculations from the tables, not published values.

What is good

A real transfer test between two hospitals. Training at one site and testing at the other, with full tables and intervals, is more demanding than a random split. The gain over ClinicalBERT is clear: AUC of 0.829 to 0.875 against 0.749 for ClinicalBERT from Minnesota to MIMIC-IV, and 0.966 to 0.980 against 0.788 the other way.

Varied comparators and error analysis. Phenotyping, BERT, ClinicalBERT, untrained LLMs: the comparison shows that fine-tuning changes the game (average AUC gain of 13.2% on MIMIC-IV and 11.6% on Minnesota over simple prompting). The authors add analyses of training-set size (with 5% of MIMIC-IV data, AUC of 0.806 internally and 0.924 on Minnesota), class imbalance and model size.

A comparison with clinicians, with an uncomfortable number. The authors show that the model alone matches the neurologists on these 300 cases (0.810 against 0.793 and 0.817), and that the combination improves both groups. That is a more useful result than "AI beats doctors".

What is less good

A label that is not video-EEG, and a risk of label leakage. The ground truth is the primary discharge diagnosis, derived from codes and a review of summaries. It is the clinical judgement the tool claims to support, not the test that settles the question. And since the notes are clinical texts close to the diagnosis, the risk of data leakage is real: a sentence like "followed by psychiatry for conversion disorder" can give away the answer. The "before diagnosis" truncation is deferred to an appendix we did not read; without it, we cannot rule out that the model reads the conclusion rather than predicts it.

A misleading accuracy metric on imbalanced classes. About 77% of MIMIC-IV cases (2,651 of 3,451) and 79% of the Minnesota cohort are epilepsy: a model that always answered "epilepsy" would already score an accuracy of 0.77 and 0.79. The 0.70 to 0.77 of EpiScreen trained on Minnesota and tested on MIMIC-IV is therefore at or below that trivial model, while the AUC stays decent: the ranking is good, the decision threshold does not transfer. Sensitivity, specificity and predictive values per threshold are not reported. It is also spectrum bias: the population consists of patients already diagnosed as either epileptic or PNES, not those arriving with "an unexplained episode".

Narrow validation, fragile clinical evidence and no reproducibility. Two US sites only, one of them private, with an unexplained selection (32,660 source patients, 10,182 retained). The clinician–AI study rests on 300 MIMIC-IV cases, four clinicians pooled into two consensuses of two (so two "readers" per group), with internet access, and no measure of real patient decisions. The gains in the journal abstract ("up to 14.3%") do not match those of the preprint (10.9%), and the metric is not specified. The saving of "8 to 12 weeks" is asserted, not measured. The explanation categories in the tables do not match those in the methods. No code or weights are mentioned, and the Minnesota cohort is private: nobody can reproduce the result.

What it changes

For the research community. The paper provides a useful baseline on a rarely addressed task, and a cross-site testing frame worth reusing. Natural next steps: labels from video-EEG, threshold calibration at each site, publication of sensitivities and specificities, tests in other languages and documentation styles, and release of the code.

For clinicians. Nothing changes today. This research tool has no regulatory clearance mentioned, and video-EEG remains the reference test. In time, an aide-mémoire that rereads the record and flags "profile suggestive of PNES" could help route patients earlier to a specialist centre, provided its false-alarm rate has been measured in the department's actual population.

For patients and the public. Diagnostic delay is a real problem for people given antiseizure drugs that do not work. But a statistical tool saying "probably psychogenic" carries a risk of stigma if the result is misused; its value will depend on how the physician integrates it, not on the score itself.

Going further

Paper published in npj Digital Medicine: doi.org/10.1038/s41746-026-03346-2 (received 23 March 2026, accepted 20 September 2026, published online 8 October 2026, early-access version). A preprint is available on arXiv: arxiv.org/abs/2603.28698. Funding: NIH (NCCIH R01AT009457, NIA R01AG078154, NCI R01CA287413); the corresponding author, Rui Zhang, is an associate editor of the journal and states he took no part in the review of this manuscript. MIMIC-IV data are available on PhysioNet under a data use agreement; the Minnesota cohort cannot be shared.

Editorial transparency: this article was produced with the help of artificial intelligence tools from the arXiv preprint, the early-access published article page and their metadata; the journal version may differ. The English, Spanish and Chinese versions are machine translations.