Clinical EEG: does a general-purpose vision-language model that "looks" at the trace rival EEG foundation models? Yes on some tasks, not on all

A team from Massachusetts General Hospital, Harvard and Binghamton University asks whether a general-purpose vision-language model, never pretrained on EEG, can read an electroencephalogram rendered as an image, as a neurologist would. On three public datasets, a fine-tuned Qwen3-VL-4B matches EEG foundation models on abnormality detection and epileptiform spike detection, but not on TUEV six-class classification in the official split, and validation remains retrospective. The result is useful for research, not yet for the clinic.

Context

EEG records the brain's electrical activity through electrodes on the scalp. It is used to find abnormalities, seizures or interictal discharges (IEDs, epileptiform spikes between seizures). Reading it is slow and needs an expert. EEG foundation models (LaBraM, CBraMod, BIOT, EEGPT and others) are networks pretrained on large amounts of unlabeled recordings, then adapted to a specific task: a foundation model is a general model reusable after a costly initial training.

The problem is that EEG corpora are much smaller than image or text corpora. The idea tested here sidesteps that gap: a vision-language model (VLM), which processes both images and text, has already seen billions of images. If EEG is drawn the way clinicians view it, as stacked traces, a VLM might exploit that without EEG pretraining.

Method

The signal is converted to microvolts, resampled (200 to 250 Hz depending on the dataset), cut into windows and arranged in a longitudinal bipolar montage, the standard clinical reading layout (18 derivations from 19 electrodes, 16 for TUEV). The default representation combines the traces (montage) with a spectrogram, an image of signal power by frequency over time. Each image comes with a fixed text prompt listing the possible labels. The main model is Qwen3-VL-4B (4 billion parameters); other sizes (2B, 8B, 32B), InternVL3-8B, Phi-3.5 Vision and Llama 3.2 Vision serve as comparisons.

Adaptation uses LoRA (low-rank adaptation, which trains only a small set of added weights) or "head-only" training (only the final classification layer is retrained). There are three main tasks: TUAB (normal versus abnormal EEG, 1,660 training subjects in the official split), TUEV (six-class EEG events, 232 training subjects) and vEpiSet (epileptiform spike detection, 67 training subjects). Three more datasets, SPaRCNet, SpikeNet2 and SeizeIT2, are used for few-shot tests, i.e. with only a handful of examples (128 or 384 windows). The main metric is balanced accuracy, the mean of each class's correct-classification rate: chance is 0.50 with two classes and about 0.17 with six. Comparators are specialized models (EEGNet, EEG-Conformer and others), EEG foundation models and published figures.

Results

ML metrics: on TUAB, Qwen3-VL-4B with LoRA reaches 0.805 balanced accuracy (AUROC 0.877), against 0.795 for BIOT, 0.828 for CBraMod and 0.831 for Reve; the authors call the task "saturated". On vEpiSet it scores 0.835 (AUROC 0.906), against 0.763 for the best cited EEG model, BIOT. On TUEV the picture is more mixed: 0.601 with the canonical split, below CBraMod (0.667), Reve (0.676) and CS-Brain (0.690); with the "EEG-Bench" split, the 8B and 32B VLMs reach 0.750 and 0.746, on par with Reve (0.739). Size matters: on canonical TUEV, balanced accuracy rises from 0.573 (2B) to 0.653 (32B). In zero-shot mode, with no examples, the VLM stays near chance on TUEV (0.169 base, 0.275 for the 32B).

In few-shot mode, with 64 examples in total on TUAB, the base VLM gets 0.648 against 0.657 for EEGPT. More telling: a VLM first adapted to a neighbouring task gains in transfer (SPaRCNet: 0.217 to 0.350; SeizeIT2: 0.607 to 0.707), whereas LaBraM and CBraMod lose ground under the same conditions. The choice of image matters too: on SpikeNet2, an elongated format (1800×256) lifts the score from 0.545 to 0.626.

Clinical translation, as an order of magnitude: a balanced accuracy of 0.805 on TUAB means that on average 19.5% of windows in each class are misclassified. Out of 1,000 normal and 1,000 abnormal windows, that is about 390 errors if they are spread evenly; the paper as I read it does not give the split between false positives and false negatives at the chosen threshold. On canonical TUEV, 0.601 across six classes leaves nearly four events in ten mislabelled on average per class. These figures describe isolated windows from a corpus, not a patient.

What is good

First, the breadth of the comparison: five VLM families and four Qwen sizes, about fifteen specialized models, five EEG foundation models, two splits for TUAB and TUEV, and standard deviations on the few-shot tests. Showing that the result depends on the split (TUEV: 0.601 versus 0.715 for the 4B) is a useful piece of honesty.

Second, precise ablations on the visual representation: montage alone, spectrogram, scalogram (wavelet transform), combinations, resolution. We learn what matters: for example, adding the scalogram to the montage raises TUEV kappa from 0.597 to 0.688 in head-only mode.

Third, transparency about the setting: NIH funding (UG3NS143749), no declared competing interests, datasets that are public or controlled-access and described one by one, and limits written into the text (visual tasks only, the compute cost of a model far larger than an EEG encoder, hyperparameter sensitivity of foundation models).

What is less good

First, a data leakage risk not ruled out in the text: the paper gives subject counts per partition but, as I read it, does not confirm that partitions are subject-disjoint. In EEG, windows from the same patient in both training and test inflate scores. The dataset table also contains an inconsistency (15 s in the table, 10 s in the text for TUAB "EEG-FM").

Second, population and scope bias: TUAB and TUEV come from the same US hospital database (Temple University), vEpiSet has only 9 test subjects, and no prospective external validation is reported. Spike detection on 9 subjects does not say how the model behaves in another department with another amplifier. There is also a variable comparator: competitor figures partly come from the literature, with training protocols that may differ, and on TUEV the foundation models keep the edge in the canonical split.

Third, a metric that hides the use case: balanced accuracy is averaged over classes and says nothing about sensitivity at a realistic alert threshold for rare seizures, or about false alarms per hour of recording. To my knowledge the paper also does not release its code or fine-tuned weights, and states no licence, which makes the result hard to reproduce.

What it changes

For the research community: a serious argument that general-purpose pretraining can be enough on "visual" tasks, with an advantage in transfer between neighbouring tasks. It remains to be seen whether the gap holds against newer EEG foundation models under identical protocols.

For clinicians: nothing changes in practice today. A model of 4 to 32 billion parameters is heavy for real-time use on the ward, and no prospective or regulatory step (CE mark, FDA) is mentioned. In the medium term, EEG reading aids that are easier to adapt to a new hospital are conceivable.

For patients and the public: AI could help triage long recordings and shorten the wait for an expert read, but this work concerns research corpora, not care decisions.

Going further

Paper: arXiv 2611.05538 (DOI 10.48550/arXiv.2611.05538), Zhou, Sergeeva, Bi, Song, Yin, Deng. Data: TUAB and TUEV (Temple University, controlled access), vEpiSet (open access), SPaRCNet and SpikeNet2 (BDSP, controlled access), SeizeIT2 (OpenNeuro ds005873, CC0). Base models: the Qwen3-VL family.

Editorial transparency note: this article was produced by Tatakoto's automated editorial pipeline from the preprint PDF; the appendices (notably cohort details) were not reviewed. The English, Spanish and Chinese versions are AI-generated. Spotted an error? Contact us.