Screening for sleep apnea from a plain face photo: what a VLM-based multimodal system does, and why its "accuracy" flatters the numbers

A Shanghai team proposes EviOSAHS, a system that gauges sleep-apnea risk from a face photo and a clinical profile by orchestrating — through prompting alone, with no training — a vision-language model and a large language model. On 642 patients with reference polysomnography, it reports 88.47% accuracy and 94.86% sensitivity. The catch: the cohort holds 584 positives to just 58 negatives, specificity is never published — it reconstructs to about 24% — and a classifier that answered "positive" for everyone would score higher accuracy (90.97%).

The context

Obstructive sleep apnea-hypopnea syndrome (OSAHS) is marked by repeated collapse of the upper airway during sleep: breathing stops (apnea) or shrinks (hypopnea) dozens of times per hour, fragmenting sleep and, over time, raising cardiovascular and metabolic risk. It is common and very widely underdiagnosed. The reference diagnosis, polysomnography (PSG), requires an instrumented overnight recording in a sleep lab: reliable but costly, capacity-limited, with long waiting lists. Hence the interest in screening tools that triage upstream which patients to send to PSG first.

Severity is measured by the AHI (apnea-hypopnea index), the number of respiratory events per hour of sleep, which defines categories (normal, mild, moderate, severe). Classic clinical questionnaires (STOP-Bang, Epworth, Berlin) are cheap but imprecise. Meanwhile, craniofacial morphology — a thick neck, a receding chin, crowded airways — correlates with risk, which fed the idea of exploiting a plain face photograph. The novelty here: instead of training a dedicated classifier, the authors orchestrate existing general-purpose models purely by writing instructions (the prompt), forcing a structured output meant to make the decision interpretable.

The method

EviOSAHS changes no model weights: it is a "zero-training" chaining (no learning on the target cohort) of two general-purpose parts. A vision-language model (VLM) — a model that "reads" an image and answers questions about it in text — and a large language model (LLM) that reasons over text. In practice, the authors use Qwen2.5-VL-7B-Instruct for vision and Qwen2.5-7B-Instruct for language, in deterministic decoding (zero temperature, for reproducible answers) on a single 80-GB GPU.

The core is "structured visual evidence decomposition." A frontal face photo is examined through seven fixed anatomical queries: neck, chin, mouth, face and neck fat, lower jaw, midface, nose. For each, the VLM sees only the image and acts as an observer, not a decision-maker: its answer is converted into a structured "evidence card" (anatomical target, observation, visibility, risk direction — support, against, uncertain —, evidence strength, confidence, summary). The seven cards are assembled, and — only at this final stage — the LLM receives the tabular clinical profile (age, sex, BMI, neck circumference, waist-hip ratio, hypertension, diabetes, heart disease, hyperlipidemia) and returns a binary verdict through a "balanced adjudication," explicitly weighing supporting, opposing and uncertain evidence. The temporal separation — vision first, clinical data later — aims to keep the model from betting everything on BMI or neck circumference. The target is a binary classification: positive label whenever the severity category exceeds "normal"; the AHI itself is removed from the prompts to avoid label leakage.

The data come from a single center: 642 subjects from a retrospective sleep-medicine cohort at the Eye & ENT Hospital of Fudan University (Shanghai), with reference PSG, registered in the Chinese Clinical Trial Registry (ChiCTR2300069223). The binary mapping yields 584 positives to 58 negatives; the original split is highly imbalanced (normal 58, mild 99, moderate 95, severe 390). The cohort is mostly male (546 men to 96 women), median age 40 (range 4 to 82), median BMI 26.5. A major methodological point: there is no training and no train/test split — all models are evaluated by direct inference on the entire cohort, with prompts "fixed before formal runs." The comparators are other ways of prompting (clinical-only, direct VLMs, early fusion, single model, ablated versions); no human expert and no validated questionnaire such as STOP-Bang serves as reference.

The results

The full system reports 88.47% accuracy, 94.86% sensitivity, an F1 of 93.74% and a false-negative rate of 5.14%, cutting, per the authors, the number of missed cases from more than 140 to 30. It beats "clinical-only" (accuracy 74.61%, sensitivity 75.34%), a directly used LLaVA-1.6 VLM (73.83%; 74.66%), early fusion (85.67%; 91.61%) and a single model (85.36%; 89.55%). Two VLMs queried "raw" collapse: InstructBLIP falls to 0% sensitivity and Qwen2.5-VL to 1.54% — they answer "negative" almost everywhere or produce unusable outputs. Ablations show where the gain comes from: without the seven visual queries, accuracy drops to 71.65% (sensitivity 71.92%); without balanced adjudication, to 73.99% (75.68%). The audit of 4,494 visual sessions (7 queries × 642 subjects) reports 100% correctly structured outputs and "high visibility" above 93%, with the nose the weak point.

Two figures, absent from the paper, change the reading. First, no specificity and no AUC are reported — the system produces only a yes/no, not a probability, which precludes any ROC curve (the AUC, area under that curve, summarizes the ability to separate positives from negatives across all thresholds). Yet specificity (the share of true negatives correctly classified) can be reconstructed from accuracy, sensitivity and prevalence: about 554 true positives and 14 true negatives, i.e. a specificity of about 24% (14 of 58) and 44 false positives. Put plainly: of the 58 truly disease-free people in this cohort, about 44 are wrongly flagged as "at risk." (Cross-check: this set of values exactly reproduces the reported F1 of 93.74%.) Second, because positives make up 90.97% of the cohort, a trivial model answering "positive" for everyone would reach 90.97% accuracy — above the system's 88.47%. Finally, robustness is questionable: randomly shuffling the images or blurring them each flips the prediction for 63 of 642 subjects (nearly 10%).

What is good

Interpretability by design. Rather than a black box spitting out a score, EviOSAHS produces seven anatomical evidence cards, legible and checkable, with risk direction, strength and confidence, and separates image observation from the final decision. For a triage tool where a clinician wants to see "why," this is a useful property — and the audit showing 100% correctly structured outputs shows the format constraint holds.

Honest ablations. The authors do not just announce a good number: they take apart their pipeline to show that most of the gain comes from the seven-query decomposition and the balanced adjudication, and they document the collapse of naively prompted VLMs (0% or 1.54% sensitivity). Exposing these failures rather than hiding them is good scientific hygiene.

Reproducibility and a low technical barrier. No training, open 7-billion-parameter models running on a single GPU, deterministic decoding, and code released on GitHub with the prompts, card schemas and evaluation code. The method can be rerun and debated — the basis of any constructive critique.

What is less good

A misleading metric on an imbalanced cohort. This is the crux. With 584 positives to 58 negatives, accuracy and F1 are inflated by prevalence, and "always positive" beats the system on accuracy (90.97% versus 88.47%). Above all, the absence of specificity and AUC hides the Achilles' heel of any screening: the reconstruction gives a specificity of about 24%, i.e. 44 false positives out of 58 true negatives. On a general, low-prevalence population — the real setting for a screening tool — this operating point would produce a flood of needless PSG referrals. Optimizing sensitivity alone on a cohort enriched in cases means measuring performance where it looks most flattering.

No train/test split, no external validation. The whole evaluation runs on the same cohort the prompts were tuned on: the result is effectively "in-sample," with a risk of overfitting the prompts and the operating point to this dataset — a prompt-era variant of data leakage. Add a strong population bias: a single hospital, a Chinese cohort, 85% male, an age range including a 4-year-old (pediatric contamination in an adult task), and non-physiological waist-hip ratios (> 2) kept as inputs. No external cohort tests generalization.

Weak comparators and unstable predictions. The most spectacularly beaten baselines (InstructBLIP at 0%, Qwen2.5-VL at 1.54%) are degenerate — straw men — and the system is never compared to a validated questionnaire such as STOP-Bang or to a published facial-photo model. As for the claimed anatomical grounding, it holds up poorly: shuffling or blurring the images flips 63 predictions out of 642, a sign of possible shortcut learning (the model leaning on spurious correlations rather than the anatomy it claims to read), all the more so as the intermediate "evidence" is never validated by an ENT specialist. Finally, note that one author is affiliated with Tencent Youtu Lab, while the statement declares no conflict of interest.

What it changes

For the research community, the study illustrates an appealing design pattern: orchestrating general-purpose models by prompt to produce structured, interpretable multimodal reasoning at no training cost. But its main value is as a cautionary tale on evaluation: reporting accuracy, sensitivity and F1 on an imbalanced, enriched cohort, without specificity, AUC or predictive value, can make a system look strong when it actually falls below a trivial baseline. The remedy is known: balanced cohorts or realistic prevalence, a calibrated operating point, dual-reported metrics, external validation.

For clinicians and decision-makers, the message is cautious: a face photo replaces neither polysomnography nor, at this stage, even a validated questionnaire. In screening, specificity is the crux — every false positive costs a PSG night and anxiety for a healthy person. Inference based on facial morphology also raises questions of demographic and ethnic bias, and of privacy, that a photo-based tool will have to face head-on.

For patients and the public, two simple ideas. No, an app cannot reliably diagnose sleep apnea from a selfie. And a headline touting "94%" hides the essential: among people who do not have the disease, this system is wrong, on this data, most of the time. A good screening test is judged on two legs — catching the sick and sparing the healthy — not on one.

To go further

The preprint "Structured Visual Evidence Decomposition for Evidence-Grounded Multimodal Screening of Obstructive Sleep Apnea-Hypopnea Syndrome" is available on arXiv (2606.00087), by Chen Zhan, Yingchen Wei, Xiaoyu Tan, Jingjing Huang and Xihe Qiu (Shanghai University of Engineering Science; Tencent Youtu Lab; Eye & ENT Hospital of Fudan University; National University of Singapore). The code is released on GitHub; the data (face images and clinical records) are not public. The study is registered in the Chinese Clinical Trial Registry (ChiCTR2300069223) and funded by the Shanghai Municipal Natural Science Foundation (23ZR1425400); the authors declare no conflict of interest. On vision-language models and their blind spots, see our decryptages on text dominating the image in clinical VLMs, on the caution and metacognition of LLM judges in the clinic, and on hidden subgroup biases in medical imaging.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. The specificity (≈ 24%) and the number of false positives are reconstructed from the reported accuracy, sensitivity and prevalence. English, Spanish and Chinese translations produced with AI assistance and reviewed.