médical IA

AMIE on video: a Google multi-agent system outperforms primary care physicians in a simulated OSCE — with the doctors' cameras off

Published on August 15, 2026 · 9 min read

A forty-author team from Google Research and Google DeepMind put AMIE, its medical conversational agent, through an objective structured clinical examination (OSCE) delivered over video: 100 scenarios enacted by 15 professional actors, with ten US primary care physicians as the comparison arm. AMIE identifies the correct top-1 diagnosis in 0.91 of cases versus 0.77 for physicians, and adding video lifts its "observation and examination" score from 0.41 to 0.74. The protocol, however, required the comparison physicians to keep their cameras off, restricted presentations to what an actor can portray, and the actors themselves — the only participants who actually lived through the consultation — found a significant difference on just 2 of the 29 criteria they scored.

The context

AMIE (Articulate Medical Intelligence Explorer) is Google's line of work on diagnostic dialogue agents. Earlier versions conversed in writing: a simulated patient types symptoms, the agent asks questions, then produces a differential diagnosis and a management plan. Those studies reported evaluation scores above those of physicians placed in the same text format.

The obvious objection is that primary care is not practised over chat. A consultation, even remote, runs on voice, on rhythm, and above all on what the clinician sees: gait, breathing, skin colour, the way a patient guards their abdomen. Video telehealth has its own body of guidance on which parts of the physical examination remain feasible at a distance — the patient self-palpates, moves, and repositions the camera under the clinician's direction.

Moving from text to real-time video raises three distinct engineering problems. Latency: good clinical reasoning needs several seconds of computation, but a twenty-second silence makes conversation unusable. Perception: a continuous audio-video stream must be interpreted, not a single still image. Robustness: rough framing, background noise, variable lighting. This paper, posted to arXiv on 10 August 2026, sets out to solve those — and to measure the result.

The method

AMIE (Video) is an asynchronous multi-agent system: several modules run in parallel at different cadences instead of one after another. Three agents, built on Gemini 3 Flash and Gemini 3.1 Pro.

The Talker holds the real-time conversation. To stay fast it runs on a minimal compute budget and looks only at the most recent five seconds of video. It also acts as orchestrator, consolidating the states produced by the two other agents, the full dialogue history and recent frames.

The Planner carries the clinical reasoning: symptom summary, differential diagnosis, management plan, and a list of live conversational milestones (information to collect, manoeuvres to request). It updates on each patient turn, but its deep reasoning is rate-limited to one invocation per ten seconds of dialogue.

The Perception agent calls Gemini 3.1 Pro over an extended video window and maintains a cumulative memory of every audio-visual cue detected. The authors first built a taxonomy of those cues, structured into 14 areas (abdominal, cardiovascular, dermatologic, HEENT, neurologic, pain behaviour, environment and setup, and so on) with a telehealth feasibility rating running from high down to "inaccessible by audio-video".

An important point: no training or fine-tuning was performed. The three agents are orchestrated by prompting — that is, by text instructions given to public Gemini models. That makes the result conceptually easier to reproduce, and makes it entirely dependent on closed models.

The clearest engineering gain is latency: asynchronous orchestration cuts mean per-turn delay from 21.4 seconds [95% CI 20.3–23.0] to 2.6 seconds [2.3–2.8] relative to an earlier sequential architecture.

The evaluation protocol. One hundred clinical scenarios were written by a Canadian organisation specialising in OSCEs, then verified by US board-certified primary care physicians. They cover five domains — cardiopulmonary, abdominal, HEENT, neurology and psychiatry, musculoskeletal — with two "no disease" cases per domain. Each scenario carries a case-specific 20-point rubric.

Each scenario was run three times: AMIE over video, AMIE text-only, and a physician over video. That is 300 consultations. Fifteen professional actors (7 women, 8 men, aged 20 to 75) played the roles. Ten US board-certified primary care physicians, with 4.5 years of post-residency practice on average (range 1–9), formed the human arm. Twenty further independent primary care physicians, markedly more experienced (14.95 years on average), scored the recordings. The study is described as partially blinded: evaluators were not told the source, but AMIE's synthetic voice remains identifiable.

One protocol detail shapes the whole reading: physicians in the human arm were required to keep their cameras off throughout the consultation, in order to "control" the comparison with AMIE, which has no avatar. They could see the patient; the patient could not see them.

The results

On differential diagnosis, AMIE (Video) places the correct diagnosis first in 0.91 of cases [0.85–0.96], against 0.77 [0.68–0.85] for physicians (p = 0.039). Across 100 scenarios that is 14 additional cases with the right leading diagnosis. At top-3 the gap narrows to 0.98 versus 0.90 and is no longer significant (p = 0.179). Differential appropriateness (0.89 vs 0.80, p = 1.05 × 10⁻³) and comprehensiveness (0.88 vs 0.73, p = 2.38 × 10⁻⁶) separate the arms more sharply.

On the case-specific 20-point rubrics, the overall score is 0.83 [0.81–0.86] for AMIE (Video), 0.75 for AMIE (Text) and 0.68 [0.64–0.72] for physicians. The domain driving the gap is "perception and examination": 0.74 versus 0.41 for the text version and 0.47 for physicians. On the isolated physical-observation criterion, 0.77 versus 0.30 (text) and 0.51 (physicians); on guided examination, 0.72 versus 0.51 and 0.39.

Across the 36 criteria scored by clinical evaluators, 26 significantly favour AMIE (Video) over physicians after Benjamini-Hochberg correction, and none significantly favours physicians. Management plan appropriateness is 0.79 versus 0.68 (p = 7.55 × 10⁻³). The "avoid confabulation" criterion is a strict tie: 0.93 for AMIE (Video), 0.95 for AMIE (Text), 0.93 for physicians (p = 1.0).

The most interesting result is the one that contradicts the others. The actors, who scored 29 criteria after each consultation, found only two significant differences from physicians — "assessing my condition" (0.91 vs 0.81) and "explaining my condition" (0.88 vs 0.79). On several relational criteria the direction reverses in favour of physicians without reaching significance: building partnership (0.66 vs 0.74), showing interest in me as a person (0.67 vs 0.77). The large gaps evaluators saw on empathy (0.82 vs 0.71) and interpersonal skills (0.96 vs 0.83) do not appear in the simulated patients' own experience.

Perception, finally, remains very uneven depending on what has to be seen. By area: pain behaviour 88%, dermatologic 75%, musculoskeletal 70%, but communication and behaviour 44% and environment and setup 46%. On specific signs the collapse is stark: nystagmus 3%, tremor 24%, affective state 29%.

What is good

The asynchronous architecture solves a real problem and proves it with a number. Going from 21.4 to 2.6 seconds of mean per-turn latency is not cosmetic: it is the difference between a demo and a conversation. Splitting Talker / Planner / Perception with a deliberately asymmetric compute budget is a clean engineering answer to the conflict between reasoning quality and response time.

The ablation honestly isolates each agent's contribution. On the 500-scenario single-turn bench, the full system scores 59%, the Perception agent alone 58%, the Talker alone 42%. The authors themselves therefore show that most of the gain comes from perception rather than from overall orchestration. In multi-turn, 87% versus 71% for Talker-only.

Perception failures are published in detail. A promotional paper would have reported the mean. This one publishes the per-sign table, including 3% on nystagmus and 24% on tremor. That granularity is what makes the work criticisable — and therefore useful.

What is less good

The comparator is biased, in several ways at once. This is the most classic failure mode in the health-AI literature, and here it is structural. The compared physicians had their cameras off — and are then marked down, on communication and empathy rubrics, for a relational deficit in a consultation where the patient could not see their face. They had 4.5 years of practice on average, while the evaluators had 15. They had no medical record, no prior history, no investigations, and no possibility of a follow-up visit — that is, none of the levers that make a real general practitioner valuable. The authors explicitly acknowledge that cameras-off reduces the ecological validity of the human arm.

The patient is an actor, and the presentation was chosen to be portrayable. The protocol deliberately excludes paediatrics, dermatology ("actors cannot demonstrate it"), obstetrics, gynaecology, chronic disease follow-up and intimate examinations. The authors also report difficulty recruiting older actors, which under-represents severe cardiovascular disease and COPD. What is tested is not general practice: it is the subset of general practice an actor can reproduce in front of a webcam. A classic population bias is layered on top, through the de facto exclusion of patients with low digital literacy.

The judges are readers, not patients — and the two disagree. Twenty-six of thirty-six criteria favour AMIE according to clinicians watching a recording; the actors who lived through the exchange retain only two. A rubric scored from a transcript mechanically rewards what an LLM does well: structure, exhaustiveness, explicit reformulation. The bias is compounded by only partial blinding — the synthetic voice is recognisable, and the text arm is structurally identifiable. That adding video does not improve diagnosis at all (0.91 versus 0.90 in text, p = 0.862) further suggests these scenarios already saturate in text modality, which the authors concede.

Finally, on independence: all forty authors are Google Research or Google DeepMind employees, the study is funded by Alphabet, and neither the code, the weights, the recordings nor the scenarios are released. The authors justify withholding by the risk of unmonitored use in medical settings — a defensible argument, but one that makes independent replication impossible. No ethics approval statement appears in the preprint.

What it changes

For the research community, the net contribution is the architecture and the audio-visual cue taxonomy, not the score. That the Perception agent alone recovers almost the entire gain (58% against 59% for the full system) is a strong signal: the bottleneck for clinical video agents is perception, not reasoning. That is a directly actionable line of work, including for teams without access to Gemini.

For clinicians, nothing changes today, and the authors say so. What deserves attention is the guided-examination result (0.72 versus 0.39): an agent that knows to ask a patient to stand, walk, press on a specific spot and reposition the camera does better, on that point, than physicians in telehealth. This is not a replacement; it is a procedural checklist that human practice executes poorly under time pressure.

For patients and the public, caution is required about what 0.91 means. It does not describe an AI's performance facing a real patient: it describes a system facing an actor performing a pre-written case, compared against a physician deprived of camera, record and investigations. The significant result worth keeping may not be that one, but the mismatch between evaluators' judgement and simulated patients' experience: what clinicians score as an empathetic consultation is not necessarily what a patient feels as one.