MedPMC: curating the medical literature to train multimodal models — +7.1 AUC points across 26 benchmarks, but still thin clinical validation

A Yale-led team turns 6.1 million PubMed Central articles into 11 million high-fidelity medical image-text pairs through a five-stage automated pipeline called MedPMC. A vision-language model trained on this corpus, MedPMC-CLIP, gains 7.1 AUC points on average across 26 benchmarks spanning 11 specialties, against the best comparable model, while using fewer than half the pairs. But the only evaluation on real patient data reduces to an image-retrieval task in dermatology, and the authors themselves warn that nothing is ready for clinical use.

The context

Medicine is multimodal: a clinician combines images, text, laboratory values, and history. Foundation models — large networks pre-trained on huge corpora, then reusable for many tasks — could provide that backbone, but their development runs into a shortage: large, high-quality medical image datasets are rare, because real clinical data stay locked behind privacy, governance, and annotation cost. An attractive alternative exists: PubMed Central (PMC), which hosts millions of articles where figures and captions already form expert-written image-text pairs.

The problem is that the literature was not written to train models. PMC-derived datasets suffer from three ills. A majority of images are not clinical (charts, schematics, molecular illustrations — up to 80% in a recent corpus). Compound figures, which juxtapose several panels under one caption, are rarely decomposed, which breaks the correspondence between an image and the text describing it. And these datasets are static: released once, they age as new diseases, modalities, and technologies appear. This gap feeds the well-known benchmark-to-bedside gap, the distance between a flattering test score and real usefulness at the bedside. The paper thus asks two questions: can the literature be automatically curated into high-fidelity data, and does that fidelity gain translate into better models?

The method

The authors start from the 6.1 million PMC articles available as of June 2024. They keep only the 5.1 million under permissive licenses (CC BY and variants, CC0), explicitly excluding those that forbid redistribution (CC BY-ND, CC BY-NC-ND). The pipeline chains five stages, each handled by a dedicated model and evaluated separately. First an initial screening: a text classifier reads the caption and the figure's in-text mentions to decide, before even downloading the image, whether it is clinically relevant — saving storage and bandwidth. Then compound-figure detection (3.1 million found). Then separation: a model predicts the bounding box of each panel; the 3.1 million compound figures decompose into 9.2 sub-figures on average, i.e. 29 million sub-images. Next, caption-panel alignment: rather than splitting the caption and then matching, the authors handle both at once with a 4-billion-parameter MLLM (multimodal large language model — a language model able to "see" images), which receives the whole figure, its panels, and the full caption, and directly produces the list of sub-captions. Cases where the counts do not match are discarded, at the cost of volume but to the benefit of fidelity: 12.5 million sub-figure/sub-caption pairs remain. A final medical-relevance filter brings it to 7.3 million pairs. In all, the MedPMC corpus holds 11 million image-text pairs.

On this corpus, the authors train MedPMC-CLIP. CLIP (Contrastive Language-Image Pre-training) is an architecture that learns to bring an image and its describing text close together in a shared space, via contrastive learning — the model is shown correct and incorrect pairs so it learns to tell them apart. The same vision encoder then initializes an MLLM (following the LLaVA-Med recipe) for reasoning tasks. Evaluation is threefold: zero-shot classification (no task-specific retraining) across 26 datasets in 11 specialties, measured by AUC (area under the ROC curve: the probability that the model ranks a positive case above a negative one); multimodal medical question answering (MMMU and OmniMedVQA benchmarks); and, above all, a "clinical" task on 10,524 dermatology photographs from the Yale New Haven Health System, where, given a written description of a lesion, one retrieves the most similar patient images (metric: Recall@5, the fraction of cases where the right image is among the top five).

The results

On curation, the quality figures are high and hand-verified: five annotators, three with medical training, judge 95.3% of MedPMC images clinically relevant, versus 19.7% for a prior PMC set. Each stage has its score: initial screening F1 of 93.2 (F1 combines precision and recall into one number), compound detection 96.5, separation mAP of 89.8, caption alignment 81.4 F1 and 85.3 ROUGE-L (a text-overlap measure), medical filter 96.5.

On the model, the effect is clear. Reusing BMC-CLIP's exact protocol and changing only the training corpus, MedPMC-CLIP gains 7.1 AUC points on average across the 26 benchmarks, with progress in 10 of the 11 specialties, and does so while training on fewer than half the pairs. Used as the vision encoder of an MLLM, it adds +1.9 points on MMMU and +16.9 points on OmniMedVQA. On hospital dermatology, it improves Recall@5 by 11.7 points against the same comparator.

The clinical translation remains modest, however, and the authors own it. The "real" task is not a diagnosis but image retrieval: finding, for a lesion description, visually similar cases — useful to a clinician who wants to compare, not to decide alone. A representation-space analysis shows that MedPMC dermatology images cover the distribution of real patient photos better than several public sets — an encouraging signal, but one that says nothing about the safety of diagnostic use.

What is good

Fidelity measured, not proclaimed. The strength is not volume but corpus cleanliness, established by human review (95.3% versus 19.7%) and by stage-by-stage benchmarks that make the pipeline auditable rather than opaque. In a field that often piles up pairs without looking at what they contain, that is rare discipline.

An experimental control that isolates the cause. By freezing BMC-CLIP's architecture and protocol and changing only the data, the study cleanly attributes the 7.1-point gain to the data themselves, not to a bigger or better-tuned model. Achieving that gain with fewer than half the pairs reinforces the message: quality over quantity.

Real, maintained openness. The framework, corpus, component-level benchmarks, and model weights are published (Hugging Face, GitHub), each record carries its source license, and the authors announce semiannual updates following the PMC snapshots (about one million additional pairs per year). That is what distinguishes reusable infrastructure from a static dataset.

What is less good

A population bias baked into the source. Published figures are not routine images: they show representative cases, publishable results, sharp and didactic examples. The corpus thus inherits a selection bias, and the model risks shortcut learning — learning the "article-figure style" (framing, annotations, arrows) rather than the pathology. The authors acknowledge it explicitly; it is the structural limit of pre-training on the literature.

Thin clinical validation, and an unruled-out leakage risk. The only test on real data is image retrieval in dermatology — neither a diagnosis nor a prospective study — and the authors state in black and white that it "does not establish deployment readiness." Add a data leakage risk: possible overlap between the pre-training corpus and the public test sets, which the authors say they "cannot fully exclude." They guard against it with an architecture-matched comparator and an independent internal cohort, but caution remains warranted on the benchmark figures.

A single comparator, strong benchmark dependence, partial openness. The gain is measured almost everywhere against a single baseline, BMC-CLIP; and the gap swings from +1.9 to +16.9 points depending on the QA benchmark, showing how much performance depends on the chosen task. The corpus is also limited to image-text (no lab values, no tabular data, no longitudinal follow-up), and a share of the articles under CC BY-NC or NC-SA licenses cannot be used commercially. Note finally that two co-authors are employed by Microsoft Research while the conflict-of-interest statement reads "none."

What it changes

For the research community, the message is that data curation deserves to be treated as an engineering problem in its own right, not as neglected preprocessing. By delivering a modular, versioned, component-by-component-evaluated infrastructure, MedPMC offers a lower-data-cost pre-training base and a reusable model for building specialized encoders or MLLMs — including in domains poorly served by public data (rare diseases, surgical imaging, pathology subdomains).

For clinicians, nothing is deployable as is. A model pre-trained on the literature is a generic starting point, to be adapted and then validated on the local data of the target institution. The authors insist: the literature complements clinical data, it does not replace them, and the real-world validation step remains entirely to be done.

For patients and the general public, the nuance matters. A model "trained on the medical literature" learns mostly from publication images — often sharper, more typical, and better captioned than the photo taken in a consultation. An excellent score on test sets does not guarantee the same performance on a real patient image: the gap between the benchmark and the bedside, which this work aims precisely to narrow, is not closed.

To go further

The preprint is available on arXiv (10.48550/arXiv.2607.07673); the corpus, benchmarks, and weights are published on Hugging Face and the code on GitHub. Study funded by the NIH/NLM (grants R01LM014604 and R00LM014024). On multimodal foundation models in oncology, see our decryptage on probing and fusion of foundation models in oncology; on a digital-pathology foundation model, that of GigaPath; and on the evaluation limits of a corpus-grounded system, our decryptage RAG and public health.

Editorial transparency: French version written and signed by the Tatakoto editorial team based on a reading of the preprint. English, Spanish and Chinese translations produced with AI assistance and reviewed.