Open-PMC

Large-scale medical vision–language pretraining from PubMed Central

If you find this project useful, please give us a star 🌟.

Open-PMC is a toolkit for training and evaluating CLIP-style medical vision–language models on large-scale image–text pairs mined from open-access PubMed Central articles. It spans the full pipeline: downloading and parsing figure–caption pairs, contrastive pretraining with mmlearn, and a zero-shot evaluation suite.

Evaluation measures how well the learned image and text embeddings align, with no task-specific fine-tuning:

• Zero-shot cross-modal retrieval — use a caption to rank images (and an image to rank captions) and report Recall@K, on Quilt-1M, MIMIC-IV-CXR, and DeepEyeNet.
• Zero-shot classification — label an image by matching it against text prompts built from each class name (top-1 accuracy), across pathology, dermatology, radiology, and MedMNIST+ datasets.

This repository hosts the code for Open-PMC (arXiv:2503.14377, MICCAI 2025 oral) and Open-PMC-18M (arXiv:2506.02738, MICCAI 2026).

News

• Jul 2026. Released Open-PMC-18M — the OpenCLIP checkpoint, models, and dataset are in the 🤗 Open-PMC-18M collection.
• May 2026. Open-PMC-18M has been accepted at MICCAI 2026! 🎉
• Sep 2025. Open-PMC was presented as an oral at MICCAI 2025 — watch the talk ▶️.
• Jun 2025. Open-PMC-18M is on arXiv.
• May 2025. Open-PMC has been accepted as an oral at MICCAI 2025! 🎉
• Mar 2025. Open-PMC is on arXiv — models and dataset are in the 🤗 OpenPMC collection.

Table of Contents

1. Installing Dependencies
2. Benchmarking
3. Evaluation
4. Results
5. Citation

Installing dependencies

We use poetry for dependency management.

# 1. Create and activate a Python 3.10 environment
python -m venv .venv && source .venv/bin/activate

# 2. Install the package with mmlearn + open_clip (from pip)
cd path/to/pmc-data-extraction
pip install --upgrade pip
poetry install --no-root --with test,open_clip,mmlearn --all-extras

Prefer building mmlearn and open_clip from source? Install without them (poetry install --no-root --with test --all-extras), then pull the submodules and install them:

git submodule update --init
pip install -e openpmcvl/experiment/mmlearn
cd openpmcvl/experiment/open_clip && make install && make install-training

Verify with python -c "import mmlearn, open_clip; print(mmlearn.__file__, open_clip.__file__)".

Benchmarking

We use mmlearn to run training and evaluation. A minimal training run:

cd pmc-data-extraction
export PYTHONPATH="./"
mmlearn_run 'hydra.searchpath=[pkg://openpmcvl.experiment.configs]' \
    +experiment=pmcoa2_matched \
    experiment_name=pmcoa2_matched_train \
    dataloader.train.batch_size=256 \
    task.encoders.text.pretrained=False \
    task.encoders.rgb.pretrained=False

Additional training/eval shell scripts are under openpmcvl/experiment/scripts.

Evaluation

Ready-to-run zero-shot retrieval and zero-shot classification scripts live in evaluation/. By default they evaluate the released Open-PMC-18M checkpoint, downloaded automatically from the Hugging Face Hub — just set the dataset's root directory:

# cross-modal retrieval
QUILT_ROOT_DIR=/data/quilt bash evaluation/zero_shot_retrieval/quilt.sh

# zero-shot classification
PCAM_ROOT_DIR=/data/pcam bash evaluation/zero_shot_classification/pcam.sh

To evaluate a different checkpoint, set CKPT to a local open_clip .pt/.bin file. See evaluation/README.md for the full list of datasets and options.

Results

Zero-shot cross-modal retrieval with Open-PMC-18M (Recall@200), produced by the scripts in evaluation/zero_shot_retrieval/:

Dataset	Image→Text R@200	Text→Image R@200
Quilt-1M (val)	25.53%	27.16%
MIMIC-IV-CXR (test)	27.47%	28.14%
DeepEyeNet (test)	19.30%	20.48%

_{Reproduce with e.g. QUILT_ROOT_DIR=… bash evaluation/zero_shot_retrieval/quilt.sh.}

Citation

If you find this code useful for your research, please consider citing:

@article{baghbanzadeh2025advancing,
  title={Advancing Medical Representation Learning Through High-Quality Data},
  author={Baghbanzadeh, Negin and Fallahpour, Adibvafa and Parhizkar, Yasaman and Ogidi, Franklin and Roy, Shuvendu and Ashkezari, Sajad and Khazaie, Vahid Reza and Colacci, Michael and Etemad, Ali and Afkanpour, Arash and Dolatabadi, Elham},
  journal={arXiv preprint arXiv:2503.14377},
  year={2025}
}

@inproceedings{baghbanzadeh2025openpmc18m,
  title={Open-PMC-18M: A High-Fidelity Large Scale Medical Dataset for Multimodal Representation Learning},
  author={Baghbanzadeh, Negin and Islam, Mohammed Saidul and Ashkezari, Sajad and Dolatabadi, Elham and Afkanpour, Arash},
  booktitle={Medical Image Computing and Computer-Assisted Intervention (MICCAI)},
  year={2026},
  note={arXiv:2506.02738}
}