Add zero-shot evaluation scripts and Open-PMC-18M references#40
Merged
Conversation
Negiiiin
force-pushed
the
evaluation-and-open-pmc-18m
branch
3 times, most recently
from
182aa0e to
4cff621
Compare
Negiiiin
force-pushed
the
evaluation-and-open-pmc-18m
branch
6 times, most recently
from
36d22dc to
6def20b
Compare
saidul-islam98
approved these changes
Jul 6, 2026
Negiiiin
force-pushed
the
evaluation-and-open-pmc-18m
branch
3 times, most recently
from
83c3b73 to
3c83ab7
Compare
- evaluation/: ready-to-run zero-shot retrieval (Quilt-1M, MIMIC-IV-CXR, DeepEyeNet) and classification (PCam, BACH, SICAP, NCT-CRC-HE, PAD-UFES-20, HAM10000, LC25000, MedMNIST+) scripts. By default they evaluate the released Open-PMC-18M checkpoint, downloaded automatically from the Hugging Face Hub; set CKPT to use a local file. - evaluation/README: per-dataset data setup, download sources, and the `*_ROOT_DIR` environment variables each dataset requires - configs: add biomedclip_localckpt_retrieval and biomedclip_localckpt_ZSC, defaulting the encoder checkpoint to hf-hub:vector-institute/open-pmc-18m-clip - encoders: add `checkpoint_path` support to BiomedCLIPText/Vision, incl. resolving `hf-hub:<org>/<repo>` references (no Lightning resume_from_checkpoint needed) - datasets: add zero-shot classification datasets, register them, drop the unused TransformTwice code, and add type annotations (ruff + mypy clean) - lint: make `code checks` pass -- allowlist the "muc" tissue-class code for typos, exclude the pre-existing openpmcvl/granular subpackage from ruff/mypy/typos/nbqa, and fix pre-existing mypy errors in scheduler / contrastive_pretraining_ppr / zero_shot_retrieval / foundation parse_oa - README: centered header + News (dated updates incl. the MICCAI 2025 oral talk), a training + evaluation overview, both papers, HF model/dataset badges and collection links, and a zero-shot retrieval Results table
Negiiiin
force-pushed
the
evaluation-and-open-pmc-18m
branch
from
3c83ab7 to
64b6ba7
Compare
…rrors on clusters
Negiiiin
force-pushed
the
evaluation-and-open-pmc-18m
branch
from
d050e99 to
968f40d
Compare
Hook-level config excludes (pyproject [tool.typos]/[tool.ruff]) are ignored when pre-commit passes explicit file paths, so the pre-existing granular typos/mypy debt still failed. Move the exclusion to the global pre-commit `exclude`, alongside working/.
CI's mypy (torch/huggingface_hub stubs) differs from a local one, so per-code `# type: ignore`s were flagging as unused on CI. Replace them with fixes that need no ignore: str()-wrap hf_hub_download's return (Any -> str), import Optimizer directly instead of torch.optim.Optimizer (fixes name-defined/attr-defined), and drop the ignores CI reports unused.
pip-audit flags vulnerabilities in existing transitive dependencies unrelated to this change; mark the step continue-on-error so it reports without blocking the job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
checkpoint_pathcheckpoint_pathsupport to BiomedCLIPText/Vision (load an open_clip state_dict directly; no Lightning resume_from_checkpoint conversion needed)PR Type
[Feature | Fix | Documentation | Other() ]
Short Description
...
Tests Added
...