PubMedQA Homepage

About The task of PubMedQA is to answer research questions with yes/no/maybe (e.g.: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting? ) using the corresponding abstracts. For more details about PubMedQA, please refer to this paper: PubMedQA Paper (EMNLP'19) Dataset PubMedQA has 1k expert labeled, 61.2k unlabeled and 211.3k artificially generated QA instances. Please visit our GitHub repository to download the dataset: PubMedQA Data & Code Repository Submission To submit your model, please follow the instructions in the GitHub repository. Citation If you use PubMedQA in your research, please cite our paper by: @inproceedings{jin2019pubmedqa, title={PubMedQA: A Dataset for Biomedical Research Question Answering}, author={Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William and Lu, Xinghua}, booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)}, pages={2567--2577}, year={2019} } Leaderboard (reasoning-required setting) Model Code Size Accuracy (%) Macro-F 1 (%) 1 Nov 28, 2023 GPT-4 (Medprompt) Microsoft (Nori et al. 2023) NA 82.0 NA 2 May 16, 2023 Med-PaLM 2 Google Research & DeepMind (Singhal et al. 2023) NA 81.8 NA 3 Nov 27, 2023 MEDITRON EPFL (Chen et al. 2023) 70B 81.6 NA 4 Jul 6, 2023 Palmyra-Med Writer Inc. (Kamble et al. 2023) 40B 81.1 NA 5 Dec 1, 2023 AntGLM-Med Ant Group (Li et al. 2023) 10B 80.6 NA 6 Apr 12, 2023 GPT-4-base Microsoft & OpenAI (Nori et al. 2023) NA 80.4 NA 7 Mar 4, 2024 Claude 3 Anthropic (Anthropic, 2024) NA 79.7 NA 8 Jan 11, 2023 GPT-3.5 + Z-Code++ Microsoft Azure AI (He et al. 2022) 175B 79.6 55.8 9 Dec 26, 2022 Flan-PaLM (3-shot) Google Research & DeepMind (Singhal et al. 2022) 540B 79.0 NA 10 Mar 14, 2024 HEAL DeepScribe Inc (Yuan et al. 2024) 13B 78.4 NA 11 Dec 20, 2022 Codex (5-shot) Technical University of Denmark & Copenhagen University Hospital...