An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition
Open accessAbstract
BACKGROUND: This article provides an overview of the first BIOASQ challenge, a competition on large-scale biomedical semantic indexing and question answering (QA), which took place between March and September 2013. BIOASQ assesses the ability of systems to semantically index very large numbers of biomedical scientific articles, and to return concise and user-understandable answers to given natural language questions by combining information from biomedical articles and ontologies. RESULTS: The 2013 BIOASQ competition comprised two tasks, Task 1a and Task 1b. In Task 1a participants were asked to automatically annotate new PUBMED documents with MESH headings. Twelve teams participated in Task 1a, with a total of 46 system runs submitted, and one of the teams performing consistently better than the MTI indexer used by NLM to suggest MESH headings to curators. Task 1b used benchmark datasets containing 29 development and 282 test English questions, along with gold standard (reference) answers, prepared by a team of biomedical experts from around Europe and participants had to automatically produce answers. Three teams participated in Task 1b, with 11 system runs. The BIOASQ infrastructure, including benchmark datasets, evaluation mechanisms, and the results of the participants and baseline methods, is publicly available. CONCLUSIONS: A publicly available evaluation infrastructure for biomedical semantic indexing and QA has been developed, which includes benchmark datasets, and can be used to evaluate systems that: assign MESH headings to published articles or to English questions; retrieve relevant RDF triples from ontologies, relevant articles and snippets from PUBMED Central; produce "exact" and paragraph-sized "ideal" answers (summaries). The results of the systems that participated in the 2013 BIOASQ competition are promising. In Task 1a one of the systems performed consistently better from the NLM's MTI indexer. In Task 1b the systems received high scores in the manual evaluation of the "ideal" answers; hence, they produced high quality summaries as answers. Overall, BIOASQ helped obtain a unified view of how techniques from text classification, semantic indexing, document and passage retrieval, question answering, and text summarization can be combined to allow biomedical experts to obtain concise, user-understandable answers to questions reflecting their real information needs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
PubMedQA and BioASQ-Y/N contain explicit specifications of the key documents.
“two biomedical research QA subsets (PubMedQA41 and BioASQ-Y/N40)”
From Leveraging long context in retrieval augmented language models for medical question answering (2025) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical Comparisons of Classifiers over Multiple Data Sets
Demšar, J. (2006) · doi:10.5555/1248547.1248548 · Cited by 11,966 · see citing works
(2004) · Cited by 9,718 · see citing works
Data mining and knowledge discovery: making sense out of data
Feyyad, U. M. (1996) · IEEE Expert · doi:10.1109/64.539013 · Cited by 4,644 · see citing works
An overview of MetaMap: historical perspective and recent advances
Aronson, A. R., & Lang, F.-M. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.002733 · Cited by 1,519 · see citing works
Tsoumakas, G., Katakis, I., & Vlahavas, I. P. (2009) · Springer eBooks · doi:10.1007/978-0-387-09823-4_34 · Cited by 1,466 · see citing works
A survey of hierarchical classification across different application domains
Junior, C. N. S., & Freitas, A. A. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0175-9 · Cited by 1,094 · see citing works
Data Mining and Knowledge Discovery
Cios, K. J., Pedrycz, W., & Świniarski, R. W. (1998) · Springer eBooks · doi:10.1007/978-1-4615-5589-6_1 · Cited by 699 · see citing works
Smith, T. F., & Waterman, M. S. (1981) · Advances in Applied Mathematics · doi:10.1016/0196-8858(81)90046-4 · Cited by 677 · see citing works
GoPubMed: exploring PubMed with the Gene Ontology
Doms, A., & Schroeder, M. D. (2005) · Nucleic Acids Research · doi:10.1093/nar/gki470 · Cited by 552 · see citing works
Self-citationTest Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 447 · see citing works
The TREC question answering track
Voorhees, E. M., & Tice, D. M. (2000) · Natural Language Engineering · doi:10.1017/s1351324901002789 · Cited by 293 · see citing works
Large scale multi-label classification via metalabeler
Tang, L., Rajan, S., & Narayanan, V. K. (2009) · International World Wide Web Conference (WWW) · doi:10.1145/1526709.1526738 · Cited by 193 · see citing works
Biomedical question answering: A survey
Athenikos, S. J., & Han, H. (2009) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2009.10.003 · Cited by 181 · see citing works
TREC genomics special issue overview
Hersh, W., & Voorhees, E. M. (2008) · Information Retrieval · doi:10.1007/s10791-008-9076-6 · Cited by 180 · see citing works
A survey and classification of semantic search approaches
Mangold, C. (2007) · International Journal of Metadata Semantics and Ontologies · doi:10.1504/ijmso.2007.015073 · Cited by 170 · see citing works
A dictionary to identify small molecules and drugs in free text
Hettne, K. M., Stierum, R. H., Schuemie, M. J., Hendriksen, P. J. M., Schijvenaars, B. J. A., van Mulligen, E. M., Kleinjans, J. C. S., & Kors, J. A. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp535 · Cited by 162 · see citing works
Recommending MeSH terms for annotating biomedical articles
Huang, M., Névéol, A., & Lu, Z. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2010-000055 · Cited by 148 · see citing works
Ihrig, C. J. (2013) · Apress eBooks · doi:10.1007/978-1-4302-5861-2_17 · Cited by 138 · see citing works
Robertson, S. E. (2006) · doi:10.1145/1183614.1183630 · Cited by 128 · see citing works
The nlm medical text indexer system for indexing biomedical literature
Mork, J. G., Yepes, A. J., & Aronson, A. R. (2013) · Cited by 82 · see citing works
A.: BioASQ: A challenge on large-scale biomedical semantic indexing and question answering
Tsatsaronis, G., Schroeder, M., Dresden, T. U., Παλιούρας, Γ., Almirantis, Y., Gaussier, É., Gallinari, P., Artières, T., Alvers, M. R., Zschunke, M., et al. (2012) · Cited by 66 · see citing works
Self-citationAbacha, A. B., & Zweigenbaum, P. (2012) · doi:10.1145/2110363.2110372 · Cited by 65 · see citing works
Galanis, D., Λάμπουρας, Γ., & Androutsopoulos, I. (2012) · Cited by 64 · see citing works
Self-citationMeSH indexing based on automatically generated summaries
Yepes, A. J., Plaza, L., Mork, J. G., Aronson, A. R., & Díaz, A. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-208 · Cited by 31 · see citing works
The ECIR 2010 large scale hierarchical classification workshop
Kosmopoulos, A., Gaussier, É., Παλιούρας, Γ., et al. (2010) · ACM SIGIR Forum · doi:10.1145/1842890.1842894 · Cited by 30 · see citing works
Answering Factoid Questions in the Biomedical Domain.
Weissenborn, D., Tsatsaronis, G., & Schroeder, M. D. (2013) · Cited by 27 · see citing works
Self-citationDeterministic binary vectors for efficient automated indexing of MEDLINE/PubMed abstracts.
Wahle, M., Widdows, D., Herskovic, J. R., et al. (2012) · PubMed · Cited by 24 · see citing works
Evaluating Multilingual Question Answering Systems at CLEF
Forner, P., Giampiccolo, D., Magnini, B., et al. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3wsyywc2crvv · Cited by 23 · see citing works
Large-Scale Semantic Indexing of Biomedical Publications.
Tsoumakas, G., Laliotis, M., Markantonatos, N., et al. (2013) · Cited by 22 · see citing works
Data Mining and Knowledge Discovery
(2013) · Data Mining and Knowledge Discovery · doi:10.1007/10618.1573-756x · Cited by 22 · see citing works
Gaudet, P. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-809633-8.20500-1 · Cited by 21 · see citing works
GoPubMed: Ontology-based literature search for the life sciences
Doms, A. (2008) · Qucosa (Saxon State and University Library Dresden) · Cited by 9 · see citing works
Two Hierarchical Text Categorization Approaches for BioASQ Semantic Indexing Challenge.
Ribadas, F. J., de Campos, L. M., Bilbao, V. M. D., et al. (2013) · Cited by 7 · see citing works
Learning to Annotate Scientific Publications
Huang, M. (2012) · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- [10]Text Analysis Conference. http://www.nist.gov/tac/.
- [33]Mork JG, Demner-Fushman D, Schmidt S, Aronson AR. Recent enhancements to the NLM medical text indexer. In: Working Notes for CLEF 2014 Conference, Sheffield, UK, September 15–18, 2014: 2014. p. 1328–1336. http://ceur-ws.org/Vol-1180/CLEF2014wn-QA-MorkEt2014.pdf.
- [35]Mork J, Jimeno-Yepes A, Aronson A. September 27th, 2013. The NLM Medical text indexer system for indexing biomedical literature. Valencia, Spain. http://ceur-ws.org/Vol-1094/bioasq2013_submission_3.pdf.
- [51]Robertson S On gmap: and other transformations CIKM 2006 NY, USA ACM
- [59]PubMed. http://www.ncbi.nlm.nih.gov/pubmed/.
- [60]PMC. http://www.ncbi.nlm.nih.gov/pmc/.
Cited by
Sorted by citations
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · see citing works
Detecting hallucinations in large language models using semantic entropy
Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024) · Nature · doi:10.1038/s41586-024-07421-0 · Cited by 934 · see citing works
“Examples of questions and answers from TriviaQA, SQuAD and BioASQ, for LLaMA 2 Chat 70B, are shown in Table 1.”
InfluentialPubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
Fact or Fiction: Verifying Scientific Claims
Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., & Hajishirzi, H. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.609 · Cited by 382 · see citing works
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
Retrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267 · see citing works
Dense Text Retrieval Based on Pretrained Language Models: A Survey
Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3637870 · Cited by 211 · see citing works
“constructed based on lexical matching models [47], [61], they tend to be more favorable for sparse retrieval methods due to a larger term overlap.”
BackgroundJin, Q., Kim, W., Chen, Q., Comeau, D. C., Yeganova, L., Wilbur, W. J., & Lu, Z. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad651 · Cited by 175 · see citing works
MeSH Now: Automatic MeSH indexing at PubMed scale via learning to rank
Mao, Y., & Lu, Z. (2017) · Journal of Biomedical Semantics · doi:10.1186/s13326-017-0123-3 · Cited by 87 · see citing works
Leveraging long context in retrieval augmented language models for medical question answering
Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01651-w · Cited by 81 · see citing works
12 years on – Is the NLM medical text indexer still useful and relevant?
Mork, J., Aronson, A., & Demner-Fushman, D. (2017) · Journal of Biomedical Semantics · doi:10.1186/s13326-017-0113-5 · Cited by 73 · see citing works
Automatic Text Summarization of Biomedical Text Data: A Systematic Review
Chaves, A., Kesiku, C., & Garcia-Zapirain, B. (2022) · Information · doi:10.3390/info13080393 · Cited by 53 · see citing works
Mitchell, J. R., Szepietowski, P., Howard, R., Reisman, P., Jones, J. D., Lewis, P., Fridley, B. L., & Rollison, D. E. (2022) · Journal of Medical Internet Research · doi:10.2196/27210 · Cited by 45 · see citing works
A Machine Learning-based Method for Question Type Classification in Biomedical Question Answering
El Alaoui, S. O., & Sarrouti, M. (2017) · Methods of Information in Medicine · doi:10.3414/me16-01-0116 · Cited by 47 · see citing works
Balde, G., Roy, S., Mondal, M., & Ganguly, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.863 · Cited by 6 · see citing works
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
Wadden, D., Shi, K., Morrison, J., Li, A., Naik, A., Singh, S., Barzilay, N., Lo, K., Hope, T., Soldaini, L., et al. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.310 · Cited by 3 · see citing works
Sabotage Evaluations for Frontier Models
Benton, J., Wagner, M., Christiansen, E., Anil, C., Perez, E., Srivastav, J., Durmus, E., Ganguli, D., Kravec, S., Shlegeris, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.21514 · Cited by 2 · see citing works
SEMA-RAG: A Self-Evolving Multi-Agent Retrieval-Augmented Generation Framework for Medical Reasoning
Huang, Y., Chen, R., & Cheng, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.917
“BioASQ-Y/N is a subset of the BioASQ Task B benchmark (Tsatsaronis et al., 2015; Krithara et al., 2023).”
A Curated Dataset for Question Answering on Laboratory Test Reference Ranges and Interpretation
Bhasuran, B., Jin, Q., Deville, A., Wu, Y., Hanna, K., Lu, Z., & He, Z. (2026) · Scientific Data · doi:10.1038/s41597-026-07554-z
- Authors
- Tsatsaronis, G., Balikas, G., Malakasiotis, P., Partalas, I., Zschunke, M., Alvers, M. R., Weissenborn, D., Krithara, A., Petridis, S., Polychronopoulos, D., Almirantis, Y., Pavlopoulos, J., Baskiotis, N., Gallinari, P., Artiéres, T., Ngomo, A.-C. N., Heino, N., Gaussier, E., Barrio-Alvers, L., Schroeder, M., Androutsopoulos, I., & Paliouras, G.
- Affiliations
- Biotechnologisches Zentrum (DE)
- Université Pierre-et-Marie-Curie (FR)
- Athens University of Economics and Business (GR)
- Université Joseph Fourier (FR)
- National Centre of Scientific Research "Demokritos" (GR)
- LIP6 (FR)
- Leipzig University (DE)
- Corresponding author
- Tsatsaronis, George
- ORCID
- Venue
- BMC Bioinformatics
- Published
- 2015-04-30
- Volume
- 16
- Issue
- 1
- Pages
- 138-138
- ISSN
- 1471-2105 (ISSN-L)
- PMC
- PMC4450488
- OpenAlex
- W1981208470
- Cited by
- 598
- References
- 35
- Citations by year
- 2026: 58 · 2025: 67 · 2024: 44 · 2023: 64 · 2022: 66 · 2021: 68 · 2020: 86 · 2019: 40 · 2018: 40 · 2017: 29 · 2016: 23 · 2015: 13
- FWCI
- 15.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Bioinformatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, cc-by)
- Qucosa (Saxon State and University Library Dresden) (submittedVersion, cc-by)
- Funders
- European Commission (2007–2013, FP7 2007-2013, ICT-2011, FP7/2007, 318652)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Semantic Web and Ontologies
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- BioASQ, question answering, PubMed, biomedical ontologies, text summarization, benchmark dataset, evaluation infrastructure, natural language queries
- MeSH terms
- Abstracting and Indexing, Algorithms, Humans, National Library of Medicine (U.S.), Natural Language Processing, Semantics, Software, United States, PubMed, Medical Subject Headings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
