CoChat
Sign Up
PreprintCited by 17Biomedical Text Mining and OntologiesTopic ModelingNatural Language Processing Techniques

BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing

Open access
Fries, Jason Alan, Weber, Leon, Seelam, Natasha, Altay, Gabriel, Datta, Debajyoti, Garda, Samuele, Kang, Myungsun, Su, Ruisi +35 more·2022·arXiv
DOI 10.48550/arxiv.2206.15076arXiv 2206.15076
Open this paper in CoChat — free accountOpen PDF

Abstract

Training and evaluating language models increasingly requires the construction of meta-datasets --diverse collections of curated data with clear provenance. Natural language prompting has recently lead to improved zero-shot generalization by transforming existing, supervised datasets into a diversity of novel pretraining tasks, highlighting the benefits of meta-dataset curation. While successful in general-domain text, translating these data-centric approaches to biomedical language modeling remains challenging, as labeled biomedical datasets are significantly underrepresented in popular data hubs. To address this challenge, we introduce BigBIO a community library of 126+ biomedical NLP datasets, currently covering 12 task categories and 10+ languages. BigBIO facilitates reproducible meta-dataset curation via programmatic access to datasets and their metadata, and is compatible with current platforms for prompt engineering and end-to-end few/zero shot language model evaluation. We discuss our process for task schema harmonization, data auditing, contribution guidelines, and outline two illustrative use cases: zero-shot evaluation of biomedical prompts and large-scale, multi-task learning. BigBIO is an ongoing community effort and is available at https://github.com/bigscience-workshop/biomedical

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.15076

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most CLaMs are evaluated on traditional NLP-style tasks such as named entity recognition, relation extraction, and document classification on either MIMIC-III or PubMed.

    “Most CLaMs are being evaluated on traditional NLP-style tasks such as named entity recognition, relation extraction, and document classification on either MIMIC-III (clinical text) or PubMed (biomedical text)73,74.”

    From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [73]

    The benchmark table in “L BIGBIO vs. Existing Benchmarks” (p. 49) lists named entity recognition, relation extraction, and document classification tasks, and “H Assessing Dataset Overlap” (pp. 29–30) discusses datasets sourced from PubMed. But the paper does not establish that most clinical language models are evaluated on those tasks or that their evaluations use either MIMIC-III or PubMed. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 95

  1. [40]Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. FLAIR: An easy-to-use framework for state-of-the-art NLP. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) , pages 54-59, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  2. [41]Simon Almgren, Sean Pavlov, and Olof Mogren. Named entity recognition in swedish medical journals with deep bidirectional character-based lstms. In Proceedings of the Fifth Workshop on Building and Evaluating Resources for Biomedical Text Mining (BioTxtM 2016) , pages 30-39. The COLING 2016 Organizing Committee, 12 2016.
  3. [43]Simon Baker, Ilona Silins, Yufan Guo, Imran Ali, Johan H"ogberg, Ulla Stenius, and Anna Korhonen. Automatic semantic classification of scientific literature according to the hallmarks of cancer. Bioinform. , 32(3):432-440, 2016.
  4. [44]Asma Ben Abacha and Dina Demner-Fushman. On the summarization of consumer health questions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2228-2234, Florence, Italy, July 2019. Association for Computational Linguistics.
  5. [45]Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering. In ACL-BioNLP 2019 , 2019.
  6. [46]Robert Bossy, Louise Del'eger, Estelle Chaix, Mouhamadou Ba, and Claire N'edellec. Bacteria biotope at BioNLP open shared tasks 2019. In Proceedings of The 5th Workshop on BioNLP Open Shared Tasks , pages 121-131, Hong Kong, China, November 2019. Association for Computational Linguistics.
  7. [48]Paul Buitelaar, Thierry Declerck, Bogdan Sacaleanu, Špela Vintar, Diana Raileanu, and Claudia Crispi. A multi-layered, xml-based approach to the integration of linguistic and semantic annotations. In Proceedings of EACL 2003 Workshop on Language Technology and the Semantic Web (NLPXML'03), Budapest, Hungary , 2003.
  8. [50]J. Gregory Caporaso, William A Baumgartner, David A Randolph, K. Bretonnel Cohen, and Lawrence Hunter. Mutationfinder: a high-performance system for extracting point mutation mentions from text. Bioinformatics , 23(14):1862-1865, Jul 2007.
  9. [51]Qingyu Chen, Alexis Allot, Robert Leaman, Rezarta Islamaj Do˘ gan, and Zhiyong Lu. Overview of the biocreative vii litcovid track: multi-label topic classification for covid-19 literature annotation. In Proceedings of the seventh BioCreative challenge evaluation workshop , 2021.
  10. [52]Shu Chen, Zeqian Ju, Xiangyu Dong, Hongchao Fang, Sicheng Wang, Yue Yang, Jiaqi Zeng, Ruisi Zhang, Ruoyu Zhang, Meng Zhou, Penghui Zhu, and Pengtao Xie. Meddialog: A large-scale medical dialogue dataset. CoRR , abs/2004.03329, 2020. · arXiv
  11. [55]Nigel Collier and Jin-Dong Kim. Introduction to the bio-entity recognition task at JNLPBA. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP) , pages 73-78, Geneva, Switzerland, August 28th and 29th 2004. COLING.
  12. [60]J Ding, D Berleant, D Nettleton, and E Wurtele. Mining MEDLINE: abstracts, sentences, or phrases? Pac Symp Biocomput , pages 326-337, 2002.
  13. [61]Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. Ncbi disease corpus: A resource for disease name recognition and concept normalization. Journal of biomedical informatics , 47:1-10, 2014.
  14. [63]Erik Faessler, Luise Modersohn, Christina Lohr, and Udo Hahn. ProGene - a large-scale, high-quality protein-gene annotated benchmark corpus. In Proceedings of the 12th Language Resources and Evaluation Conference , pages 4585-4596, Marseille, France, May 2020. European Language Resources Association.
  15. [64]Katrin Fundel, Robert Küffner, and Ralf Zimmer. Relex-relation extraction using dependency parse trees. Bioinformatics , 23(3):365-371, 2007.
  16. [67]Luis Gasco, Anastasios Nentidis, Anastasia Krithara, Darryl Estrada-Zavala, Renato Toshiyuki Murasaki, Elena Primo-Peña, Cristina Bojo Canales, Georgios Paliouras, Martin Krallinger, et al. Overview of bioasq 2021-mesinesp track. evaluation of advance hierarchical classification techniques for scientific literature, patents and clinical trials. CEUR Workshop Proceedings, 2021.
  17. [68]Martin Gerner, Goran Nenadic, and Casey M Bergman. Linnaeus: a species name identification system for biomedical literature. BMC bioinformatics , 11(1):1-17, 2010.
  18. [69]Aitor Gonzalez-Agirre, Montserrat Marimon, Ander Intxaurrondo, Obdulia Rabal, Marta Villegas, and Martin Krallinger. Pharmaconer: Pharmacological substances, compounds and proteins named entity recognition track. In Proceedings of The 5th Workshop on BioNLP Open Shared Tasks , pages 1-10, Hong Kong, China, November 2019. Association for Computational Linguistics.
  19. [70]Natalia Grabar, Vincent Claveau, and Cl'ement Dalloux. CAS: French corpus with clinical cases. In Proceedings of the Ninth International Workshop on Health Text Mining and Information Analysis , pages 122-128, Brussels, Belgium, October 2018. Association for Computational Linguistics.
  20. [71]Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Heal. , 3(1):2:1-2:23, 2022.
  21. [72]Harsha Gurulingappa, Roman Klinger, Martin Hofmann-Apitius, and Juliane Fluck. An empirical evaluation of resources for the identification of diseases and adverse effects in biomedical literature. In LREC Workshop on Building and Evaluating Resources for Biomedical Text Mining , 2010.
  22. [73]Sam Henry, Kevin Buchan, Michele Filannino, Amber Stubbs, and Ozlem Uzuner. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J. Am. Medical Informatics Assoc. , 27(1):3-12, 2020.
  23. [74]María Herrero-Zazo, Isabel Segura-Bedmar, Paloma Martínez, and Thierry Declerck. The ddi corpus: An annotated corpus with pharmacological substances and drug-drug interactions. Journal of Biomedical Informatics , 46(5):914-920, 2013.
  24. [75]Rezarta Islamaj, Robert Leaman, Sun Kim, Dongseop Kwon, Chih-Hsuan Wei, Donald C Comeau, Yifan Peng, David Cissel, Cathleen Coss, Carol Fisher, et al. Nlm-chem, a new resource for chemical entity recognition in pubmed full text literature. Scientific Data , 8(1):1-12, 2021.
  25. [76]Peter Jansen, Mihai Surdeanu, and Peter Clark. Discourse complements lexical semantics for non-factoid answer reranking. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 977-986, Baltimore, Maryland, June 2014. Association for Computational Linguistics.
  26. [77]Antonio J Jimeno-Yepes, Bridget T McInnes, and Alan R Aronson. Exploiting mesh indexing in medline to generate a data set for word sense disambiguation. BMC bioinformatics , 12(1):1-14, 2011.
  27. [79]Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2567-2577, 2019.
  28. [80]Sarvnaz Karimi, Alejandro Metke-Jimenez, Madonna Kemp, and Chen Wang. Cadec: A corpus of adverse drug event annotations. Journal of biomedical informatics , 55:73-81, 2015.
  29. [81]Hrant Khachatrian, Lilit Nersisyan, Karen Hambardzumyan, Tigran Galstyan, Anna Hakobyan, Arsen Arakelyan, Andrey Rzhetsky, and Aram Galstyan. BioRelEx 1.0: Biological relation extraction benchmark. In Proceedings of the 18th BioNLP Workshop and Shared Task , pages 176-190, Florence, Italy, August 2019. Association for Computational Linguistics.
  30. [83]Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In AAAI , 2018.
  31. [84]Baeksoo Kim, Wonjun Choi, and Hyunju Lee. A corpus of plant-disease relations in the biomedical domain. PLoS One , 14(8):e0221582, 2019.
  32. [85]Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Jun'ichi Tsujii. Overview of BioNLP'09 shared task on event extraction. In Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task , pages 1-9, Boulder, Colorado, June 2009. Association for Computational Linguistics.
  33. [86]Jin-Dong Kim, Yue Wang, Toshihisa Takagi, and Akinori Yonezawa. Overview of genia event task in bionlp shared task 2011. In Proceedings of the BioNLP Shared Task 2011 Workshop , BioNLP Shared Task '11, page 7-15, USA, 2011. Association for Computational Linguistics.
  34. [87]Jin-Dong Kim, Yue Wang, and Yamamoto Yasunori. The Genia event extraction shared task, 2013 edition - overview. In Proceedings of the BioNLP Shared Task 2013 Workshop , pages 8-15, Sofia, Bulgaria, August 2013. Association for Computational Linguistics.
  35. [88]Jung-jae Kim, Xu Han, Vivian Lee, and Dietrich Rebholz-Schuhmann. GRO task: Populating the gene regulation ontology with events and relations. In Proceedings of the BioNLP Shared Task 2013 Workshop , pages 50-57, Sofia, Bulgaria, August 2013. Association for Computational Linguistics.
  36. [89]Corinna Kol'arik, Roman Klinger, Christoph M Friedrich, Martin Hofmann-Apitius, and Juliane Fluck. Chemical names: Terminological resources and corpora annotation. In LREC Workshop on Building and Evaluating Resources for Biomedical Text Mining , 2008.
  37. [90]Jan A Kors, Simon Clematide, Saber A Akhondi, Erik M van Mulligen, and Dietrich Rebholz-Schuhmann. A multilingual gold-standard corpus for biomedical concept recognition: the Mantra GSC. Journal of the American Medical Informatics Association , 22(5):948-956, 05 2015.
  38. [91]Neema Kotonya and Francesca Toni. Explainable automated fact-checking for public health claims. arXiv preprint arXiv:2010.09926 , 2020. · arXiv
  39. [92]Martin Krallinger, Obdulia Rabal, Florian Leitner, Miguel Vazquez, David Salgado, Zhiyong Lu, Robert Leaman, Yanan Lu, Donghong Ji, Daniel M. Lowe, Roger A. Sayle, Riza Theresa Batista-Navarro, Rafal Rak, Torsten Huber, Tim Rockt"aschel, S'ergio Matos, David Campos, Buzhou Tang, Hua Xu, Tsendsuren Munkhdalai, Keun Ho Ryu, S. V. Ramanan, Senthil Nathan, Slavko Zitnik, Marko Bajec, Lutz Weber, Matthias Irmer, Saber A. Akhondi, Jan A. Kors, Shuo Xu, Xin An, Utpal Kumar Sikdar, Asif Ekbal, Masaharu Yoshioka, Thaer M. Dieb, Miji Choi, Karin Verspoor, Madian Khabsa, C. Lee Giles, Hongfang Liu, Komandur Elayavilli Ravikumar, Andre Lamurias, Francisco M. Couto, Hong-Jie Dai, Richard Tzong-Han Tsai, Caglar Ata, Tolga Can, Anabel Usi'e, Rui Alves, Isabel Segura-Bedmar, Paloma Mart'inez, Julen Oyarzabal, and Alfonso Valencia. The chemdner corpus of chemicals and drugs and its annotation principles. Journal of Cheminformatics , 7(1):S2, Jan 2015.
  40. [93]Rabal-O. Lourenço A. Krallinger, M. Effective transfer learning for identifying similar questions: Matching user questions to covid-19 faqs. KDD'20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 3458-3465, 2020.
  41. [94]Vishesh Kumar, Amber Stubbs, Stanley Shaw, and Özlem Uzuner. Creation of a new longitudinal corpus of clinical narratives. Journal of Biomedical Informatics , 58:S6-S10, 2015. Supplement: Proceedings of the 2014 i2b2/UTHealth Shared-Tasks and Workshop on Challenges in Natural Language Processing for Clinical Data.
  42. [95]Fabr'ıcio Kury, Alex Butler, Chi Yuan, Li-heng Fu, Yingcheng Sun, Hao Liu, Ida Sim, Simona Carini, and Chunhua Weng. Chia, a large annotated corpus of clinical trial eligibility criteria. Scientific data , 7(1):1-11, 2020.
  43. [98]Nut Limsopatham and Nigel Collier. Normalising medical concepts in social media texts by learning semantic representation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1014-1023, Berlin, Germany, August 2016. Association for Computational Linguistics.
  44. [100]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.
  45. [102]Ling Luo, Po-Ting Lai, Chih-Hsuan Wei, Cecilia N Arighi, and Zhiyong Lu. Biored: A comprehensive biomedical relation extraction dataset. arXiv preprint arXiv:2204.04263 , 2022. · arXiv
  46. [103]Ling Luo, Po-Ting Lai, Chih-Hsuan Wei, Cecilia N. Arighi, and Zhiyong Lu. Biored: A comprehensive biomedical relation extraction dataset. CoRR , abs/2204.04263, 2022. · arXiv
  47. [104]Montserrat Marimon, Aitor Gonzalez-Agirre, Ander Intxaurrondo, Heidy Rodriguez, Jose Lopez Martin, Marta Villegas, and Martin Krallinger. Automatic de-identification of medical texts in spanish: the meddocan track, corpus, guidelines, methods and evaluation of results. In IberLEF SEPLN , pages 618-638, 2019.
  48. [105]Antonio Miranda-Escalada, Eulàlia Farré, and Martin Krallinger. Named entity recognition, concept normalization and clinical coding: Overview of the cantemist track for cancer text mining in spanish, corpus, guidelines, methods and results. IberLEF SEPLN , pages 303-323, 2020.
  49. [108]Anastasios Nentidis, Konstantinos Bougiatiotis, Anastasia Krithara, Georgios Paliouras, and Ioannis Kakadiaris. Results of the fifth edition of the BioASQ challenge. BioNLP 2017, 2007.
  50. [109]Aurélie Névéol, Cyril Grouin, Jeremy Leixa, Sophie Rosset, and Pierre Zweigenbaum. The QUAERO French medical corpus: A ressource for medical entity recognition and normalization. In Proc of BioTextMining Work , pages 24-30, 2014.
  51. [110]Mariana Neves, Alexander Damaschun, Andreas Kurtz, and Ulf Leser. Annotating and evaluating text for stem cell research. In Proceedings of the Third Workshop on Building and Evaluation Resources for Biomedical Text Mining (BioTxtM 2012) at Language Resources and Evaluation (LREC). Istanbul, Turkey , pages 16-23. Citeseer, 2012.
  52. [111]Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, and Byron Wallace. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 197-207, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  53. [112]Tomoko Ohta, Sampo Pyysalo, Makoto Miwa, Jin-Dong Kim, and Jun'ichi Tsujii. Event extraction for post-translational modifications. In Proceedings of the 2010 Workshop on Biomedical Natural Language Processing , pages 19-27, Uppsala, Sweden, July 2010. Association for Computational Linguistics.
  54. [113]Tomoko Ohta, Sampo Pyysalo, Rafal Rak, Andrew Rowley, Hong-Woo Chun, Sung-Jae Jung, Sung-Pil Choi, Sophia Ananiadou, and Jun'ichi Tsujii. Overview of the pathway curation (PC) task of BioNLP shared task 2013. In Proceedings of the BioNLP Shared Task 2013 Workshop , pages 67-75, Sofia, Bulgaria, August 2013. Association for Computational Linguistics.
  55. [114]Tomoko Ohta, Sampo Pyysalo, and Jun'ichi Tsujii. Overview of the epigenetics and post-translational modifications (EPI) task of BioNLP shared task 2011. In Proceedings of BioNLP Shared Task 2011 Workshop , pages 16-25, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
  56. [115]Tomoko Ohta, Sampo Pyysalo, Jun'ichi Tsujii, and Sophia Ananiadou. Open-domain anatomical entity mention detection. volume W12-43. Association for Computational Linguistics, 2012.
  57. [116]Tomoko Ohta, Yuka Tateisi, and Jin-Dong Kim. The genia corpus: An annotated research abstract corpus in molecular biology domain. In Proceedings of the Second International Conference on Human Language Technology Research , HLT '02, page 82-86, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc.
  58. [117]Serguei Pakhomov, Bridget McInnes, Terrence Adam, Ying Liu, Ted Pedersen, and Genevieve B Melton. Semantic similarity and relatedness between clinical terms: an experimental study. In AMIA annual symposium proceedings , volume 2010, page 572. American Medical Informatics Association, 2010.
  59. [118]Dimitris Pappas, Petros Stavropoulos, Ion Androutsopoulos, and Ryan McDonald. BioMRC: A dataset for biomedical machine reading comprehension. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing , pages 140-149, Online, July 2020. Association for Computational Linguistics.
  60. [119]Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, M Hassan Murad, and Chitta Baral. In-boxbart: Get instructions into biomedical multi-task learning. arXiv preprint arXiv:2204.07600 , 2022. · arXiv
  61. [120]Ted Pedersen, Serguei VS Pakhomov, Siddharth Patwardhan, and Christopher G Chute. Measures of semantic similarity and relatedness in the biomedical domain. Journal of biomedical informatics , 40(3):288-299, 2007.
  62. [121]Sampo Pyysalo and Sophia Ananiadou. Anatomical entity mention recognition at literature scale. Bioinformatics , 30(6):868-875, 2014.
  63. [122]Sampo Pyysalo, Filip Ginter, Juho Heimonen, Jari Bj"orne, Jorma Boberg, Jouni J"arvinen, and Tapio Salakoski. Bioinfer: a corpus for information extraction in the biomedical domain. BMC bioinformatics , 8(1):1-24, 2007.
  64. [123]Sampo Pyysalo, Tomoko Ohta, and Sophia Ananiadou. Overview of the cancer genetics (CG) task of BioNLP shared task 2013. In Proceedings of the BioNLP Shared Task 2013 Workshop , pages 58-66, Sofia, Bulgaria, August 2013. Association for Computational Linguistics.
  65. [124]Sampo Pyysalo, Tomoko Ohta, Jin-Dong Kim, and Jun'ichi Tsujii. Static relations: a piece in the biomedical information extraction puzzle. In Proceedings of the BioNLP 2009 Workshop , pages 1-9, Boulder, Colorado, June 2009. Association for Computational Linguistics.
  66. [125]Sampo Pyysalo, Tomoko Ohta, Makoto Miwa, Han-Cheol Cho, Jun'ichi Tsujii, and Sophia Ananiadou. Event extraction across multiple levels of biological organization. Bioinformatics , 28(18):i575-i581, 2012.
  67. [126]Sampo Pyysalo, Tomoko Ohta, Rafal Rak, Dan Sullivan, Chunhong Mao, Chunxia Wang, Bruno Sobral, Jun'ichi Tsujii, and Sophia Ananiadou. Overview of the infectious diseases (ID) task of BioNLP shared task 2011. In Proceedings of BioNLP Shared Task 2011 Workshop , pages 26-35, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
  68. [127]Sampo Pyysalo, Tomoko Ohta, and Jun'ichi Tsujii. Overview of the entity relations (rel) supporting task of bionlp shared task 2011. In Proceedings of the BioNLP Shared Task 2011 Workshop , BioNLP Shared Task '11, page 83-88, USA, 2011. Association for Computational Linguistics.
  69. [128]Paolo Rosso, Julio Gonzalo, Raquel Martinez, Soto Montalvo, and Jorge Carrillo de Albornoz. Proceedings of the third workshop on evaluation of human language technologies for iberian languages (ibereval 2018). volume 2150 of CEUR Workshop Proceedings . CEUR-WS.org, 2018.
  70. [129]Claudia Schulz, Josh Levy-Kramer, Camille Van Assel, Miklos Kepes, and Nils Hammerla. Biomedical concept relatedness - a large EHR-based benchmark. In Proceedings of the 28th International Conference on Computational Linguistics , pages 6565-6575, Barcelona, Spain (Online), dec 2020. International Committee on Computational Linguistics.
  71. [130]MJ Shardlow, N Nguyen, G Owen, C O'Donovan, A Leach, J McNaught, S Turner, and S Ananiadou. A new corpus to support text mining for the curation of metabolites in the ChEBI database. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) , pages 280-285, May 2018.
  72. [133]Yoshinobu Kano Tomoko Ohkuma Shoko Wakamiya, Mizuki Morita and Eiji Aramaki. Overview of the ntcir-13 medweb task. Proceedings of the 13th NTCIR Conference on Evaluation of Information Access Technologies (NTCIR-13) , 2017.
  73. [134]Felipe Soares, Viviane Moreira, and Karin Becker. A large parallel corpus of full-text scientific articles. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC2018) , 2018.
  74. [135]Hakime Öztürk So˘ gancıo˘ glu, Gizem and Arzucan Özgür. Biosses: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics , 33(14):i49-i58, 2017.
  75. [136]Amber Stubbs, Michele Filannino, Ergin Soysal, Samuel Henry, and Ozlem Uzuner. Cohort selection for clinical trials: n2c2 2018 shared task track 1. J. Am. Medical Informatics Assoc. , 26(11):1163-1171, 2019.
  76. [137]Amber Stubbs, Christopher Kotfila, and Özlem Uzuner. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/uthealth shared task track 1. Journal of Biomedical Informatics , 58:S11-S19, 2015.
  77. [141]Thinh Hung Truong, Mai Hoang Dao, and Dat Quoc Nguyen. Covid-19 named entity recognition for vietnamese. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , page 2146-2153, 2021.
  78. [142]George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics , 16(1):138, 2015.
  79. [143]Ozlem Uzuner. Recognizing obesity and comorbidities in sparse data. Journal of the American Medical Informatics Association , 16(4):561-570, 07 2009.
  80. [144]Ozlem Uzuner, Andreea Bodnari, Shuying Shen, Tyler Forbush, John Pestian, and Brett R South. Evaluating the state of the art in coreference resolution for electronic medical records. Journal of the American Medical Informatics Association , 19(5):786-791, 02 2012.
  81. [145]Ozlem Uzuner, Ira Goldstein, Yuan Luo, and Isaac Kohane. Identifying patient smoking status from medical discharge records. Journal of the American Medical Informatics Association , 15(1):14-24, 01 2008.
  82. [146]Özlem Uzuner, Yuan Luo, and Peter Szolovits. Evaluating the state-of-the-art in automatic deidentification. Journal of the American Medical Informatics Association , 14(5):550-563, 09 2007.
  83. [147]Ozlem Uzuner, Imre Solti, and Eithon Cadag. Extracting medication information from clinical text. J. Am. Medical Informatics Assoc. , 17(5):514-518, 2010.
  84. [148]Ozlem Uzuner, Brett R. South, Shuying Shen, and Scott L. DuVall. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. J. Am. Medical Informatics Assoc. , 18(5):552-556, 2011.
  85. [149]Erik M. van Mulligen, Annie Fourrier-Reglat, David Gurwitz, Mariam Molokhia, Ainhoa Nieto, Gianluca Trifiro, Jan A. Kors, and Laura I. Furlong. The eu-adr corpus: Annotated drugs, diseases, targets, and their relationships. Journal of Biomedical Informatics , 45(5):879-884, 2012. Text Mining and Natural Language Processing in Pharmacogenomics.
  86. [151]Veronika Vincze, Gy"orgy Szarvas, Rich'ard Farkas, Gy"orgy M'ora, and J'anos Csirik. The bioscope corpus: biomedical texts annotated for uncertainty, negation and their scopes. BMC bioinformatics , 9(11):1-9, 2008.
  87. [152]David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. pages 7534-7550, 2020.
  88. [153]Xuan Wang, Xiangchen Song, Yingjun Guan, Bangzheng Li, and Jiawei Han. Comprehensive named entity recognition on CORD-19 with distant or weak supervision. CoRR , abs/2003.12218, 2020. · arXiv
  89. [154]M Weeber, J G Mork, and A R Aronson. Developing a test collection for biomedical word sense disambiguation. Proc AMIA Symp , pages 746-750, 2001.
  90. [157]Chih-Hsuan Wei, Bethany R Harris, Hung-Yu Kao, and Zhiyong Lu. tmvar: a text mining approach for extracting sequence variants in biomedical literature. Bioinformatics , 29(11):1433-1439, 2013.
  91. [159]Chih-Hsuan Wei, Lon Phan, Juliana Feltz, Rama Maiti, Tim Hefferon, and Zhiyong Lu. tmvar 2.0: integrating genomic variant information from literature with dbsnp and clinvar for precision medicine. Bioinformatics , 34(1):80-87, 2018.
  92. [160]Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text , pages 94-106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
  93. [161]Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics , 6:287302, 2018.
  94. [163]Markus Zlabinger, Marta Sabou, Sebastian Hofst"atter, and Allan Hanbury. Effective crowd-annotation of participants, interventions, and outcomes in the text of clinical trial reports. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 3064-3074, Online, November 2020. Association for Computational Linguistics.
  95. [164]Maryam Zolnoori, Kin Wah Fung, Timothy B. Patrick, Paul Fontelo, Hadi Kharrazi, Anthony Faiola, Yi Shuan Shirley Wu, Christina E. Eldredge, Jake Luo, Mike Conway, Jiaxi Zhu, Soo Kyung Park, Kelly Xu, Hamideh Moayyed, and Somaieh Goudarzvand. A systematic approach for developing a corpus of patient reported adverse drug events: A case study for SSRI and SNRI medications. Journal of Biomedical Informatics , 90, 2019.

Cited by

Sorted by citations

Authors
Fries, J. A., Weber, L., Seelam, N., Altay, G., Datta, D., Garda, S., Kang, M., Su, R., Kusa, W., Cahyawijaya, S., Barth, F., Ott, S., Samwald, M., Bach, S., Biderman, S., Sänger, M., Wang, B., Callahan, A., Periñán, D. L., Gigant, T., Haller, P., Chim, J., Posada, J. D., Giorgi, J. M., Sivaraman, K. R., Pàmies, M., Nezhurina, M., Martin, R., Cullan, M., Freidank, M., Dahlberg, N., Mishra, S., Bose, S., Broad, N. M., Labrak, Y., Deshmukh, S. S., Kiblawi, S., Singh, A., Vu, M. C., Neeraj, T., Golde, J., del Moral, A. V., & Beilharz, B.
Affiliations
  • Stanford University (US)
  • Max Delbrück Center (DE)
  • Tempus Labs (United States) (US)
  • University of Virginia (US)
  • Humboldt-Universität zu Berlin (DE)
  • Immuneering (United States) (US)
  • TU Wien (AT)
  • Hong Kong University of Science and Technology (HK)
  • University of Vienna (AT)
  • Brown University (US)
  • Booz Allen Hamilton (United States) (US)
  • Laboratoire Informatique d'Avignon (FR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2206.15076
arXiv
2206.15076
OpenAlex
W4283773284
Cited by
17
References
15
Citations by year
2025: 1 · 2024: 9 · 2023: 6 · 2022: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • HAL (submittedVersion)
  • arXiv (Cornell University)
Funders
Deutsche Forschungsgemeinschaft (and 11)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
biomedical natural language processing, meta-dataset, data curation, prompt engineering, zero-shot learning, language model evaluation, multi-task learning, biomedical datasets, data auditing, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review