Pressing needs of biomedical text mining in biocuration and beyond: opportunities and challenges
Open accessAbstract
Text mining in the biomedical sciences is rapidly transitioning from small-scale evaluation to large-scale application. In this article, we argue that text-mining technologies have become essential tools in real-world biomedical research. We describe four large scale applications of text mining, as showcased during a recent panel discussion at the BioCreative V Challenge Workshop. We draw on these applications as case studies to characterize common requirements for successfully applying text-mining techniques to practical biocuration needs. We note that system 'accuracy' remains a challenge and identify several additional common difficulties and potential research directions including (i) the 'scalability' issue due to the increasing need of mining information from millions of full-text articles, (ii) the 'interoperability' issue of integrating various text-mining systems into existing curation workflows and (iii) the 'reusability' issue on the difficulty of applying trained systems to text genres that are not seen previously during development. We then describe related efforts within the text-mining community, with a special focus on the BioCreative series of challenge workshops. We believe that focusing on the near-term challenges identified in this work will amplify the opportunities afforded by the continued adoption of text-mining tools. Finally, in order to sustain the curation ecosystem and have text-mining systems adopted for practical benefits, we call for increased collaboration between text-mining researchers and various stakeholders, including researchers, publishers and biocurators.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The reduced precision achieved by S&B when applied to an unknown corpus reflects previous experiences from mining gene and protein symbols from text.
“The reduced precision achieved by S&B when applied to an unknown corpus reflects previous experiences from mining gene and protein symbols from text [54].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [54]
The Research community section (page unavailable) reports that performance in BioCreative Gene Normalization challenges dropped significantly on full texts compared with abstracts. The Reusability section also discusses difficulty generalizing text-mining systems to previously unseen text. These passages support the analogy, but they do not specifically report reduced precision or evaluate S&B on an unknown corpus. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Structure, function and diversity of the healthy human microbiome
The Human Microbiome Project Consortium (2012) · Nature · doi:10.1038/nature11234 · Cited by 11,726 · see citing works
Database resources of the National Center for Biotechnology Information: update
Wheeler, D. L. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkh073 · Cited by 10,691 · see citing works
UniProt: a hub for protein information
The UniProt Consortium (2015) · Nucleic Acids Research · doi:10.1093/nar/gku989 · Cited by 5,106 · see citing works
Database resources of the National Center for Biotechnology Information
Sayers, E. W., Beck, J., Bolton, E. E., Brister, J., Chan, J., Comeau, D. C., Connor, R., DiCuccio, M., Farrell, C., Feldgarden, M., et al. (2014) · Nucleic Acids Res. · doi:10.1093/nar/gku1130 · Cited by 2,584 · see citing works
Self-citationControl of brain development, function, and behavior by the microbiome
Sampson, T. R., & Mazmanian, S. K. (2015) · Cell Host & Microbe · doi:10.1016/j.chom.2015.04.011 · Cited by 1,093 · see citing works
Ferrucci, D. A., & Lally, A. (2004) · Natural Language Engineering · doi:10.1017/s1351324904003523 · Cited by 888 · see citing works
Critical assessment of methods of protein structure prediction (CASP) — round x
Moult, J., Fidelis, K. A., Kryshtafovych, A., Schwede, T. F., & Tramontano, A. L. (2013) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.24452 · Cited by 673 · see citing works
PubTator: a web-based text mining tool for assisting biocuration
Wei, C.-H., Kao, H. Y., & Lu, Z. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt441 · Cited by 567 · see citing works
Self-citationA large‐scale experiment to assess protein structure prediction methods
Moult, J., Pedersen, J. T., Judson, R., & Fidelis, K. (1995) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.340230303 · Cited by 555 · see citing works
High-fat-diet-mediated dysbiosis promotes intestinal carcinogenesis independently of obesity
Schulz, M. D., Atay, Ç., Heringer, J., Romrig, F., Schwitalla, S., Aydın, B., Ziegler, P. K., Varga, J., Reindl, W., Pommerenke, C., et al. (2014) · Nature · doi:10.1038/nature13398 · Cited by 452 · see citing works
Self-citationThe environment ontology: contextualising biological and biomedical entities
Buttigieg, P. L., Morrison, N., Smith, B., Mungall, C., & Lewis, S. (2013) · Journal of Biomedical Semantics · doi:10.1186/2041-1480-4-43 · Cited by 359 · see citing works
LINNAEUS: A species name identification system for biomedical literature
Gerner, M., Nenadić, G., & Bergman, C. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-85 · Cited by 337 · see citing works
The microbiome and regulation of mucosal immunity
McDermott, A. J., & Huffnagle, G. B. (2013) · Immunology · doi:10.1111/imm.12231 · Cited by 335 · see citing works
Overview of BioCreative II gene normalization
Morgan, A. A., Lu, Z., Wang, X., Cohen, A. M., Fluck, J., Ruch, P., Divoli, A., Fundel, K., Leaman, R., Hakenberg, J., et al. (2008) · Genome biology · doi:10.1186/gb-2008-9-s2-s3 · Cited by 327 · see citing works
Self-citationUsing the Metagenomics RAST Server (MG-RAST) for Analyzing Shotgun MetagenomesPreprint
Glass, E. M., Wilkening, J., Wilke, A., Antonopoulos, D. A., & Meyer, F. (2010) · Cold Spring Harbor Protocols · doi:10.1101/pdb.prot5368 · Cited by 304 · see citing works
TaggerOne: joint named entity recognition and normalization with semi-Markov Models
Leaman, R., & Lu, Z.-Y. (2016) · Bioinform. · doi:10.1093/bioinformatics/btw343 · Cited by 297 · see citing works
Self-citationCommunity challenges in biomedical text mining over 10 years: success, failure and the future
Huang, C., & Lu, Z. (2015) · Briefings in Bioinformatics · doi:10.1093/bib/bbv024 · Cited by 225 · see citing works
Self-citationThe Protein Information Resource: an integrated public resource of functional annotation of proteins
Wu, C. H. (2002) · Nucleic Acids Research · doi:10.1093/nar/30.1.35 · Cited by 212 · see citing works
VIROME: a standard operating procedure for analysis of viral metagenome sequences
Wommack, K. E., Bhavsar, J., Polson, S. W., Chen, J., Dumas, M., Srinivasiah, S., Furman, M., Jamindar, S., & Nasko, D. J. (2012) · Standards in Genomic Sciences · doi:10.4056/sigs.2945050 · Cited by 200 · see citing works
Self-citationThe gene normalization task in BioCreative III
Lu, Z., Kao, H., Wei, C.-H., Huang, M., Liu, J., Kuo, C.-J., Hsu, C., Tsai, R. T., Dai, H.-J., Okazaki, N., et al. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s8-s2 · Cited by 180 · see citing works
Self-citationBioC: a minimalist approach to interoperability for biomedical text processing
Comeau, D. C., Islamaj, R., Ciccarese, P., Cohen, K. B., Krallinger, M., Leitner, F., Lu, Z., Peng, Y., Rinaldi, F., Torii, M., et al. (2013) · Database · doi:10.1093/database/bat064 · Cited by 176 · see citing works
Self-citationCohen, K. B., Johnson, H. L., Verspoor, K., Roeder, C., & Hunter, L. E. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-492 · Cited by 174 · see citing works
Text mining for the biocuration workflow
Hirschman, L., Burns, G., Krallinger, M., Arighi, C. N., Cohen, K. B., Valencia, A., Wu, C. H., Chatr‐aryamontri, A., Dowell, K. G., Huala, E., et al. (2012) · Database · doi:10.1093/database/bas020 · Cited by 172 · see citing works
Self-citationPredicting gene function in a hierarchical context with an ensemble of classifiers
Guan, Y., Myers, C. L., Hess, D. C., Barutçuoğlu, Z., Caudy, A. A., & Troyanskaya, O. G. (2008) · Genome biology · doi:10.1186/gb-2008-9-s1-s3 · Cited by 149 · see citing works
Recommending MeSH terms for annotating biomedical articles
Huang, M., Aurélie Névéol, A., & Lu, Z. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2010-000055 · Cited by 149 · see citing works
Self-citationEurope PMC: a full-text literature database for the life sciences and platform for innovation
The Europe PMC Consortium (2014) · Nucleic Acids Research · doi:10.1093/nar/gku1061 · Cited by 145 · see citing works
Temporal variation selects for diet–microbe co-metabolic traits in the gut of Gorilla spp
Gomez, A. M., Rothman, J. M., Petrželková, K. J., Yeoman, C. J., Vlčková, K., Umaña, J. D., Carr, M., Modrý, D., Todd, A. F., Torralba, M., et al. (2015) · The ISME Journal · doi:10.1038/ismej.2015.146 · Cited by 114 · see citing works
Boué, S., Talikka, M., Westra, J. W., Hayes, W., Fabio, A. D., Park, J., Schlage, W. K., Sewer, A., Fields, B., Ansari, S., et al. (2015) · Database · doi:10.1093/database/bav030 · Cited by 103 · see citing works
Overview of the interactive task in BioCreative V
Wang, Q., Syed-Abdul, S., Almeida, L., Ananiadou, S., Balderas-Martínez, Y. I., Batista-Navarro, R., Campos, D., Chilton, L., Chou, H.-J., Contreras, G., et al. (2016) · Database · doi:10.1093/database/baw119 · Cited by 100 · see citing works
Self-citationMartin, F. P., Thomson, T. M., Sewer, A., Drubin, D. A., Mathis, C., Weisensee, D., Pratt, D. R., Hoeng, J., & Peitsch, M. C. (2012) · BMC Systems Biology · doi:10.1186/1752-0509-6-54 · Cited by 100 · see citing works
Mucosal Immunity and the Microbiome
Neish, A. S. (2014) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201306-161mg · Cited by 99 · see citing works
Catlett, N. L., Bargnesi, A. J., Ungerer, S., et al. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-340 · Cited by 96 · see citing works
Self-citationWei, C.-H., Harris, B. R., Li, D., Berardini, T., Huala, E., Kao, H.-Y., & Lu, Z. (2012) · Database · doi:10.1093/database/bas041 · Cited by 94 · see citing works
Self-citationReflect: augmented browsing for the life scientist
Pafilis, E., O’Donoghue, S. I., Jensen, L. J., Horn, H., Kuhn, M., Brown, N. P., & Schneider, R. (2009) · Nature Biotechnology · doi:10.1038/nbt0609-508 · Cited by 91 · see citing works
SR4GN: A Species Recognition Software Tool for Gene Normalization
Wei, C.-H., Kao, H., & Lu, Z. (2012) · PLoS ONE · doi:10.1371/journal.pone.0038460 · Cited by 87 · see citing works
Self-citationBioCreative III interactive task: an overview
Arighi, C. N., Roberts, P. M., Agarwal, S., Bhattacharya, S., Cesareni, G., Chatr‐aryamontri, A., Clematide, S., Gaudet, P., Giglio, M. G., Harrow, I., et al. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s8-s4 · Cited by 82 · see citing works
Self-citationBiocuration workflows and text mining: overview of the BioCreative 2012 Workshop Track II
Lu, Z., & Hirschman, L. (2012) · Database · doi:10.1093/database/bas043 · Cited by 73 · see citing works
Self-citationtagtog: interactive and text-mining-assisted annotation of gene mentions in PLOS full-text articles
Cejuela, J. M., McQuilton, P., Ponting, L., et al. (2014) · Database · doi:10.1093/database/bau033 · Cited by 71 · see citing works
An overview of the BioCreative 2012 Workshop Track III: interactive text mining task
Arighi, C. N., Carterette, B., Cohen, K. B., Krallinger, M., Wilbur, W. J., Fey, P., Dodson, R. J., Cooper, L., Slyke, C. E. V., Dahdul, W., et al. (2013) · Database · doi:10.1093/database/bas056 · Cited by 71 · see citing works
Self-citationGeneView: a comprehensive semantic search engine for PubMed
Thomas, P. E., Starlinger, J., Vowinkel, A., Arzt, S., & Leser, U. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks563 · Cited by 70 · see citing works
The Importance of Biological Databases in Biological Discovery
Baxevanis, A. D. (2006) · Current Protocols in Bioinformatics · doi:10.1002/0471250953.bi0101s13 · Cited by 63 · see citing works
The Importance of Biological Databases in Biological Discovery
Baxevanis, A. D., & Bateman, A. G. (2015) · Current Protocols in Bioinformatics · doi:10.1002/0471250953.bi0101s50 · Cited by 61 · see citing works
Thomson, T. M., Sewer, A., Martin, F. P., Belcastro, V., Frushour, B. P., Gebel, S., Park, J., Schlage, W. K., Talikka, M., Vasilyev, D. M., et al. (2013) · Toxicology and Applied Pharmacology · doi:10.1016/j.taap.2013.07.007 · Cited by 60 · see citing works
Auken, K. M. V., Jaffery, J., Chan, J., et al. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-228 · Cited by 58 · see citing works
Overview of the gene ontology task at BioCreative IV
Mao, Y., Auken, K. M. V., Li, D., Arighi, C. N., McQuilton, P., Hayman, G. T., Tweedie, S., Schaeffer, M. L., Laulederkind, S. J. F., Wang, S. J., et al. (2014) · Database · doi:10.1093/database/bau086 · Cited by 57 · see citing works
Self-citationGerner, M., Sarafraz, F., Bergman, C., & Nenadić, G. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts332 · Cited by 57 · see citing works
Evolution of orthologous tandemly arrayed gene clusters
Tremblay-Savard, O., Bertrand, D., & El-Mabrouk, N. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s9-s2 · Cited by 48 · see citing works
INTRINSIC EVALUATION OF TEXT MINING TOOLS MAY NOT PREDICT PERFORMANCE ON REALISTIC TASKS
Caporaso, J. G., DESHPANDE, N., Fink, J. L., et al. (2007) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789812776136_0061 · Cited by 44 · see citing works
Pafilis, E., Buttigieg, P. L., Ferrell, B. D., Pereira-Flores, E., Schnetzer, J., Arvanitidis, C., & Jensen, L. J. (2016) · Database · doi:10.1093/database/baw005 · Cited by 43 · see citing works
Beyond accuracy: creating interoperable and scalable text-mining web services
Wei, C.-H., Leaman, R., & Lu, Z. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btv760 · Cited by 40 · see citing works
Self-citationAuken, K. M. V., Fey, P., Berardini, T., et al. (2012) · Database · doi:10.1093/database/bas040 · Cited by 39 · see citing works
SourceData: a semantic platform for curating and searching figures
Liechti, R., George, N., Götz, L., El-Gebali, S., Chasapi, A., Crespo, I., Xénarios, I., & Lemberger, T. (2017) · Nature Methods · doi:10.1038/nmeth.4471 · Cited by 33 · see citing works
Self-citationWiegers, T. C., Davis, A. P., & Mattingly, C. J. (2014) · Database · doi:10.1093/database/bau050 · Cited by 29 · see citing works
Systematic Characterizations of Text Similarity in Full Text Biomedical Publications
Sun, Z. H., Errami, M., Long, T., et al. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012704 · Cited by 26 · see citing works
The eFIP system for text mining of protein interaction networks of phosphorylated proteins
Tudor, C. O., Arighi, C. N., Wang, Q., et al. (2012) · Database · doi:10.1093/database/bas044 · Cited by 26 · see citing works
Self-citationPafilis, E., Pletscher-Frankild, S., Schnetzer, J., et al. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv045 · Cited by 26 · see citing works
BioC interoperability track overview
Comeau, D. C., Batista-Navarro, R., Dai, H.-J., et al. (2014) · Database · doi:10.1093/database/bau053 · Cited by 19 · see citing works
Erratum: Temporal variation selects for diet–microbe co-metabolic traits in the gut of Gorilla spp
Gomez, A. M., Rothman, J. M., Petrželková, K. J., et al. (2016) · The ISME Journal · doi:10.1038/ismej.2015.252 · Cited by 8 · see citing works
Leveraging open source web resources to improve retrieval of low text content items
Singhal, A. (2014) · University of Minnesota Digital Conservancy (University of Minnesota) · Cited by 5 · see citing works
Self-citationLemberger, T. (2014) · Molecular Systems Biology · doi:10.1002/msb.201305048 · Cited by 4 · see citing works
Self-citationSourceData - a semantic platform for curating and searching figuresPreprint
Liechti, R., George, N., El-Gebali, S., et al. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/058529 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [16]Matis-Mitchell S. Roberts P. Tudor C.O. Arighi C.N. (2013) BioCreative IV interactive task. Proceedings of the Fourth BioCreative Challenge Evaluation Workshop, vol 1. Bethesda, MD, pp. 190–203.
- [19]Singhal A. Kasturi R. Sivakumar V. (2013) Leveraging web intelligence for finding interesting research datasets. Proceedings of the International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT). 2013 IEEE/WIC/ACM. IEEE, vol. 1, pp. 321– 328.
- [25]Fluck J. Madan S. Ansari S. (2014) BELIEF-a Semiautomatic Workflow for BEL Network Creation. Proceedings of the 6th International Symposium on Semantic Mining in Biomedicine (SMBM), University of Aveiro, Portugal, 109– 113.
- [42]Verspoor K. Baumgartner W. Jr Roeder C. (2009) Abstracting the types away from a UIMA type system. In C. Chiarcos, R. Eckhart de Castilho, M. Stede (Eds.), Form to Meaning: Processing Texts Automatically, pp. 249–256.
Cited by
Sorted by citations
PubTator central: automated concept annotation for biomedical full text articles
Wei, C.-H., Allot, A., Leaman, R., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz389 · Cited by 400 · see citing works
Self-citationLabbé, C., Grima, N., Gautier, T., Favier, B., & Byrne, J. A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0213266 · Cited by 55 · see citing works
Gobeill, J., Caucheteur, D., Michel, P.-A., Mottin, L., Pasche, E., & Ruch, P. (2020) · Nucleic Acids Research · doi:10.1093/nar/gkaa328 · Cited by 20 · see citing works
- Authors
- Singhal, A., Leaman, R., Catlett, N., Lemberger, T., McEntyre, J., Polson, S., Xenarios, I., Arighi, C., & Lu, Z.
- Affiliations
- National Institutes of Health (US)
- Selventa (United States) (US)
- European Molecular Biology Organization (DE)
- European Bioinformatics Institute (GB)
- Delaware Biotechnology Institute (US)
- Corresponding author
- Arighi, Cecilia
- Lu, Zhiyong
- ORCID
- Venue
- Database
- Publisher
- Oxford University Press (OUP)
- Published
- 2016
- Volume
- 2016
- Pages
- baw161
- ISSN
- 1758-0463 (ISSN-L)
- PMC
- PMC5199160
- OpenAlex
- W2564387948
- Cited by
- 48
- References
- 61
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 6 · 2023: 2 · 2022: 5 · 2021: 4 · 2020: 8 · 2019: 7 · 2018: 8 · 2017: 4
- FWCI
- 3.29
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- Locations
- Database (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Funders
- National Science Foundation (1356374)
- U.S. Department of Energy (DESC0010838)
- Wellcome Trust (098231/Z/12/Z)
- Australian Government
- Robert Bosch Stiftung
- SystemsX.ch
- National Institutes of Health (GM103446)
- Staatssekretariat für Bildung, Forschung und Innovation
- National Institute of General Medical Sciences (R13GM109648, P20GM103446)
- U.S. National Library of Medicine
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- biomedical text mining, biocuration, scalability, interoperability, reusability, system accuracy, curation workflows, stakeholder collaboration
- MeSH terms
- Biomedical Research, Data Mining, Data Curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
