Scaling up data curation using deep learning: An application to literature triage in genomic variation resources
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Manually curating biomedical knowledge from publications is necessary to build a knowledge based service that provides highly precise and organized information to users. The process of retrieving relevant publications for curation, which is also known as document triage, is usually carried out by querying and reading articles in PubMed. However, this query-based method often obtains unsatisfactory precision and recall on the retrieved results, and it is difficult to manually generate optimal queries. To address this, we propose a machine-learning assisted triage method. We collect previously curated publications from two databases UniProtKB/Swiss-Prot and the NHGRI-EBI GWAS Catalog, and used them as a gold-standard dataset for training deep learning models based on convolutional neural networks. We then use the trained models to classify and rank new publications for curation. For evaluation, we apply our method to the real-world manual curation process of UniProtKB/Swiss-Prot and the GWAS Catalog. We demonstrate that our machine-assisted triage method outperforms the current query-based triage methods, improves efficiency, and enriches curated content. Our method achieves a precision 1.81 and 2.99 times higher than that obtained by the current query-based triage methods of UniProtKB/Swiss-Prot and the GWAS Catalog, respectively, without compromising recall. In fact, our method retrieves many additional relevant publications that the query-based method of UniProtKB/Swiss-Prot could not find. As these results show, our machine learning-based method can make the triage process more efficient and is being implemented in production so that human curators can focus on more challenging tasks to improve the quality of knowledge bases.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Lee et al. (13), used the machine learning (Convolutional Neural Networks) text classification method for a literature classification step for the GWAS Catalog database curation, reducing by two-thirds the curators' workload of reading newly published papers, without compromising recall.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Several computational methods have been proposed as an effective alternative to keyword-based query methods for literature recommendation.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Finally, while recent studies including our own (13,25) have shown promise in improving classification performance with deep learning, such algorithms are not implemented due to the bottleneck of model efficiency in practical use.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
UniProt: the Universal Protein knowledgebase
Apweiler, R. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh131 · Cited by 7,498
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
UniProt: the universal protein knowledgebase
The UniProt Consortium (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1099 · Cited by 4,436
Convolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.5555/303568.303704 · Cited by 4,231
Multi-column deep neural networks for image classification
Ciresan, D., Meier, U., & Schmidhuber, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248110 · Cited by 3,720
A census of human cancer genes
Futreal, P. A., Coin, L., Marshall, M., Down, T., Hubbard, T., Wooster, R., Rahman, N., & Stratton, M. R. (2004) · Nature Reviews Cancer · doi:10.1038/nrc1299 · Cited by 3,184
Face recognition: a convolutional neural-network approach
Lawrence, S., Giles, C. L., Tsoi, A. C., & Back, A. D. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.554195 · Cited by 3,092
The new NHGRI-EBI Catalog of published genome-wide association studies (GWAS Catalog)
MacArthur, J., Bowler, E., Cerezo, M., Gil, L., Hall, P., Hastings, E., Junkins, H., McMahon, A., Milano, A., Morales, J., et al. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1133 · Cited by 2,161
Self-citationBias in algorithmic filtering and personalization
Bozdag, E. (2013) · Ethics and Information Technology · doi:10.1007/s10676-013-9321-6 · Cited by 649
Understanding intermediate layers using linear classifier probes
Alain, G., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1610.01644 · Cited by 581
PubTator: a web-based text mining tool for assisting biocuration
Wei, C.-H., Kao, H.-Y., & Lu, Z. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt441 · Cited by 566
Self-citationDistributional Semantics Resources for Biomedical Text Processing
Pyysalo, S., Ginter, F., Moen, H., Salakoski, T., & Ananiadou, S. (2013) · Research Explorer (The University of Manchester) · Cited by 548
Classifying Relations by Ranking with Convolutional Neural Networks
dos Santos, C., Xiang, B., & Zhou, B. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1061 · Cited by 498
How to Train good Word Embeddings for Biomedical NLP
Chiu, B., Crichton, G., Korhonen, A., & Pyysalo, S. (2016) · Proceedings of the 15th Workshop on Biomedical Natural Language Processing · doi:10.18653/v1/w16-2922 · Cited by 344
Singh, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7475259 · Cited by 252
Manual curation is not sufficient for annotation of genomic databases
Baumgartner, W. A., Cohen, K. B., Fox, L. M., Acquaah-Mensah, G., & Hunter, L. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm229 · Cited by 229
Zhao, Z., Yang, Z., Luo, L., Lin, H., & Wang, J. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw486 · Cited by 223
Classifying Relations by Ranking with Convolutional Neural NetworksPreprint
dos Santos, C. N., Xiang, B., & Zhou, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06580 · Cited by 189
Text mining for the biocuration workflow
Hirschman, L., Burns, G. A. P. C., Krallinger, M., Arighi, C., Cohen, K. B., Valencia, A., Wu, C. H., Chatr-Aryamontri, A., Dowell, K. G., Huala, E., et al. (2012) · Database · doi:10.1093/database/bas020 · Cited by 168
tmVar: a text mining approach for extracting sequence variants in biomedical literature
Wei, C.-H., Harris, B. R., Kao, H.-Y., & Lu, Z. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt156 · Cited by 168
Self-citationZhang, Y., & Wallace, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.03820 · Cited by 150
On expert curation and scalability: UniProtKB/Swiss-Prot as a case study
Poux, S., Arighi, C. N., Magrane, M., Bateman, A., Wei, C.-H., Lu, Z., Boutet, E., Bye-A-Jee, H., Famiglietti, M. L., Roechert, B., et al. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx439 · Cited by 135
Self-citationCuration of characterized glycoside hydrolases of Fungal origin
Murphy, C., Powlowski, J. B., Wu, M. H., Butler, G., & Tsang, A. (2011) · Database · doi:10.1093/database/bar020 · Cited by 115
Perspective: Sustaining the big-data ecosystem
Bourne, P. E., Lorsch, J. R., & Green, E. D. (2015) · Nature · doi:10.1038/527s16a · Cited by 112
Doughty, E., Kertesz-Farkas, A., Bodenreider, O., Thompson, G., Adadey, A., Peterson, T., & Kann, M. G. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btq667 · Cited by 105
Wei, C.-H., Phan, L., Feltz, J., Maiti, R., Hefferon, T., & Lu, Z. (2018) · Bioinformatics · doi:10.1093/bioinformatics/btx541 · Cited by 93
Self-citationBiocuration workflows and text mining: overview of the BioCreative 2012 Workshop Track II
Lu, Z., & Hirschman, L. (2012) · Database · doi:10.1093/database/bas043 · Cited by 73
Self-citationExpert curation in UniProtKB: a case study on dealing with conflicting and erroneous data
Poux, S., Magrane, M., Arighi, C. N., Bridge, A., O'Donovan, C., Laiho, K., & The UniProt Consortium (2014) · Database · doi:10.1093/database/bau016 · Cited by 62
Self-citationA SNPshot of PubMed to associate genetic variants with drugs, diseases, and adverse reactions
Hakenberg, J., Voronov, D., Nguyên, V. H., Liang, S., Anwar, S., Lumpkin, B., Leaman, R., Tari, L., & Baral, C. R. (2012) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2012.04.006 · Cited by 50
Challenge: Processing web texts for classifying job offers
Amato, F., Boselli, R., Cesarini, M., Mercorio, F., Mezzanzanica, M., Moscato, V., Persia, F., & Picariello, A. (2015) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icosc.2015.7050852 · Cited by 48
Deep learning of mutation-gene-drug relations from the literature
Lee, K., Kim, B., Choi, Y., Kim, S., Shin, W., Lee, S., Park, S., Kim, S., Tan, A. C., & Kang, J. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2029-1 · Cited by 46
Self-citationGenetic Variations and Diseases in UniProtKB/Swiss‐Prot: The Ins and Outs of Expert Manual Curation
Famiglietti, M. L., Estreicher, A., Gos, A., Bolleman, J., Géhant, S., Breuza, L., Bridge, A., Poux, S., Redaschi, N., Bougueleret, L., et al. (2014) · Human Mutation · doi:10.1002/humu.22594 · Cited by 45
Self-citationHybrid curation of gene–mutation relations combining automated extraction and crowdsourcing
Burger, J. D., Doughty, E., Khare, R., Wei, C.-H., Mishra, R., Aberdeen, J., Tresner-Kirsch, D., Wellner, B., Kann, M. G., Lu, Z., et al. (2014) · Database · doi:10.1093/database/bau094 · Cited by 44
Self-citationYepes, A. J., & Verspoor, K. M. (2014) · F1000Research · doi:10.12688/f1000research.3-18.v2 · Cited by 40
Machine Learning for Biomedical Literature Triage
Almeida, H., Meurs, M., Kosseim, L., Butler, G., & Tsang, A. (2014) · PLoS ONE · doi:10.1371/journal.pone.0115892 · Cited by 40
Van Auken, K., Fey, P., Berardini, T. Z., Dodson, R., Cooper, L., Li, D., Chan, J., Li, Y., Basu, S., Muller, H.-M., et al. (2012) · Database · doi:10.1093/database/bas040 · Cited by 39
Curation accuracy of model organism databases
Keseler, I. M., Skrzypek, M. S., Weerasinghe, D., Chen, A. Y., Fulcher, C., Li, G., Lemmer, K. C., Mladinich, K., Chow, E. D., Sherlock, G., et al. (2014) · Database · doi:10.1093/database/bau058 · Cited by 33
A hybrid human and machine resource curation pipeline for the Neuroscience Information Framework
Bandrowski, A. E., Cachat, J., Li, Y., Muller, H. M., Sternberg, P. W., Ciccarese, P., Clark, T., Marenco, L. N., Wang, R., Astakhov, V., et al. (2012) · Database · doi:10.1093/database/bas005 · Cited by 27
Yepes, A. J., & Verspoor, K. M. (2014) · F1000Research · doi:10.12688/f1000research.3-18.v1 · Cited by 24
Verspoor, K. M., Heo, G. E., Kang, K. Y., & Song, M. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0294-3 · Cited by 19
Opening the Black Box of Deep Neural Networks via Information
Shwartz-Ziv, R., & Tishby, N. (2017) · arXiv · doi:10.48550/arxiv.1703.00810
Research works printed but not linked: 1
- Information NCfB. Entrez Programming Utilities 2010. Available from: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/.
Cited by
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553
The Role of Machine Learning in the Understanding and Design of Materials
Moosavi, S. M., Jablonka, K. M., & Smit, B. (2020) · Journal of the American Chemical Society · doi:10.1021/jacs.0c09105 · Cited by 433
PubTator central: automated concept annotation for biomedical full text articles
Wei, C.-H., Allot, A., Leaman, R., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz389 · Cited by 400
“Third-party researchers have used PubTa-tor in a variety of use cases, including biocuration support (11–13), gene prioritization (14,15), genetic disease analysis (16), literature-based knowledge discovery (17,18) and downstream text mining (7,19–24).”
BackgroundMethodSelf-citationLitCovid: an open database of COVID-19 literature
Chen, Q., Allot, A., & Lu, Z. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkaa952 · Cited by 253
“…support human review, we developed an ensemble of automated document classifiers that include both traditional machine-learning models, such as support vector machines, using bag-of-words features and recent deep-learning models, such as convolutional neural networks, using word embeddings (12–14).”
BackgroundSelf-citationDeep learning in electron microscopy
Ede, J. M. (2021) · Machine Learning Science and Technology · doi:10.1088/2632-2153/abd614 · Cited by 143
Metadata
- Authors
- Lee, K., Famiglietti, M. L., McMahon, A., Wei, C.-H., MacArthur, J. A. L., Poux, S., Breuza, L., Bridge, A., Cunningham, F., Xenarios, I., & Lu, Z.
- Affiliations
- National Institutes of Health (US)
- SIB Swiss Institute of Bioinformatics (CH)
- European Bioinformatics Institute (GB)
- Corresponding author
- Lu, Zhiyong
- ORCID
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-08-13
- Volume
- 14
- Issue
- 8
- Pages
- e1006390
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC6107285
- OpenAlex
- W2887377515
- Cited by
- 47
- References
- 46
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 4 · 2023: 7 · 2022: 5 · 2021: 6 · 2020: 12 · 2019: 6 · 2018: 2
- FWCI
- 2.31
- Type
- journal article
- Language
- en
- Update notices
- new_version (2018-08-23)
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS Computational Biology (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- SERVAL (Université de Lausanne) (acceptedVersion, other-oa)
- IRIS (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. National Library of Medicine (Intramural Research Program)
- National Institutes of Health (UniProt 5U41HG007822-02)
- National Human Genome Research Institute (U41HG007823)
- European Molecular Biology Organization
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- document ranking, UniProtKB/Swiss-Prot, GWAS Catalog, convolutional neural networks, query-based retrieval, knowledge base construction
- MeSH terms
- Machine Learning, Deep Learning, Publications, Information Storage and Retrieval, Genomics, Databases, Genetic, Databases, Protein, Knowledge Bases, Data Curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.