Scaling up data curation using deep learning: An application to literature triage in genomic variation resources
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Manually curating biomedical knowledge from publications is necessary to build a knowledge based service that provides highly precise and organized information to users. The process of retrieving relevant publications for curation, which is also known as document triage, is usually carried out by querying and reading articles in PubMed. However, this query-based method often obtains unsatisfactory precision and recall on the retrieved results, and it is difficult to manually generate optimal queries. To address this, we propose a machine-learning assisted triage method. We collect previously curated publications from two databases UniProtKB/Swiss-Prot and the NHGRI-EBI GWAS Catalog, and used them as a gold-standard dataset for training deep learning models based on convolutional neural networks. We then use the trained models to classify and rank new publications for curation. For evaluation, we apply our method to the real-world manual curation process of UniProtKB/Swiss-Prot and the GWAS Catalog. We demonstrate that our machine-assisted triage method outperforms the current query-based triage methods, improves efficiency, and enriches curated content. Our method achieves a precision 1.81 and 2.99 times higher than that obtained by the current query-based triage methods of UniProtKB/Swiss-Prot and the GWAS Catalog, respectively, without compromising recall. In fact, our method retrieves many additional relevant publications that the query-based method of UniProtKB/Swiss-Prot could not find. As these results show, our machine learning-based method can make the triage process more efficient and is being implemented in production so that human curators can focus on more challenging tasks to improve the quality of knowledge bases.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Lee et al. (13), used the machine learning (Convolutional Neural Networks) text classification method for a literature classification step for the GWAS Catalog database curation, reducing by two-thirds the curators' workload of reading newly published papers, without compromising recall.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Several computational methods have been proposed as an effective alternative to keyword-based query methods for literature recommendation.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Finally, while recent studies including our own (13,25) have shown promise in improving classification performance with deep learning, such algorithms are not implemented due to the bottleneck of model efficiency in practical use.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
UniProt: the Universal Protein knowledgebase
Apweiler, R. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh131 · Cited by 7,498
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
UniProt: the universal protein knowledgebase
The UniProt Consortium (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1099 · Cited by 4,436
Convolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.5555/303568.303704 · Cited by 4,231
Multi-column deep neural networks for image classification
Ciresan, D., Meier, U., & Schmidhuber, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248110 · Cited by 3,720
Research works printed but not linked: 3
- Johnson R, Zhang T, editors. Deep Pyramid Convolutional Neural Networks for Text Categorization Proceedings of ACL; 2017.
- Lai S, Xu L, Liu K, Zhao J, editors. Recurrent Convolutional Neural Networks for Text Classification AAAI; 2015.
- Information NCfB. Entrez Programming Utilities 2010. Available from: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/.
Cited by
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553
The Role of Machine Learning in the Understanding and Design of Materials
Moosavi, S. M., Jablonka, K. M., & Smit, B. (2020) · Journal of the American Chemical Society · doi:10.1021/jacs.0c09105 · Cited by 433
PubTator central: automated concept annotation for biomedical full text articles
Wei, C.-H., Allot, A., Leaman, R., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz389 · Cited by 400
“Third-party researchers have used PubTa-tor in a variety of use cases, including biocuration support (11–13), gene prioritization (14,15), genetic disease analysis (16), literature-based knowledge discovery (17,18) and downstream text mining (7,19–24).”
BackgroundMethodSelf-citationLitCovid: an open database of COVID-19 literature
Chen, Q., Allot, A., & Lu, Z. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkaa952 · Cited by 253
“…support human review, we developed an ensemble of automated document classifiers that include both traditional machine-learning models, such as support vector machines, using bag-of-words features and recent deep-learning models, such as convolutional neural networks, using word embeddings (12–14).”
BackgroundSelf-citationDeep learning in electron microscopy
Ede, J. M. (2021) · Machine Learning Science and Technology · doi:10.1088/2632-2153/abd614 · Cited by 144
Deep learning in bioinformatics
Yousef, M., & Allmer, J. (2023) · TURKISH JOURNAL OF BIOLOGY · doi:10.55730/1300-0152.2671 · Cited by 71
“On the other hand, deep learning can learn higher-level features directly from the data (Lee et al., 2018).”
BackgroundChen, Q., Lee, K., Yan, S., Kim, S., Wei, C.-H., & Lu, Z. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007617 · Cited by 62
“For instance, the use of embeddings have shown promising performance in biomedical applications such as biomedical document classification [18], sentence retrieval [19], and question answering [20].”
BackgroundSelf-citationLitSuggest: a web-based system for literature recommendation and curation using machine learning
Allot, A., Lee, K., Chen, Q., Luo, L., & Lu, Z. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab326 · Cited by 61
Self-citationA roadmap for the functional annotation of protein families: a community perspective
de Crécy-lagard, V., Amorin de Hegedus, R., Arighi, C., Babor, J., Bateman, A., Blaby, I., Blaby-Haas, C., Bridge, A. J., Burley, S. K., Cleveland, S., et al. (2022) · Database · doi:10.1093/database/baac062 · Cited by 56
“Both PubTator and LitSuggest are already being used in production pipelines by many biocuration groups including Swiss-Prot (36), Rhea (37) and the NHGRI-EBI GWAS catalog (38).”
MethodSelf-citationPredicting Heart Failure Readmission from Clinical Notes Using Deep Learning
Liu, X., Chen, Y., Bae, J. P., Li, H., Johnston, J. A., & Sanger, T. M. (2019) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm47256.2019.8983095 · Cited by 59
“CNNs were typically used for image processing [14]; they have recently achieved good results in NLP [10],[15],[16],[17].”
BackgroundMethodTejani, A. S., Ng, Y. S., Xi, Y., Fielding, J. R., Browning, T. G., & Rayan, J. C. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.220007 · Cited by 53
Building deep learning models for evidence classification from the open access biomedical literature
Burns, G., Li, X., & Peng, N. (2019) · Database · doi:10.1093/database/baz034 · Cited by 33
“Recent work has demonstrated deep learning can accelerate triage in genome-wide association studies based on input data from PubMed and Pubtator services (15).”
BackgroundXu, W., Lu, Z., Wang, X., Cheung, M. H., Lin, M., Li, C., Dong, Y., Liang, C., & Chen, Y. (2020) · Nutrition and Diabetes · doi:10.1038/s41387-020-00134-z · Cited by 33
Self-citationSequencing-based genome-wide association studies reporting standards
McMahon, A., Lewis, E., Buniello, A., Cerezo, M., Hall, P., Sollis, E., Parkinson, H., Hindorff, L. A., Harris, L. W., & MacArthur, J. A. L. (2021) · Cell Genomics · doi:10.1016/j.xgen.2021.100005 · Cited by 32
“1% scored as “damaging” or “deleterious” by all five functional prediction algorithms; (3) PTVs included in (1) plus missense variants with MAF<0.”
BackgroundSelf-citationLee, K., Wei, C.-H., & Lu, Z. (2020) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa142 · Cited by 25
Self-citationGobeill, J., Caucheteur, D., Michel, P.-A., Mottin, L., Pasche, E., & Ruch, P. (2020) · Nucleic Acids Research · doi:10.1093/nar/gkaa328 · Cited by 20
“A couple of works recently focused on literature triage based on deep learning algorithms, based on convolutional neural networks (22,23), resulting on higher precisions in result sets.”
BackgroundArtificial Intelligence Transforming Post-Translational Modification Research
Kim, D. N., Yin, T., Zhang, T., Im, A. K., Cort, J., Rozum, J. C., Pollock, D. D., Qian, W., & Feng, S. (2024) · Bioengineering · doi:10.3390/bioengineering12010026 · Cited by 25
“DL is well suited for PTM applications (Figure 2) with its advantage of leveraging large quantities of data effectively [62,63].”
Long, E., Wan, P., Chen, Q., Lu, Z., & Choi, J. (2023) · Cell Genomics · doi:10.1016/j.xgen.2023.100320 · Cited by 23
“99 times higher than current query-based approach in curating the most comprehensive GWAS database, the NHGRI-EBI GWAS Catalog.(43) This deep-learning approach can efficiently identify relevant literature and thus significantly reduced the number of papers that required review in the manual curation process.”
BackgroundSelf-citationMELINDA: A Multimodal Dataset for Biomedical Experiment Method Classification
Wu, T.-L., Singh, S., Paul, S., Burns, G., & Peng, N. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17657 · Cited by 23
Moon, J.-H., Posada-Quintero, H. F., & Chon, K. H. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118930 · Cited by 18
Harerimana, G., Kim, G. I., Kim, J. W., & Jang, B. (2023) · IEEE Access · doi:10.1109/access.2023.3320179 · Cited by 10
Jiang, X., Li, P., Kadin, J. A., Blake, J. A., Ringwald, M., & Shatkay, H. (2020) · Database · doi:10.1093/database/baaa024 · Cited by 16
“Hence, automated biomedical document classification is a topic of much recent research (1,5,15,20).”
BackgroundUPCLASS: a deep learning-based classifier for UniProtKB entry publications
Teodoro, D., Knafou, J., Naderi, N., Pasche, E., Gobeill, J., Arighi, C. N., & Ruch, P. (2020) · Database · doi:10.1093/database/baaa026 · Cited by 14
“used a convolutional neural network (CNN) model, supported by word2vec representations, in the triage phase of genomic variation resources, outperforming the precision of SVM models by up to 3% [21].”
MethodBritan, A., Cusin, I., Hinard, V., Mottin, L., Pasche, E., Gobeill, J., de Laval, V. R., Gleizes, A., Teixeira, D., Michel, P.-A., et al. (2018) · Database · doi:10.1093/database/bay129 · Cited by 10
“Our work (29) and that of other groups (43) indicate that machine-learning assisted triage method could improve the document retrieval process, outperforming manual curators at least for specific tasks.”
BackgroundKnowledge-based Conversational Search
Vakulenko, S. (2019) · arXiv (Cornell University) · doi:10.34726/hss.2019.72687 · Cited by 14
Optimising biomedical relationship extraction with BioBERTPreprint
Giles, O., Karlsson, A., Masiala, S., White, S. R., Cesareni, G., Perfetto, L., Mullen, J., Hughes, M. J., Harland, L., & MALONE, J. M. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.09.01.277277 · Cited by 11
“relationships from text therefore warrants research [7].”
Knafou, J., Haas, Q., Borissov, N., Counotte, M. J., Low, N. N., Imeri, H., Ipekci, A. M., Buitrago‐García, D., Heron, L., Amini, P., et al. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02247-9 · Cited by 15
Using deep learning to identify translational research in genomic medicine beyond bench to bedside
Hsu, Y.-Y., Clyne, M., Wei, C.-H., Khoury, M. J., & Lu, Z. (2019) · Database · doi:10.1093/database/baz010 · Cited by 6
“(13) employed the CNN to identify publications that are relevant for variant curation.”
MethodSelf-citationGabella, C., Duvaud, S., & Durinx, C. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab478 · Cited by 7
“The UniProtKB/Swiss-Prot database contains >500 000 protein sequences curated by experts with the support of advanced machine learning techniques [9].”
Assigning species information to corresponding genes by a sequence labeling framework
Luo, L., Wei, C.-H., Lai, P., Chen, Q., Islamaj, R., & Lu, Z. (2022) · Database · doi:10.1093/database/baac090 · Cited by 6
“Many text-mining studies [4-6] rely highly on the automatic extraction of gene names in the text.”
BackgroundSelf-citationHarnessing deep learning in bioinformatics- opportunities, challenges, and ethical considerations
Rani, N., Chaudhary, P., Bhoomika, B., Kumar, U., Nikhil, Khatana, K., Kumar, N., Mavi, A. K., Mathur, M., & Singh, S. (2025) · Vita Scientia · doi:10.65116/vitascientia.v1i1.rev.01 · Cited by 3
Gabud, R., Lapitan, P. G., Mariano, V. Y., Mendoza, E., Pampolina, N. M., Clariño, M. A. A. D., & Batista-Navarro, R. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1371411 · Cited by 3
Arabi-Jeshvaghani, F., Javadi‐Zarnaghi, F., Löchel, H. F., Martin, R., & Heider, D. (2023) · Infection · doi:10.1007/s15010-023-02100-0 · Cited by 3
“The overwhelming sheer amount of published data in recent years has driven these challenges up dramatically [43].”
BackgroundMaraver, P., Tecuatl, C., & Ascoli, G. A. (2023) · Brain Informatics · doi:10.1186/s40708-023-00202-x · Cited by 2
“[ 12] used title, abstract, journal information, and publication type over three data sets with good precision ( > 0.91 ) and recall ( > 0.93 ) on the balanced data sets.”
MethodInfluentialPSB 2019 Workshop on Text Mining and Visualization for Precision Medicine
Gonzalez‐Hernandez, G., Lu, Z., Leaman, R., Weissenbacher, D., Boland, M. R., Chen, Y., Du, J., Fluck, J., Greene, C. S., Holmes, J., et al. (2018) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789813279827_0042 · Cited by 1
Self-citationKnafou, J., Haas, Q., Borissov, N., Counotte, M. J., Low, N. N., Imeri, H., Ipekci, A. M., Buitrago‐García, D., Heron, L., Amini, P., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.18.524571 · Cited by 1
Literature Triage on Genomic Variation Publications by Knowledge-enhanced Multi-channel CNNPreprint
Lv, C., Lu, Q., & Zhang, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04044
“proposed a CNN-based model for literature triage in genomic variation resources [12].”
BackgroundMethodInfluentialUmeton, R., Bellucci, G., Bigi, R., Romano, S., Buscarinu, M. C., Reniè, R., Rinaldi, V., Umeton, R. P., Morena, E., Romano, C., et al. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.03.12.434773
Implementation of Literature Database for Human Health Risk Assessment of Electromagnetic Fields
Lee, S.-W., Kim, N., Choi, H., & Kim, E. (2021) · Journal of Digital Contents Society · doi:10.9728/dcs.2021.22.5.817
Deep Semantic Biomedical Document Representation Method
Gabsi, I., Kammoun, H., Souidi, D., & Amous, I. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-4985-0_31
Challenges in Curating Real-Time Data During a Crisis: The Case of the COVID-19 Pandemic in Alberta
Tchoh, B. K., Barnard, S., Budac, R., Ingram, K., Chartier, R., Baumgart, D. C., & Rockwell, G. (2025) · Digital Studies / Le champ numérique · doi:10.16995/dscn.15498
Umeton, R., Bellucci, G., Bigi, R., Romano, S., Buscarinu, M. C., Reniè, R., Rinaldi, V., Umeton, R. P., Morena, E., Romano, C., et al. (2021) · The Journal of the American Medical Association (JAMA) Network (American Medical Association)
Gu, S., Qiao, G., & Zhu, J. (2026) · Future Foods · doi:10.1016/j.fufo.2026.100990
SIB Swiss Institute of Bioinformatics (2025) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.14937183
FAIRsharing record for: LitCovid
FAIRsharing Team (2022) · FAIRsharing.org · doi:10.25504/fairsharing.a01ca1
How does SIB support open software and databases?
Zahn‐Zabal, M., & Duvaud, S. (2025) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.17184970
Bénichou, L., Rebholz‐Schuhmann, D., Waterhouse, R. M., Agosti, D., & Naderi, N. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.19679289
Metadata
- Authors
- Lee, K., Famiglietti, M. L., McMahon, A., Wei, C.-H., MacArthur, J. A. L., Poux, S., Breuza, L., Bridge, A., Cunningham, F., Xenarios, I., & Lu, Z.
- Affiliations
- National Institutes of Health (US)
- SIB Swiss Institute of Bioinformatics (CH)
- European Bioinformatics Institute (GB)
- Corresponding author
- Lu, Zhiyong
- ORCID
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-08-13
- Volume
- 14
- Issue
- 8
- Pages
- e1006390
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC6107285
- OpenAlex
- W2887377515
- Cited by
- 47
- References
- 46
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 4 · 2023: 7 · 2022: 5 · 2021: 6 · 2020: 12 · 2019: 6 · 2018: 2
- FWCI
- 2.31
- Type
- journal article
- Language
- en
- Update notices
- new_version (2018-08-23)
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS Computational Biology (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- SERVAL (Université de Lausanne) (acceptedVersion, other-oa)
- IRIS (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. National Library of Medicine (Intramural Research Program)
- National Institutes of Health (UniProt 5U41HG007822-02)
- National Human Genome Research Institute (U41HG007823)
- European Molecular Biology Organization
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- document ranking, UniProtKB/Swiss-Prot, GWAS Catalog, convolutional neural networks, query-based retrieval, knowledge base construction
- MeSH terms
- Machine Learning, Deep Learning, Publications, Information Storage and Retrieval, Genomics, Databases, Genetic, Databases, Protein, Knowledge Bases, Data Curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.