The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Scaling up data curation using deep learning: An application to literature triage in genomic variation resources
Open accessAbstract
Manually curating biomedical knowledge from publications is necessary to build a knowledge based service that provides highly precise and organized information to users. The process of retrieving relevant publications for curation, which is also known as document triage, is usually carried out by querying and reading articles in PubMed. However, this query-based method often obtains unsatisfactory precision and recall on the retrieved results, and it is difficult to manually generate optimal queries. To address this, we propose a machine-learning assisted triage method. We collect previously curated publications from two databases UniProtKB/Swiss-Prot and the NHGRI-EBI GWAS Catalog, and used them as a gold-standard dataset for training deep learning models based on convolutional neural networks. We then use the trained models to classify and rank new publications for curation. For evaluation, we apply our method to the real-world manual curation process of UniProtKB/Swiss-Prot and the GWAS Catalog. We demonstrate that our machine-assisted triage method outperforms the current query-based triage methods, improves efficiency, and enriches curated content. Our method achieves a precision 1.81 and 2.99 times higher than that obtained by the current query-based triage methods of UniProtKB/Swiss-Prot and the GWAS Catalog, respectively, without compromising recall. In fact, our method retrieves many additional relevant publications that the query-based method of UniProtKB/Swiss-Prot could not find. As these results show, our machine learning-based method can make the triage process more efficient and is being implemented in production so that human curators can focus on more challenging tasks to improve the quality of knowledge bases.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Lee et al. (13), used the machine learning (Convolutional Neural Networks) text classification method for a literature classification step for the GWAS Catalog database curation, reducing by two-thirds the curators' workload of reading newly published papers, without compromising recall.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Several computational methods have been proposed as an effective alternative to keyword-based query methods for literature recommendation.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Finally, while recent studies including our own (13,25) have shown promise in improving classification performance with deep learning, such algorithms are not implemented due to the bottleneck of model efficiency in practical use.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
UniProt: the Universal Protein knowledgebase
Apweiler, R. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh131 · Cited by 7,498
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
UniProt: the universal protein knowledgebase
The UniProt Consortium (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1099 · Cited by 4,436
Convolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.5555/303568.303704 · Cited by 4,231
Multi-column deep neural networks for image classification
Ciresan, D., Meier, U., & Schmidhuber, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248110 · Cited by 3,720
Research works printed but not linked: 1
- Information NCfB. Entrez Programming Utilities 2010. Available from: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/.
Cited by
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553
The Role of Machine Learning in the Understanding and Design of Materials
Moosavi, S. M., Jablonka, K. M., & Smit, B. (2020) · Journal of the American Chemical Society · doi:10.1021/jacs.0c09105 · Cited by 433
PubTator central: automated concept annotation for biomedical full text articles
Wei, C.-H., Allot, A., Leaman, R., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz389 · Cited by 400
“Third-party researchers have used PubTa-tor in a variety of use cases, including biocuration support (11–13), gene prioritization (14,15), genetic disease analysis (16), literature-based knowledge discovery (17,18) and downstream text mining (7,19–24).”
BackgroundMethodSelf-citationLitCovid: an open database of COVID-19 literature
Chen, Q., Allot, A., & Lu, Z. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkaa952 · Cited by 253
“…support human review, we developed an ensemble of automated document classifiers that include both traditional machine-learning models, such as support vector machines, using bag-of-words features and recent deep-learning models, such as convolutional neural networks, using word embeddings (12–14).”
BackgroundSelf-citationDeep learning in electron microscopy
Ede, J. M. (2020) · Machine Learning Science and Technology · doi:10.1088/2632-2153/abd614 · Cited by 142
Metadata
- Authors
- Lee, K., Famiglietti, M. L., McMahon, A., Wei, C.-H., MacArthur, J. A. L., Poux, S., Breuza, L., Bridge, A., Cunningham, F., Xenarios, I., & Lu, Z.
- Affiliations
- National Institutes of Health (US)
- SIB Swiss Institute of Bioinformatics (CH)
- European Bioinformatics Institute (GB)
- Corresponding author
- Lu, Zhiyong
- ORCID
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-08-13
- Volume
- 14
- Issue
- 8
- Pages
- e1006390
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC6107285
- OpenAlex
- W2887377515
- Cited by
- 47
- References
- 46
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 4 · 2023: 7 · 2022: 5 · 2021: 6 · 2020: 12 · 2019: 6 · 2018: 2
- FWCI
- 2.31
- Type
- journal article
- Language
- en
- Update notices
- new_version (2018-08-23)
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS Computational Biology (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- SERVAL (Université de Lausanne) (acceptedVersion, other-oa)
- IRIS (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. National Library of Medicine (Intramural Research Program)
- National Institutes of Health (UniProt 5U41HG007822-02)
- National Human Genome Research Institute (U41HG007823)
- European Molecular Biology Organization
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- document ranking, UniProtKB/Swiss-Prot, GWAS Catalog, convolutional neural networks, query-based retrieval, knowledge base construction
- MeSH terms
- Machine Learning, Deep Learning, Publications, Information Storage and Retrieval, Genomics, Databases, Genetic, Databases, Protein, Knowledge Bases, Data Curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Scaling up data curation using deep learning: An application to literature triage in genomic variation resources. https://app.cochat.ai/papers/10.1371/journal.pcbi.1006390
- The paper
- Lee, K., Famiglietti, M. L., McMahon, A., Wei, C.-H., MacArthur, J. A. L., Poux, S., Breuza, L., Bridge, A., Cunningham, F., Xenarios, I., & Lu, Z. (2018). Scaling up data curation using deep learning: An application to literature triage in genomic variation resources. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1006390
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.