LitCovid: an open database of COVID-19 literature
Open accessAbstract
Since the outbreak of the current pandemic in 2020, there has been a rapid growth of published articles on COVID-19 and SARS-CoV-2, with about 10 000 new articles added each month. This is causing an increasingly serious information overload, making it difficult for scientists, healthcare professionals and the general public to remain up to date on the latest SARS-CoV-2 and COVID-19 research. Hence, we developed LitCovid (https://www.ncbi.nlm.nih.gov/research/coronavirus/), a curated literature hub, to track up-to-date scientific information in PubMed. LitCovid is updated daily with newly identified relevant articles organized into curated categories. To support manual curation, advanced machine-learning and deep-learning algorithms have been developed, evaluated and integrated into the curation workflow. To the best of our knowledge, LitCovid is the first-of-its-kind COVID-19-specific literature resource, with all of its collected articles and curated data freely available. Since its release, LitCovid has been widely used, with millions of accesses by users worldwide for various information needs, such as evidence synthesis, drug discovery and text and data mining, among others.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The evaluation on the LitCovid dataset shows the model achieved high performance in classifying COVID-19 related articles because many COVID-19 related articles have explicit terms and the dataset is large enough.
“This evaluation shows that the model achieved high performance in classifying COVID-19 related articles because many COVID-19 related articles have explicit terms to describe COVID-19 and SARS-COV-2, and the dataset is large enough for machine learning models.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [8]
The Results section, “Evaluation performance of automated methods in curation assistance” (page not provided), reports a document-classification F1-score of 0.99 on LitCovid evaluation data, with 63,998 training and 16,000 test articles. It does not attribute that performance to articles using explicit COVID-19 terms or state that the dataset was large enough to produce the result. (LLM, conf=0.88)
Experimental — accuracy not yet measuredLitSuggest identified about 30% more COVID-19 related articles than keyword searches.
“We found that LitSuggest identified about 30% more COVID-19 related articles than keyword searches, suggesting that machine learning models handle the high variability and ambiguity of terms describing COVID-19 and SARS-COV-2 in the literature (8) more effectively.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [8]
The Results section (‘Comparison of LitCovid versus keyword query’) reports that LitCovid contained 593 COVID-19 papers versus 457 retrieved by a PubMed keyword query—approximately 30% more. The Methods section (‘Document triage’) describes LitSuggest as a system that streamlines manual review and curation; the paper does not attribute the 30% difference to LitSuggest alone. (LLM, conf=0.88)
Experimental — accuracy not yet measuredExisting query-based search systems such as PubMed often return suboptimal results, missing some publications while returning some that are irrelevant.
“As such, existing query-based search systems such as PubMed and many others often return suboptimal results (8,12): missing some publications while returning some that are irrelevant.”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [8]
LitSuggest models achieved an F1-score of 0.99 on identifying COVID-19 related papers.
“The performance of LitSuggest models on identifying COVID-19 related papers (See evaluation section) was evaluated in detail, and the models achieved an F1-score of 0.99 (the training and testing datasets are also publicly available) (8).”
From LitSuggest: a web-based system for literature recommendation and curation using machine learning (2021) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
PubTator: a web-based text mining tool for assisting biocuration
Wei, C.-H., Kao, H. Y., & Lu, Z. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt441 · Cited by 567 · see citing works
Self-citationBioWordVec, improving biomedical word embeddings with subword information and MeSH
Zhang, Y., Chen, Q., Yang, Z., Lin, H., & Lu, Z. (2019) · Scientific Data · doi:10.1038/s41597-019-0055-0 · Cited by 455 · see citing works
Self-citationPubTator central: automated concept annotation for biomedical full text articles
Wei, C.-H., Allot, A., Leaman, R., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz389 · Cited by 400 · see citing works
Self-citationKeep up with the latest coronavirus research
Chen, Q., Allot, A., & Lu, Z. (2020) · Nature · doi:10.1038/d41586-020-00694-1 · Cited by 295 · see citing works
Self-citationPandemic publishing poses a new COVID-19 challenge
Palayew, A., Norgaard, O., Safreed-Harmon, K., Andersen, T. H., Rasmussen, L. N., & Lazarus, J. V. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0911-0 · Cited by 258 · see citing works
A real-time dashboard of clinical trials for COVID-19
Thorlund, K., Dron, L., Park, J., Hsu, G. G., Forrest, J. I., & Mills, E. J. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30086-8 · Cited by 227 · see citing works
Infectious Diseases Society of America Guidelines on the Diagnosis of COVID-19 (June 2020)
Hanson, K. E., Caliendo, A. M., Arias, C. A. D., Englund, J. A., Lee, M. J., Loeb, M. B., Patel, R., Alayli, A. E., Kalot, M. A., Falck-Ytter, Y. T., et al. (2020) · Clinical Infectious Diseases · doi:10.1093/cid/ciaa760 · Cited by 207 · see citing works
Best Match: New relevance search for PubMed
Fiorini, N., Canese, K., Starchenko, G., Kireev, E., Kim, W., Miller, V., Osipov, M., Kholodov, M., Ismagilov, R., Mohan, S., et al. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2005343 · Cited by 175 · see citing works
Self-citationGalmés, S., Serra, F., & Palou, A. (2020) · Nutrients · doi:10.3390/nu12092738 · Cited by 165 · see citing works
On expert curation and scalability: UniProtKB/Swiss-Prot as a case study
Poux, S., Arighi, C. N., Magrane, M., Bateman, A., Wei, C.-H., Lu, Z., Boutet, E., Bye-A-Jee, H., Famiglietti, M. L., Roechert, B., et al. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx439 · Cited by 135 · see citing works
Self-citationBiocuration: Distilling data into knowledge
International Society for Biocuration (2018) · PLoS Biology · doi:10.1371/journal.pbio.2002846 · Cited by 104 · see citing works
How user intelligence is improving PubMed
Fiorini, N., Leaman, R., Lipman, D. J., et al. (2018) · Nature Biotechnology · doi:10.1038/nbt.4267 · Cited by 79 · see citing works
Integrating the Probabilistic Models BM25/BM25F into LucenePreprint
Pérez-Iglesias, J., Pérez-Agüera, J. R., Fresno, V., et al. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0911.5046 · Cited by 71 · see citing works
Chen, Q., Lee, K., Yan, S., Kim, S., Wei, C.-H., & Lu, Z. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007617 · Cited by 62 · see citing works
Self-citationLitSense: making sense of biomedical literature at sentence level
Allot, A., Chen, Q., Kim, S., Alvarez, R. V., Comeau, D. C., Wilbur, W. J., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz289 · Cited by 57 · see citing works
Self-citationThe worldwide clinical trial research response to the COVID-19 pandemic - the first 100 days
Janiaud, P., Axfors, C., Hooft, J. V. ’., Saccilotto, R. T., Agarwal, A., Appenzeller‐Herzog, C., Contopoulos‐Ioannidis, D. G., Danchev, V., Dirnagl, U., Ewald, H., et al. (2020) · F1000Research · doi:10.12688/f1000research.26707.2 · Cited by 49 · see citing works
Lee, K., Famiglietti, M. L., McMahon, A., Wei, C.-H., MacArthur, J. A. L., Poux, S., Breuza, L., Bridge, A., Cunningham, F., Xenarios, I., et al. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006390 · Cited by 47 · see citing works
Self-citationThe worldwide clinical trial research response to the COVID-19 pandemic - the first 100 days
Janiaud, P., Axfors, C., Hooft, J. V. ’., Saccilotto, R. T., Agarwal, A., Appenzeller‐Herzog, C., Contopoulos‐Ioannidis, D. G., Danchev, V., Dirnagl, U., Ewald, H., et al. (2020) · F1000Research · doi:10.12688/f1000research.26707.1 · Cited by 44 · see citing works
Chen, Q., Britto, R., Erill, I., Jeffery, C. J., Liberzon, A., Magrane, M., Onami, J.-I., Robinson‐Rechavi, M., Sponarova, J., Zobel, J., et al. (2020) · Genomics Proteomics & Bioinformatics · doi:10.1016/j.gpb.2018.11.006 · Cited by 27 · see citing works
Self-citationCAiRE-COVID: A Question Answering and Multi-Document Summarization System for COVID-19 Research.
Su, D., Xu, Y., Yu, T., Siddique, F. B., Barezi, E. J., & Fung, P. (2020) · Europe PMC (PubMed Central) · Cited by 24 · see citing works
EVIDENCEMINER: Textual Evidence Discovery for Life Sciences
Wang, X., Guan, Y., Liu, W., et al. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.8 · Cited by 14 · see citing works
Leaman, R., Wei, C.-H., Allot, A., & Lu, Z. (2020) · PLoS Biology · doi:10.1371/journal.pbio.3000716 · Cited by 13 · see citing works
Self-citationChakraborti, S., & Srinivasan, N. (2020) · ChemRxiv · doi:10.26434/chemrxiv.12057846.v1 · Cited by 7 · see citing works
Navigating the landscape of COVID-19 research through literature analysis: A bird's eye viewPreprint
Yeganova, L., Islamaj, R., Chen, Q., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.03397 · Cited by 6 · see citing works
Self-citationChen, Q., Britto, R., Erill, I., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/788034 · Cited by 5 · see citing works
Self-citationBIP4COVID19: Releasing impact measures for articles relevant to COVID-19Preprint
Vergoulis, T., Kanellos, I., Chatzopoulos, S., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.04.11.037093 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [7]Wang L.L., Lo K., Chandrasekhar Y., Reas R., Yang J., Eide D., Funk K., Kinney R., Liu Z., Merrill W. CORD-19: the Covid-19 open research dataset. ACL NLP-COVID Workshop. 2020; 2020:
- [14]Chen Q., Peng Y., Lu Z. BioSentVec: creating sentence embeddings for biomedical texts. 2019 IEEE International Conference on Healthcare Informatics (ICHI). 2019; 1– 5.
Cited by
Sorted by citations
Database resources of the national center for biotechnology information
Sayers, E. W., Bolton, E. E., Brister, J. R., Canese, K., Chan, J., Comeau, D. C., Connor, R., Funk, K., Kelly, C., Kim, S., et al. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkab1112 · Cited by 2,812 · see citing works
Self-citationMore than 50 long-term effects of COVID-19: a systematic review and meta-analysis
Lopez-Leon, S., Wegman-Ostrosky, T., Perelman, C., Sepulveda, R., Rebolledo, P. A., Cuapio, A., & Villapol, S. (2021) · Scientific Reports · doi:10.1038/s41598-021-95565-8 · Cited by 2,757 · see citing works
LitSuggest: a web-based system for literature recommendation and curation using machine learning
Allot, A., Lee, K., Chen, Q., Luo, L., & Lu, Z. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab326 · Cited by 61 · see citing works
“The first corpus is derived from LitCovid (see Use Case 1), and contains publications related to the new coron-avirus pandemic, excluding publications mentioning previous coronaviruses, annotated as relevant or irrelevant.”
BackgroundMethodInfluentialSelf-citationByrne, J. A., Park, Y., Richardson, R. A. K., Pathmendra, P., Sun, M., & Stoeger, T. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac1139 · Cited by 46 · see citing works
“org, 107) that have never appeared in the title or abstract of any publication in the relevant literature (MeSH term on Glioblastoma, MeSH term on Alzheimer’s disease, LitCOVID (108)).”
- Authors
- Chen, Q., Allot, A., & Lu, Z.
- Affiliations
- National Institutes of Health (US)
- Corresponding author
- Lu, Zhiyong
- Venue
- Nucleic Acids Research
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-11-09
- Volume
- 49
- Issue
- D1
- Pages
- D1534-D1540
- ISSN
- 0305-1048 (ISSN-L)
- 1362-4962
- PMC
- PMC7778958
- OpenAlex
- W3105753785
- Cited by
- 253
- References
- 27
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 29 · 2023: 54 · 2022: 81 · 2021: 61 · 2020: 4
- FWCI
- 50.08
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- Locations
- Nucleic Acids Research (publishedVersion, public-domain)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Intramural Research Program of the National Library of Medicine
- National Institutes of Health
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Health Sciences › Medicine › Infectious Diseases › SARS-CoV-2 and COVID-19 Research
- Keywords
- COVID-19 pandemic, SARS-CoV-2, literature curation, information overload, deep learning, PubMed, evidence synthesis, drug discovery, text and data mining
- MeSH terms
- Machine Learning, COVID-19, SARS-CoV-2, Humans, Publications, Databases, Factual, Internet, PubMed, Data Mining, Pandemics, Data Curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
