CoChat
Sign Up
Cited by 62Recommender Systems and TechniquesInformation Retrieval and Search BehaviorBiomedical Text Mining and Ontologies

LitSuggest: a web-based system for literature recommendation and curation using machine learning

Open access
Allot, Alexis, Lee, Kyubum, Chen, Qingyu, Luo, Ling, Lu, Zhiyong·2021·Nucleic Acids Research
DOI 10.1093/nar/gkab326PMC PMC8262723
Open this paper in CoChat — free accountOpen PDF

Abstract

Searching and reading relevant literature is a routine practice in biomedical research. However, it is challenging for a user to design optimal search queries using all the keywords related to a given topic. As such, existing search systems such as PubMed often return suboptimal results. Several computational methods have been proposed as an effective alternative to keyword-based query methods for literature recommendation. However, those methods require specialized knowledge in machine learning and natural language processing, which can make them difficult for biologists to utilize. In this paper, we propose LitSuggest, a web server that provides an all-in-one literature recommendation and curation service to help biomedical researchers stay up to date with scientific literature. LitSuggest combines advanced machine learning techniques for suggesting relevant PubMed articles with high accuracy. In addition to innovative text-processing methods, LitSuggest offers multiple advantages over existing tools. First, LitSuggest allows users to curate, organize, and download classification results in a single interface. Second, users can easily fine-tune LitSuggest results by updating the training corpus. Third, results can be readily shared, enabling collaborative analysis and curation of scientific literature. Finally, LitSuggest provides an automated personalized weekly digest of newly published articles for each user's project. LitSuggest is publicly available at https://www.ncbi.nlm.nih.gov/research/litsuggest.

Integrity signals

Registry

Crossref · 10.1093/nar/gkab326

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

17 of 20 cited sources checked · 7 supported · 0 flagged · accuracy not yet measured

  • Findingcitation

    LitSuggest models achieved an F1-score of 0.99 on identifying COVID-19 related papers.

    “The performance of LitSuggest models on identifying COVID-19 related papers (See evaluation section) was evaluated in detail, and the models achieved an F1-score of 0.99 (the training and testing datasets are also publicly available) (8).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [8] Chen, 2020, LitCovid: an open database of COVID-19 literature, Nucleic Acids Res. source
  • Interpretationcitation

    Each celery worker uses the stacking ensemble learning method to provide the best document classification results, suitable for variable number and nature of training datasets.

    “Each celery worker uses the 'stacking ensemble learning method' to provide the best document classification results, suitable for the variable number of publications and variable nature of training datasets submitted by our users, which might not be suitable for one specific classifier.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [20] Chen, 2020, Deep learning with sentence embeddings pre-trained on biomedical corpora improves the performance of finding similar sentences in electronic medical records, BMC Med. Inform. Decis. Mak. source

      The Methods section describes a stacking ensemble, and the Results section reports that it performed best on the paper’s clinical sentence-similarity task. The paper does not evaluate document classification, Celery workers, or suitability for users’ varying publication counts and training datasets. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Findingown data

    Compared to BioReader, LitSuggest shows superior classification performance on the hcc r11 corpus.

    “Compared to BioReader in terms of classification performance (Table 3), LitSuggest shows superior results on the hcc r11 corpus.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [18] Simon, 2019, BioReader: a text mining tool for performing classification of biomedical literature, BMC Bioinformatics source

      The Results and discussion section reports BioReader’s classification performance against MedlineRanker, but does not compare it with LitSuggest on the hcc r11 corpus. Searches for LitSuggest and hcc r11 found no passage addressing that comparison in our copy. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Findingown data

    LitSuggest identified about 30% more COVID-19 related articles than keyword searches.

    “We found that LitSuggest identified about 30% more COVID-19 related articles than keyword searches, suggesting that machine learning models handle the high variability and ambiguity of terms describing COVID-19 and SARS-COV-2 in the literature (8) more effectively.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [8] Chen, 2020, LitCovid: an open database of COVID-19 literature, Nucleic Acids Res. source

      The Results section (‘Comparison of LitCovid versus keyword query’) reports that LitCovid contained 593 COVID-19 papers versus 457 retrieved by a PubMed keyword query—approximately 30% more. The Methods section (‘Document triage’) describes LitSuggest as a system that streamlines manual review and curation; the paper does not attribute the 30% difference to LitSuggest alone. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Findingreasoning

    LitSuggest offers more useful features compared to MedlineRanker and BioReader, as shown in Table 2.

    “In addition to generating high-quality classification results, LitSuggest also offers more useful features when compared to similar tools such as MedlineRanker (17) and BioReader (18), as shown in Table 2.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [17] Fontaine, 2009, MedlineRanker: flexible ranking of biomedical literature, Nucleic Acids Res. source
    • cites [18] Simon, 2019, BioReader: a text mining tool for performing classification of biomedical literature, BMC Bioinformatics source
  • Classified as probably not a substantial claim. 18
  • May combine two claims. 15
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

    Premisecitation

    Lee et al. used convolutional neural networks for literature classification in GWAS Catalog curation, reducing curators' workload by two-thirds without compromising recall.

    “For example, Lee et al. (13), used the machine learning (Convolutional Neural Networks) text classification method for a literature classification step for the GWAS Catalog database curation, reducing by two-thirds the curators' workload of reading newly published papers, without compromising recall.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [13] Lee, 2018, Scaling up data curation using deep learning: An application to literature triage in genomic variation resources, PLoS Comput. Biol. source

      The Methods describe CNN classifiers trained for GWAS Catalog literature triage, and the Results (Utility assessment; page unavailable) say the method returned one-third as many papers for curators to review. But it missed one curatable publication found by the query-based method. The Discussion says recall was not *significantly* compromised, rather than uncompromised. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingreasoning

    The evaluation on the LitCovid dataset shows the model achieved high performance in classifying COVID-19 related articles because many COVID-19 related articles have explicit terms and the dataset is large enough.

    “This evaluation shows that the model achieved high performance in classifying COVID-19 related articles because many COVID-19 related articles have explicit terms to describe COVID-19 and SARS-COV-2, and the dataset is large enough for machine learning models.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [8] Chen, 2020, LitCovid: an open database of COVID-19 literature, Nucleic Acids Res. source

      The Results section, “Evaluation performance of automated methods in curation assistance” (page not provided), reports a document-classification F1-score of 0.99 on LitCovid evaluation data, with 63,998 training and 16,000 test articles. It does not attribute that performance to articles using explicit COVID-19 terms or state that the dataset was large enough to produce the result. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    The stacking ensemble method has achieved the best performance in previous biomedical text mining open challenge tasks and is known to show stable results in text classification.

    “This method has achieved the best performance in previous biomedical text mining open challenge tasks (20) and is known to show stable results in text classification methods (21,22).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [20] Chen, 2020, Deep learning with sentence embeddings pre-trained on biomedical corpora improves the performance of finding similar sentences in electronic medical records, BMC Med. Inform. Decis. Mak. source
    • cites [21] Xia, 2011, Ensemble of feature sets and classification algorithms for sentiment classification, Inform. Sci. source
    • cites [22] Fung, 2006 source

  • Classified as probably not a substantial claim.

    Premisecitation

    Several computational methods have been proposed as an effective alternative to keyword-based query methods for literature recommendation.

    “Several computational methods have been proposed as an effective alternative to keyword-based query methods for literature recommendation.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [12] Poux, 2017, On expert curation and scalability: UniProtKB/Swiss-Prot as a case study, Bioinformatics source

      Section 3.1 describes several text-mining approaches integrated into PubTator to help curators identify and triage relevant articles, and Section 4.4 says text-mining tools help with literature triage. The paper does not establish that these methods are an effective alternative to keyword-based queries for literature recommendation; it does not make that comparison. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [13] Lee, 2018, Scaling up data curation using deep learning: An application to literature triage in genomic variation resources, PLoS Comput. Biol. source

      The Introduction describes keyword-based PubMed queries and a prior machine-learning triage method, mycoSORT. The Abstract and Discussion report that this paper’s deep-learning method outperformed query-based triage for two genomic-variation resources. The paper does not establish that several computational methods are effective alternatives to keyword-based queries for literature recommendation more broadly. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [14] Hsu, 2019, Using deep learning to identify translational research in genomic medicine beyond bench to bedside, Database source

      The Introduction and Utility assessment (pages not specified) describe computational methods for classifying and prioritizing relevant articles. They do not establish these methods as alternatives to keyword-based queries: in the assessed workflow, PubMed queries still retrieve articles, and the classifier ranks the results for review. The paper also concerns genomic-research curation rather than literature recommendation generally. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    LitSuggest is a machine learning-based document recommendation and management system for biomedical literature.

    “LitSuggest is a machine learning-based document recommendation and management system for biomedical literature.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Classification accuracy is often limited by the absence of abstracts, forcing classification based only on title and bibliographic fields.

    “Classification accuracy is also often limited by the fact that for some publications the abstract is not available, and the classification must be based only on the title and related bibliographic fields.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingreasoning

    Classification results can be readily shared, enabling collaborative analysis and curation of scientific literature.

    “Third, results can be readily shared, enabling collaborative analysis and curation of scientific literature.”
    • Also stated in: § INTRODUCTION · plain
      “Third, classification results can be readily shared through a URL, enabling collaborative curation of scientific literature.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Managing, curating, annotating, storing, and sharing retrieved publications is often a cumbersome task for researchers.

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingnone given

    LitSuggest automatically scans every week for new literature relevant to the project's model and adds the result as a new job on the project page.

    “When the user checks the appropriate checkbox, LitSuggest automatically scans every week for new literature relevant to the project's model, then adds the result as a new job on the project page.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    To date, 3132 articles have been curated in the liver cancer project.

    “To date, we have curated 3132 articles.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Training may take between a few seconds to several hours depending on the size of the training set.

    “Depending on the size of the training set, training may take between a few seconds to several hours.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    LitSuggest allows users to curate, organize, and download classification results in a single interface.

    “First, LitSuggest allows users to curate, organize, and download classification results in a single interface.”
    • Also stated in: § INTRODUCTION · plain
      “First, LitSuggest allows users to easily curate classified publications, organize classification tasks into jobs and projects, and download classification results with related annotations.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Managing, curating, annotating, storing, and sharing retrieved publications is often a cumbersome task for researchers.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Users can share any of their projects or classification jobs with other users by sharing the URL.

    “Users can easily share any of their projects or classification jobs with other users by sharing the URL.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    LitSuggest currently only operates on abstracts and does not support full texts, which might impact recall if pertinent information is only in the full text.

    “Second, Lit-Suggest currently only operates on abstracts and does not yet support full texts, which might impact the recall if pertinent information is only present in the full text.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    MedlineRanker does not support a negative training set, and so was not evaluated.

    “MedlineRanker does not support a negative training set, and so was not evaluated.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingnone given

    LitSuggest has been used as a literature curation tool in cooperation with the NCI team for liver cancer, epidemiology, and health disparities.

    “LitSuggest has been used as a literature curation tool in cooperation with the NCI team.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    Users can train a model on a set of publications relevant for a topic and then apply this learned model to classify and rank new articles.

    “Users can train a model on a set of publications relevant for that topic and then apply this learned model to classify and rank new articles.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    For the hcc r11 corpus, most false positive articles do not discuss health disparities, and false negatives describe health disparities but are misclassified as negatives.

    “For the second corpus (hcc r11), according to the error analysis of NCI (National Cancer Institute) team, most false positive articles do not discuss health disparities, and false negatives describe health disparities but are misclassified as negatives.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    If the number of negative examples supplied by the user is not enough compared to the number of positive examples, additional negative PMIDs will be generated.

    “If the number of negative examples supplied by the user is not enough compared to the number of positive examples, additional negative PMIDs will be generated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    LitSuggest is publicly available at https://www.ncbi.nlm.nih.gov/research/litsuggest.

    “LitSuggest is publicly available at https://www.ncbi.nlm.nih.gov/research/litsuggest.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Collaborators can see the results of classification jobs and curate relevant publications, but cannot edit project properties, retrain the model, or start new classification jobs.

    “Their collaborators will then be able to see the results of classification jobs and curate relevant publications, but they will not be able to edit the properties of the projects, retrain the model or start new classification jobs within the project.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    If no negative examples are supplied by the user, negative publications will be automatically generated.

    “If no negative examples are supplied by the user, negative publications will be automatically generated.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The major difference between LitSuggest and other tools is that LitSuggest allows storing and reusing both training and classification results, offers powerful curation and collaboration features, and an automated digest.

    “The major difference between LitSuggest and the other tools is that LitSuggest allows to store and reuse both training and classification results, offers powerful curation and collaboration features as well as an automated digest.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    LitSuggest achieved precision 0.995, recall 0.993, F1 0.994, AUC 0.994 on LitCovid corpus.

    “LitCovid | 31 998 + 32 000 | 8000 + 8000 | 0.995 | 0.993 | 0.994 | 0.994”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    LitSuggest achieved precision 0.831, recall 0.758, F1 0.793, AUC 0.837 on hcc r11 corpus.

    “hcc r11 | 573 + 1099 | 149 + 270 | 0.831 | 0.758 | 0.793 | 0.837”
    • Also stated in: § CONCLUSION · plain
      “LitSuggest | 0.831 | 0.758 | 0.793 | 0.837”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    BioReader achieved precision 0.824, recall 0.614, F1 0.704, AUC 0.770 on hcc r11 corpus.

    “BioReader | 0.824 | 0.614 | 0.704 | 0.770”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The architecture allows LitSuggest to be easily scaled to support additional workload by adding new VMs with additional celery workers.

    “This architecture allows LitSuggest to be easily scaled to support additional workload: new VMs with additional celery workers can be added easily as needed.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    LitSuggest provides a rich environment to classify relevant articles and review annotations, which speeds up the literature curation process.

    “LitSuggest provides a rich environment to classify relevant articles and review annotations, which speeds up the literature curation process.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The performance on the hcc r11 corpus is lower than on the LitCovid corpus because the hcc corpus is small and detecting health disparities is challenging with limited training data.

    “The performance is lower than the Lit-Covid corpus because the size of the hcc corpus is small, and developing a high-performance machine learning algorithm to detect health disparities is a challenge with limited training data, because the health disparity studies encompass many subjects (including gender, age, race, ethnicity, education, income, etc.).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The score distributions demonstrate how confidently the models classify the articles (e.g., a score of 0.9 suggests the model is very confident).

    “The score distributions demonstrate how confidently the models classify the articles (e.g. a score of 0.9 suggests the model is very confident when classifying the article), thereby facilitating user decisions (for example, whether re-training a model is needed).”
    Quote in the paperNot a substantial claim
29 of 29

  • The paper relies on this without support.

    It is challenging for a user to design optimal search queries using all the keywords related to a given topic.

    “However, it is challenging for a user to design optimal search queries using all the keywords related to a given topic.”

  • The paper relies on this without support.

    Machine learning and natural language processing methods require specialized knowledge, making them difficult for biologists to utilize.

    “However, those methods require specialized knowledge in machine learning and natural language processing, which can make it difficult for biologists to utilize.”

  • The paper relies on this without support.

    Managing, curating, annotating, storing, and sharing retrieved publications is often a cumbersome task for researchers.

    “In addition, retrieving relevant publications is only the first part of the literature review pipeline, hence managing, curating, annotating, storing, and sharing the list of retrieved publications is often a cumbersome task for researchers.”

  • The paper relies on this without support.

    There is a strong desire for more user-friendly systems with additional features such as reusing models, curating and sharing classification results.

    “A handful of web applications have been developed to address this issue, such as MedlineRanker (17) and BioReader (18), but there is a strong desire for more user-friendly systems with additional features such as reusing models, curating and sharing classification results.”
4 of 4

Cites

Flagged first, then by citations

2 of the 30 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Allot, A., Lee, K., Chen, Q., Luo, L., & Lu, Z.
Affiliations
National Institutes of Health (US)
Corresponding author
Lu, Zhiyong
ORCID
Venue
Nucleic Acids Research
Publisher
Oxford University Press (OUP)
Published
2021-05-05
Volume
49
Issue
W1
Pages
W352-W358
ISSN
  • 0305-1048 (ISSN-L)
  • 1362-4962
DOI
10.1093/nar/gkab326
PMC
PMC8262723
OpenAlex
W3157132663
Cited by
61
References
30
Citations by year
2026: 7 · 2025: 8 · 2024: 19 · 2023: 8 · 2022: 15 · 2021: 4
FWCI
10.40
Type
journal article
Language
en
Open access
Licence
public-domain
Locations
  • Nucleic Acids Research (publishedVersion, public-domain)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
National Institutes of Health
Topics
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
Keywords
literature recommendation, literature curation, PubMed search, query refinement, recommender systems, biomedical literature
MeSH terms
Machine Learning, COVID-19, Humans, Liver Neoplasms, Publications, Software, Internet, Healthcare Disparities, Data Curation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review