CoChat
Sign Up
PreprintCited by 14Topic ModelingInformation Retrieval and Search BehaviorSoftware Engineering Research

LitLLM: A Toolkit for Scientific Literature Review

Open access
Agarwal, Shubham, Sahu, Gaurav, Puri, Abhay, Laradji, Issam H., Dvijotham, Krishnamurthy DJ, Stanley, Jason, Charlin, Laurent, Pal, Christopher·2024·arXiv
DOI 10.48550/arxiv.2402.01788arXiv 2402.01788
Open this paper in CoChat — free accountOpen PDF

Abstract

Conducting literature reviews for scientific papers is essential for understanding research, its limitations, and building on existing work. It is a tedious task which makes an automatic literature review generator appealing. Unfortunately, many existing works that generate such reviews using Large Language Models (LLMs) have significant limitations. They tend to hallucinate-generate non-factual information-and ignore the latest research they have not been trained on. To address these limitations, we propose a toolkit that operates on Retrieval Augmented Generation (RAG) principles, specialized prompting and instructing techniques with the help of LLMs. Our system first initiates a web search to retrieve relevant papers by summarizing user-provided abstracts into keywords using an off-the-shelf LLM. Authors can enhance the search by supplementing it with relevant papers or keywords, contributing to a tailored retrieval process. Second, the system re-ranks the retrieved papers based on the user-provided abstract. Finally, the related work section is generated based on the re-ranked results and the abstract. There is a substantial reduction in time and effort for literature review compared to traditional methods, establishing our toolkit as an efficient alternative. Our project page including the demo and toolkit can be accessed here: https://litllm.github.io

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.01788

Cites

0 citation errors · 4 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Previous studies on building LMs to synthesize scientific literature use either small-scale, single-domain human evaluation or oversimplified multiple-choice QA set-ups.

    “Previous studies on building LMs to synthesize scientific literature use either small-scale, single-domain human evaluation8,9 or oversimplified multiple-choice QA set-ups10.”

    From Synthesizing scientific literature with retrieval-augmented language models (2026) · cites this paper as [8]

    Section 5 (User Experience; page unavailable) describes a preliminary evaluation in which five researchers used LitLLM to write literature reviews, supporting the small-scale human-evaluation part of the claim. Our copy does not say the evaluation was confined to a single domain. The multiple-choice QA clause is attached to a different citation, not this source. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Previous retrieval-augmented pipelines for scientific literature review relied on proprietary, black-box systems and limited evaluations, commonly small-scale human evaluation or simplified set-ups such as multiple-choice QA.

    “Although these studies and our research all explore the potential of retrieval-augmented LMs in automating literature synthesis, previous works often relied on proprietary, black-box systems and limited evaluations, which commonly entail small-scale human evaluation or simplified set-ups such as multiple-choice QA.”

    From Synthesizing scientific literature with retrieval-augmented language models (2026) · cites this paper as [8]

    The Introduction describes earlier literature-review systems using ChatGPT, which supports an example of reliance on a proprietary model. But the five-researcher preliminary evaluation in §5 is LitLLM’s own, not evidence about previous works; our copy does not substantiate the broader claim that earlier pipelines commonly used small-scale human evaluations or multiple-choice QA. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

  • Evaluations for literature synthesis are limited, typically focusing on narrow, single-discipline studies or simplified tasks such as multiple-choice question answering.

    “Evaluations for literature synthesis are also limited, typically focusing on narrow, single-discipline studies8,9 or simplified tasks such as multiple-choice question answering10.”

    From Synthesizing scientific literature with retrieval-augmented language models (2026) · cites this paper as [8]

    Citation 10 is attached to the claim about simplified evaluations such as multiple-choice question answering. The User Experience section describes a preliminary evaluation with five researchers writing literature reviews, but our copy does not characterize literature-synthesis evaluations as typically using multiple-choice questions. Searches for that task and for evaluation benchmarks found no supporting passage; no page numbers are available in our copy. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 4

  1. Jingshan Huang and Ming Tan. 2023. The role of chat- gpt in scientific communication: writing better sci- entific review articles. American Journal of Cancer Research, 13(4):1148.
  2. Hang Li, Jindong Gu, Rajat Koner, Sahand Sharifzadeh, and Volker Tresp. 2023. Do dall-e and flamingo understand each other? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1999-2010.
  3. Leigang Qu, Meng Liu, Jianlong Wu, Zan Gao, and Liqiang Nie. 2021. Dynamic modality interaction modeling for image-text retrieval. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval.
  4. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Ye- ung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text founda- tion models. Trans. Mach. Learn. Res., 2022.

Cited by

Sorted by citations

Related

Authors
Agarwal, S., Sahu, G., Puri, A., Laradji, I. H., Dvijotham, K. D., Stanley, J., Charlin, L., & Pal, C.
Affiliations
ServiceNow (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.01788
arXiv
2402.01788
OpenAlex
W4391591686
Cited by
13
References
0
Citations by year
2026: 4 · 2025: 5 · 2024: 4
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
retrieval-augmented generation, large language models, abstract summarization, keyword extraction, hallucination mitigation, scientific information retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review