Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index; 8 of the 9 sources its claims cite were checked for support.
Abstract
In this paper we show that OpenAI's Large Language Model (LLM) GPT perform remarkably well when used for title and abstract eligibility screening of scientific articles and within a (systematic) literature review workflow. We evaluated GPT on screening data from a systematic review study on electric vehicle charging infrastructure demand with almost 12,000 records using the same eligibility criteria as human screeners. We tested 3 different versions of this model that were tasked to distinguishing between relevant and irrelevant content by responding with a relevance probability between 0 and 1. For the latest GPT-4 model (tested in November 2023) and probability cutoff 0.5 the recall rate is 100%, meaning no relevant papers were missed and using this mode for screening would have saved 50% of the time that would otherwise be spent on manual screening. Experimenting with a higher cut of threshold can save more time. With threshold chosen so that recall is still above 95% for GPT-4 (where up to 5% of relevant papers might be missed), the model could save 75% of the time spent on manual screening. If automation technologies can replicate manual screening by human experts with effectiveness, accuracy, and precision, the work and cost savings are significant. Furthermore, the value of a comprehensive list of relevant literature, rather quickly available at the start of a research project, is hard to understate. However, as this study only evaluated the performance on one systematic review and one prompt, we caution that more test and methodological development is needed, and outline the next steps to properly evaluate rigor and effectiveness of LLMs for eligibility screening.
Overview
CoChat Summary
TL;DR
This validation study tested three OpenAI GPT versions for title and abstract screening in an environmental systematic review on electric vehicle charging infrastructure demand, using 11,984 manually screened records and the review’s eligibility criteria. GPT-4-1106 achieved 100% recall at a 0.5 relevance-probability cutoff, potentially saving more than 50% of manual screening time, and >95% recall at a higher cutoff could save 75%, though the authors caution that more testing is needed.
Key findings
- At a 0.5 probability cutoff, GPT-4-1106 had recall 1.00 and WSS 0.55, saving more than 50% of records from human screening; at cutoff 0.8, recall was 0.96 and WSS 0.75, saving 75% while allowing up to 5% false negatives.
- Model versions improved: gpt-3.5-0311 screened out 1,100 of 11,984 records (9.5%) with no false negatives at cutoff 0.3, first false negative at 0.4; gpt-3.5-0613 screened out 2,300 (18%) before first false negative at cutoff 0.7; gpt-4-1106 screened out 6,700 (55%) with first errors at cutoff 0.7. False positives decreased from 10,700 to 9,600 to 5,300 across the three models.
- The models were less conservative than human title/abstract screeners, excluding records that humans included at title/abstract stage but later excluded at full-text screening.
Methods
- Validation study comparing LLM screening decisions with human screening in a completed systematic review on electric vehicle charging infrastructure demand; 11,984 records were manually screened by one reviewer, with a 100-record consistency check among three reviewers yielding 88% interrater agreement.
- Three GPT versions (gpt-3.5-0311, gpt-3.5-0613, gpt-4-1106) were used via the OpenAI API with the same prompt, no fine-tuning, no additional training, and no examples; the prompt asked the model to apply eligibility criteria and return a relevance probability from 0 to 1 plus a justification.
- Performance was compared at different probability cutoffs against human title/abstract decisions and final full-text included records, using Type I false positives, Type II false negatives, specificity, precision, recall, F-measure, and Work Saved over Sampling (WSS).
Limitations
- Stated: the study evaluated only one systematic review, one prompt, and one field; robust a priori cutoff selection remains unresolved; LLM opacity, retraining, version deprecation, and reproducibility are concerns; grey literature could not be tested because most lacked abstracts; human errors limit evaluation; more multi-field, multi-prompt, and multi-metric testing is needed.
- Stated: model performance may depend on abstract style and reporting quality, and robustness was only informally checked on a small batch of 100 records.
- Evident: the human reference standard relied on a single reviewer for the full 11,984-record title/abstract screening, with only a 100-record consistency check, so the benchmark itself may contain errors.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
4 Valid · 3 Not valid · 1 Not checked · 29 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“The only exception we are aware of was a recent study by Nguyen-Trung and colleagues [17], which assessed the application of LLMs to assist with rapid reviews in a case study focused on land management and climate resilience.”
§ Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis > Introduction
What the sources say
- Supportedcites [17] Nguyen-Trung K, Saeri AK, Kaufman S. Applying ChatGPT and AI-Powered Tools to Accelerate Evidence Reviews. Geraedts V, editor. Human Behavior and Emerging Technologies. 2024;2024:8815424. source
Section 2.1 ('Outline of a Rapid Review Methodology With AI-Powered Tools') states that the authors designed and employed the eight steps of a rapid review (drawing on Bragge et al. and Speckemeier et al.) and applied ChatGPT/GPT for Google Sheets, ChatPDF and Casper AI to Steps 2 and 4, arguing that generative AI could improve rapid-review efficiency. Section 2.2 ('Research Topics and Case Study') and Table 2 confirm the case study concerned land managers' participation in environmental markets in the Goulburn Broken catchment, Victoria, Australia, with the primary research question 'Does land managers' participation in environmental markets build climate resilience?' The 'only exception we are aware of' framing is the citing authors' own statement and not attributed to this source. Reference 26 (Kaufman & Nguyen-Trung, Environmental Markets and Climate Resilience for the Goulburn Broken Region) corroborates the land-management/climate-resilience focus. (LLM, conf=0.90)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, these tools, sometimes referred to as discriminative artificial intelligence (AI) technologies, generally require human researchers providing large amount of sufficiently diverse data for training and active learning [1, 3, 5, 6].”
§ Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis > Introduction
What the sources say
- Partly supportedcites [1] van Dinter R, Tekinerdogan B, Catal C. Automation of systematic literature reviews: A systematic literature review. Inf Softw Technol. 2021;136:106589. sourceExperimental — accuracy not yet measured
Section 4.6.1 (p. 10) says supervised learning is the primary technique for automating systematic reviews and that SVMs have been used for active learning. Section 5.11 (p. 12) also notes that features often need domain- or dataset-specific hand-crafting. These passages do not establish that classifiers generally require researchers to provide large amounts of sufficiently diverse training data. (LLM, conf=0.84)
- Partly supportedcites [3] de la Torre-López J, Ramírez A, Romero JR. Artificial intelligence to automate the systematic review of scientific literature. Computing. 2023;105:2171–94. sourceExperimental — accuracy not yet measured
Section 4.2 says supervised classifiers require labelled training data, and §4.2.2 describes human reviewers providing labels during active learning (page numbers unavailable). But the paper does not say researchers must provide large amounts of sufficiently diverse data; it says active learning can proceed without a pre-existing labelled dataset, and §5.3 reports a classifier achieving 92% performance with only 5% of papers labelled. (LLM, conf=0.88)
- Partly supportedcites [5] Chai KEK, Lines RLJ, Gucciardi DF, Ng L. Research screener: a machine learning tool to semi-automate abstract screening for systematic reviews. Syst Reviews. 2021;10:93. sourceExperimental — accuracy not yet measured
In the passage labelled Abstract > Supplementary Information (page unavailable), the paper says some existing screening methods require reviewers to screen hundreds to thousands of articles to train the program. It also describes classifiers trained on reviewers’ decisions and examples of active learning. It does not say that existing classifiers generally require large amounts of sufficiently diverse data; the large-training-set limitation is qualified as applying to some methods. (LLM, conf=0.85)
- Partly supportedcites [6] Tsafnat G, Glasziou P, Karystianis G, Coiera E. Automated screening of research studies for systematic reviews using study characteristics. Syst Reviews. 2018;7:64. sourceExperimental — accuracy not yet measured
The Background (page unavailable) says supervised screening classifiers are trained on human reviewers’ inclusion and exclusion decisions and require many such decisions before they become reliable. It does not say the data must be sufficiently diverse or that active learning is required. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, we show how performance depends on the relevance probability cutoff.”
§ Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis > Discussion
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the second model version tested - gpt-3.5-0613, the performance increased to 2,300 titles and abstracts (18%) before the first false negative error (Fig. 2d) and at a cutoff probability of 0.7.”
§ Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis > Results of title and abstract screening using GPT API
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the first version tested, gpt-3.5-0311 successfully screens out 1,100 of 11,984 titles and abstracts (9.5%) without any false negative errors (occurring when model excludes relevant studies) at a 0.3 probability cutoff.”
§ Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis > Results of title and abstract screening using GPT API
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
14 more claims
Create a free account to see all 37 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 15
- It may state two separate claims as one. 10
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis > Results of title and abstract screening using GPT API
“| Specificity = TN / (TN + FP) | 0.00 | 0.11 | 0.29 | 0.32 | 0.49 | 0.55 | 0.55 | 0.62 | 0.79 | 0.97 | 1.00 | | Recall (Sensitivity)R = TP/(TP + FN) | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.99 | 0.96 | 0.49 | 0.05 |”
§ Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis > Results of title and abstract screening using GPT API
“| WSS = (TN + FN)/N-(1.0-R) | 0 | 0.11 | 0.28 | 0.32 | 0.48 | 0.55 | 0.55 | 0.61 | 0.75 | 0.48 | 0.04 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“the performance increased to 2,300 titles and abstracts (18%)”
§ Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis > Results of title and abstract screening using GPT API
Reported 18 · recomputed 19.192256
Citations
Cites
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 3×
“Finally, we also calculated Specificity (S), Precision (P), Recall (R) and Work Saved over Sampling (WSS) (according to Cohen et al. [21]).”
InfluentialLiving systematic reviews: 2. Combining human and machine effort
Thomas, J., Noel-Storr, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., Glasziou, P., Shemilt, I., Synnot, A., Turner, T., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.011 · Cited by 355 · cited 1×
“Any savings in laborious human screening not only enables elimination of human fatigue and bias [24], but would allow a larger number of primary research studies to be included in the screening, or, perhaps more useful– more time for human experts to focus on synthesis and interpretation of results.”
ChatGPT and Environmental Research
Zhu, J.-J., Jiang, J., Yang, M., & Ren, Z. J. (2023) · Environmental Science & Technology · doi:10.1021/acs.est.3c01818 · Cited by 243
Automated Paper Screening for Clinical Reviews Using Large Language Models
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2023) · arXiv · doi:10.48550/arxiv.2305.00844 · Cited by 230
Automated paper screening for clinical reviews using large language models: Data analysis study
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2024) · Journal of Medical Internet Research · doi:10.2196/48996 · Cited by 205
Qureshi, R., Shaughnessy, D., Gill, K. A. R., Robinson, K. A., Li, T., & Agai, E. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02243-z · Cited by 201 · cited 2×
“Reproducibility of the LLM application thus need to be ensured in order to increase usability of such models in scientific applications in general and in systematic evidence syntheses in particular [7, 22, 23].”
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
Artificial intelligence to automate the systematic review of scientific literature
de la Torre-López, J., Ramírez, A., & Romero, J. R. (2023) · Computing · doi:10.1007/s00607-023-01181-x · Cited by 170
Research works printed but not linked: 1
- Xylia M, Olsson E, Macura B, Nykvist B. Estimating charging infrastructure demand for electric vehicles: a systematic review. Energy Strategy Reviews. Under reivew;Under review.
Cited by
Nawrath, M., Merlina, A., Knight, J., Welch, S. A., Rashidian, M., & Seifert-Dähnn, I. (2026) · Information · doi:10.3390/info17050501 · Cited by 8
“Consistent with this, prior work has recommended high recall thresholds for title-abstract screening even when this increases downstream workload [21].”
InfluentialHu, J., Yao, X., Huang, Z., Liang, D., Yang, D., Liu, D., Li, J., Zhang, Y., & Ma, X. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3792787 · Cited by 3
“Prior work suggested that the GPT-4 models are promising in the title and abstract screening task [104, 122].”
de Araújo, P. R., de Rezende, J. A. M., Faria, D. R. M., & Gomes, O. D. S. M. (2026) · Sensors · doi:10.3390/s26020747 · Cited by 7
Zrubka, M., Ahadli, T., & Zrubka, Z. (2026) · IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI) · doi:10.1109/sami68106.2026.11420686 · Cited by 1
Zrubka, M., Alexy, M., György, K. T., & Zrubka, Z. (2025) · IEEE International Symposium on Computational Intelligence and Informatics (CINTI) · doi:10.1109/cinti67731.2025.11311831 · Cited by 2
“In contrast to classic ML review tools, large language models (LLMs) do not require additional training on a categorized subset of articles [7], as they come out of the box pre-trained on large datasets [8].”
Influential
Metadata
- Authors
- Nykvist, B., Macura, B., Xylia, M., & Olsson, E.
- Affiliations
- Lund University (SE)
- Stockholm Environment Institute (SE)
- Corresponding author
- Nykvist, Björn
- Venue
- Environmental Evidence
- Published
- 2025-04-23
- Volume
- 14
- Issue
- 1
- Pages
- 7-7
- ISSN
- 2047-2382 (ISSN-L)
- PMC
- PMC12016299
- OpenAlex
- W4409709722
- Cited by
- 17
- References
- 23
- Citations by year
- 2026: 12 · 2025: 5
- FWCI
- 10.87
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Environmental Evidence (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Lund University Publications (Lund University) (submittedVersion)
- Funders
- Lund University
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- GPT-4, large language models, eligibility screening, title and abstract screening, systematic reviews, electric vehicle charging infrastructure, precision and recall, prompt engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis. https://app.cochat.ai/papers/10.1186/s13750-025-00360-x
- The paper
- Nykvist, B., Macura, B., Xylia, M., & Olsson, E. (2025). Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis. Environmental Evidence. https://doi.org/10.1186/s13750-025-00360-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.