CoChat

Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis

Open access
Nykvist, Björn, Macura, Biljana, Xylia, Maria, Olsson, Erik · 2025 · Environmental Evidence

The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index; 8 of the 9 sources its claims cite were checked for support.

Abstract

In this paper we show that OpenAI's Large Language Model (LLM) GPT perform remarkably well when used for title and abstract eligibility screening of scientific articles and within a (systematic) literature review workflow. We evaluated GPT on screening data from a systematic review study on electric vehicle charging infrastructure demand with almost 12,000 records using the same eligibility criteria as human screeners. We tested 3 different versions of this model that were tasked to distinguishing between relevant and irrelevant content by responding with a relevance probability between 0 and 1. For the latest GPT-4 model (tested in November 2023) and probability cutoff 0.5 the recall rate is 100%, meaning no relevant papers were missed and using this mode for screening would have saved 50% of the time that would otherwise be spent on manual screening. Experimenting with a higher cut of threshold can save more time. With threshold chosen so that recall is still above 95% for GPT-4 (where up to 5% of relevant papers might be missed), the model could save 75% of the time spent on manual screening. If automation technologies can replicate manual screening by human experts with effectiveness, accuracy, and precision, the work and cost savings are significant. Furthermore, the value of a comprehensive list of relevant literature, rather quickly available at the start of a research project, is hard to understate. However, as this study only evaluated the performance on one systematic review and one prompt, we caution that more test and methodological development is needed, and outline the next steps to properly evaluate rigor and effectiveness of LLMs for eligibility screening.

Overview

CoChat Summary

TL;DR

This validation study tested three OpenAI GPT versions for title and abstract screening in an environmental systematic review on electric vehicle charging infrastructure demand, using 11,984 manually screened records and the review’s eligibility criteria. GPT-4-1106 achieved 100% recall at a 0.5 relevance-probability cutoff, potentially saving more than 50% of manual screening time, and >95% recall at a higher cutoff could save 75%, though the authors caution that more testing is needed.

Key findings

  • At a 0.5 probability cutoff, GPT-4-1106 had recall 1.00 and WSS 0.55, saving more than 50% of records from human screening; at cutoff 0.8, recall was 0.96 and WSS 0.75, saving 75% while allowing up to 5% false negatives.
  • Model versions improved: gpt-3.5-0311 screened out 1,100 of 11,984 records (9.5%) with no false negatives at cutoff 0.3, first false negative at 0.4; gpt-3.5-0613 screened out 2,300 (18%) before first false negative at cutoff 0.7; gpt-4-1106 screened out 6,700 (55%) with first errors at cutoff 0.7. False positives decreased from 10,700 to 9,600 to 5,300 across the three models.
  • The models were less conservative than human title/abstract screeners, excluding records that humans included at title/abstract stage but later excluded at full-text screening.

Methods

  • Validation study comparing LLM screening decisions with human screening in a completed systematic review on electric vehicle charging infrastructure demand; 11,984 records were manually screened by one reviewer, with a 100-record consistency check among three reviewers yielding 88% interrater agreement.
  • Three GPT versions (gpt-3.5-0311, gpt-3.5-0613, gpt-4-1106) were used via the OpenAI API with the same prompt, no fine-tuning, no additional training, and no examples; the prompt asked the model to apply eligibility criteria and return a relevance probability from 0 to 1 plus a justification.
  • Performance was compared at different probability cutoffs against human title/abstract decisions and final full-text included records, using Type I false positives, Type II false negatives, specificity, precision, recall, F-measure, and Work Saved over Sampling (WSS).

Limitations

  • Stated: the study evaluated only one systematic review, one prompt, and one field; robust a priori cutoff selection remains unresolved; LLM opacity, retraining, version deprecation, and reproducibility are concerns; grey literature could not be tested because most lacked abstracts; human errors limit evaluation; more multi-field, multi-prompt, and multi-metric testing is needed.
  • Stated: model performance may depend on abstract style and reporting quality, and robustness was only informally checked on a small batch of 100 records.
  • Evident: the human reference standard relied on a single reviewer for the full 11,984-record title/abstract screening, with only a 100-record consistency check, so the benchmark itself may contain errors.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13750-025-00360-x

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Performance across the three models at probability cutoff 0.5. False negative rate displays number to clarify that the value is zero for two of the models
Fig. 1 Performance across the three models at probability cutoff 0.5. False negative rate displays number to clarify that the value is zero for two of the modelsCC BY · PubMed Central
Fig. 2 Performance in terms of number of records screened out by GPT, and the false positive errors (Type I) and false negative error (Type II) made as a function of the relevance probability. Columns in the panel show results for three versions of GPT model API used: Top row GPT3.5 as of 1st March 2023, second row, same model as of 13th June, and third row GPT4 as of 6th Nov 2023. Left hand side a, c, e, shows results benchmarked towards the results from title and abstract screening stage conducted by humans, and right-hand side b, d, f, compared to final set of included records in the review after full-text screening by humans. Numbers are shown above dark blue Type II errors lines for clarity
Fig. 2 Performance in terms of number of records screened out by GPT, and the false positive errors (Type I) and false negative error (Type II) made as a function of the relevance probability. Columns in the panel show results for three versions of GPT model API used: Top row GPT3.5 as of 1st March 2023, second row, same model as of 13th June, and third row GPT4 as of 6th Nov 2023. Left hand side a, c, e, shows results benchmarked towards the results from title and abstract screening stage conducted by humans, and right-hand side b, d, f, compared to final set of included records in the review after full-text screening by humans. Numbers are shown above dark blue Type II errors lines for clarityCC BY · PubMed Central

Verification

Claims the paper makes

4 Valid · 3 Not valid · 1 Not checked · 29 Own results and interpretations

14 more claims

Create a free account to see all 37 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 15
  • It may state two separate claims as one. 10

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 26 references

Research works printed but not linked: 1

  1. Xylia M, Olsson E, Macura B, Nykvist B. Estimating charging infrastructure demand for electric vehicles: a systematic review. Energy Strategy Reviews. Under reivew;Under review.

Cited by

See all 17

Metadata

Authors
Nykvist, B., Macura, B., Xylia, M., & Olsson, E.
Affiliations
  • Lund University (SE)
  • Stockholm Environment Institute (SE)
Corresponding author
Nykvist, Björn
Venue
Environmental Evidence
Published
2025-04-23
Volume
14
Issue
1
Pages
7-7
ISSN
2047-2382 (ISSN-L)
OpenAlex
W4409709722
Cited by
17
References
23
Citations by year
2026: 12 · 2025: 5
FWCI
10.87
Type
journal article
Language
en
Open access
Licence
Locations
  • Environmental Evidence (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Lund University Publications (Lund University) (submittedVersion)
Funders
Lund University
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
GPT-4, large language models, eligibility screening, title and abstract screening, systematic reviews, electric vehicle charging infrastructure, precision and recall, prompt engineering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis. https://app.cochat.ai/papers/10.1186/s13750-025-00360-x
The paper
Nykvist, B., Macura, B., Xylia, M., & Olsson, E. (2025). Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis. Environmental Evidence. https://doi.org/10.1186/s13750-025-00360-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.