Harnessing the Power of ChatGPT for Automating Systematic Review Process: Methodology, Case Study, Limitations, and Future Directions

Most studies focused on narrow aspects of systematic reviews, such as Boolean queries or only evaluating performance on titles and abstracts screening.

“Most studies focused on narrow aspects of systematic reviews, such as Boolean queries (S. Wang et al., 2023) or only evaluating performance on titles and abstracts screening (Alshami et al., 2023; Guo et al., 2023; Syriani et al., 2023).”

Guo et al. and Alshami et al. achieved specificity scores close to ours (90% and 93%, respectively, vs our 92%).

“They achieved a specificity score close to ours (90% and 93%, respectively, vs our 92%).”

Large language models rely on semantic interpretation, so their performance may be influenced by textual ambiguity or the framing of prompts, with prompt quality directly shaping model outputs.

“Because large language models rely on semantic interpretation, their performance may be influenced by textual ambiguity or the framing of prompts, with prompt quality directly shaping model outputs (Alshami et al., 2023).”

Prior studies and our findings suggest that GPT outperforms traditional active learning methods in terms of time efficiency.

“Despite these limitations, prior studies and our findings suggest that GPT outperforms traditional active learning methods in terms of time efficiency (Alshami et al., 2023).”

ISLaR's ability to identify correct values that human reviewers missed highlights a potential benefit of automation in terms of better reproducibility and lower risk of bias.

“highlighting a potential benefit of automation in terms of better reproducibility and lower risk of bias (Alshami et al., 2023; Guo et al., 2024; Matsui et al., 2024).”

Guo et al. (2023) and Alshami et al. (2023) found that specificity was the strongest metric for title/abstract screening, as we did.

“For instance, Guo et al. (2023) and Alshami et al. (2023) found that specificity was the strongest metric for title/abstract screening, as we did.”

Their sensitivity score was slightly higher than ours (76% and 84%, respectively, vs our 67%; Alshami et al., 2023 used a human-in-the-loop method).

“Their sensitivity score was slightly higher than ours (76% and 84%, respectively, vs our 67%; although note that Alshami et al., 2023 used a human-in-the-loop method).”

Applications of large language models in systematic literature reviews include construction of literature search terms, article screening, data extraction, and article content synthesis/analysis.

“Applications of LLMs in SLRs include construction of literature search terms (Alshami et al., 2023; Wang Z. et al., 2025; Wang et al., 2023), article screening (Akinseloyin et al., 2024; Alshami et al., 2023; Guo et al., 2024; Khraisha et al., 2024; Kohandel Gargari et al., 2024; Landschaft et al., 2024; Li et al., 2024; Matsui et al., 2024; Syriani et al., 2023; Tran et al., 2024; Wang Z. et al.,”

ISLaR's time savings and performance metrics were broadly in line with those reported in other studies of LLM-based SLR automation.

“ISLaR’s time savings and performance metrics were broadly in line with those reported in other studies of LLM-based SLR automation (Alshami et al., 2023; Dunn et al., 2022; Gartlehner et al., 2024; Guo et al., 2024; Khraisha et al., 2024; Kohandel Gargari et al., 2024; Landschaft et al., 2024; Li et al., 2024; Matsui et al., 2024; Tran et al., 2024; Wang Z. et al., 2025).”

Some prior studies have methodological shortcomings, such as Mahuli et al. (2023) not providing an objective evaluation of GPT's performance, or Alshami et al. (2023) not testing GPT's autonomous performance.

“Some have methodological shortcomings, such as Mahuli et al., (2023), who did not provide an objective evaluation of GPT's performance, or Alshami et al., (2023), who did not test GPT's autonomous performance, instead relying on a 'human-in-the-loop' approach.”

The finding that ISLaR's specificity was lower than its sensitivity is consistent with several other studies, although a smaller number of studies have reported both high sensitivity and high specificity.

“This finding is consistent with several other studies (Alshami et al., 2023; Guo et al., 2024; Khraisha et al., 2024; Kohandel Gargari et al., 2024; Matsui et al., 2024; Tran et al., 2024), although a smaller number of studies have reported both high sensitivity and high specificity (Landschaft et al., 2024; Li et al., 2024).”

Our results for sensitivity and specificity are consistent with previous studies.

“Our results for sensitivity, and specificity are consistent with previous studies.”