Automated Paper Screening for Clinical Reviews Using Large Language Models

Other studies, such as Guo et al. (2023), did not consider imbalance nor incorporate chance agreement, potentially inflating GPT's accuracy.

“Other studies, such as Guo et al., (2023), did not consider imbalance nor incorporate chance agreement in their interpretation of the results, thereby potentially inflating GPT's accuracy.”

One possible reason for the limitations of existing AI tools is that they split text into fixed segments, which hinders their ability to understand context, especially in longer texts.

“One possible reason for these limitations is that they split text into fixed segments, which has been argued to hinder their ability to understand context, especially in longer texts (Guo et al., 2023).”

Guo et al. and Alshami et al. achieved specificity scores close to ours (90% and 93%, respectively, vs our 92%).

“They achieved a specificity score close to ours (90% and 93%, respectively, vs our 92%).”

Our results for sensitivity and specificity are consistent with previous studies.

“Our results for sensitivity, and specificity are consistent with previous studies.”

Guo et al. (2023) and Alshami et al. (2023) found that specificity was the strongest metric for title/abstract screening, as we did.

“For instance, Guo et al. (2023) and Alshami et al. (2023) found that specificity was the strongest metric for title/abstract screening, as we did.”

Most studies focused on narrow aspects of systematic reviews, such as Boolean queries or only evaluating performance on titles and abstracts screening.

“Most studies focused on narrow aspects of systematic reviews, such as Boolean queries (S. Wang et al., 2023) or only evaluating performance on titles and abstracts screening (Alshami et al., 2023; Guo et al., 2023; Syriani et al., 2023).”

Some prior studies may have included contaminated data, such as Guo et al. (2023) and Syriani et al. (2023), using datasets published before or in 2021, when GPT was trained, potentially biasing results in favour of GPT.

“A few may have included contaminated data, such as Guo et al., (2023) and Syriani et al., (2023), who used datasets for systematic reviews that published their results before or in 2021, when GPT was trained, which may bias the results in favour of GPT.”