Applying ChatGPT and AI‐Powered Tools to Accelerate Evidence Reviews
Open accessPublished version of a preprint
Abstract
Artificial intelligence (AI) tools have been used to improve the productivity of evidence review and synthesis since at least 2016, with EPPI‐Reviewer and Abstrackr being two prominent examples. However, since the release of ChatGPT by OpenAI in late 2022, the use of generative AI for research, especially for text‐based data analysis, has exploded. In this article, we used a critical reflection approach to document and evaluate the capacity of different generative AI tools such as ChatGPT, GPT for Google Sheets and Docs, Casper AI, and ChatPDF to assist in the early stages of a rapid evidence review process. Our results demonstrate that these tools can boost research productivity in formulating search strings and screening literature, but they have some notable weaknesses, including producing inconsistent results and occasional errors. We recommend that researchers exercise caution when using generative AI technologies by designing a thorough research strategy and review protocol to ensure effective monitoring and quality control.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The only exception known to the authors was a recent study by Nguyen-Trung and colleagues, which assessed the application of LLMs to assist with rapid reviews in a case study focused on land management and climate resilience.
“The only exception we are aware of was a recent study by Nguyen-Trung and colleagues [17], which assessed the application of LLMs to assist with rapid reviews in a case study focused on land management and climate resilience.”
From Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis (2025) · cites this paper as [17]
Cites
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · see citing works
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842 · see citing works
Chatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355 · see citing works
Partners in Cognition: Extending Human Intelligence with Intelligent Technologies
SALOMON, G., PERKINS, D. N., & GLOBERSON, T. (1991) · Educational Researcher · doi:10.3102/0013189x020003002 · Cited by 948 · see citing works
Role of Chat GPT in public health
Biswas, S. S. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03172-7 · Cited by 673 · see citing works
A Conversation on Artificial Intelligence, Chatbots, and Plagiarism in Higher Education
King, M. R., & chatGPT (2023) · Cellular and Molecular Bioengineering · doi:10.1007/s12195-022-00754-8 · Cited by 602 · see citing works
Exploring the Use of Artificial Intelligence for Qualitative Data Analysis: The Case of ChatGPT
Morgan, D. L. (2023) · International Journal of Qualitative Methods · doi:10.1177/16094069231211248 · Cited by 510 · see citing works
Practising Critical Reflection: A Resource Handbook
Fook, J., & Gardner, F. (2007) · Cited by 410 · see citing works
Potential Use of Chat GPT in Global Warming
Biswas, S. S. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03171-8 · Cited by 291 · see citing works
How Chat GPT Can Transform Autodidactic Experiences and Open Education?
Firat, M. (2023) · doi:10.31219/osf.io/9ge8m · Cited by 280 · see citing works
Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591703 · Cited by 275 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268 · see citing works
Tools to support the automation of systematic reviews: a scoping review.
Khalil, H., Ameen, D., & Zarnegar, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.005 · Cited by 190 · see citing works
Supporting Systematic Reviews Using Text Mining
Ananiadou, S., Rea, B., Okazaki, N., Procter, R., & Thomas, J. (2009) · Social Science Computer Review · doi:10.1177/0894439309332293 · Cited by 144 · see citing works
Using ChatGPT to conduct a literature review
Haman, M., & Školník, M. (2024) · Accountability in Research · doi:10.1080/08989621.2023.2185514 · Cited by 144 · see citing works
Tsou, A. Y., Treadwell, J. R., Erinoff, E., & Schoelles, K. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01324-7 · Cited by 117 · see citing works
e-Science and its implications
Hey, T., & Trefethen, A. (2003) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2003.1224 · Cited by 93 · see citing works
Cheap, Quick, and Rigorous: Artificial Intelligence and the Systematic Literature Review
Atkinson, C. F. (2023) · Social Science Computer Review · doi:10.1177/08944393231196281 · Cited by 73 · see citing works
Critical Reflection in Context
Fook, J., & Gardner, F. (2012) · Routledge eBooks · doi:10.4324/9780203094662 · Cited by 60 · see citing works
Kebede, M. M., Le Cornet, C., & Fortner, R. T. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1589 · Cited by 50 · see citing works
Using Critical Reflection to Research Possibilities for Change
Morley, C. (2013) · The British Journal of Social Work · doi:10.1093/bjsw/bct004 · Cited by 35 · see citing works
Methodological guidance for rapid reviews in healthcare: A scoping review
Speckemeier, C., Niemann, A., Wasem, J., Buchberger, B., & Neusser, S. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1555 · Cited by 34 · see citing works
EPPI-Reviewer 3.5: software for research synthesis
Thomas, J., Brunton, J., & Graziosi, S. (2007) · IOE EPrints · Cited by 25 · see citing works
Morley, C. (2011) · The British Journal of Social Work · doi:10.1093/bjsw/bcr153 · Cited by 22 · see citing works
Goyanes, M., Lopezosa, C., & Jordá, B. (2024) · doi:10.31235/osf.io/8mr2f · Cited by 20 · see citing works
ChatGPT in Thematic Analysis: Can AI become a research assistant in qualitative research?Preprint
Nguyen‐Trung, K. (2024) · doi:10.31219/osf.io/vefwc · Cited by 9 · see citing works
Self-citationBragge, P., Waddell, A., Kellner, P., Delafosse, V., Marten, R., Nordström, A., & Demaio, S. (2023) · BMJ Global Health · doi:10.1136/bmjgh-2023-011683 · Cited by 7 · see citing works
Nguyen‐Trung, K., Saeri, A. K., & Kaufman, S. (2024) · Qualitative Research Journal · doi:10.1108/qrj-11-2023-0168 · Cited by 6 · see citing works
Self-citationFrom Centuries to Hours: The Journey of Research into Practice
Bragge, P. (2022) · Digital Government Research and Practice · doi:10.1145/3529166 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical UsagePreprint
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v1 · Cited by 429 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 173 · see citing works
AI and its implications for research in higher education: a critical dialogue
Butson, R., & Spronken-Smith, R. (2024) · Higher Education Research & Development · doi:10.1080/07294360.2023.2280200 · Cited by 130 · see citing works
ChatGPT in thematic analysis: Can AI become a research assistant in qualitative research?
Nguyen‐Trung, K. (2025) · Quality & Quantity · doi:10.1007/s11135-025-02165-z · Cited by 122 · see citing works
Self-citationAcademic publisher guidelines on AI usage: A ChatGPT supported thematic analysis
Perkins, M., & Roe, J. (2023) · F1000Research · doi:10.12688/f1000research.142411.2 · Cited by 111 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741 · Cited by 88 · see citing works
Cheap, Quick, and Rigorous: Artificial Intelligence and the Systematic Literature Review
Atkinson, C. F. (2023) · Social Science Computer Review · doi:10.1177/08944393231196281 · Cited by 73 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 62 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v2 · Cited by 47 · see citing works
Nykvist, B., Macura, B., Xylia, M., & Olsson, E. (2025) · Environmental Evidence · doi:10.1186/s13750-025-00360-x · Cited by 17 · see citing works
“The only exception we are aware of was a recent study by Nguyen-Trung and colleagues [17], which assessed the application of LLMs to assist with rapid reviews in a case study focused on land management and climate resilience.”
The role of ChatGPT in developing systematic literature searches: an evidence summary
Parisi, V., & Sutton, A. (2024) · Journal of EAHIL · doi:10.32384/jeahil20623 · Cited by 14 · see citing works
Understanding Cyclists’ Visual Behavior Using Eye-Tracking Technology: A Systematic Review
Kchour, F., Cafiso, S. D., & Pappalardo, G. (2024) · Sensors · doi:10.3390/s25010022 · Cited by 13 · see citing works
AI as research partner: key implications of using AI for data visualisation in qualitative research
Phan, A. N. Q., & Le, C. (2025) · International Journal of Social Research Methodology · doi:10.1080/13645579.2025.2518562 · Cited by 26 · see citing works
Assessing the Impact of ChatGPT on EFL Students’ Writing Productivity and Proficiency
Alwasidi, M. A., & Al-Khalifah, K. S. (2025) · Journal of Language Teaching and Research · doi:10.17507/jltr.1603.29 · Cited by 11 · see citing works
Selvakumar, P., Satyanarayana, P., Lahoti, Y., Saipogu, L., Mohit, M., & Manjunath, T. C. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-3246-8.ch005 · Cited by 9 · see citing works
Mtotywa, M. M., Mowers, J.-L., Ndou, W., Moleko, T. V. Q., & Ledwaba, M. (2026) · Informatics · doi:10.3390/informatics13030043 · Cited by 13 · see citing works
ChatGPT in Thematic Analysis: Can AI become a research assistant in qualitative research?Preprint
Nguyen‐Trung, K. (2024) · doi:10.31219/osf.io/vefwc · Cited by 9 · see citing works
Self-citationRethinking evidence-informed policy and practice in the age of generative artificial intelligence
Cong‐Lem, N. (2025) · London Review of Education · doi:10.14324/lre.23.1.16 · Cited by 6 · see citing works
Cabanillas-García, J. L. (2025) · Informatics · doi:10.3390/informatics12030061 · Cited by 5 · see citing works
Machine Learning-Assisted Health Economics and Policy Reviews: A Comparative Assessment
Cavallaro, L., Ardito, V., Drummond, M., & Ciani, O. (2025) · Applied Health Economics and Health Policy · doi:10.1007/s40258-025-00963-y · Cited by 5 · see citing works
Asrifan, A., Widyaningrum, A., Ramli, R. B., Riyanti, R., & Sultan, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-4576-5.ch013 · Cited by 3 · see citing works
Courvoisier, D. S., Buitrago‐García, D., Buclin, C. P., Bürgisser, N., Iudici, M., & Mongin, D. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10054 · Cited by 3 · see citing works
Matthes, J., & Vranken, S. (2025) · Communication and Change · doi:10.1007/s44382-025-00019-8 · Cited by 3 · see citing works
Shereefdeen, H., Thaivalappil, A., Young, I., & MacKay, M. (2025) · INQUIRY The Journal of Health Care Organization Provision and Financing · doi:10.1177/00469580251399374 · Cited by 3 · see citing works
Inteligência artificial no levantamento bibliográfico em bases de dados científicos
Picalho, A. C., de Oliveira, G. R., & Cativelli, A. S. (2025) · RDBCI Revista Digital de Biblioteconomia e Ciência da Informação · doi:10.20396/rdbci.v23i00.8678378 · Cited by 3 · see citing works
AI-Driven Document Analysis: Employing Streamlit, Faiss, Nvidia Nemo
Singh, N. T., Kaur, H., Dhiman, J., Aryan, A., Rani, J., & Wadhwa, M. (2025) · International Conference on Inventive Computing and Informatics (ICICI) · doi:10.1109/icici65870.2025.11069902 · Cited by 3 · see citing works
Leveraging Artificial Intelligence (AI) in Competitive Intelligence (CI) Research
Hair, J. F., & Sabol, M. A. (2024) · Revista Inteligência Competitiva · doi:10.24883/eaglesustainable.v15i.469 · Cited by 4 · see citing works
Ethical Implications and Governance of AI in Research
Caton, L., & Hwang, Y. (2025) · Springer eBooks · doi:10.1007/978-3-031-94809-1_8 · Cited by 2 · see citing works
Student Perspectives on the Impact of GenAI
Lewis, C., Funnell, P., Fisher, N., & Elston, P. (2025) · Springer eBooks · doi:10.1007/978-3-031-94809-1_2 · Cited by 2 · see citing works
Zrubka, M., Alexy, M., György, K. T., & Zrubka, Z. (2025) · IEEE International Symposium on Computational Intelligence and Informatics (CINTI) · doi:10.1109/cinti67731.2025.11311831 · Cited by 2 · see citing works
Natto, Z. S. (2026) · BMC Medical Education · doi:10.1186/s12909-026-08576-2 · Cited by 2 · see citing works
Braune, J., Conradty, C., Bogner, F. X., & Paul, J. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1770878 · Cited by 2 · see citing works
Ciro, J. B., & Franco-Rico, J. (2025) · INTER DISCIPLINA · doi:10.22201/ceiich.24485705e.2025.36.90833 · Cited by 2 · see citing works
AI-Enabled Tools: Shaping the Future of Technology
Chinimilli, V. R. P., Bhansali, P. K., Devi, J. S., Ledalla, S., & Gade, M. S. L. (2025) · ICST Transactions on Scalable Information Systems · doi:10.4108/eetsis.8213 · Cited by 1 · see citing works
Azeroual, O. (2025) · Information - Wissenschaft & Praxis · doi:10.1515/iwp-2025-2050 · Cited by 1 · see citing works
Reconfiguring Inquiry: The Epistemic and Methodological Impacts of NLP in Higher Education Research
Gao, C. Y., Wang, X., & Yu, K. (2026) · Palgrave studies in global higher education · doi:10.1007/978-3-032-07781-3_13 · Cited by 3 · see citing works
Zrubka, M., Ahadli, T., & Zrubka, Z. (2026) · IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI) · doi:10.1109/sami68106.2026.11420686 · Cited by 1 · see citing works
AI-Supported Knowledge-Building Action Networks (KBANs) for Education
Kelly, A. E. (2026) · Springer international handbooks of education · doi:10.1007/978-3-031-88898-4_13-1 · Cited by 1 · see citing works
Artificial Intelligence (AI) in Competitive Intelligence (CI) Research
Hair, J. F., & Sabol, M. A. (2025) · SDGs Studies Review. · doi:10.37497/sdgs.v6istudies.43 · Cited by 1 · see citing works
van As, J. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1414-3.ch003
The Role of AI Platforms in Enhancing Entrepreneurship Education: A Theoretical Analysis
Park, T. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-06982-5_8
Morató, J. V. (2025) · European Public & Social Innovation Review · doi:10.31637/epsir-2026-2108
Javed, S., Rong, Y., & Abbasi, B. N. (2025) · Educational Technology Research and Development · doi:10.1007/s11423-025-10580-0
Park, S., & Choe, H. (2026) · The Journal of Korean Association of Computer Education · doi:10.32431/kace.2026.29.3.009
Azeroual, O., & Khan, R. R. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-1433-2.ch003
Ocks, Y., & Salubi, O. G. (2026) · South African journal of information management · doi:10.4102/sajim.v28i1.2125
Lai, H., & Li, Y. (2026) · Telematics and Informatics · doi:10.1016/j.tele.2026.102447
- Authors
- Nguyen-Trung, K., Saeri, A. K., & Kaufman, S.
- Affiliations
- Massachusetts Institute of Technology (US)
- Monash University (AU)
- Venue
- Human Behavior and Emerging Technologies
- Publisher
- Wiley
- Published
- 2024-01
- Volume
- 2024
- Issue
- 1
- ISSN
- 2578-1863 (ISSN-L)
- OpenAlex
- W4405096679
- Cited by
- 47
- References
- 29
- Citations by year
- 2026: 10 · 2025: 25 · 2024: 6 · 2023: 6
- FWCI
- 11.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Human Behavior and Emerging Technologies (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- Monash University
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- generative AI, ChatGPT, rapid review, AI-assisted literature screening, critical reflection, research quality control, ChatPDF
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
