Cheap, Quick, and Rigorous: Artificial Intelligence and the Systematic Literature Review
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
The systematic literature review (SLR) is the gold standard in providing research a firm evidence foundation to support decision-making. Researchers seeking to increase the rigour, transparency, and replicability of their SLRs are provided a range of guidelines towards these ends. Artificial Intelligence (AI) and Machine Learning Techniques (MLTs) developed with computer programming languages can provide methods to increase the speed, rigour, transparency, and repeatability of SLRs. Aimed towards researchers with coding experience, and who want to utilise AI and MLTs to synthesise and abstract data obtained through a SLR, this article sets out how computer languages can be used to facilitate unsupervised machine learning for synthesising and abstracting data sets extracted during a SLR. Utilising an already known qualitative method, Deductive Qualitative Analysis, this article illustrates the supportive role that AI and MLTs can play in the coding and categorisation of extracted SLR data, and in synthesising SLR data. Using a data set extracted during a SLR as a proof of concept, this article will include the coding used to create a well-established MLT, Topic Modelling using Latent Dirichlet allocation. This technique provides a working example of how researchers can use AI and MLTs to automate the data synthesis and abstraction stage of their SLR, and aide in increasing the speed, frugality, and rigour of research projects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
The Discovery of Grounded Theory; Strategies for Qualitative Research
Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968) · Nursing Research · doi:10.1097/00006199-196807000-00014 · Cited by 39,449
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223 · cited 2×
“Additionally, the Preferred Reporting Items for Systematic reviews and Meta-Analyses – Protocols (PRISMA-P) checklist is a 15 item guideline that aims at aiding the reporting of review protocols prior to their conduction (Moher et al., 2015).”
MethodAn introduction to latent semantic analysis
Landauer, T. K., Foltz, P. W., & Laham, D. (1998) · Discourse Processes · doi:10.1080/01638539809545028 · Cited by 4,880 · cited 1×
“Other topic models include; Non Negative Matrix Factorization, developed by Liu et al. (2006); Latent Semantic Analysis, developed by Landauer et al. (1998); Parallel Latent Dirichlet allocation, developed by Wang et al. (2009); and Pachinko Allocation Model, developed by Li and McCallum (2006).”
MethodDwivedi, Y. K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., Duan, Y., Dwivedi, R., Edwards, J., Eirug, A., et al. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.08.002 · Cited by 4,611 · cited 1×
“Not since the industrial revolution have new technologies, such as the ones provided through AI and Machine Learning Techniques (MLTs) automated what were once labour-intensive tasks (Dwivedi et al., 2021).”
BackgroundArtificial Intelligence in Education: A Review
Chen, L., Chen, P., & Lin, Z. (2020) · IEEE Access · doi:10.1109/access.2020.2988510 · Cited by 4,133 · cited 1×
“Characterised by cognitive abilities, learning, adaptability, and decision-making, human intelligence through AI has allowed for the automation of tasks that would normally take a person signi fi cant time to complete (Chen et al., 2020).”
BackgroundA few useful things to know about machine learning
Domingos, P. (2012) · Communications of the ACM · doi:10.1145/2347736.2347755 · Cited by 3,326 · cited 1×
“Domingos (2012) points out that algorithms themselves may introduce biases.”
BackgroundSiddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
Scientific procedures and rationales for systematic literature reviews (SPAR‐4‐SLR)
Paul, J., Lim, W. M., O’Cass, A., Hao, A. W., & Bresciani, S. (2021) · International Journal of Consumer Studies · doi:10.1111/ijcs.12695 · Cited by 2,364 · cited 2×
“There are a number of reasons to undertake a SLR: (1) to advertise research being undertaken and to avoid repeating research already conducted; (2) to guide the planning of new research; and (3) to underpin claims of originality when new research is contrasted against old (Paul et al., 2021).”
BackgroundFive steps to conducting a systematic review
Khan, K. S., Kunz, R., Kleijnen, J., & Antes, G. (2003) · Journal of the Royal Society of Medicine · doi:10.1177/014107680309600304 · Cited by 1,873 · cited 2×
“The next stage is where relevant works are identi fi ed, normally through the screening of titles and abstracts, followed by sourcing full texts of included studies and performing quality assessment on the included studies (Khan et al., 2003).”
BackgroundChatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157 · cited 3×
“…of the active learning process utilised in ASReview; (3) the binary label of (1 for relevant vs. 0 for irrelevant) is then used to train a new model, following this a new article is presented to the researcher based on the previous articles inclusion or exclusion (van de Schoot et al., 2021).”
BackgroundMethodInfluentialSupervised Machine Learning Algorithms: Classification and Comparison
Babcock University, F.Y, O., J.E.T, A., O, A., J. O, H., O, O., & J, A. (2017) · International Journal of Computer Trends and Technology · doi:10.14445/22312803/ijctt-v48p126 · Cited by 932
The SAGE Handbook of Current Developments in Grounded Theory
Bryant, A., & Charmaz, K. (2019) · doi:10.4135/9781526485656 · Cited by 614
Machine learning and natural language processing in mental health: systematic review
Le Glaz, A., Haralambous, Y., Kim-Dufor, D.-H., Lenca, P., Billot, R., Ryan, T. C., Marsh, J., DeVylder, J., Walter, M., Berrouiguet, S., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/15708 · Cited by 576 · cited 2×
“If data is input that is noisy, incomplete, or has been extracted through processes that have introduced selective biases, then the results themselves may be erroneous (Le Glaz et al., 2021).”
BackgroundMethodCan ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591703 · Cited by 275 · cited 1×
“Wang et al. (2023) provide a comprehensive guide on how to prompt Chat GPT to formulate search strategies to be utilised in SLRs.”
MethodOn behalf of the founding members of the ICASR group, Beller, E., Clark, J., Tsafnat, G., Adams, C., Diehl, H., Lund, H., Ouzzani, M., Thayer, K., Thomas, J., et al. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0740-7 · Cited by 191 · cited 5×
“The solution, according to Beller et al. (2018), is that every technique used for automation should be shared, in particular, by making code evaluation data and corporas available.”
BackgroundMethodInfluentialArtificial Intelligence-Enabled Science Poetry
Kirmani, A. R. (2023) · ACS Energy Letters · doi:10.1021/acsenergylett.2c02758 · Cited by 124 · cited 1×
“A public tool created by OpenAI, Chat GPT is based on the Generative Pre-Trained language model (Kirmani, 2022).”
MethodO’Connor, A. M., Tsafnat, G., Gilbert, S. B., Thayer, K. A., & Wolfe, M. S. (2018) · Systematic Reviews · doi:10.1186/s13643-017-0667-4 · Cited by 69
Applying ChatGPT and AI‐Powered Tools to Accelerate Evidence Reviews
Nguyen-Trung, K., Saeri, A. K., & Kaufman, S. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/8815424 · Cited by 47 · cited 1×
“AI tools have been utilised in reviews and evidence synthesis since 2016 (see EPPI-Reviewer and Abstracker for examples) (Nguyen-Trung et al., 2023).”
Method
Cited by
Applying ChatGPT and AI‐Powered Tools to Accelerate Evidence Reviews
Nguyen-Trung, K., Saeri, A. K., & Kaufman, S. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/8815424 · Cited by 47
Aslam, A., Postolache, O., Oliveira, S., & Pereira, J. D. (2025) · Sensors · doi:10.3390/s25030807 · Cited by 10
Lai, H., & Li, Y. (2026) · Telematics and Informatics · doi:10.1016/j.tele.2026.102447
Metadata
- Authors
- Atkinson, C. F.
- Venue
- Social Science Computer Review
- Publisher
- SAGE Publications
- Published
- 2023-08-26
- Volume
- 42
- Issue
- 2
- Pages
- 376-393
- ISSN
- 0894-4393 (ISSN-L)
- 1552-8286
- OpenAlex
- W4386194145
- Cited by
- 71
- References
- 20
- FWCI
- 3.76
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.