Diffusion of innovation in systematic review methodology: Why is study selection not yet assisted by automation?
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Introduction Systematic reviews, the foundation of much Evidence-Based medicine, are suffering from increasing ‘data deluge’: reviewers often need to manually assess many thousands of titles and abstracts to determine their relevance. Automation has been advanced as a potential solution; but given that its efficacy was first demonstrated in 2006, why is it not yet widely used? The Diffusion of Innovations framework by EM Rogers is used to structure an exploration of why this might be the case. Discussion According to Rogers, five characteristics affect the rate of adoption of innovations: those perceived as having greater relative advantage, compatibility, trialability and observability, and less complexity, will be adopted more rapidly than others. The relative advantage of automation has been demonstrated empirically, though usually in narrowly focused reviews in clinical areas, rather than more challenging areas for automation, such as public health. Detailed methods and procedures for their use have yet to be established, addressing transparency, replicability and reporting practices. While issues concerning the compatibility of new technology with existing infrastructure are probably surmountable, the use of automation may challenge contemporary notions of what constitutes a systematic search and how publication bias is addressed using sensitive search techniques. The remaining factors are interrelated. The technologies are complex, both to understand and to deploy. This affects the trialability of automation: technical expertise is required and there are thus few opportunities for reviewers to observe others using these technologies. Conclusion Further technical and empirical work is needed where systematic reviewers work with information and computer scientists to develop solutions which have a demonstrative relative advantage and which are clearly compatible with the needs of systematic reviewers and their users. Such work may have a significant role to play in addressing the deluge of new research publications which threaten to overwhelm systematic review processes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Recent studies (Van Altena et al. 2019), suggest that this limited usage primarily stems from usability issues, in addition to a few other relevant factors: (i) steep learning curve , as researchers may be unfamiliar with the tools' functionalities (Scott et al. 2021), (ii) misalignment with user requirements , as many of these software deviate from the guidelines set forth by SLR protocols and”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Thomas 2013
In the citing paper
“Despite their promise, the adoption of ML tools among reviewers has been slow [6–8].”
From Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools (2019) · cites this paper as [8]
In the citing paper
“Of these issues, we hypothesize that the issue of compatibility with current practice (Characteristic 2) is most closely related to our concept of trust and set-up challenges [23].”
From A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? (2019) · cites this paper as [23]
Citations
Cites
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · An Integrated Approach to Communication Theory and Research · doi:10.4324/9780203710753-35 · Cited by 37,322
An Introduction to Systematic Reviews
Gough, D., Oliver, S., & Thomas, J. (2017) · doi:10.4135/9781036234942 · Cited by 1,239
Self-citationSeventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351
Watt, A., Cameron, A., Sturm, L., Lathlean, T., Babidge, W., Blamey, S., Facey, K., Hailey, D., Norderhaug, I., & Maddern, G. (2008) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462308080185 · Cited by 226 · cited 1×
“Diffusion of innovation in systematic review methodology: Why is study selection not yet assisted by automation? OA Evidence-Based Medicine 2013 Oct 21;1(2):12.”
BackgroundApplications of text mining within systematic reviews
Thomas, J., McNaught, J., & Ananiadou, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.27 · Cited by 212
Self-citationReducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190
Self-citationShemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
Self-citationIp, S., D'Ambrosio, C., Patel, K., Obadan, N., Kitsios, G. D., Chung, M., & Balk, E. M. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-20 · Cited by 148
Cited by
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
“Whilst the logic behind applying text mining to the screening stage of systematic reviews has intuitive appeal, there are obvious concerns that might be raised by the systematic review community [21].”
BackgroundSelf-citationSearching for studies: a guide to information retrieval for Campbell systematic reviews
Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. K., Hammerstrøm, K., & Sathe, N. (2017) · Campbell Systematic Reviews · doi:10.4073/cmg.2016.1 · Cited by 371
Self-citationO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Self-citationRobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317
“This should alleviate a common concern about automation technologies: that they act as black boxes.(18) Elsewhere, we have described a prototype tool which presents the model predictions to the user directly within the original PDF document.”
BackgroundEvidence & Gap Maps: A tool for promoting evidence informed policy and strategic research agendas
Snilstveit, B., Vojtkova, M., Bhavsar, A., Stevenson, J., & Gaarder, M. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.05.015 · Cited by 323
Metadata
- Authors
- Thomas, J.
- Corresponding author
- Thomas, JM
- Venue
- OA Evidence-Based Medicine
- Publisher
- Open Access Publishing London
- Published
- 2013-10
- Volume
- 1
- Issue
- 2
- ISSN
- 2053-2636 (ISSN-L)
- OpenAlex
- W1997248429
- Cited by
- 44
- References
- 15
- Citations by year
- 2024: 6 · 2023: 2 · 2022: 2 · 2021: 4 · 2020: 8 · 2019: 5 · 2018: 4 · 2017: 4 · 2016: 3 · 2015: 5 · 2013: 1
- FWCI
- 3.27
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- OA Evidence-Based Medicine (publishedVersion, cc-by)
- Funders
- Medical Research Council (MR/J005037/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- diffusion of innovations, study selection, title and abstract screening, technology adoption, relative advantage, compatibility, trialability, observability, complexity, publication bias, systematic search, data deluge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Diffusion of innovation in systematic review methodology: Why is study selection not yet assisted by automation? https://app.cochat.ai/papers/10.13172/2053-2636-1-2-1109
- The paper
- Thomas, J. (2013). Diffusion of innovation in systematic review methodology: Why is study selection not yet assisted by automation? OA Evidence-Based Medicine. https://doi.org/10.13172/2053-2636-1-2-1109
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.