Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed
Open accessThe DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Abstract
Background: The demand for high-quality systematic literature reviews (SRs) for evidence-based medical decision-making is growing. SRs are costly and require the scarce resource of highly skilled reviewers. Automation technology has been proposed to save workload and expedite the SR workflow. We aimed to provide a comprehensive overview of SR automation studies indexed in PubMed, focusing on the applicability of these technologies in real world practice. Methods: In November 2022, we extracted, combined, and ran an integrated PubMed search for SRs on SR automation. Full-text English peer-reviewed articles were included if they reported studies on SR automation methods (SSAM), or automated SRs (ASR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, and the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results, and Google Scholar citations of SR automation studies. Results: From 5321 records screened by title and abstract, we included 123 full text articles, of which 108 were SSAM and 15 ASR. Automation was applied for search (19/123, 15.4%), record screening (89/123, 72.4%), full-text selection (6/123, 4.9%), data extraction (13/123, 10.6%), risk of bias assessment (9/123, 7.3%), evidence synthesis (2/123, 1.6%), assessment of evidence quality (2/123, 1.6%), and reporting (2/123, 1.6%). Multiple SR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SR topics. In published ASR, we found examples of automated search, record screening, full-text selection, and data extraction. In some ASRs, automation fully complemented manual reviews to increase sensitivity rather than to save workload. Reporting of automation details was often incomplete in ASRs. Conclusions: Automation techniques are being developed for all SR stages, but with limited real-world adoption. Most SR automation tools target single SR stages, with modest time savings for the entire SR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SR automation techniques in real-world practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Cumpston, M., Li, T., Page, M. J., Chandler, J., Welch, V. A., Higgins, J. P., & Thomas, J. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000142 · Cited by 7,218
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537
Murad, M. H., Asi, N., Alsawas, M., & Alahdab, F. (2016) · Evidence Based Medicine · doi:10.1136/ebmed-2016-110401 · Cited by 1,388
How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838 · cited 1×
“Although the Cochrane Handbook recommends that SRs are updated biannually [16], 23% of SRs can become out-dated within 2 years due to the omission of new evidence that could impact their conclusions [17].”
BackgroundA full systematic review was completed in 2 weeks using automation tools: a case study
Clark, J., Glasziou, P., Del Mar, C., Bannach-Brown, A., Stehlik, P., & Scott, A. M. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.008 · Cited by 687 · cited 10×
“…EPPI Reviewer [80, 128, 132, 136], RobotAnalyst [91, 94, 113, 131], Distiller SR [94, 99, 126], Rayyan [103, 131], Systematic Review Accelerator [18, 135], RCT Tagger [77, 78], SWIFT Review [125, 126], SyRF [92, 111], ASR (automated systematic review) [67], ASReview [133], Aggre-gator [58],…”
BackgroundMethodInfluentialO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 6×
“The feasibility of automation differs by stages of the SR workflow [21, 22], with search, record screening, full-text selection, data extraction, risk of bias assessment, evidence synthesis, and reporting being the most prominent examples [16, 21].”
BackgroundMethodResultInfluentialSystematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · cited 1×
“The feasibility of automation differs by stages of the SR workflow [21, 22], with search, record screening, full-text selection, data extraction, risk of bias assessment, evidence synthesis, and reporting being the most prominent examples [16, 21].”
MethodKohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0115-5 · Cited by 471 · cited 1×
“…( N = 26) [28], retrieval of high-quality clinical studies ( N = 10) [30], SR software packages including those with automation features [153, 154], reviews using AI-based automation ( N = 12) [155], a living review of automated data extraction tools ( N = 53) [156], or the syntheses of…”
BackgroundAutomation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 7×
“The performance of automation tools varies largely across review topics [32, 39, 54, 122, 130].”
BackgroundMethodInfluentialViewing systematic reviews and meta-analysis in social research through different lenses
Davis, J., Mengersen, K., Bennett, S., & Mazerolle, L. (2014) · SpringerPlus · doi:10.1186/2193-1801-3-511 · Cited by 364 · cited 1×
“Beyond medicine, systematic reviews are often performed in disciplines including engineering [11–13] or the social sciences [14, 15].”
BackgroundSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 6×
“…126], SyRF [92, 111], ASR (automated systematic review) [67], ASReview [133], Aggre-gator [58], ATCER [63], Cochrane RCT Classifier [72], Covidence [131], Curious Snake [83], DoCTER [65], GAP Screener [52], MetaPreg [74], Research Screener [102], revtools [134], RobotAnalyst, and TeMMPo [124].”
BackgroundMethodInfluentialRobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317 · cited 4×
“…[122], ExaCT for automatic extraction of clinical trial data [138], and RobotReviewer for automated assessment of risk of bias in clinical trials [145], and five reported methodological innovation, such as completing a SR in 2 weeks [18], reducing workload in extreme reviews with 1 million…”
MethodInfluentialAutomating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287 · cited 4×
“…illustrated by the study of Dinter et al. [29], which, despite including automation studies in all stages of the SR workflow and extending the electronic search with a manual snow-ball technique, yielded fewer reports than earlier reviews focusing on a more specific aspect of SR automation [28].”
MethodInfluentialHoffmann, F., Allers, K., Rombey, T., Helbach, J., Hoffmann, A., Mathes, T., & Pieper, D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.022 · Cited by 278
revtools: An R package to support article screening for evidence synthesis
Westgate, M. J. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1374 · Cited by 261
Thomas, J., McDonald, S., Noel-Storr, A., Shemilt, I., Elliott, J., Mavergames, C., & Marshall, I. J. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.11.003 · Cited by 253
Error rates of human reviewers during abstract screening in systematic reviews
Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H. (2020) · PLOS ONE · doi:10.1371/journal.pone.0227742 · Cited by 217
SWIFT-Review: a text-mining workbench for systematic review
Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., et al. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0263-z · Cited by 196 · cited 8×
“Further nine SSAMs [18, 54, 84, 86, 122, 129, 134, 138, 145] and two ASRs [80, 136] received over 100 citations.”
BackgroundMethodInfluentialChai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
Reducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190 · cited 5×
“Further nine SSAMs [18, 54, 84, 86, 122, 129, 134, 138, 145] and two ASRs [80, 136] received over 100 citations.”
BackgroundMethodInfluentialExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188 · cited 4×
“Further nine SSAMs [18, 54, 84, 86, 122, 129, 134, 138, 145] and two ASRs [80, 136] received over 100 citations.”
BackgroundMethodInfluentialBannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0942-7 · Cited by 185 · cited 2×
“…131], Distiller SR [94, 99, 126], Rayyan [103, 131], Systematic Review Accelerator [18, 135], RCT Tagger [77, 78], SWIFT Review [125, 126], SyRF [92, 111], ASR (automated systematic review) [67], ASReview [133], Aggre-gator [58], ATCER [63], Cochrane RCT Classifier [72], Covidence [131],…”
BackgroundInfluentialText Categorization Models for High-Quality Article Retrieval in Internal Medicine
Aphinyanaphongs, Y. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1641 · Cited by 177 · cited 1×
“For example, the seminal article by Aphinyanaphongs from 2005 [159], which is considered by many authors as the inaugural paper for automated record screening, was excluded during our record screening due to the lack of specific reference to systematic reviews.”
BackgroundShemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
Valizadeh, A., Moassefi, M., Nakhostin-Ansari, A., Hosseini Asl, S. H., Saghab Torbati, M., Aghajani, R., Maleki Ghorbani, Z., & Faghani, S. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01631-8 · Cited by 159 · cited 1×
“[32, 41, 52, 54–56, 59, 61, 68, 72, 82, 84, 85, 87, 89, 92, 93, 95, 96, 100, 106, 108, 110, 112, 114, 115, 119–122], followed by term-frequency/inverse document frequency (TF-IDF) ( N = 16/89, 18.0”
InfluentialPrioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148
Gates, A., Johnson, C., & Hartling, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0707-8 · Cited by 148 · cited 1×
“…Review Project (DERP) [32, 41, 54, 68, 77, 87, 88, 93, 107, 120–122 , The off-the shelf or freeware screening automation software were Abstrackr [90, 94, 97, 98, 105, 113, 132], EPPI Reviewer [80, 128, 132, 136], RobotAnalyst [91, 94, 113, 131], Distiller SR [94, 99, 126], Rayyan [103, 131],…”
MethodHamel, C., Kelly, S. E., Thavorn, K., Rice, D. B., Wells, G. A., & Hutton, B. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01129-1 · Cited by 140
Noel-Storr, A. H., Dooley, G., Wisniewski, S., Glanville, J., Thomas, J., Cox, S., Featherstone, R., & Foxlee, R. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.08.008 · Cited by 136 · cited 1×
“Two papers introduced curated article collections, such as Cochrane CENTRAL [44], and the Realtime Data Synthesis and Analysis (REDASA) COVID-19 dataset [48], which were assembled using various automation techniques.”
BackgroundGates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1222-2 · Cited by 133 · cited 4×
“…shelf or freeware screening automation software were Abstrackr [90, 94, 97, 98, 105, 113, 132], EPPI Reviewer [80, 128, 132, 136], RobotAnalyst [91, 94, 113, 131], Distiller SR [94, 99, 126], Rayyan [103, 131], Systematic Review Accelerator [18, 135], RCT Tagger [77, 78], SWIFT Review [125, 126],…”
BackgroundMethodInfluentialTsou, A. Y., Treadwell, J. R., Erinoff, E., & Schoelles, K. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01324-7 · Cited by 122 · cited 2×
“…41, 54, 68, 77, 87, 88, 93, 107, 120–122 , The off-the shelf or freeware screening automation software were Abstrackr [90, 94, 97, 98, 105, 113, 132], EPPI Reviewer [80, 128, 132, 136], RobotAnalyst [91, 94, 113, 131], Distiller SR [94, 99, 126], Rayyan [103, 131], Systematic Review…”
MethodStatistical stopping criteria for automated screening in systematic reviews
Callaghan, M. W., & Müller-Hansen, F. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01521-4 · Cited by 121 · cited 1×
“[32, 41, 52, 54–56, 59, 61, 68, 72, 82, 84, 85, 87, 89, 92, 93, 95, 96, 100, 106, 108, 110, 112, 114, 115, 119–122], followed by term-frequency/inverse document frequency (TF-IDF) ( N = 16/89, 18.0”
InfluentialCleo, G., Scott, A. M., Islam, F., Julien, B., & Beller, E. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1069-6 · Cited by 112 · cited 3×
“…freeware screening automation software were Abstrackr [90, 94, 97, 98, 105, 113, 132], EPPI Reviewer [80, 128, 132, 136], RobotAnalyst [91, 94, 113, 131], Distiller SR [94, 99, 126], Rayyan [103, 131], Systematic Review Accelerator [18, 135], RCT Tagger [77, 78], SWIFT Review [125, 126], SyRF [92,…”
BackgroundMethodData extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
A new algorithm for reducing the workload of experts in performing systematic reviews
Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., & O'Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.004325 · Cited by 100
Cross-Topic Learning for Work Prioritization in Systematic Review Creation and Update
Cohen, A. M., Ambert, K., & McDonagh, M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3162 · Cited by 88 · cited 1×
“…EPPI Centre database [60, 83, 84, 86, 89] or those from the Oregon Drug Effectiveness Review Project (DERP) [32, 41, 54, 68, 77, 87, 88, 93, 107, 120–122 , The off-the shelf or freeware screening automation software were Abstrackr [90, 94, 97, 98, 105, 113, 132], EPPI Reviewer [80, 128, 132,…”
MethodValidity of data extraction in evidence synthesis practice of adverse events: reproducibility study
Xu, C., Yu, T., Furuya-Kanamori, L., Lin, L., Zorzela, L., Zhou, X., Dai, H., Loke, Y., & Vohra, S. (2022) · BMJ · doi:10.1136/bmj-2021-069155 · Cited by 82 · cited 1×
“SR automation using artificial intelligence (AI) and advanced computing technologies has the potential to speed up the review process, reduce the workload of researchers, prevent human errors, and facilitate reproducibility by diminishing the role of human judgement [18–20].”
MethodScott, A. M., Forbes, C., Clark, J., Carter, M., Glasziou, P., & Munn, Z. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.030 · Cited by 81 · cited 1×
“Despite some positive experiences, the uptake of SR automation tools is still limited [26, 27].”
BackgroundCohen, A. M., Smalheiser, N. R., McDonagh, M. S., Yu, C., Adams, C. E., Davis, J. M., & Yu, P. S. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocu025 · Cited by 78 · cited 2×
“[59, 61, 76, 88, 119, 123], or semantic annotation to the Unified Medical Language System (UMLS) ( N = 6 /89, 6.7”
BackgroundInfluentialBui, D. D. A., Del Fiol, G., Hurdle, J. F., & Jonnalagadda, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.10.014 · Cited by 70
Cowie, K., Rahmatullah, A., Hardy, N., Holub, K., & Kallmes, K. (2022) · JMIR Medical Informatics · doi:10.2196/33219 · Cited by 68
Machine learning to assist risk-of-bias assessments in systematic reviews
Millard, L. A., Flach, P. A., & Higgins, J. P. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv306 · Cited by 65
A clustering approach for topic filtering within systematic literature reviews
Weißer, T., Saßmannshausen, T., Ohrndorf, D., Burggräf, P., & Wagner, J. (2020) · MethodsX · doi:10.1016/j.mex.2020.100831 · Cited by 64
Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., Schmid, C. H., Bertram, L., Lill, C. M., Cohen, J. T., & Trikalinos, T. A. (2012) · Genetics in Medicine · doi:10.1038/gim.2012.7 · Cited by 62 · cited 1×
“[32, 41, 52, 54–56, 59, 61, 68, 72, 82, 84, 85, 87, 89, 92, 93, 95, 96, 100, 106, 108, 110, 112, 114, 115, 119–122], followed by term-frequency/inverse document frequency (TF-IDF) ( N = 16/89, 18.0”
InfluentialImproving reference prioritisation with PICO recognition.
Brockmeier, A. J., Ju, M., Przybyła, P., & Ananiadou, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0992-8 · Cited by 62
Clark, J., McFarlane, C., Cleo, G., Ishikawa Ramos, C., & Marshall, S. (2021) · JMIR Medical Education · doi:10.2196/24418 · Cited by 54 · cited 9×
“In another SRs, risk of bias assessment using seven domains of the Cochrane Collab-oration’s RoB1 tool needed 23 h and 40 min for 16 studies (5340 s per study), while RobotReviewer finished in 2 h and 12 min assessing four risk of bias domains (495 s per study), saving 4845 s per study [135].”
BackgroundMethodInfluentialSoboczenski, F., Trikalinos, T. A., Kuiper, J., Bias, R. G., Wallace, B. C., & Marshall, I. J. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0814-z · Cited by 53 · cited 3×
“RobotReviewer was used in an additional five studies [18, 135, 147–149].”
BackgroundMethodPICO element detection in medical text without metadata: are first sentences enough?
Huang, K.-C., Chiang, I.-J., Xiao, F., Liao, C.-C., Liu, C. C.-H., & Wong, J.-M. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.07.009 · Cited by 52
Mindfulness for smoking cessation
Jackson, S., Brown, J., Norris, E., Livingstone-Banks, J., Hayes, E., & Lindson, N. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013696.pub2 · Cited by 45 · cited 4×
“…(published in 2016) introduced Rayyan, a leading SR platform ( N = 9015) [87], followed by an ASR on mindfulness for smoking cessation ( N = 526) [49], a study introducing Curious snake, a freeware active learning-based screening automation tool ( N = 323) [83], the seminal study from Cohen et…”
BackgroundMethodInfluentialBekhuis, T., Tseytlin, E., Mitchell, K. J., & Demner-Fushman, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086277 · Cited by 45 · cited 1×
“[45, 53, 57, 60, 63, 66, 68, 73, 76, 83, 91, 109, 115, 116, 122, 123], topic models ( N = 10/89, 11.2”
InfluentialAutomated screening of research studies for systematic reviews using study characteristics
Tsafnat, G., Glasziou, P., Karystianis, G., & Coiera, E. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0724-7 · Cited by 44
Systematic review toolbox: a catalogue of tools to support systematic reviews
Marshall, C., & Brereton, P. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745824 · Cited by 44
Automatic classification of literature in systematic reviews on food safety using machine learning
van den Bulk, L. M., Bouzembrak, Y., Gavai, A., Liu, N., van den Heuvel, L. J., & Marvin, H. J. P. (2022) · Current Research in Food Science · doi:10.1016/j.crfs.2021.12.010 · Cited by 38
Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews
Gates, A., Gates, M., DaRosa, D., Elliott, S. A., Pillay, J., Rahman, S., Vandermeer, B., & Hartling, L. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01528-x · Cited by 31
Comparing machine and human reviewers to evaluate the risk of bias in randomized controlled trials
Armijo‐Olivo, S., Craig, R., & Campbell, S. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1398 · Cited by 31
Karystianis, G., Thayer, K., Wolfe, M., & Tsafnat, G. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.04.004 · Cited by 31
PICO entity extraction for preclinical animal literature.
Wang, Q., Liao, J., Lapata, M., & Macleod, M. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02074-4 · Cited by 29
Walker, V. R., Schmitt, C. P., Wolfe, M. S., Nowak, A. J., Kulesza, K., Williams, A. R., Shin, R., Cohen, J., Burch, D., Stout, M. D., et al. (2022) · Environment International · doi:10.1016/j.envint.2021.107025 · Cited by 28
Discriminating between empirical studies and nonempirical works using automated text classification
Langlois, A., Nie, J., Thomas, J., Hong, Q. N., & Pluye, P. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1317 · Cited by 26 · cited 4×
“[45, 53, 57, 60, 63, 66, 68, 73, 76, 83, 91, 109, 115, 116, 122, 123], topic models ( N = 10/89, 11.2”
BackgroundMethodInfluentialAutomatic endpoint detection to support the systematic review process.
Blake, C., & Lucic, A. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.05.004 · Cited by 22 · cited 1×
“…using SWIFT Review [125], extraction of geographic locations from the manuscript [141], extraction of endpoints as comparative claim sentences [142], data extraction from ClinicalTrials.gov for meta-analyses [143], and convenience tools to highlight relevant sentences [74], or extract data…”
MethodMapping Phenomena Relevant to Adolescent Emotion Regulation: A Text-Mining Systematic Review
van Lissa, C. J. (2022) · Adolescent Research Review · doi:10.1007/s40894-021-00160-7 · Cited by 22
Muller, A. E., Ames, H. M. R., Jardim, P. S. J., & Rose, C. J. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1541 · Cited by 21 · cited 1×
“…93, 107, 120–122 , The off-the shelf or freeware screening automation software were Abstrackr [90, 94, 97, 98, 105, 113, 132], EPPI Reviewer [80, 128, 132, 136], RobotAnalyst [91, 94, 113, 131], Distiller SR [94, 99, 126], Rayyan [103, 131], Systematic Review Accelerator [18, 135], RCT Tagger…”
MethodThe use of bibliography enriched features for automatic citation screening
Olorisade, B. K., Brereton, P., & Andras, P. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103202 · Cited by 21
Tsubota, T., Bollegala, D., Zhao, Y., Jin, Y., & Kozu, T. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104185 · Cited by 11 · cited 3×
“[69, 71, 79, 81, 118].”
BackgroundInfluentialZrubka, Z., Gulacsi, L., & Pentek, M. (2022) · 2022 IEEE 26th International Conference on Intelligent Engineering Systems (INES) · doi:10.1109/ines56734.2022.9922639 · Cited by 9
Self-citation
Research works printed but not linked: 94
- Rubin A, Bellamy J. Practitioner’s guide to using research for evidence-based practice. Hoboken: Wiley; 2012.
- Melissa LR, Tara JB, Carrie P, David M, Lex MB, Jamie JK, et al. Systematic review search strategies are poorly described and not reproducible: a cross-sectional meta-research study. J Clin Epidemiol. 2024;166:111229. · DOI
- Holgyesi A, Toth B, Kozlovszky M, Kuti J, Weszl M, Balazs G, et al. Epidemiology and patients’ self-reported knowledge of implantable medical devices: results of a cross-sectional survey in Hungary. PLoS One. 2023;18(4):e0284577. · DOI
- Zah V, Burrel A, Asche C, Zrubka Z. Paying for digital health interventions – what evidence is needed? Acta Polytech Hung. 2022;19(9):179–99. · DOI
- Moga K, Holgyesi A, Zrubka Z, Pentek M, Haidegger T. Augmented or mixed reality enhanced head-mounted display navigation for in vivo spine surgery: a systematic review of clinical outcomes. J Clin Med. 2023;12(11):3788. · doi:10.3390/jcm12113788
- Czere JT, Gulacsi L, Zrubka Z, Pentek M, editors. Quality assessment of clinical trials with artificial intelligence based chatbots in healthcare: points to consider in the protocol development for a systematic literature review. In: SACI 2022 - IEEE 16th international symposium on applied computational intelligence and informatics, proceedings. 2022. · doi:10.1109/saci55618.2022.9919543
- Motahari-Nezhad H, Fgaier M, Mahdi Abid M, Pentek M, Gulacsi L, Zrubka Z. Digital biomarker-based studies: scoping review of systematic reviews. JMIR Mhealth Uhealth. 2022;10(10):e35722. · DOI
- Motahari-Nezhad H, Al-Abdulkarim H, Fgaier M, Abid MM, Péntek M, Gulácsi L, et al. Digital biomarker–based interventions: systematic review of systematic reviews. J Med Internet Res. 2022;24(12):e41042. · DOI
- Méndez-Gurrola II, Ramírez-Reyes A, Barranco-Gutiérrez AI. A review and perspective on the main machine learning methods applied to physical sciences. Acta Polytech Hung. 2022;19(10):205–20. · DOI
- Tran TA, Péntek M, Motahari-Nezhad H, Abonyi J, Kovács L, Gulácsi L, et al. Heart rate variability measurement to assess acute work-content-related stress of workers in industrial manufacturing environment—a systematic scoping review. IEEE Trans Syst Man Cybern Syst. 2023;53(11):6685–92. · DOI
- Varga J. Defining the economic role and benefits of micro, small and medium-sized enterprises in the 21st century with a systematic review of the literature. Acta Polytech Hung. 2021;18(11):209–28. · DOI
- Llewellyn A, Whittington C, Stewart G, Higgins JP, Meader N. The use of Bayesian networks to assess the quality of evidence from research synthesis: 2. Inter-rater reliability and comparison with standard GRADE Assessment. PLoS One. 2015;10(12):e0123511. · DOI
- Stewart GB, Higgins JP, Schunemann H, Meader N. The use of Bayesian networks to assess the quality of evidence from research synthesis: 1. PLoS One. 2015;10(3):e0114497. · DOI
- Burgard T, Bittermann A. Reducing literature screening workload with machine learning. Z Psychol. 2023;231(1):3–15.
- Abdelkader W, Navarro T, Parrish R, Cotoi C, Germini F, Iorio A, et al. Machine learning approaches to retrieve high-quality, clinically relevant evidence from the biomedical literature: systematic review. JMIR Med Inform. 2021;9(9):e30401. · DOI
- Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203. · DOI · Paper page
- Oertelt-Prigione S, Wiedmann S, Endres M, Nolte CH, Regitz-Zagrosek V, Heuschmann P. Stroke and myocardial infarction: a comparative systematic evaluation of gender-specific analysis, funding and authorship patterns in cardiovascular research. Cerebrovasc Dis. 2011;31(4):373–81. · DOI
- O’Mara-Eves A, Brunton G, McDaid D, Kavanagh J, Oliver S, Thomas J. Techniques for identifying cross-disciplinary and ‘hard-to-detect’ evidence for systematic review. Res Synth Methods. 2014;5(1):50–9. · DOI
- Thompson J, Davis J, Mazerolle L. A systematic method for search term selection in systematic reviews. Res Synth Methods. 2014;5(2):87–97. · DOI
- Bui DD, Jonnalagadda S, Del Fiol G. Automatically finding relevant citations for clinical guideline development. J Biomed Inform. 2015;57:436–45. · DOI · Paper page
- Mwachui MA, Crump L, Hartskeerl R, Zinsstag J, Hattendorf J. Environmental and behavioural determinants of leptospirosis transmission: a systematic review. PLoS Negl Trop Dis. 2015;9(9):e0003843. · doi:10.1371/journal.pntd.0003843
- Sampson M, de Bruijn B, Urquhart C, Shojania K. Complementary approaches to searching MEDLINE may be sufficient for updating systematic reviews. J Clin Epidemiol. 2016;78:108–15. · DOI
- Rathbone J, Albarqouni L, Bakhit M, Beller E, Byambasuren O, Hoffmann T, et al. Expediting citation screening using PICo-based title-only screening for identifying studies in scoping searches and rapid reviews. Syst Rev. 2017;6(1):233. · DOI · Paper page
- Lanera C, Minto C, Sharma A, Gregori D, Berchialla P, Baldi I. Extending PubMed searches to ClinicalTrials.gov through a machine learning approach for systematic reviews. J Clin Epidemiol. 2018;103:22–30. · DOI
- Schmitz T, Bukowski M, Koschmieder S, Schmitz-Rode T, Farkas R. Potential Technologies Review: a hybrid information retrieval framework to accelerate demand-pull innovation in biomedical engineering. Res Synth Methods. 2019;10(3):420–39. · DOI
- Alharbi A, Stevenson M. Refining Boolean queries to identify relevant studies for systematic review updates. J Am Med Inform Assoc. 2020;27(11):1658–66. · DOI
- Brunskill A. A Microsoft Excel approach to reduce errors and increase efficiency in systematic searching. Med Ref Serv Q. 2020;39(1):15–26. · DOI
- Thiabaud A, Triulzi I, Orel E, Tal K, Keiser O. Social, behavioral, and cultural factors of HIV in Malawi: semi-automated systematic review. J Med Internet Res. 2020;22(8):e18747. · doi:10.2196/18747
- Paynter R, Fiordalisi C, Stoeger E, Erinoff E, Featherstone R, Voisin C, et al. A prospective comparison of evidence synthesis search strategies developed with and without text-mining tools. Rockville: Agency for Healthcare Research and Quality (US); 2021. Contract no.: report no.: 21-EHC008. · DOI
- Vaghela U, Rabinowicz S, Bratsos P, Martin G, Fritzilas E, Markar S, et al. Using a secure, continually updating, web source processing pipeline to support the real-time data synthesis and analysis of scientific literature: development and validation study. J Med Internet Res. 2021;23(5):e25714. · DOI
- O’Keefe H, Rankin J, Wallace SA, Beyer F. Investigation of text-mining methodologies to aid the construction of search strategies in systematic reviews of diagnostic test accuracy-a case study. Res Synth Methods. 2023;14(1):79–98. · DOI
- Yu W, Clyne M, Dolan SM, Yesupriya A, Wulf A, Liu T, et al. GAPscreener: an automatic tool for screening human genetic association literature in PubMed using the support vector machine technique. BMC Bioinformatics. 2008;9:205. · DOI · Paper page
- Bekhuis T, Demner-Fushman D. Towards automating the initial screening phase of a systematic review. Stud Health Technol Inform. 2010;160(Pt 1):146–50.
- Frunza O, Inkpen D, Matwin S, Klement W, O’Blenis P. Exploiting the systematic review protocol for classification of medical abstracts. Artif Intell Med. 2011;51(1):17–25. · DOI
- Shao W, Adams CE, Cohen AM, Davis JM, McDonagh MS, Thakurta S, et al. Aggregator: a machine learning approach to identifying MEDLINE articles that derive from the same underlying clinical trial. Methods. 2015;74:65–70. · DOI · Paper page
- Mo Y, Kontonatsios G, Ananiadou S. Supporting systematic reviews using LDA-based document representations. Syst Rev. 2015;4:172. · DOI · Paper page
- Almeida H, Meurs MJ, Kosseim L, Tsang A. Data sampling and supervised learning for HIV literature screening. IEEE Trans Nanobioscience. 2016;15(4):354–61. · DOI
- Cawley M, Beardslee R, Beverly B, Hotchkiss A, Kirrane E, Sams R 2nd, et al. Novel text analytics approach to identify relevant literature for human health risk assessments: a pilot study with health effects of in utero exposures. Environ Int. 2020;134:105228. · DOI
- Lanera C, Berchialla P, Sharma A, Minto C, Gregori D, Baldi I. Screening PubMed abstracts: is class imbalance always a challenge to machine learning? Syst Rev. 2019;8(1):317. · DOI
- Odintsova VV, Roetman PJ, Ip HF, Pool R, Van der Laan CM, Tona KD, et al. Genomics of human aggression: current state of genome-wide studies and an automated systematic review tool. Psychiatr Genet. 2019;29(5):170–90. · DOI · Paper page
- Ambalavanan AK, Devarakonda MV. Using the contextual language model BERT for multi-criteria classification of scientific articles. J Biomed Inform. 2020;112:103578. · DOI
- Dhrangadhariya A, Hilfiker R, Schaer R, Muller H. Machine learning assisted citation screening for systematic reviews. Stud Health Technol Inform. 2020;270:302–6.
- Aum S, Choe S. srBERT: automatic article classification model for systematic review using BERT. Syst Rev. 2021;10(1):285. · DOI · Paper page
- van Altena AJ, Spijker R, Leeflang MMG, Olabarriaga SD. Training sample selection: impact on screening automation in diagnostic test accuracy reviews. Res Synth Methods. 2021;12(6):831–41. · DOI
- Ajiji P, Cottin J, Picot C, Uzunali A, Ripoche E, Cucherat M, et al. Feasibility study and evaluation of expert opinion on the semi-automated meta-analysis and the conventional meta-analysis. Eur J Clin Pharmacol. 2022;78(7):1177–84. · DOI
- Ishankulov T, Danilov G, Kotik K, Orlov Y, Shifrin M, Potapov A. The classification of scientific abstracts using text statistical features. Stud Health Technol Inform. 2022;290:263–7.
- Li X, Zhang A, Al-Zaidy R, Rao A, Baral S, Bao L, et al. Automating document classification with distant supervision to increase the efficiency of systematic reviews: a case study on identifying studies with HIV impacts on female sex workers. PLoS One. 2022;17(6):e0270034. · DOI
- Proescholdt R, Hsiao TK, Schneider J, Cohen AM, McDonagh MS, Smalheiser NR. Testing a filtering strategy for systematic reviews: evaluating work savings and recall. AMIA Jt Summits Transl Sci Proc. 2022;2022:406–13.
- Schneider J, Hoang L, Kansara Y, Cohen AM, Smalheiser NR. Evaluation of publication type tagging as a strategy to screen randomized controlled trial articles in preparing systematic reviews. JAMIA Open. 2022;5(1):ooac015. · DOI
- Viner R, Russell S, Saulle R, Croker H, Stansfield C, Packer J, et al. School closures during social lockdown and mental health, health behaviors, and well-being among children and adolescents during the first COVID-19 wave: a systematic review. JAMA Pediatr. 2022;176(4):400–9. · DOI · Paper page
- Hempel S, Shetty KD, Shekelle PG, Rubenstein LV, Danz MS, Johnsen B, et al. AHRQ Methods for Effective Health Care. Machine learning methods in systematic reviews: identifying quality improvement intervention evaluations. Rockville: Agency for Healthcare Research and Quality (US); 2012.
- Dunn AG, Arachi D, Bourgeois FT. Identifying clinical study types from PubMed metadata: the active (machine) learning approach. Stud Health Technol Inform. 2015;216:867–71.
- Hashimoto K, Kontonatsios G, Miwa M, Ananiadou S. Topic detection using paragraph vectors to support active learning in systematic reviews. J Biomed Inform. 2016;62:59–65. · DOI · Paper page
- Ji X, Ritter A, Yen PY. Using ontology-based semantic similarity to facilitate the article screening process for systematic reviews. J Biomed Inform. 2017;69:33–42. · DOI
- Kontonatsios G, Brockmeier AJ, Przybyla P, McNaught J, Mu T, Goulermas JY, et al. A semi-supervised approach using label propagation to support citation screening. J Biomed Inform. 2017;72:67–76. · DOI · Paper page
- Norman CR, Leeflang MMG, Porcher R, Neveol A. Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy. Syst Rev. 2019;8(1):243. · DOI · Paper page
- Giummarra MJ, Lau G, Gabbe BJ. Evaluation of text mining to reduce screening workload for injury-focused systematic reviews. Inj Prev. 2020;26(1):55–60. · DOI · Paper page
- Li Z, Gurgel H, Dessay N, Hu L, Xu L, Gong P. Semi-supervised text classification framework: an overview of dengue landscape factors and satellite earth observation. Int J Environ Res Public Health. 2020;17(12):4509. · DOI
- Yamada T, Yoneoka D, Hiraike Y, Hino K, Toyoshiba H, Shishido A, et al. Deep neural network for reducing the screening workload in systematic reviews for clinical guidelines: algorithm validation study. J Med Internet Res. 2020;22(12):e22422. · DOI · Paper page
- Gaskins NJ, Bray E, Hill JE, Doherty PJ, Harrison A, Connell LA. Factors influencing implementation of aerobic exercise after stroke: a systematic review. Disabil Rehabil. 2021;43(17):2382–96. · DOI
- Pham B, Jovanovic J, Bagheri E, Antony J, Ashoor H, Nguyen TT, et al. Text mining to support abstract screening for knowledge syntheses: a semi-automated workflow. Syst Rev. 2021;10(1):156. · DOI · Paper page
- Carey N, Harte M, Mc CL. A text-mining tool generated title-abstract screening workload savings: performance evaluation versus single-human screening. J Clin Epidemiol. 2022;149:53–9. · DOI
- Yang JJ, Cohen AM, McDonagh MS. SYRIAC: the systematic review information automated collection system a data warehouse for facilitating automated biomedical text classification. AMIA Annu Symp Proc. 2008;2008:825–9.
- Surian D, Dunn AG, Orenstein L, Bashir R, Coiera E, Bourgeois FT. A shared latent space matrix factorisation method for recommending new trial evidence for systematic review updates. J Biomed Inform. 2018;79:32–40. · DOI
- Xiong Z, Liu T, Tse G, Gong M, Gladding PA, Smaill BH, et al. A machine learning aided systematic review and meta-analysis of the relative risk of atrial fibrillation in patients with diabetes mellitus. Front Physiol. 2018;9:835. · DOI · Paper page
- Currie GL, Angel-Scott HN, Colvin L, Cramond F, Hair K, Khandoker L, et al. Animal models of chemotherapy-induced peripheral neuropathy: a machine-assisted systematic review and meta-analysis. PLoS Biol. 2019;17(5):e3000243. · DOI
- Norman CR, Gargon E, Leeflang MMG, Neveol A, Williamson PR. Evaluation of an automatic article selection method for timelier updates of the Comet Core Outcome Set database. Database (Oxford). 2019;2019:baz109. · DOI
- Reddy SM, Patel S, Weyrich M, Fenton J, Viswanathan M. Comparison of a traditional systematic review approach with review-of-reviews and semi-automation as strategies to update the evidence. Syst Rev. 2020;9(1):243. · DOI · Paper page
- Stansfield C, Stokes G, Thomas J. Applying machine classifiers to update searches: analysis from two case studies. Res Synth Methods. 2022;13(1):121–33. · DOI
- Surian D, Bourgeois FT, Dunn AG. The automation of relevant trial registration screening for systematic review updates: an evaluation study on a large dataset of ClinicalTrials.gov registrations. BMC Med Res Methodol. 2021;21(1):281. · DOI
- Dalal SR, Shekelle PG, Hempel S, Newberry SJ, Motala A, Shetty KD. AHRQ Methods for Effective Health Care. A pilot study using machine learning and domain knowledge to facilitate comparative effectiveness review updating. Rockville: Agency for Healthcare Research and Quality (US); 2012.
- Qin X, Liu J, Wang Y, Liu Y, Deng K, Ma Y, et al. Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviews. J Clin Epidemiol. 2021;133:121–9. · DOI · Paper page
- Cohen AM. Optimizing feature representation for automated systematic review work prioritization. AMIA Annu Symp Proc. 2008;2008:121–5.
- Cohen AM, Ambert K, McDonagh M. A prospective evaluation of an automated classification system to support evidence-based medicine and systematic review. AMIA Annu Symp Proc. 2010;2010:121–5.
- Li D, Wang Z, Wang L, Sohn S, Shen F, Murad MH, et al. A text-mining framework for supporting systematic reviews. Am J Inf Manag. 2016;1(1):1–9.
- Lewis SJ, Gardner M, Higgins J, Holly JMP, Gaunt TR, Perks CM, et al. Developing the WCRF International/University of Bristol methodology for identifying and carrying out systematic reviews of mechanisms of exposure-cancer associations. Cancer Epidemiol Biomarkers Prev. 2017;26(11):1667–75. · DOI
- Lam J, Howard BE, Thayer K, Shah RR. Low-calorie sweeteners and health outcomes: a demonstration of rapid evidence mapping (rEM). Environ Int. 2019;123:451–8. · DOI
- Carlson LM, Angrish M, Shirke AV, Radke EG, Schulz B, Kraft A, et al. Systematic evidence map for over one hundred and fifty per- and polyfluoroalkyl substances (PFAS). Environ Health Perspect. 2022;130(5):56001. · DOI · Paper page
- Halamoda-Kenzaoui B, Rolland E, Piovesan J, Puertas Gallardo A, Bremer-Hoffmann S. Toxic effects of nanomaterials for health applications: how automation can support a systematic review of the literature? J Appl Toxicol. 2022;42(1):41–51. · DOI · Paper page
- Ji X, Yen PY. Using MEDLINE elemental similarity to assist in the article screening process for systematic reviews. JMIR Med Inform. 2015;3(3):e28. · DOI
- Mytton J, Ingram J, Manns S, Thomas J. Facilitators and barriers to engagement in parenting programs: a qualitative systematic review. Health Educ Behav. 2014;41(2):127–37. · DOI · Paper page
- Trypsteen W, Kiselinova M, Vandekerckhove L, De Spiegelaere W. Diagnostic utility of droplet digital PCR for HIV reservoir quantification. J Virus Erad. 2016;2(3):162–9. · DOI
- Gates A, Gates M, Sim S, Elliott S, Pillay J, Hartling L. Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool. Rockville: Agency for Healthcare Research and Quality (US); 2021. Contract no.: report no.: 21-EHC006.
- De Menezes A, Nunes AC, Pimenta DN, Lotta G, Nkya T, Krieger MM, et al. Examining the intersection between gender, community health workers, and vector control policies: a text mining literature review. Am J Trop Med Hyg. 2022;106(3):768–74.
- Pradhan R, Hoaglin DC, Cornell M, Liu W, Wang V, Yu H. Automatic extraction of quantitative data from ClinicalTrials.gov to conduct meta-analyses. J Clin Epidemiol. 2019;105:92–100. · DOI · Paper page
- Cramond F, O’Mara-Eves A, Doran-Constant L, Rice AS, Macleod M, Thomas J. The development and evaluation of an online application to assist in the extraction of data from graphs for use in systematic reviews. Wellcome Open Res. 2018;3:157. · DOI
- Jardim PSJ, Rose CJ, Ames HM, Echavez JFM, Van de Velde S, Muller AE. Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system. BMC Med Res Methodol. 2022;22(1):167. · DOI · Paper page
- Wang Q, Liao J, Lapata M, Macleod M. Risk of bias assessment in preclinical literature using natural language processing. Res Synth Methods. 2022;13(3):368–80. · DOI
- Sanchez-Graillet O, Witte C, Grimm F, Grautoff S, Ell B, Cimiano P. Synthesizing evidence from clinical trials with dynamic interactive argument trees. J Biomed Semantics. 2022;13(1):16. · DOI
- Van der Mierden S, Tsaioun K, Bleich A, Leenaars CHC. Software tools for literature screening in systematic reviews in biomedical research. Altex. 2019;36(3):508–17.
- Blaizot A, Veettil SK, Saidoung P, Moreno-Garcia CF, Wiratunga N, Aceves-Martins M, et al. Using artificial intelligence methods for systematic review in health sciences: a systematic review. Res Synth Methods. 2022;13(3):353–62. · DOI · Paper page
- Feng Y, Liang S, Zhang Y, Chen S, Wang Q, Huang T, et al. Automated medical literature screening using artificial intelligence: a systematic review and meta-analysis. J Am Med Inform Assoc. 2022;29(8):1425–32. · DOI · Paper page
- Santos AOD, da Silva ES, Couto LM, Reis GVL, Belo VS. The use of artificial intelligence for automating or semi-automating biomedical literature analyses: a scoping review. J Biomed Inform. 2023;142:104389. · DOI · Paper page
- Burrell A, Zrubka Z, Champion A, Zah V, Vinuesa L, Holtorf A-P, et al. How useful are digital health terms for outcomes research? An ISPOR special interest group report. Value Health. 2022;25(9):1469–79. · DOI
Cited by
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
INSPECT-SR: a tool for assessing trustworthiness of randomised controlled trials
Wilkinson, J., Heal, C., Flemyng, E., Antoniou, G. A., Aburrow, T., Alfirevic, Z., Avenell, A., Barbour, V., Berghella, V., Bishop, D. V. M., et al. (2025) · medRxiv · doi:10.1101/2025.09.03.25334905 · Cited by 15
Xu, S., Zhao, Z., Liu, X., & Meng, X.-L. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03138-w · Cited by 8
Robleto, E., & Shehadeh, L. A. (2025) · American Journal of Physiology. Heart and Circulatory Physiology · doi:10.1152/ajpheart.00374.2025 · Cited by 2
Metadata
- Authors
- Tóth, B., Berek, L., Gulácsi, L., Péntek, M., & Zrubka, Z.
- Venue
- Systematic Reviews
- Published
- 2024-07-08
- Volume
- 13
- Issue
- 1
- Pages
- 174-174
- ISSN
- 2046-4053 (ISSN-L)
- OpenAlex
- W4400421957
- Cited by
- 48
- References
- 68
- FWCI
- 6.59
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Funders
- Nemzeti Kutatási, Fejlesztési és Innovaciós Alap (TKP2021-NKTA-36)
- Óbuda University
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.