Systematic review toolbox: a catalogue of tools to support systematic reviews
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Systematic review is a widely used research method in software engineering, and in other disciplines, for identifying and analysing empirical evidence. The method is data intensive and time consuming, and hence is usually supported by a wide range of software-based tools. However, systematic reviewers have found that finding and selecting tools can be quite challenging.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we identified machine learning systems that are available for use in practice at the time of writing, through manual screening of records in SR Toolbox1 on January 3, 2019, to identify all systematic review tools which incorporated machine learning [11].”
From Toward systematic review automation: a practical guide to using machine learning tools in research synthesis (2019) · cites this paper as [11]
In the citing paper
“Around 160 tools assist the reviewer in either one specific step (during record search [43], screening [39, 44, 45], data extraction [45], risk of bias assessment/ critical appraisal, etc.), or guide the user through several steps or the whole SR process [46–48].”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [42]
In the citing paper
“software repositories or toolboxes such as the SR toolbox [42, 70] exist to promote and share already available tools that assist the SR process”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [42]
In the citing paper
“Topic models have only recently been advocated for investigating free text in ecology and evolution 13,14 , for example, despite 15 years of testing in computer science 15 .”
From Software support for environmental evidence synthesis (2018) · cites this paper as [15]
Citations
Cites
The automation of systematic reviews
Tsafnat, G., Dunn, A., Glasziou, P., & Coiera, E. (2013) · BMJ · doi:10.1136/bmj.f139 · Cited by 131
Tools to Support Systematic Literature Reviews in Software Engineering: A Mapping Study
Marshall, C., & Brereton, P. (2013) · 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.32 · Cited by 87 · cited 1×
“Within software engineering, a broader mapping study of SR tools was performed, which identified a predominance of visualisation and text mining tools used to support study selection, data extraction and data synthesis [4].”
MethodSelf-citationMarshall, C., Brereton, P., & Kitchenham, B. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745827 · Cited by 46 · cited 1×
“EASE '15, April 27 - 29, 2015, Nanjing, China Copyright 2015 ACM 978-1-4503-3350-4/15/04…$15.00 http://dx.doi.org/10.1145/2745802.2745824 [5].”
InfluentialSelf-citationFelizardo, K. R., MacDonell, S. G., Mendes, E., & Maldonado, J. C. (2012) · Journal of Software · doi:10.4304/jsw.7.2.450-461 · Cited by 27
Marshall, C., Brereton, O. P., & Kitchenham, B. (2015) · Keele Research Repository (Keele University) · doi:10.25405/ncl.34394610 · Cited by 7
Self-citation
Research works printed but not linked: 1
- Kitchenham , B. A. , & Charters , S. ( 2007 ). Guidelines for performing systematic literature reviews in software engineering . Keele University and University of Durham , EBSE Technical Report Kitchenham, B. A., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Keele University and University of Durham, EBSE Technical Report
Cited by
McGuinness, L. A., & Higgins, J. P. T. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1411 · Cited by 5,579
Searching for and selecting studies
Lefebvre, C., Glanville, J., Briscoe, S., Littlewood, A., Marshall, C., Metzendorf, M., Noel‐Storr, A., Rader, T., Shokraneh, F., Thomas, J., et al. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch4 · Cited by 647
Self-citationMarshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
Harrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0897-3 · Cited by 449
revtools: An R package to support article screening for evidence synthesis
Westgate, M. J. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1374 · Cited by 261
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
On the pragmatic design of literature studies in software engineering: an experience-based guideline
Kuhrmann, M., Fernández, D. M., & Daneva, M. (2017) · Empirical Software Engineering · doi:10.1007/s10664-016-9492-y · Cited by 161
Searching for studies: A guide to information retrieval for Campbell systematic reviews
MacDonald, H., Comer, C., Foster, M., Labelle, P. R., Marsalis, S., Nyhan, K., Premji, Z., Rogers, M., Splenda, R., Stansfield, C., et al. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1433 · Cited by 98
Thielen, F., Van Mastrigt, G., Burgers, L., Bramer, W., Majoie, H., Evers, S., & Kleijnen, J. (2016) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2016.1246962 · Cited by 132
O’Connor, A. M., Tsafnat, G., Gilbert, S. B., Thayer, K. A., Shemilt, I., Thomas, J., Glasziou, P., & Wolfe, M. S. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0975-y · Cited by 110
How to conduct systematic reviews more expeditiously?
Tsertsvadze, A., Chen, Y.-F., Moher, D., Sutcliffe, P., & McCarthy, N. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0147-7 · Cited by 104
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
“Additionally software repositories or toolboxes such as the SR toolbox [42, 70] exist to promote and share already available tools that assist the SR process.”
BackgroundMethodRobledo, S., Grisales Aguirre, A. M., Hughes, M., & Eggers, F. (2023) · Journal of Small Business Management · doi:10.1080/00472778.2021.1955125 · Cited by 80
“Next, we turned to the systematic review toolbox 1 web page (Marshall & Brereton, 2015) to increase the pool of ML literature tools, which revealed 22 hits.”
MethodSantos, Á. O. D., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 64
Tóth, B., Berek, L., Gulácsi, L., Péntek, M., & Zrubka, Z. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02592-3 · Cited by 48
Nunn, J., & Chang, S. (2020) · WikiJournal of Medicine · doi:10.15347/wjm/2020.005 · Cited by 65
Software support for environmental evidence synthesis
Westgate, M. J., Haddaway, N. R., Cheng, S. H., McIntosh, E. J., Marshall, C., & Lindenmayer, D. B. (2018) · Nature Ecology & Evolution · doi:10.1038/s41559-018-0502-x · Cited by 51
Self-citationStern, C., Munn, Z., Porritt, K., Lockwood, C., Peters, M. D. J., Bellman, S., Stephenson, M., & Jordan, Z. (2018) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.12314 · Cited by 35
Orel, E., Ciglenečki, I., Thiabaud, A., Temerev, A., Calmy, A., Keiser, O., & Merzouki, A. (2023) · Journal of Medical Internet Research · doi:10.2196/39736 · Cited by 44
Walker, V. R., Schmitt, C. P., Wolfe, M. S., Nowak, A. J., Kulesza, K., Williams, A. R., Shin, R., Cohen, J., Burch, D., Stout, M. D., et al. (2022) · Environment International · doi:10.1016/j.envint.2021.107025 · Cited by 28
Priest, N., Doery, K., Truong, M., Guo, S., Perry, R., Trenerry, B., Karlsen, S., Kelly, Y. J., & Paradies, Y. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-043722 · Cited by 29
The Development of a Living Knowledge System and Implications for Future Systematic Searching
Grbin, L., Nichols, P., Russell, F., Fuller‐Tyszkiewicz, M., & Olsson, C. A. (2022) · Journal of the Australian Library and Information Association · doi:10.1080/24750158.2022.2087954 · Cited by 28
revtools: An R package to support article screening for evidence synthesisPreprint
Westgate, M. J. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/262881 · Cited by 24
Tool Features to Support Systematic Reviews in Software Engineering - A Cross Domain Study
Marshall, C., Kitchenham, B. A., & Brereton, P. (2018) · Keele Research Repository (Keele University) · doi:10.5277/e-inf180104 · Cited by 20
Self-citationKahili-Heede, M. K., & Hillgren, K. (2021) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2021.1263 · Cited by 17
Ndifon, E. M. (2022) · Research in Ecology · doi:10.30564/re.v4i2.4746 · Cited by 14
Scaling Systematic Literature Reviews with Machine Learning Pipelines
Goldfarb-Tarrant, S., Robertson, A., Lazić, J., Tsouloufi, T. K., Donnison, L. M., & Smyth, K. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.21 · Cited by 14
Towards Sustainability of Systematic Literature Reviews
dos Santos, V., Iwazaki, A. Y., Felizardo, K. R., de Souza, É. F., & Nakagawa, E. Y. (2021) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3475716.3484192 · Cited by 20
Porifera: A Collaborative Tool to Support Systematic Literature Review and Systematic Mapping Study
de Campos, T. P., Damasceno, E. F., & Valentim, N. (2022) · Brazilian Symposium on Software Engineering · doi:10.1145/3555228.3555273 · Cited by 12
Proposal and Evaluation of a Collaborative IS to Support Systematic Reviews and Mapping Studies
de Campos, T. P., Damasceno, E. F., & Valentim, N. (2022) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3535511.3535531 · Cited by 13
Tuono, R. D. M., Louokdom, J. S., Chetcha, B. C., Nya, P. C. B., Pieme, C. A., & Tagny, C. T. (2023) · Health Science Reports · doi:10.1002/hsr2.1609 · Cited by 8
Beyond the spreadsheet: reflections on tool support for literature studies
Tell, P., Cholewa, J. B., Nellemann, P., & Kuhrmann, M. (2016) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/2915970.2916011 · Cited by 7
Louokdom, J. S., Tuono, R. D. M., Njopwouo, M. S., & Tayou, C. T. (2026) · The Scientific World JOURNAL · doi:10.1155/tswj/8864775 · Cited by 4
Perspective: Towards Automated Tracking of Content and Evidence Appraisal of Nutrition Research
Yang, C., Hawwash, D., Baets, B. D., Bouwman, J., & Lachat, C. K. (2020) · Advances in Nutrition · doi:10.1093/advances/nmaa057 · Cited by 5
Have we reached the beginning of the end for review papers?
Smyth, B., & Cunningham, P. (2026) · Quantitative Science Studies · doi:10.1162/qss.a.488 · Cited by 3
Tuono, R. D. M., Louokdom, J. S., Njopwouo, M. S., & Tagny, C. T. (2025) · Thrombosis Journal · doi:10.1186/s12959-025-00804-x · Cited by 2
SyROCCo: enhancing systematic reviews using machine learning
Fang, Z., Arana‐Catania, M., van Lier, F.-A., Velarde, J. O., Bregazzi, H., Airoldi, M., Carter, E., & Procter, R. (2024) · Data & Policy · doi:10.1017/dap.2024.33 · Cited by 3
Robin: A Systematic Literature Mapping Management Tool
Abdullah, M., Töpfer, M., & Bureš, T. (2024) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa64295.2024.00051 · Cited by 1
Automation Tool for the Planning Phase of the Systematic Review
Zumaeta, R., Cuayla, E., & Mauricio, D. (2019) · International Engineering, Sciences and Technology Conference (IESTEC) · doi:10.1109/iestec46403.2019.00-62 · Cited by 2
Rodgers, J. B., & Ritskes-Hoitinga, M. J. (2021) · Laboratory animal science and medicine · doi:10.1007/978-3-030-66147-2_10 · Cited by 1
Exploring the synergies between biocuration and ontology alignment automation.
Dearing, D. D., & Goan, T. (2017) · OM@ISWC
Tool support for the search process of systematic literature reviews
Voigt, D. (2020) · OPUS Publication Server of the University of Stuttgart (University of Stuttgart) · doi:10.18419/opus-11300
Pauzi, Z., & Capiluppi, A. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-64182-4_6
Teaching Systematic Literature Reviews: Strategies and Best Practices
Genero, M., & Piattini, M. G. (2024) · Springer eBooks · doi:10.1007/978-3-031-71769-7_20
Decoupes, R., Cataldo, C., Busani, L., Mathieu Roche, A., & Teisseire, M. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1526820
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Figshare · doi:10.6084/m9.figshare.14361540
Metadata
- Authors
- Marshall, C., & Brereton, P.
- Affiliations
- Keele University (GB)
- Keele University, Staffordshire UK
- Venue
- Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2015-04-27
- Pages
- 1-6
- OpenAlex
- W2060536346
- Cited by
- 44
- References
- 5
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 7 · 2023: 4 · 2022: 5 · 2021: 8 · 2020: 4 · 2019: 5 · 2018: 4 · 2017: 2 · 2016: 2 · 2015: 1
- FWCI
- 1.46
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Newcastle University ePrints (Newcastle Univesity)
- Funders
- Keele University
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- software engineering, tool selection, evidence synthesis, research software tools
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.