Tools to support systematic reviews in software engineering: a cross-domain survey using semi-structured interviews
The DOI matches its Crossref record; 19 of the 20 references it cites were found in a registry or an index.
Abstract
Background: A number of software tools are being developed to support systematic reviewers within the software engineering domain. However, at present, we are not sure which aspects of the review process can most usefully be supported by such tools or what characteristics of the tools are most important to reviewers. Aim: The aim of the study is to explore the scope and practice of tool support for systematic reviewers in other disciplines. Method: Researchers with experience of performing systematic reviews in Healthcare and the Social Sciences were surveyed. Qualitative data was collected through semi-structured interviews and data analysis followed an inductive approach. Results: 13 interviews were carried out. 21 software tools categorised into one of seven types were identified. Reference managers were the most commonly mentioned tools. Features considered particularly important by participants were support for multiple users, support for data extraction and support for tool maintenance. The features and importance levels identified by participants were compared with those proposed for tools to support systematic reviews in software engineering. Conclusions: Many problems faced by systematic reviewers in other disciplines are similar to those faced in software engineering. There is general consensus across domains that improved tools are needed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These features were chosen in part based on previous studies on user preferences for systematic review software functionality [24, 25].”
From Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools (2018) · cites this paper as [25]
In the citing paper
“From a user perspective, we believe that CADIMA stands out in terms of ease of use, support for multiple users, support for on- or off-line data extraction, commitment to ongoing maintenance and financing, therefore meeting the criteria rated as most important by users of systematic review software in a recent study [25].”
From Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools (2018) · cites this paper as [25]
Citations
Cites
(1990) · Cited by 24,217
Qualitative Data Analysis: A Methods Sourcebook
Miles, M. B., Huberman, A. M., & Saldaña, J. (1988) · Cited by 20,701
Qualitative evaluation and research methods
Ellis, D. (1990) · International Journal of Information Management · doi:10.1016/0268-4012(90)90041-p · Cited by 17,943
The Good Research Guide: For Small-Scale Social Research Projects
Denscombe, M. (1998) · Cited by 4,557
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.07.009 · Cited by 2,394 · cited 2×
“The features (as summarised in Table 1) were based on the experiences of performing SRs in SE reported in the literature [1, 2, 3, 9], a preliminary screening of candidate tools and discussion amongst the researchers who performed the feature analysis.”
BackgroundMethodSelf-citationQualitative Research in Information Systems
Myers, M. D. (1997) · MIS Quarterly · doi:10.2307/249422 · Cited by 2,365
Studying Software Engineers: Data Collection Techniques for Software Field Studies
Lethbridge, T. C., Sim, S. E., & Singer, J. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-1290-x · Cited by 494
A Student's Guide to Methodology
Clough, P., & Nutbrown, C. (2012) · SAGE Publications Ltd eBooks · doi:10.4135/9781529682564 · Cited by 370 · cited 1×
“Semi-structured interviews rely heavily on the communication skills of the interviewer [15].”
BackgroundExperiences using systematic review guidelines
Staples, M., & Niazi, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.09.046 · Cited by 313 · cited 1×
“The features (as summarised in Table 1) were based on the experiences of performing SRs in SE reported in the literature [1, 2, 3, 9], a preliminary screening of candidate tools and discussion amongst the researchers who performed the feature analysis.”
MethodGomm, R. (2008) · doi:10.1007/978-0-230-22911-2 · Cited by 268 · cited 1×
“In addition, participant‟s responses can be influenced by what they think the situation requires [17].”
Background
Research works printed but not linked: 1
- Robson , C. ( 2002 ). Real word research . Oxford : Blackwell . Robson, C. (2002). Real word research. Oxford: Blackwell.
Cited by
Kohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0115-5 · Cited by 471
“…we believe that CADIMA stands out in terms of ease of use, support for multiple users, support for on-or off-line data extraction, commitment to ongoing maintenance and financing, therefore meeting the criteria rated as most important by users of systematic review software in a recent study [25].”
BackgroundMethodArtificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
“Most researchers continue to depend on manual methods, often supported by software like Microsoft Excel, or reference management tools [145] such as Zotero 14 and Mendeley 15 .”
On the pragmatic design of literature studies in software engineering: an experience-based guideline
Kuhrmann, M., Fernández, D. M., & Daneva, M. (2017) · Empirical Software Engineering · doi:10.1007/s10664-016-9492-y · Cited by 161
“A group around Marshall conducted research on tool support for literature studies [30,31,32,33].”
BackgroundMethodCierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
Finding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83
FAST2: An intelligent assistant for finding relevant papers
Yu, Z., & Menzies, T. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.021 · Cited by 80
Santoso, N. R., Sulistyaningtyas, I. D., & Pratama, B. P. (2022) · Journal of Communication Inquiry · doi:10.1177/01968599221095182 · Cited by 142
“The open-ended questions allowed for considerable freedom in the sequencing and various responses from the informants (Marshall et al., 2015) to get informants’ thoughts, insights, and perspectives from personal and sensitive issues.”
BackgroundSystematic review toolbox: a catalogue of tools to support systematic reviews
Marshall, C., & Brereton, P. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745824 · Cited by 44
“EASE '15, April 27 - 29, 2015, Nanjing, China Copyright 2015 ACM 978-1-4503-3350-4/15/04…$15.00 http://dx.doi.org/10.1145/2745802.2745824 [5].”
InfluentialSelf-citationIdentification of SLR tool needs – results of a community workshop
Hassler, E., Carver, J. C., Hale, D., & Al-Zubidy, A. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2015.10.011 · Cited by 35
Vision for SLR tooling infrastructure: Prioritizing value-added requirements
Al-Zubidy, A., Carver, J. C., Hale, D. P., & Hassler, E. E. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.06.007 · Cited by 27
Collecting qualitative data for information systems studies: The reality in practice
Iyamu, T. (2018) · Education and Information Technologies · doi:10.1007/s10639-018-9718-2 · Cited by 51
“The qualitative research often focuses on investigating and understanding of complex phenomena from social and cultural perspectives (Marshall et al. 2015).”
BackgroundInfluentialThoth: A Web-based Tool to Support Systematic Reviews
Marchezan, L., Bolfe, G., Rodrigues, E., Bernardino, M., & Basso, F. P. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870160 · Cited by 21
“However, there is also studies claiming for their improvements [8].”
BackgroundMethodInfluentialA Model-Based Approach to Systematic Review of Research Literature
Barat, S., Clark, T., Barn, B. S., & Kulkarni, V. (2017) · Innovations in Software Engineering Conference (ISEC) · doi:10.1145/3021460.3021462 · Cited by 25
“concluded their comparative study on SR tools [21] with a similar observation.”
BackgroundReducing the effort for systematic reviews in software engineering
Osborne, F., Muccini, H., Lago, P., & Motta, E. (2019) · Data Science · doi:10.3233/ds-190019 · Cited by 21
“[31] carried out an interview survey with experts in other domains (i.”
BackgroundSantoso, N. R., Sulistyaningtyas, I. D., & Pratama, B. P. (2023) · Communication & Society · doi:10.15581/003.36.2.187-204 · Cited by 34
Tool Features to Support Systematic Reviews in Software Engineering - A Cross Domain Study
Marshall, C., Kitchenham, B. A., & Brereton, P. (2018) · Keele Research Repository (Keele University) · doi:10.5277/e-inf180104 · Cited by 20
“[19] In this study this analysis was extended to consider the impact of participant experience as shown in Table 24.”
BackgroundMethodResultInfluentialSelf-citationGonzalez-Toral, S., Freire, R., Gualán, R., & Saquicela, V. (2019) · Latin American Computer Conference (CLEI) · doi:10.1109/clei47609.2019.235079 · Cited by 16
“carried out various systematic mapping studies [10], [19], [20] to identify and classify tools aiming to support the SLR process.”
BackgroundHow to Read Less: Better Machine Assisted Reading Methods for Systematic Literature Reviews.Preprint
Yu, Z., Kraft, N. A., & Menzies, T. (2016) · arXiv (Cornell University) · Cited by 9
Felizardo, K. R., Ramos, A. M., Melo, C. D. O., de Souza, É. F., Vijaykumar, N. L., & Nakagawa, E. Y. (2021) · Journal of the Brazilian Computer Society · doi:10.1186/s13173-021-00109-7 · Cited by 11
How do Secondary Studies in Software Engineering report Automated Searches? A Preliminary Analysis
Singh, P., Galster, M., & Singh, K. (2018) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3210459.3210474 · Cited by 9
“Search strategies and digital libraries: Marshall et al. [17] investigated tool support for SLRs and found that support for automated searches is a “highly desirable” SLR tool feature.”
Dong, X., Tang, C., Lian, Y., & Tang, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01309 · Cited by 9
“Study Design Since the goal of this study is to both “explore” and “measure” the needs of entrepreneurial followers in entrepreneurship cultivation, it not only demands a more qualitative design (Marshall et al., 2015), but also requires a quantitative consideration.”
BackgroundEmbedded System Design Student’s Learning Readiness Instruments: Systematic Literature Review
Elfadil, N., & Ibrahim, I. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.799683 · Cited by 9
“The Likert scale (Azhar et al., 2012), semantic differential (Marshall et al., 2015), visual analog, and numerical response formats (Mundy et al., 2015) are only few of the instruments that have been employed in social science.”
BackgroundSoftware Engineering Research at the International Conference on Software Engineering in 2016
Theisen, C., Dunaiski, M., Williams, L. A., & Visser, W. (2018) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/3149485.3149496 · Cited by 12
“generated tools in support of some of these practices [20].”
BackgroundNapoleão, B. M., Petrillo, F. D. S., & Hallé, S. (2021) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00015 · Cited by 7
“There are several initiatives [3] and available tools [12] for automating or semi-automating activities of the SLR process.”
MethodBeyond the spreadsheet: reflections on tool support for literature studies
Tell, P., Cholewa, J. B., Nellemann, P., & Kuhrmann, M. (2016) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/2915970.2916011 · Cited by 7
“Previous work, in particular [7, 11], is used afterwards as an evaluation tool to assess the solutions via a feature analysis [5].”
BackgroundMethodInfluentialEstablishing a Search String to Detect SecondaryStudies in Software Engineering
Napoleão, B. M., Felizardo, K. R., de Souza, É. F., Petrillo, F. D. S., Hallé, S., Vijaykumar, N. L., & Nakagawa, E. Y. (2021) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · doi:10.5281/zenodo.4638201 · Cited by 4
ReViz: A Tool for Automatically Generating Citation Graphs and Variants
Groppe, S., & Hartung, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64452-9_10 · Cited by 4
Yünkül, E. (2022) · Journal of Educational Technology and Online Learning · doi:10.31681/jetol.1118048 · Cited by 3
Managing the State of the Art of Engineering: Learning from Medicine
Silva, É. R., Bartholo, R., & Júnior, D. P. (2018) · Philosophy of engineering and technology · doi:10.1007/978-3-319-91029-1_15 · Cited by 3
Santoso, N. R. (2022) · Jurnal Studi Komunikasi (Indonesian Journal of Communications Studies) · doi:10.25139/jsk.v6i1.4137 · Cited by 4
“The questions were open-ended allowing various and rich responses from the informants (Marshall et al., 2015).”
BackgroundEvidence-Based Systematic Literature Reviews in the Cloud
Ruiz‐Rube, I., Person, T., Mota, J. M., Dodero, J. M., & González-Toro, Á. R. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-03496-2_12 · Cited by 2
Wang, W. M., Lünnemann, P., Neumeyer, S., Hayka, H., & Stark, R. (2016) · IFIP advances in information and communication technology · doi:10.1007/978-3-319-45390-3_26 · Cited by 2
QUALITATIVE ASSESSMENT OF SYSTEMATIC LITERATURES IN SOFTWARE ENGINEERING
Okesola, O., Okokpujie, K., Adebiyi, A. A., & Ayo, C. K. (2018) · Covenant University Repository (Covenant University) · Cited by 4
Testing Reading Tactics for Automated Reading Assistance: Is it Useful to Apply Old Knowledge?Preprint
Yu, Z., & Menzies, T. (2017) · arXiv (Cornell University) · Cited by 2
Faroun, H., Zary, N., Baqer, K., Alkhaja, F., Gad, K., Alameddine, M. S., & Suwaidi, H. S. A. (2022) · JMIR Research Protocols · doi:10.2196/42278 · Cited by 1
Reducing the Effort for Systematic Reviews in Software Engineering
Osborne, F., Muccini, H., Lago, P., & Motta, E. (2019) · Open Research Online (The Open University) · doi:10.48550/arxiv.1908.06676 · Cited by 1
“[31] carried out an interview survey with experts in other domains (i.”
BackgroundProposal of an Implementation Methodology of ICT Processes
Canedo, E. D., da Costa, R. P., Amaral, L. H. V., Coutinho, M., Nze, G. D. A., & de Sousa, R. T. (2019) · Information · doi:10.3390/info10110327 · Cited by 1
“Semi-structured interviews have as main objective to understand the meanings that the interviewees attribute to the issues and situations related to the topics of interest [28].”
BackgroundMethodPemberdayaan Ibu-ibu Kelompok Emping Desa Kalirejo Melalui In House Training and Workshop
Nisa, K., & Prasetyo, I. (2020) · Journal of Millennial Community · doi:10.24114/jmic.v2i1.16714
How do Colombian Software Companies evaluate Software Product Quality
Charry, W. P. (2020) · UTS ePRESS (University of Technology Sydney)
Pauzi, Z., & Capiluppi, A. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-64182-4_6
Ara, E., Seddiky, M. A., Basit, A., & Balaly, M. H. (2024) · World Journal of Vaccines · doi:10.4236/wjv.2024.142002 · Cited by 1
Remote Teaching's Legacy Perceptions: An Interview Study with Brazilian Education Professionals
Lima, M. D. S., David, J., & Maciel, P. (2024) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2024.242706
Teaching Systematic Literature Reviews: Strategies and Best Practices
Genero, M., & Piattini, M. G. (2024) · Springer eBooks · doi:10.1007/978-3-031-71769-7_20
LitFlow: An Integrated, AI-Augmented Systematic Literature Review Platform
Näscher, H.-H., Strohmann, T., & Brocke, J. V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-28570-6_30 · Cited by 1
Oyatola, O. S. C., Bachelier, B., Oliviera, T. D., Laboisse, S., & Diogo, R. V. C. (2026) · Agronomy Journal · doi:10.1002/agj2.70505
Metadata
- Authors
- Marshall, C., Brereton, P., & Kitchenham, B.
- Affiliations
- Keele University (GB)
- Keele University, Staffordshire UK
- ORCID
- Kitchenham, Barbara
- Venue
- Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2015-04-27
- Pages
- 1-6
- OpenAlex
- W1986251315
- Cited by
- 46
- References
- 20
- Citations by year
- 2026: 2 · 2024: 5 · 2023: 1 · 2022: 6 · 2021: 3 · 2020: 3 · 2019: 7 · 2018: 10 · 2017: 4 · 2016: 3 · 2015: 2
- FWCI
- 3.08
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- systematic reviews, software engineering, review tools, reference management software, data extraction, tool maintenance, multi-user support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.