(Re)considering the Concept of Literature Review Reproducibility
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Literature reviews play a key role in academic research by describing, understanding, explaining, and testing the constructs and theories within a particular topic area. In recent years, various commentaries, debates, and editorials in the information systems (IS) field’s top journals have highlighted the importance of a trustworthy literature review process, including detailed discussions on systematicity and transparency. Although the reproducibility of a literature review has also been noted as important, it remains less recognized because of several terminology-related issues. This ambiguity could result in misunderstandings regarding the degree of trust that should be placed in a literature review’s process. In this research essay, we seek to clarify what makes a literature review reproducible, how it is distinct from related concepts, and when achieving it is desirable and feasible. We propose a series of clarifications and remedies to assist scholars within and outside the IS field in the preparation of stand-alone reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Synthesizing information systems knowledge: A typology of literature reviews
Paré, G., Trudel, M.-C., Jaana, M., & Kitsiou, S. (2015) · Information & Management · doi:10.1016/j.im.2014.08.008 · Cited by 1,987
Self-citationShea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A., & Boers, M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.009 · Cited by 1,755 · cited 1×
“Similarly, Paré et al. (2016) (cf. Shea et al., 2009; Liberati et al., 2009) define transparency as “the completeness with which a review is presented and whether important methodological aspects about its design and execution are clearly or explicitly reported” (p. 497).”
BackgroundReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 6×
“From this perspective, reproducibility is achieved when data from an existing study are made available to an independent party for reanalysis in order to duplicate the study’s results (Bollen et al., 2015; Cassey & Blackburn, 2006; Goodman et al., 2016; Peng, 2011).”
BackgroundMethodInfluentialA Guide to Conducting a Standalone Systematic Literature Review
Okoli, C. (2015) · Communications of the Association for Information Systems · doi:10.17705/1cais.03743 · Cited by 1,413
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · cited 4×
“Secondly, it distinguishes this activity from the concept of reproducibility, in terms of findings that could be duplicated by an independent party, based on existing data (e.g., Bollen et al., 2015; Cassey & Blackburn, 2006; Goodman et al., 2016; Peng, 2011).”
BackgroundMethodInfluentialPublication Bias in Meta‐Analysis
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch1 · Cited by 603 · cited 1×
“Similarly, a common technique in literature reviews, particularly meta-analyses, is to collect unpublished studies in order to minimize the risk of publication bias (Rothstein, Sutton, & Borenstein, 2005; Schmidt & Hunter, 2015).”
MethodA Framework for Guiding and Evaluating Literature Reviews
Templier, M., & Paré, G. (2015) · Communications of the Association for Information Systems · doi:10.17705/1cais.03706 · Cited by 553
Self-citationMaking literature reviews more reliable through application of lessons from systematic reviews
Haddaway, N. R., Woodcock, P., Macura, B., & Collins, A. (2015) · Conservation Biology · doi:10.1111/cobi.12541 · Cited by 416 · cited 1×
“This could include review techniques, such as those used in biology, where care is taken to mitigate potential bias in the literature search process and to equally consider the evidence as a whole, rather than to focus on specific studies (Haddaway et al., 2015).”
BackgroundTemplier, M., & Paré, G. (2018) · European Journal of Information Systems · doi:10.1080/0960085x.2017.1398880 · Cited by 180 · cited 11×
“A variety of views on either systematicity, transparency, or both have figured prominently in the recent IS literature review guidance of Boell & Cecez-Kecmanovic (2015a, 2015b), Paré et al. (2016), Rowe (2014), Templier and Paré (2018), and vom Brocke et al. (2015).”
BackgroundMethodResultInfluentialSelf-citationReview and Theory Symbiosis: An Introspective Retrospective
Baylor University and Lund University, & Leidner, D. (2018) · Journal of the Association for Information Systems · doi:10.17705/1jais.00501 · Cited by 102 · cited 10×
“For example, the Journal of the Association for Information Systems has data policies in place, including the requirement that “all authors using empirical datasets have to make them available on request for checking by senior editors or reviewers after care has been taken to anonymize the data.””
BackgroundInfluential
Cited by
Dwivedi, Y. K., Hughes, D. L., Coombs, C., Constantiou, I., Duan, Y., Edwards, J. S., Gupta, B., Lal, B., Misra, S., Prashant, P., et al. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2020.102211 · Cited by 1,232
Artificial intelligence in E-Commerce: a bibliometric study and literature review
Bawack, R. E., Wamba, S. F., Carillo, K. D. A., & Akter, S. (2022) · Electronic Markets · doi:10.1007/s12525-022-00537-z · Cited by 372
Normalising the “new normal”: Changing tech-driven work practices under pandemic time pressure
Carroll, N., & Conboy, K. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2020.102186 · Cited by 359
Artificial intelligence and the conduct of literature reviews
Wagner, G., Lukyanenko, R., & Paré, G. (2022) · Journal of Information Technology · doi:10.1177/02683962211048201 · Cited by 296
Leong, L.-Y., Hew, T.-S., Ooi, K.-B., Metri, B., & Dwivedi, Y. K. (2023) · Information Systems Frontiers · doi:10.1007/s10796-022-10337-7 · Cited by 80
“3 was applied in this study to gather the pertinent data (Cram et al., 2020; Templier & Paré, 2015).”
Method
Metadata
- Authors
- University of Waterloo, Cram, W. A., Templier, M., Laval, U., Pare, G., & Montréal, H.
- Venue
- Journal of the Association for Information Systems
- Publisher
- Association for Information Systems
- Published
- 2020-09-01
- Volume
- 21
- Issue
- 5
- Pages
- 1103-1114
- ISSN
- 1536-9323 (ISSN-L)
- OpenAlex
- W3090054056
- Cited by
- 55
- References
- 10
- FWCI
- 4.12
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.