Computational Literature Reviews: Method, Algorithms, and Roadmap
The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
The substantial volume, continued growth, and resulting complexity of the scientific literature not only increases the need for systematic, replicable, and rigorous literature reviews, but also highlights the natural limits of human researchers’ information processing capabilities. In search of a solution to this dilemma, computational techniques are beginning to support human researchers in synthesizing large bodies of literature. However, actionable methodological guidance on how to design, conduct, and document such computationally augmented literature reviews is lacking to date. We respond by introducing and defining computational literature reviews (CLRs) as a new review method and put forward a six-step roadmap, covering the CLR process from identifying the review objectives to selecting algorithms and reporting findings. We make the CLR method accessible to novice and expert users alike by identifying critical design decisions and typical challenges for each step and provide practical guidelines for tailoring the CLR method to four conceptual review goals. As such, we present CLRs as a literature review method where the choice, design, and implementation of a CLR are guided by specific review objectives, methodological capabilities, and resource constraints of the human researcher.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Exploration and Exploitation in Organizational Learning
March, J. G. (1991) · Organization Science · doi:10.1287/orsc.2.1.71 · Cited by 21,942
Bibliometric Methods in Management and Organization
Zupic, I., & Čater, T. (2015) · Organizational Research Methods · doi:10.1177/1094428114562629 · Cited by 7,495
Griffiths, T. L., & Steyvers, M. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0307752101 · Cited by 6,049 · cited 3×
“…explicating is a conceptual goal that can be found in numerous CLRs, including the first machine-drafted book by Writer (2019), which summarizes scientific articles in the field of lithium-ion batteries, or Griffiths and Steyvers (2004) Sciences ( PNAS ) to describe the corpus of PNAS publications.”
BackgroundMethodInfluentialBlei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516 · cited 3×
“For instance, a CLR using LDA yields a list of topics with associated terms, as well as indicators highlighting how strongly each article within the text corpus is related to each topic (e.g., Blei, 2012).”
BackgroundMethodText as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297 · cited 2×
“Ultimately, and despite major advances in text mining and other areas of machine learning, today’s computational content analysis techniques still have a long way to go before they can match the human ability to understand meaning (Grimmer & Stewart, 2013).”
BackgroundSiddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550 · cited 3×
“Finally, as we explained at the outset, conducting literature reviews involves “a critique of the extent, nature, and quality of evidence in relation to a particular research question” (Siddaway et al., 2019, p. 1).”
BackgroundMethodInfluentialDahlander, L., & Gann, D. M. (2010) · Research Policy · doi:10.1016/j.respol.2010.01.013 · Cited by 2,435
Writing Integrative Literature Reviews: Using the Past and Present to Explore the Future
Torraco, R. J. (2016) · Human Resource Development Review · doi:10.1177/1534484316671606 · Cited by 2,031 · cited 2×
“Future researchers could present and communicate their CLR findings using one of the following four forms of synthesis: (a) research agenda, (b) taxonomy, (c) alternative models, and (d) meta-theory (Post et al., 2020; Torraco, 2016).”
MethodMeta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews
Rosenthal, R., & DiMatteo, M. R. (2001) · Annual Review of Psychology · doi:10.1146/annurev.psych.52.1.59 · Cited by 1,780 · cited 1×
“The scientific knowledge available to humankind has increased exponentially since the beginning of the 20th century (Rosenthal & DiMatteo, 2001).”
BackgroundSystematic Reviews: Rationale for systematic reviews
Mulrow, C. D. (1994) · BMJ · doi:10.1136/bmj.309.6954.597 · Cited by 1,677
A Framework for Conceptual Contributions in Marketing
MacInnis, D. J. (2011) · Journal of Marketing · doi:10.1509/jmkg.75.4.136 · Cited by 1,547 · cited 7×
“Identifying refers to establishing a new “construct, theory, procedure, domain, discipline, or aspect of science that has yet to be apprehended” (MacInnis, 2011, p. 143).”
BackgroundMethodInfluentialtopicmodels : An R Package for Fitting Topic Models
Grün, B., & Hornik, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v040.i13 · Cited by 1,072
EDITOR'S COMMENTS: CONSTRUCT CLARITY IN THEORIES OF MANAGEMENT AND ORGANIZATION.
Suddaby, R. (2010) · Academy of Management Review · doi:10.5465/amr.2010.51141319 · Cited by 984
Short, J. C., Broberg, J. C., Cogliser, C. C., & Brigham, K. H. (2010) · Organizational Research Methods · doi:10.1177/1094428109335949 · Cited by 644
Schmiedel, T., Müller, O., & vom Brocke, J. (2019) · Organizational Research Methods · doi:10.1177/1094428118773858 · Cited by 319 · cited 2×
“…as extant attempts to guide future researchers adopted a narrow technical focus on individual text mining algorithms such as topic modeling (e.g., Schmiedel et al., 2019), their requirements for text preprocessing (e.g., Ru¨diger et al., 2017), or programming languages (e.g., Hornik & Gru¨n,…”
BackgroundText mining for information systems researchers: An annotated topic modeling tutorial
University of Liechtenstein, Debortoli, S., Müller, O., IT University of Copenhagen, Junglas, I., Florida State University, vom Brocke, J., & University of Liechtenstein (2016) · Communications of the Association for Information Systems · doi:10.17705/1cais.03907 · Cited by 199
General purpose computer-assisted clustering and conceptualization
Grimmer, J., & King, G. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1018067108 · Cited by 179 · cited 2×
“Hence, CAC appears well suited to support envisioning by exposing human researchers to diverse and potentially novel ways of grouping scientific texts in a domain (Grimmer & King, 2011).”
BackgroundMethodWriting Nonempirical Articles for Journal of Management: General Thoughts and Suggestions
Cropanzano, R. (2009) · Journal of Management · doi:10.1177/0149206309344118 · Cited by 176 · cited 3×
“…extracting critical evidence (Briner et al., 2009), detecting subtle cues (Denyer & Tranfield, 2009), and recognizing latent patterns (Simon, 1990) from large and complex scientific text corpora will, at best, become more costly and lengthy and, at worst, become infeasible (Cropanzano, 2009).”
BackgroundAutomatic text classification to support systematic reviews in medicine
García Adeva, J. J., Pikatza Atxa, J. M., Ubeda Carrillo, M., & Ansuategi Zengotitabengoa, E. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.08.047 · Cited by 125 · cited 2×
“For example, Adeva et al. (2014) applied ATC to screen 1,941 medical research articles associated with 61,752 unique terms based on the article’s title, author list, journal, keywords, and abstract.”
MethodLithium-Ion Batteries: A Machine-Generated Summary of Current Research
Writer, B. (2019) · doi:10.1007/978-3-030-16800-1 · Cited by 72
Cited by
Marzi, G., Balzano, M., Caputo, A., & Pellegrini, M. M. (2025) · International Journal of Management Reviews · doi:10.1111/ijmr.12381 · Cited by 577
Artificial intelligence and the conduct of literature reviews
Wagner, G., Lukyanenko, R., & Paré, G. (2022) · Journal of Information Technology · doi:10.1177/02683962211048201 · Cited by 296
Klarin, A. (2024) · International Journal of Consumer Studies · doi:10.1111/ijcs.13031 · Cited by 292
Advancing literature review methodology through rigour, generativity, scope and transparency
Fan, D., Breslin, D., Callahan, J. L., & Iszatt‐White, M. (2022) · International Journal of Management Reviews · doi:10.1111/ijmr.12291 · Cited by 283
Leadership in the digital era: A review of who, what, when, where, and why
Banks, G. C., Dionne, S. D., Mast, M. S., & Sayama, H. (2022) · The Leadership Quarterly · doi:10.1016/j.leaqua.2022.101634 · Cited by 155
Metadata
- Authors
- Antons, D., Breidbach, C. F., Joshi, A. M., & Salge, T. O.
- Venue
- Organizational Research Methods
- Publisher
- SAGE Publications
- Published
- 2021-03-09
- Volume
- 26
- Issue
- 1
- Pages
- 107-138
- ISSN
- 1094-4281 (ISSN-L)
- 1552-7425
- OpenAlex
- W3135734885
- Cited by
- 147
- References
- 20
- FWCI
- 11.16
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.