The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
An automated approach to identifying search terms for systematic reviews using keyword co‐occurrence networks
Abstract
Systematic review, meta‐analysis and other forms of evidence synthesis are critical to strengthen the evidence base concerning conservation issues and to answer ecological and evolutionary questions. Synthesis lags behind the pace of scientific publishing, however, due to time and resource costs which partial automation of evidence synthesis tasks could reduce. Additionally, current methods of retrieving evidence for synthesis are susceptible to bias towards studies with which researchers are familiar. In fields that lack standardized terminology encoded in an ontology, including ecology and evolution, research teams can unintentionally exclude articles from the review by omitting synonymous phrases in their search terms. To combat these problems, we developed a quick, objective, reproducible method for generating search strategies that uses text mining and keyword co‐occurrence networks to identify the most important terms for a review. The method reduces bias in search strategy development because it does not rely on a predetermined set of articles and can improve search recall by identifying synonymous terms that research teams might otherwise omit. When tested against the search strategies used in published environmental systematic reviews, our method performs as well as the published searches and retrieves gold‐standard hits that replicated versions of the original searches do not. Because the method is quasi‐automated, the amount of time required to develop a search strategy, conduct searches, and assemble results is reduced from approximately 17–34 hr to under 2 hr. To facilitate use of the method for environmental evidence synthesis, we implemented the method in the R package litsearchr, which also contains a suite of functions to improve efficiency of systematic reviews by automatically deduplicating and assembling results from separate databases.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096 · cited 1×
“To stem terms and capture additional word forms when searching databases, litsearchr uses a stemming algo - rithm (Porter, 1980) to reduce each word to its root form if its stem is at least four characters long, which is the length suggested by in - formation specialists to balance efficiency with…”
MethodPrimary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“…to evidence synthesis, whereby a formal strategy is used to study the way in which a topic has been stud - ied, originated in clinical fields (Glass, 1976; Hunt, 1997; Smith & Glass, 1977), but have been co‐opted by ecologists in recognition of the need for more rigorous methods of reviewing…”
BackgroundThe well-built clinical question: A key to evidence-based decisions
(1995) · ACP Journal Club · doi:10.7326/acpjc-1995-123-3-a12 · Cited by 2,931
Meta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154 · cited 1×
“…whereby a formal strategy is used to study the way in which a topic has been stud - ied, originated in clinical fields (Glass, 1976; Hunt, 1997; Smith & Glass, 1977), but have been co‐opted by ecologists in recognition of the need for more rigorous methods of reviewing the litera - ture…”
MethodHandbook of Meta-analysis in Ecology and Evolution
(2013) · doi:10.1515/9781400846184 · Cited by 1,183
Guidelines for systematic review in Conservation and environmental management
PULLIN, A. S., & STEWART, G. B. (2006) · Conservation Biology · doi:10.1111/j.1523-1739.2006.00485.x · Cited by 1,137
Automatic Keyword Extraction from Individual Documents
Rose, S., Engel, D., Cramer, N., & Cowley, W. (2010) · Text Mining · doi:10.1002/9780470689646.ch1 · Cited by 1,095
Haddaway, N. R., Macura, B., Whaley, P., & Pullin, A. S. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0121-7 · Cited by 946
Su, H.-N., & Lee, P.-C. (2010) · Scientometrics · doi:10.1007/s11192-010-0259-8 · Cited by 916
A systematic approach to searching: an efficient and complete method to develop literature searches
Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018) · Journal of the Medical Library Association · doi:10.5195/jmla.2018.283 · Cited by 863
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 3×
“…on search strategy development and it - eratively test their search strategies against a set of known articles to select more specific, yet still comprehensive, keywords to main - tain high precision without sacrificing the total number of suitable articles retrieved (O’Mara‐Eves et al., 2015).”
BackgroundMethodEffectiveness in Conservation Practice: Pointers from Medicine and Public Health
Pullin, A. S., & Knight, T. M. (2001) · Conservation Biology · doi:10.1046/j.1523-1739.2001.99499.x · Cited by 481
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · cited 1×
“…for automation of the most time‐intensive tasks while still maintaining the standards of conventional search methods (Delaney & Tams, 2018; O’Mara‐Eves, Thomas, McNaught, Miwa, & Ananiadou, 2015; Paisley & Foster, 2018; Tsafnat et al., 2014; Tsertsvadze, Chen, Moher, Sutcliffe, & McCarthy, 2015).”
MethodGurevitch, J., Curtis, P. S., & Jones, M. H. (2001) · Advances in Ecological Research · doi:10.1016/s0065-2504(01)32013-5 · Cited by 421
Making literature reviews more reliable through application of lessons from systematic reviews
Haddaway, N. R., Woodcock, P., Macura, B., & Collins, A. (2015) · Conservation Biology · doi:10.1111/cobi.12541 · Cited by 416 · cited 1×
“…select keywords based on their own knowledge of the field and fail to specify how the search strategy was developed, it is also susceptible to selection bias (Haddaway, Woodcock, Macura, & Collins, 2015), irreproducibility, and low recall if researchers do not select a comprehensive set of terms.”
BackgroundRadhakrishnan, S., Erbis, S., Isaacs, J. A., & Kamarthi, S. (2017) · PLOS ONE · doi:10.1371/journal.pone.0172778 · Cited by 374
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Doing more good than harm – Building an evidence-base for conservation and environmental management
Pullin, A. S., & Knight, T. M. (2009) · Biological Conservation · doi:10.1016/j.biocon.2009.01.010 · Cited by 297
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287
How Science Takes Stock: The Story of Meta-Analysis
Thacker, S. B. (1998) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.279.3.244 · Cited by 217
The benefits of systematic mapping to evidence-based environmental management
Haddaway, N. R., Bernes, C., Jonsson, B.-G., & Hedlund, K. (2016) · Ambio · doi:10.1007/s13280-016-0773-x · Cited by 191
Rathbone, J., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0067-6 · Cited by 184 · cited 1×
“…ad - vancements have enabled automation of key tasks such as re - moving duplicate articles, prioritizing articles for screening, and extracting data from tables and figures (Jonnalagadda, Goyal, & Huffman, 2015; Przybyla et al., 2018; Rathbone, Hoffmann, & Glasziou, 2015; Shemilt et al., 2014).”
MethodShemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
Prioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148
Predicting the time needed for environmental systematic reviews and systematic maps
Haddaway, N. R., & Westgate, M. J. (2019) · Conservation Biology · doi:10.1111/cobi.13231 · Cited by 120
Routine development of objectively derived search strategies
Hausner, E., Waffenschmidt, S., Kaiser, T., & Simon, M. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-19 · Cited by 118
How to conduct systematic reviews more expeditiously?
Tsertsvadze, A., Chen, Y.-F., Moher, D., Sutcliffe, P., & McCarthy, N. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0147-7 · Cited by 104 · cited 1×
“…for automation of the most time‐intensive tasks while still maintaining the standards of conventional search methods (Delaney & Tams, 2018; O’Mara‐Eves, Thomas, McNaught, Miwa, & Ananiadou, 2015; Paisley & Foster, 2018; Tsafnat et al., 2014; Tsertsvadze, Chen, Moher, Sutcliffe, & McCarthy, 2015).”
MethodBernes, C., Bråthen, K. A., Forbes, B. C., Speed, J. D., & Moen, J. (2015) · Environmental Evidence · doi:10.1186/s13750-014-0030-3 · Cited by 103
A fresh approach to evidence synthesis
Sutherland, W. J., & Wordley, C. F. R. (2018) · Nature · doi:10.1038/d41586-018-05472-8 · Cited by 94
Samson, D. J., & Schoelles, K. M. (2012) · Journal of General Internal Medicine · doi:10.1007/s11606-012-2007-7 · Cited by 81
Citation analysis as a literature search method for systematic reviews
Belter, C. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23605 · Cited by 77
Evaluation of a new method for librarian‐mediated literature searches for systematic reviews
Bramer, W. M., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1279 · Cited by 71
Hausner, E., Guddat, C., Hermanns, T., Lampert, U., & Waffenschmidt, S. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.016 · Cited by 63
Knot selection for least-squares and penalized splines
Spiriti, S., Eubank, R. L., Smith, P. W., & Young, D. L. (2012) · Journal of Statistical Computation and Simulation · doi:10.1080/00949655.2011.647317 · Cited by 52
Hausner, E., Guddat, C., Hermanns, T., Lampert, U., & Waffenschmidt, S. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.05.002 · Cited by 47
Stansfield, C., O’Mara-Eves, A., & Thomas, J. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1250 · Cited by 41
Delaney, A., & Tamás, P. A. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1282 · Cited by 40
4. Gathering Data: Searching Literature and Selection Criteria
Côté, I. M., Curtis, P. S., Rothstein, H. R., & Stewart, G. B. (2013) · Princeton University Press eBooks · doi:10.1515/9781400846184-006 · Cited by 35
Gathering Data: Searching Literature and Selection Criteria
Côté, I. M., Curtis, P. S., Rothstein, H. R., & Stewart, G. B. (2013) · Princeton University Press eBooks · doi:10.23943/princeton/9780691137285.003.0004 · Cited by 30
Laverick, J. H., Andradi‐Brown, D. A., Exton, D. A., Bongaerts, P., Bridge, T. C. L., Lesser, M. P., Pyle, R. L., Slattery, M., Wagner, D., & Rogers, A. D. (2016) · Environmental Evidence · doi:10.1186/s13750-016-0068-5 · Cited by 27
An empirical assessment of a systematic search process for systematic reviews
Zhang, H., Babar, M. A., Bai, X., Li, J., & Huang, L. (2011) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1049/ic.2011.0007 · Cited by 24
Testing a Citation and Text-Based Framework for Retrieving Publications for Literature Reviews
Sarol, M. J., Liu, L., & Schneider, J. (2018) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 9
Innovation in information retrieval methods for evidence synthesis studies
Paisley, S., & Foster, M. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1322 · Cited by 5
Cited by
McGuinness, L. A., & Higgins, J. P. T. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1411 · Cited by 5,579
Conducting systematic literature reviews and bibliometric analyses
Linnenluecke, M. K., Marrone, M., & Singh, A. K. (2020) · Australian Journal of Management · doi:10.1177/0312896219877678 · Cited by 1,649
Pagano, T. P., Loureiro, R. B., Lisboa, F. V. N., Peixoto, R. M., Guimarães, G. A. S., Cruz, G. O. R., Araujo, M. M., Santos, L. L., Cruz, M. A. S., Oliveira, E. L. S., et al. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7010015 · Cited by 311
revtools: An R package to support article screening for evidence synthesis
Westgate, M. J. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1374 · Cited by 261
Resource use during systematic review production varies widely: a scoping review
Nussbaumer-Streit, B., Ellen, M., Klerings, I., Sfetcu, R., Riva, N., Mahmić-Kaknjo, M., Poulentzas, G., Martinez, P., Baladia, E., Ziganshina, L. E., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.019 · Cited by 98
Metadata
- Authors
- Grames, E. M., Stillman, A. N., Tingley, M. W., & Elphick, C. S.
- Venue
- Methods in Ecology and Evolution
- Publisher
- Wiley
- Published
- 2019-08-11
- Volume
- 10
- Issue
- 10
- Pages
- 1645-1654
- ISSN
- 2041-210X (ISSN-L)
- OpenAlex
- W2963483681
- Cited by
- 377
- References
- 44
- FWCI
- 4.36
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.