An exploration of crowdsourcing citation screening for systematic reviews
Open accessThe DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Systematic reviews are increasingly used to inform health care decisions, but are expensive to produce. We explore the use of crowdsourcing (distributing tasks to untrained workers via the web) to reduce the cost of screening citations. We used Amazon Mechanical Turk as our platform and 4 previously conducted systematic reviews as examples. For each citation, workers answered 4 or 5 questions that were equivalent to the eligibility criteria. We aggregated responses from multiple workers into an overall decision to include or exclude the citation using 1 of 9 algorithms and compared the performance of these algorithms to the corresponding decisions of trained experts. The most inclusive algorithm (designating a citation as relevant if any worker did) identified 95% to 99% of the citations that were ultimately included in the reviews while excluding 68% to 82% of irrelevant citations. Other algorithms increased the fraction of irrelevant articles excluded at some cost to the inclusion of relevant studies. Crowdworkers completed screening in 4 to 17 days, costing $460 to $2220, a cost reduction of up to 88% compared to trained experts. Crowdsourcing may represent a useful approach to reducing the cost of identifying literature for systematic reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“We report Fleiss Kappa scores for each review (calculated independently for each question) in Table 5.(22) One can observe quite a bit of variance; agreement ranges from poor to moderate.”
MethodLintott, C. J., Schawinski, K., Slosar, A., Land, K., Bamford, S., Thomas, D., Raddick, M. J., Nichol, R. C., Szalay, A., Andreescu, D., et al. (2008) · Monthly Notices of the Royal Astronomical Society · doi:10.1111/j.1365-2966.2008.13689.x · Cited by 1,488
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · cited 1×
“Researchers can no longer keep up with the primary literature, and this hinders the practice of evidence‐based care.(2) This has motivated interest in methods to modernize certain aspects of the systematic review process (eg, via automation).”
BackgroundO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 1×
“Methods for semiautomating this step using data mining have been proposed as a potential means of reducing the workload.(3)”
MethodEstimating Time to Conduct a Meta-analysis From Number of Citations Retrieved
Allen, I. E. (1999) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.282.7.634 · Cited by 163 · cited 1×
“can require thousands of person‐hours.(1) The exponential expansion of the biomedical literature base has imposed an increased burden on reviewers who have to screen more citations to find relevant articles, thus multiplying costs.”
BackgroundThe automation of systematic reviews
Tsafnat, G., Dunn, A., Glasziou, P., & Coiera, E. (2013) · BMJ · doi:10.1136/bmj.f139 · Cited by 131
Cited by
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T., Mulrow, C. D., Shamseer, L., & Moher, D. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.11.010 · Cited by 211
“These developments include novel guidance on how to identify studies for inclusion (9-11), synthesise and present findings when meta-analysis is not possible or appropriate (12, 13), report and synthesise intervention characteristics of included studies (14, 15), and enhance the reproducibility of meta-analytic result…”
BackgroundResource use during systematic review production varies widely: a scoping review
Nussbaumer-Streit, B., Ellen, M., Klerings, I., Sfetcu, R., Riva, N., Mahmić-Kaknjo, M., Poulentzas, G., Martinez, P., Baladia, E., Ziganshina, L. E., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.019 · Cited by 98
Hennessy, E. A., Johnson, B. T., & Keenan, C. (2019) · Applied Psychology: Health and Well-Being · doi:10.1111/aphw.12169 · Cited by 91
Noel-Storr, A., Dooley, G., Affengruber, L., & Gartlehner, G. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.09.024 · Cited by 61
Automation tools to support undertaking scoping reviews
Khalil, H., Pollock, D., McInerney, P., Evans, C., Moraes, E. B., Godfrey, C. M., Alexander, L., Tricco, A., Peters, M. D. J., Pieper, D., et al. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1731 · Cited by 43
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices
Ramírez, J., Sayin, B., Baez, M., Casati, F., Cernuzzi, L., Benatallah, B., & Demartini, G. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479531 · Cited by 4
“For example, requesters may not need to interact with workers to study task design in the context of classification tasks, relying entirely on leveraging clear guidelines to articulate what they expect from workers [50, 60].”
Background
Metadata
- Authors
- Mortensen, M. L., Adam, G. P., Trikalinos, T. A., Kraska, T., & Wallace, B. C.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2017-07-04
- Volume
- 8
- Issue
- 3
- Pages
- 366-386
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2728712790
- Cited by
- 76
- References
- 6
- FWCI
- 2.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.