An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
BACKGROUND AND OBJECTIVES: Filtering the deluge of new research to facilitate evidence synthesis has proven to be unmanageable using current paradigms of search and retrieval. Crowdsourcing, a way of harnessing the collective effort of a "crowd" of people, has the potential to support evidence synthesis by addressing this information overload created by the exponential growth in primary research outputs. Cochrane Crowd, Cochrane's citizen science platform, offers a range of tasks aimed at identifying studies related to health care. Accompanying each task are brief, interactive training modules, and agreement algorithms that help ensure accurate collective decision-making.The aims of the study were to evaluate the performance of Cochrane Crowd in terms of its accuracy, capacity, and autonomy and to examine contributor engagement across three tasks aimed at identifying randomized trials. STUDY DESIGN AND SETTING: Crowd accuracy was evaluated by measuring the sensitivity and specificity of crowd screening decisions on a sample of titles and abstracts, compared with "quasi gold-standard" decisions about the same records using the conventional methods of dual screening. Crowd capacity, in the form of output volume, was evaluated by measuring the number of records processed by the crowd, compared with baseline. Crowd autonomy, the capability of the crowd to produce accurate collectively derived decisions without the need for expert resolution, was measured by the proportion of records that needed resolving by an expert. RESULTS: The Cochrane Crowd community currently has 18,897 contributors from 163 countries. Collectively, the Crowd has processed 1,021,227 records, helping to identify 178,437 reports of randomized controlled trials (RCTs) for Cochrane's Central Register of Controlled Trials. The sensitivity for each task was 99.1% for the RCT identification task (RCT ID), 99.7% for the RCT identification task of trials from ClinicalTrials.gov (CT ID), and 97.7% for the identification of RCTs from the International Clinical Trials Registry Platform (ICTRP ID). The specificity for each task was 99% for RCT ID, 98.6% for CT ID, and 99.1% for CT ICTRP ID. The capacity of the combined Crowd and machine learning workflow has increased fivefold in 6 years, compared with baseline. The proportion of records requiring expert resolution across the tasks ranged from 16.6% to 19.7%. CONCLUSION: Cochrane Crowd is sufficiently accurate and scalable to keep pace with the current rate of publication (and registration) of new primary studies. It has also proved to be a popular, efficient, and accurate way for a large number of people to play an important voluntary role in health evidence production. Cochrane Crowd is now an established part of Cochrane's effort to manage the deluge of primary research being produced.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bornmann, L., & Mutz, R. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23329 · Cited by 1,576 · cited 1×
“Over the last two decades, published health research output has more than doubled [1,2].”
BackgroundBrett, J., Staniszewska, S., Mockford, C., Herron-Marx, S., Hughes, J., Tysall, C., & Suleman, R. (2014) · The Patient - Patient-Centered Outcomes Research · doi:10.1007/s40271-014-0065-0 · Cited by 839
How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838 · cited 1×
“This bottleneck in the evidence production process can cause delay and contributes to often lengthy production times for systematic reviews and other evidence syntheses, such as guidelines and technology assessments, leaving important clinical questions unanswered, and possibly resulting in reliance on out-of-date, an…”
BackgroundBorah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · cited 1×
“This information deluge is putting health evidence production systems under strain, as systematic reviewers often need to sift through large numbers of records, identified from sensitive searches performed across these and other databases, in search of eligible studies [3].”
BackgroundLiving Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap
Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001603 · Cited by 596 · cited 1×
“This bottleneck in the evidence production process can cause delay and contributes to often lengthy production times for systematic reviews and other evidence syntheses, such as guidelines and technology assessments, leaving important clinical questions unanswered, and possibly resulting in reliance on out-of-date, an…”
BackgroundSelf-citationA brief history of research synthesis
Chalmers, I., Hedges, L. V., & Cooper, H. (2002) · Evaluation & the Health Professions · doi:10.1177/0163278702025001003 · Cited by 552
Thomas, J., McDonald, S., Noel-Storr, A., Shemilt, I., Elliott, J., Mavergames, C., & Marshall, I. J. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.11.003 · Cited by 253
Self-citationStakeholder involvement in systematic reviews: a scoping review
Pollock, A., Campbell, P., Struthers, C., Synnot, A., Nunn, J., Hill, S., Goodare, H., Morris, J., Watts, C., & Morley, R. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0852-0 · Cited by 180 · cited 1×
“It has also been challenging to offer prospective contributors to Cochrane meaningful ways to get involved with producing Cochrane systematic reviews; particularly those with little or no experience of health research [8,9].”
BackgroundConsumer involvement in systematic reviews of comparative effectiveness research
Kreis, J., Puhan, M. A., Schünemann, H. J., & Dickersin, K. (2013) · Health Expectations · doi:10.1111/j.1369-7625.2011.00722.x · Cited by 88
Noel-Storr, A., Dooley, G., Affengruber, L., & Gartlehner, G. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.09.024 · Cited by 61
Self-citation
Cited by
Thomas, J., McDonald, S., Noel-Storr, A., Shemilt, I., Elliott, J., Mavergames, C., & Marshall, I. J. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.11.003 · Cited by 253
Self-citationDecentralisation in Clinical Trials and Patient Centricity: Benefits and Challenges
Sinha, S. D., Chary Sriramadasu, S., Raphael, R., & Roy, S. (2024) · Pharmaceutical Medicine · doi:10.1007/s40290-024-00518-x · Cited by 26
Marshall, I. J., Trikalinos, T. A., Soboczenski, F., Yun, H. S., Kell, G., Marshall, R., & Wallace, B. C. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.08.013 · Cited by 23
“[23–25] These projects also monitor research databases (using a combination of machine learning identification of RCTs and crowdsourcing) and notify Cochrane review groups (who each typically manage tens of systematic reviews on a common clinical theme) when new research is published relevant to J…”
BackgroundKristjansson, E., Dignam, M., Rizvi, A., Osman, M., Magwood, O., Olarte, D., Cohen, J. F., Krasevec, J., Grover, T., Labelle, P. R., et al. (2025) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd014794.pub2 · Cited by 4
Metadata
- Authors
- Noel-Storr, A., Dooley, G., Elliott, J., Steele, E., Shemilt, I., Mavergames, C., Wisniewski, S., McDonald, S., Murano, M., Glanville, J., Foxlee, R., Beecher, D., Ware, J., & Thomas, J.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2021-05
- Volume
- 133
- Pages
- 130-139
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W3125296572
- Cited by
- 132
- References
- 12
- FWCI
- 12.29
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials. https://app.cochat.ai/papers/10.1016/j.jclinepi.2021.01.006
- The paper
- Noel-Storr, A., Dooley, G., Elliott, J., Steele, E., Shemilt, I., Mavergames, C., Wisniewski, S., McDonald, S., Murano, M., Glanville, J., Foxlee, R., Beecher, D., Ware, J., & Thomas, J. (2021). An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials. Journal of Clinical Epidemiology. https://doi.org/10.1016/j.jclinepi.2021.01.006
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.