The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Scientific Utopia III: Crowdsourcing Science
Open accessPublished version of a preprint
Abstract
Most scientific research is conducted by small teams of investigators who together formulate hypotheses, collect data, conduct analyses, and report novel findings. These teams operate independently as vertically integrated silos. Here we argue that scientific research that is horizontally distributed can provide substantial complementary value, aiming to maximize available resources, promote inclusiveness and transparency, and increase rigor and reliability. This alternative approach enables researchers to tackle ambitious projects that would not be possible under the standard model. Crowdsourced scientific initiatives vary in the degree of communication between project members from largely independent work curated by a coordination team to crowd collaboration on shared activities. The potential benefits and challenges of large-scale collaboration span the entire research process: ideation, study design, data collection, data analysis, reporting, and peer review. Complementing traditional small science with crowdsourced approaches can accelerate the progress of science and improve the quality of scientific research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
The Matthew Effect in Science: The reward and communication systems of science are considered.
Merton, R. K. (1968) · Science · doi:10.1126/science.159.3810.56 · Cited by 6,720 · cited 3×
“Further, although merit matters in science, there are also Matthew effects (Bol et al., 2018; Clauset et al., 2015; Merton, 1968; Petersen et al., 2011; Wahls, 2018).”
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“…consciously or unconsciously choose statistical specifications that yield desired results, in particular statistically significant results, in support of a favored theory (Bakker et al., 2012; Ioannidis, 2005; Ioannidis & Trikalinos, 2007; Simmons et al., 2011; Simonsohn, Nelson, & Simmons, 2014).”
BackgroundResultRunning experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
The Increasing Dominance of Teams in Production of Knowledge
Wuchty, S., Jones, B. F., & Uzzi, B. (2007) · Science · doi:10.1126/science.1136099 · Cited by 3,615
Reagent and laboratory contamination can critically impact sequence-based microbiome analyses
Salter, S. J., Cox, M. J., Turek, E. M., Calus, S. T., Cookson, W. O., Moffatt, M. F., Turner, P., Parkhill, J., Loman, N. J., & Walker, A. W. (2014) · BMC Biology · doi:10.1186/s12915-014-0087-z · Cited by 3,463
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 2×
“…articles have been published since 1965 (Pan et al., 2016)—and evidence that many published findings are less robust than initially thought (Begley & Ellis, 2012; Errington et al., 2014; Open Science Collaboration, 2015; Prinz, Schlange, & Asadullah, 2011), researchers must determine how…”
Background
Research works printed but not linked: 112
- 10.1038/506422a · doi:10.1038/506422a
- 10.1073/pnas.1708285114. · DOI
- 10.1177/1350508411403531 · DOI · Paper page
- 10.1017/S0003055416000058 · DOI · Paper page
- 10.1002/mpr.225 · DOI · Paper page
- 10.1080/13642531003637775 · DOI
- 10.1073/pnas.1719557115. · DOI · Paper page
- 10.1126/scitranslmed.3001399 · DOI · Paper page
- 10.1177/1745691617739421 · DOI
- 10.1087/20150211 · DOI · Paper page
- 10.1287/mnsc.2014.1909 · DOI · Paper page
- 10.3389/fncom.2014.00082 · DOI
- 10.3389/fnins.2012.00149 · DOI · Paper page
- 10.1016/j.neuroimage.2012.07.004 · DOI · Paper page
- Cavalier D., 2016, The rightful place of science: Citizen science
- 10.1016/j.cortex.2012.12.016 · DOI · Paper page
- Chargaff E., 1978, Heraclitean fire: Sketches from a life before nature
- Christensen C. M., 2014, Harvard Business Review, 92, 60
- 10.1038/msb4100040 · DOI · Paper page
- 10.1126/sciadv.1400005 · DOI · Paper page
- 10.1086/225553 · doi:10.1086/225553
- 10.1007/s13164-018-0400-9 · DOI · Paper page
- Davis R., Espinosa J., Glass C., Green M. R., Massague J., Pan D., Dang C. V. (2018). Reproducibility project: Cancer biology. Retrieved from https://elifesciences.org/collections/9b1e83d1/reproducibility-project-cancer-biology
- 10.1086/699976 · DOI · Paper page
- 10.1093/restud/rdx033 · DOI · Paper page
- Ebersole C. R., 2018, Experimentally examining the consequences of preregistered analyses
- 10.5194/tc-8-1289-2014 · DOI
- 10.1016/j.jesp.2018.06.004 · DOI
- 10.7554/eLife.04333 · DOI · Paper page
- 10.3389/fpsyg.2015.01152 · DOI
- Feyerabend P., 1982, Science in a free society
- 10.1016/j.joep.2018.10.009 · DOI
- 10.1177/1745691612460686 · DOI · Paper page
- 10.1511/2014.111.460 · DOI · Paper page
- 10.1177/1745691612459057 · DOI
- 10.1126/science.1224988 · DOI
- 10.1038/nj7444-259a · DOI · Paper page
- 10.1038/466685a · DOI · Paper page
- 10.1073/pnas.0707962104 · DOI · Paper page
- 10.1037/a0015566 · DOI · Paper page
- 10.1177/1049731506292570 · DOI
- 10.1007/978-3-642-17572-5_45 · DOI · Paper page
- 10.1177/1740774507079441 · DOI · Paper page
- Jia M., 2018, The crowdsourced generation, evaluation, and testing of research hypotheses
- Kanefsky B., 2001, Proceedings of the 32nd Annual Lunar and Planetary Science Conference
- 10.1371/journal.pbio.1002456doi:10.1371/journal.pbio.1002456 · doi:10.1371/journal.pbio.1002456doi:10.1371/journal.pbio.1002456
- 10.1145/1460563.1460572 · DOI · Paper page
- Kittur A., 2009, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1495, 10.1145/1518701.1518928 · DOI
- Kittur A., 2011, Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, 43, 10.1145/2047196.2047202 · DOI · Paper page
- 10.1027/1864-9335/a000178 · DOI · Paper page
- 10.1037/amp0000288 · doi:10.1037/amp0000288
- 10.1002/fee.1436 · DOI · Paper page
- 10.1037/a0036260 · DOI · Paper page
- Landy J. F., 2018, Crowdsourcing hypothesis tests
- Larrick R. P., 2012, Frontiers in social psychology: Social judgment and decision making, 227
- 10.1177/2515245918787489 · DOI · Paper page
- 10.1155/2012/493598 · DOI
- 10.1038/546009a · DOI
- 10.1016/j.endeavour.2014.03.002 · doi:10.1016/j.endeavour.2014.03.002
- 10.1093/clipsy/6.1.95 · DOI
- 10.1177/1745691612460688 · DOI · Paper page
- 10.1037/a0036677 · DOI · Paper page
- 10.1136/bmj.g3058 · DOI
- 10.7554/eLife.15802 · DOI · Paper page
- 10.1177/2515245918777487 · DOI
- 10.1186/s40168-016-0168-z · DOI · Paper page
- 10.1038/542409a · DOI
- 10.1027/1864-9335/a000202 · DOI
- 10.1037/1089-2680.11.2.99 · DOI · Paper page
- 10.1177/2515245918797607 · DOI · Paper page
- 10.1016/j.respol.2018.07.019 · DOI
- 10.1142/p455 · doi:10.1142/p455
- 10.1080/00335557743000053 · DOI · Paper page
- 10.1037/0033-2909.131.5.763 · DOI · Paper page
- 10.1080/1047840X.2012.692215 · DOI · Paper page
- 10.1177/1745691618755704 · DOI
- Olmsted D., 1834, American Journal of Science and Arts, 26, 354
- 10.1016/j.joi.2018.06.005 · DOI · Paper page
- 10.1146/annurev-genom-091212-153520 · DOI
- 10.1093/geronb/gbs077 · DOI
- 10.1111/j.1540-5885.2011.00893.x · DOI · Paper page
- 10.4007/annals.2012.175.3.6 · doi:10.4007/annals.2012.175.3.6
- Polymath D. H. J., 2014, Algebra & Number Theory, 9, 2067
- Price A., 2012, Journal of the American Association of Variable Star Observers, 40, 614
- 10.1038/nn0405-397 · DOI
- 10.1038/nrg.2016.69 · DOI · Paper page
- Salganik M. J., 2017, Bit by bit: Social research in the digital age
- 10.1038/515009a · DOI · Paper page
- Schweinsberg M., 2018, Crowdsourcing data analysis: Gender, status, and science
- 10.1016/j.jesp.2015.10.001 · DOI
- 10.1037/0022-3514.51.3.515 · doi:10.1037/0022-3514.51.3.515 · Paper page
- 10.1177/2515245917747646 · DOI · Paper page
- 10.1177/0956797613480366 · DOI · Paper page
- Sobel D., 2007, Longitude: The true story of a lone genius who solved the greatest scientific problem of his time
- 10.1038/nature17620 · DOI
- 10.1046/j.1365-2575.2002.00116.x · DOI · Paper page
- 10.1037/bul0000169 · DOI · Paper page
- 10.1016/j.tics.2017.06.007 · DOI · Paper page
- 10.1196/annals.1407.021 · DOI · Paper page
- Surowiecki J., 2005, The wisdom of crowds
- 10.1090/S0025-5718-2011-02542-1 · DOI
- Thelen B. A., 2008, Park Science, 25, 74
- 10.1038/sdata.2016.82 · DOI
- 10.1017/S0140525X18000894 · DOI
- 10.1126/science.317.5844.1496b · DOI
- 10.1525/collabra.177 · DOI
- Wahls W. P. (2018). High cost of bias: Diminishing marginal returns on NIH grant 3 funding to institutions. Unpublished manuscript. · DOI
- 10.1177/01461672992512005 · DOI · Paper page
- 10.1017/CBO9780511803932 · DOI · Paper page
- 10.1038/387341a0 · DOI · Paper page
- 10.1093/bioinformatics/btr323 · DOI
- 10.1002/job.178 · doi:10.1002/job.178
Cited by
Small Effects: The Indispensable Foundation for a Cumulative Psychological Science
Götz, F. M., Gosling, S. D., & Rentfrow, P. J. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620984483 · Cited by 538
Addressing climate change with behavioral science: A global intervention tournament in 63 countries
Vlasceanu, M., Doell, K. C., Bak-Coleman, J. B., Todorova, B., Berkebile-Weinberg, M. M., Grayson, S. J., Patel, Y., Goldwert, D., Pei, Y., Chakroff, A., et al. (2024) · Science Advances · doi:10.1126/sciadv.adj5778 · Cited by 269
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
Self-citationThe Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
Estimating the replicability of Brazilian biomedical science
Initiative, T. B. R., Amaral, O. B., Carneiro, C. F. D., Neves, K., Sampaio, A. P. W., Gomes, B. V., Abreu, M. B. D., Tan, P. B., Mota, G. P. S., Goulart, R. N., et al. (2025) · bioRxiv · doi:10.1101/2025.04.02.645026 · Cited by 13
“This suggests that part of the irreplicability observed in our study is due to factors specific to published experiments that were not present in replications; nevertheless, a sizable amount of variation was still observed among replications, as can be seen in Figure 3A .”
Metadata
- Authors
- Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2019-07-01
- Volume
- 14
- Issue
- 5
- Pages
- 711-733
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2781247022
- Cited by
- 174
- References
- 56
- FWCI
- 24.48
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Scientific Utopia III: Crowdsourcing Science. https://app.cochat.ai/papers/10.1177/1745691619850561
- The paper
- Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019). Scientific Utopia III: Crowdsourcing Science. Perspectives on Psychological Science. https://doi.org/10.1177/1745691619850561
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
