The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
The Accuracy of Citizen Science Data: A Quantitative Review
Open accessAbstract
Citizen science involves volunteers who participate in scientific research by collecting data, monitoring sites, and even taking part in the whole process of scientific inquiry (Roy et al. 2012, Scyphers et al. 2015). In the past two decades, citizen science (also called participatory or community-based monitoring) has gained tremendous popularity (Bonney et al. 2009, Danielsen et al. 2014), due in part to the increasing realization among scientists of the benefits of engaging volunteers (Silvertown 2009, Danielsen et al. 2014, Aceves-Bueno et al. 2015, Scyphers et al. 2015). In particular, the cost-effectiveness of citizen science data offers the potential for scientists to tackle research questions with large spatial and/or temporal scales (Brossard et al. 2005, Holck 2007, Levrel et al. 2010, Szabo et al. 2010, Belt and Krausman 2012). Today, citizen science projects span a wide range of research topics concerning the preservation of marine and terrestrial environments, from invasive species monitoring (e.g., Scyphers et al. 2015) to ecological restoration and from local indicators of climate change to water quality monitoring (Silvertown 2009). They include well-known conservation examples like the Audubon Christmas Bird Count (Butcher et al. 1990) and projects of the Cornell Lab of Ornithology (Bonney et al. 2009). Despite the growth in the number of citizen science projects, scientists remain concerned about the accuracy of citizen science data (Danielsen et al. 2005, Crall et al. 2011, Gardiner et al. 2012, Law et al. 2017). Some studies evaluating data quality have found volunteer data to be more variable than professionally collected data (Harvey et al. 2002, Uychiaoco et al. 2005, Belt and Krausman 2012, Moyer-Horner et al. 2012) and others that volunteers' performance is comparable to that of professionals or scientists (Hoyer et al. 2001, 2012, Canfield et al. 2002, Oldekop et al. 2011). For example, Danielsen et al. (2005) concluded that the 16 comparative cases studies they reviewed only provided cautious support for volunteers' ability to detect changes in populations, habitats, or patterns of resource use. In a more recent review, Dickinson et al. (2010) found that the potential of citizen scientists to produce datasets with error and bias is poorly understood. The evidence of problems with citizen science data accuracy (e.g., Hochachka et al. 2012, Vermeiren et al. 2016) indicates a need for a more systematic analysis of the accuracy of citizen science data derived from individual studies of accuracy. To our knowledge, despite useful qualitative reviews (e.g., Lewandowski and Specht 2015), there are to date no reviews that combine the case studies to quantitatively evaluate the data quality of citizen science. In this paper, we conduct a quantitative review of citizen science data in the areas of ecology and environmental science. We focus on the universe of peer-reviewed studies in which researchers compare citizen science data to reference data either as part of validation mechanisms in a citizen science project or by designing experiments to test whether volunteers can collect sufficiently accurate data. We code the authors' qualitative assessments of data accuracy and we code the quantitative assessments of data accuracy. This enables us to evaluate both whether the authors believe the data to be accurate enough to achieve the goals of the program and the degree of accuracy reflected in the quantitative comparisons. We then use a linear regression model to assess correlates of accuracy. With citizen science playing an increasingly important role in expanding our scientific knowledge and enhancing the management of the environment, we conclude with recommendations for assessing data quality and for designing citizen science tasks that are more likely to produce accurate data. This study uses the case survey method to compile the set of studies published before 2014 that directly compare citizen science data with reference data. The goal of this method is to supplement qualitative, in-depth case studies with a quantitative analysis. As with all large-n studies, this prioritizes generalizability over detailed analysis of each case. It supplements existing published case studies and qualitative reviews (e.g., Freitag et al. 2016, Kosmala et al. 2016). We used a "snowball" approach to identify studies published before 2014 that compare citizen science data with some sort of reference data. Beginning with the 16 studies reviewed in Danielsen et al. (2005), we performed a cited reference search on Google Scholar (http://scholar.google.com/) for papers that cited these 16 studies. Next, we identified every paper cited in this group of papers that compared citizen science data to reference data and again performed a cited reference search on this new group of papers. We repeated this process iteratively until we encountered no new case studies, giving confidence that we had identified the univer
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy
Bonney, R., Cooper, C. B., Dickinson, J., Kelling, S., Phillips, T., Rosenberg, K. V., & Shirk, J. (2009) · BioScience · doi:10.1525/bio.2009.59.11.9 · Cited by 2,529
A new dawn for citizen science
Silvertown, J. (2009) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2009.03.017 · Cited by 2,183
Citizen Science as an Ecological Research Tool: Challenges and Benefits
Dickinson, J. L., Zuckerberg, B., & Bonter, D. N. (2010) · Annual Review of Ecology, Evolution, and Systematics · doi:10.1146/annurev-ecolsys-102209-144636 · Cited by 1,914
Assessing data quality in citizen science
Kosmala, M., Wiggins, A., Swanson, A., & Simmons, B. (2016) · Frontiers in Ecology and the Environment · doi:10.1002/fee.1436 · Cited by 985
Scientific knowledge and attitude change: The impact of a citizen science project
Brossard, D., Lewenstein, B., & Bonney, R. (2005) · International Journal of Science Education · doi:10.1080/09500690500069483 · Cited by 687
Monitoring Matters: Examining the Potential of Locally-based Approaches
Danielsen, F., Burgess, N. D., & Balmford, A. (2005) · Biodiversity and Conservation · doi:10.1007/s10531-005-8375-0 · Cited by 583
Data-intensive science applied to broad-scale citizen science
Hochachka, W. M., Fink, D., Hutchinson, R. A., Sheldon, D., Wong, W.-K., & Kelling, S. (2012) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2011.11.006 · Cited by 412
Assessing citizen science data quality: an invasive species case study
Crall, A. W., Newman, G. J., Stohlgren, T. J., Holfelder, K. A., Graham, J., & Waller, D. M. (2011) · Conservation Letters · doi:10.1111/j.1755-263x.2011.00196.x · Cited by 386
Lessons from lady beetles: accuracy of monitoring data from US and UK citizen‐science programs
Gardiner, M. M., Allee, L. L., Brown, P. M., Losey, J. E., Roy, H. E., & Smyth, R. R. (2012) · Frontiers in Ecology and the Environment · doi:10.1890/110185 · Cited by 294
eBird: Engaging Birders in Science and Conservation
Wood, C., Sullivan, B., Iliff, M., Fink, D., & Kelling, S. (2011) · PLoS Biology · doi:10.1371/journal.pbio.1001220 · Cited by 250
Mechanisms for Data Quality and Validation in Citizen Science
Wiggins, A., Newman, G., Stevenson, R. D., & Crowston, K. (2011) · 2011 IEEE Seventh International Conference on e-Science Workshops · doi:10.1109/esciencew.2011.27 · Cited by 220
Estimation of reef fish length by divers and by stereo-video
Harvey, E. S., Fletcher, D., & Shortis, M. R. (2002) · Fisheries Research · doi:10.1016/s0165-7836(01)00356-3 · Cited by 214
Lewandowski, E., & Specht, H. (2015) · Conservation Biology · doi:10.1111/cobi.12481 · Cited by 206
Leveraging the power of place in citizen science for effective conservation decision making
Newman, G., Chandler, M., Clyde, M., McGreavy, B., Haklay, M., Ballard, H., Gray, S., Scarpino, R., Hauptfeld, R., Mellor, D., et al. (2017) · Biological Conservation · doi:10.1016/j.biocon.2016.07.019 · Cited by 184
Danielsen, F., Pirhofer‐Walzl, K., Adrian, T., Kapijimpanga, D. R., Burgess, N. D., Jensen, P., Bonney, R. E., Funder, M., Landa, A. M., Levermann, N., et al. (2013) · Conservation Letters · doi:10.1111/conl.12024 · Cited by 173
Aceves-Bueno, E., Adeleye, A. S., Bradley, D., Tyler Brandt, W., Callery, P., Feraud, M., Garner, K. L., Gentry, R., Huang, Y., McCullough, I., et al. (2015) · Ecosystems · doi:10.1007/s10021-015-9842-4 · Cited by 161
Self-citationSzabo, J. K., Vesk, P. A., Baxter, P. W. J., & Possingham, H. P. (2010) · Ecological Applications · doi:10.1890/09-0877.1 · Cited by 156
Strategies Employed by Citizen Science Programs to Increase the Credibility of Their Data
Freitag, A., Meyer, R., & Whiteman, L. (2016) · Citizen Science Theory and Practice · doi:10.5334/cstp.6 · Cited by 140
Levrel, H., Fontaine, B., Henry, P., Jiguet, F., Julliard, R., Kerbiriou, C., & Couvet, D. (2010) · Ecological Economics · doi:10.1016/j.ecolecon.2010.03.001 · Cited by 101
Uychiaoco, A. J., Arceo, H. O., Green, S. J., la Cruz, M. D. L. T., Gaite, P. A., & Aliño, P. M. (2005) · Biodiversity and Conservation · doi:10.1007/s10531-005-8414-x · Cited by 99
Participatory forest monitoring: an assessment of the accuracy of simple cost–effective methods
Holck, M. H. (2007) · Biodiversity and Conservation · doi:10.1007/s10531-007-9273-4 · Cited by 89
Scyphers, S. B., Powers, S. P., Akins, J. L., Drymon, J. M., Martin, C. W., Schobernd, Z. H., Schofield, P. J., Shipp, R. L., & Switzer, T. S. (2014) · Conservation Letters · doi:10.1111/conl.12127 · Cited by 85
An assessment of the use of volunteers for terrestrial invertebrate biodiversity surveys
Lovell, S., Hamer, M., Slotow, R., & Herbert, D. (2009) · Biodiversity and Conservation · doi:10.1007/s10531-009-9642-2 · Cited by 73
Crowdsourcing as a Tool for Research: Implications of Uncertainty
Law, E., Gajos, K. Z., Wiggins, A., Gray, M. L., & Williams, A. C. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998197 · Cited by 70
Evaluating population estimates of mountain goats based on citizen science
Belt, J. J., & Krausman, P. R. (2012) · Wildlife Society Bulletin · doi:10.1002/wsb.139 · Cited by 29
Vermeiren, P., Muñoz, C. C., Zimmer, M., & Sheaves, M. J. (2016) · Ecological Indicators · doi:10.1016/j.ecolind.2016.01.031 · Cited by 24
Data quality in crowdsourcing for biodiversity research: issues and examples
Jacobs, C. (2016) · Ubiquity Press eBooks · doi:10.5334/bax.f · Cited by 19
Cited by
Citizen science and sustainability transitions
Sauermann, H., Vohland, K., Antoniou, V., Balázs, B., Göbel, C., Karatzas, K., Mooney, P., Perelló, J., Ponti, M., Samson, R., et al. (2020) · Research Policy · doi:10.1016/j.respol.2020.103978 · Cited by 302
Quantification and exposure assessment of microplastics in Australian indoor house dust
Soltani, N. S., Taylor, M. P., & Wilson, S. P. (2021) · Environmental Pollution · doi:10.1016/j.envpol.2021.117064 · Cited by 277
Casana, J. (2020) · Journal of Field Archaeology · doi:10.1080/00934690.2020.1713285 · Cited by 90
Schacher, A., Roger, E., Williams, K. J., Stenson, M. P., Sparrow, B., & Lacey, J. (2023) · Remote Sensing · doi:10.3390/rs15051407 · Cited by 12
Metadata
- Authors
- Aceves‐Bueno, E., Adeleye, A. S., Feraud, M., Huang, Y., Tao, M., Yang, Y., & Anderson, S. E.
- Venue
- The Bulletin of the Ecological Society of America
- Publisher
- Wiley
- Published
- 2017-09-29
- Volume
- 98
- Issue
- 4
- Pages
- 278-290
- ISSN
- 0012-9623 (ISSN-L)
- 2327-6096
- OpenAlex
- W2761027789
- Cited by
- 338
- References
- 29
- FWCI
- 8.97
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.