The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Crowdsourcing for Cognitive Science – The Utility of Smartphones
Open accessAbstract
By 2015, there will be an estimated two billion smartphone users worldwide. This technology presents exciting opportunities for cognitive science as a medium for rapid, large-scale experimentation and data collection. At present, cost and logistics limit most study populations to small samples, restricting the experimental questions that can be addressed. In this study we investigated whether the mass collection of experimental data using smartphone technology is valid, given the variability of data collection outside of a laboratory setting. We presented four classic experimental paradigms as short games, available as a free app and over the first month 20,800 users submitted data. We found that the large sample size vastly outweighed the noise inherent in collecting data outside a controlled laboratory setting, and show that for all four games canonical results were reproduced. For the first time, we provide experimental validation for the use of smartphones for data collection in cognitive science, which can lead to the collection of richer data sets and a significant cost reduction as well as provide an opportunity for efficient phenotypic screening of large populations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A common feature was that task sessions were kept as short as possible to make them more convenient and to increase likelihood of completion [28,32].”
From Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy (2016) · cites this paper as [32]
In the citing paper
“Wii Tests [21], Shapebuilder [37], The Great Brain Experiment [32,39], BAM-COG [28], and Tap the Hedgehog [27] were all found to produce output measures/scores that correlated fairly well with their non–gamelike counterparts”
From Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy (2016) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Temporary suppression of visual processing in an RSVP task: An attentional blink?
Raymond, J. E., Shapiro, K. L., & Arnell, K. M. (1992) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.18.3.849 · Cited by 2,286
Temporary suppression of visual processing in an RSVP task: an attentional blink?
Raymond, J. E., Shapiro, K. L., & Arnell, K. M. (1992) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.18.3.849 · Cited by 1,649
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
On the ability to inhibit simple and choice reaction time responses: A model and a method.
Logan, G. D., Cowan, W. B., & Davis, K. A. (1984) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.10.2.276 · Cited by 1,542
A two-stage model for multiple target detection in rapid serial visual presentation.
Chun, M. M., & Potter, M. C. (1995) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.21.1.109 · Cited by 1,465
Response inhibition in the stop-signal paradigm
Verbruggen, F., & Logan, G. D. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.07.005 · Cited by 1,373
reCAPTCHA: Human-Based Character Recognition via Web Security Measures
von Ahn, L., Maurer, B., McMillen, C. P., Abraham, D., & Blum, M. (2008) · Science · doi:10.1126/science.1160379 · Cited by 1,116
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Top-down suppression deficit underlies working memory impairment in normal aging
Gazzaley, A., Cooney, J. W., Rissman, J., & D'Esposito, M. (2005) · Nature Neuroscience · doi:10.1038/nn1543 · Cited by 934
Human Research and Data Collection via the Internet
Birnbaum, M. H. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.141601 · Cited by 927
Happiness is greater in natural environments
MacKerron, G., & Mourato, S. (2013) · Global Environmental Change · doi:10.1016/j.gloenvcha.2013.03.010 · Cited by 899
Development of inhibitory control across the life span.
Williams, B. R., Ponesse, J. S., Schachar, R. J., Logan, G. D., & Tannock, R. (1999) · Developmental Psychology · doi:10.1037/0012-1649.35.1.205 · Cited by 889
Horse-race model simulations of the stop-signal procedure
Band, G. P. H., van der Molen, M. W., & Logan, G. D. (2003) · Acta Psychologica · doi:10.1016/s0001-6918(02)00079-3 · Cited by 687
Algorithm discovery by protein folding game players
Khatib, F., Cooper, S., Tyka, M. D., Xu, K., Makedon, I., Popović, Z., Baker, D. A., & Players, F. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1115898108 · Cited by 593
Sprouse, J. (2010) · Behavior Research Methods · doi:10.3758/s13428-010-0039-7 · Cited by 513
Thinking like a trader selectively reduces individuals' loss aversion
Sokol-Hessner, P., Hsu, M., Curley, N. G., Delgado, M. R., Camerer, C. F., & Phelps, E. A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0806761106 · Cited by 452
Potter, M. C., Chun, M. M., Banks, B. S., & Muckenhoupt, M. (1998) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037//0278-7393.24.4.979 · Cited by 321
A systematic review of cognitive decline in the general elderly population
Park, H. L., O'Connell, J. E., & Thomson, R. G. (2003) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.1023 · Cited by 279
Inhibitory attentional mechanisms and aging.
Kane, M. J., Hasher, L., Stoltzfus, E. R., Zacks, R. T., & al, E. (1994) · Psychology and Aging · doi:10.1037/0882-7974.9.1.103 · Cited by 269
Potter, M. C., Chun, M. M., Banks, B. S., et al. (1998) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.24.4.979 · Cited by 227
Aron, A. R., & Verbruggen, F. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02216.x · Cited by 187
Dufau, S., Duñabeitia, J. A., Moret‐Tatay, C., McGonigal, A., Peeters, D., Alario, F. ‐., Balota, D. A., Brysbaert, M., Carreiras, M., Ferrand, L., et al. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024974 · Cited by 177
Stop and Go: The Neural Basis of Selective Movement Prevention
Coxon, J. P., Stinear, C. M., & Byblow, W. D. (2008) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2009.21081 · Cited by 112
Galaxy Zoo Volunteers Share Pain and Glory of Research
Clery, D. (2011) · Science · doi:10.1126/science.333.6039.173 · Cited by 105
Dissociating distractor-filtering at encoding and during maintenance.
McNab, F., & Dolan, R. J. (2014) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/a0036013 · Cited by 49
Self-citationIndividual but not fragile: Individual differences in task control predict Stroop facilitation
Kalanthroff, E., & Henik, A. (2013) · Consciousness and Cognition · doi:10.1016/j.concog.2013.01.010 · Cited by 42
Smittenaar, P., Prichard, G., FitzGerald, T. H. B., Diedrichsen, J., & Dolan, R. J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086850 · Cited by 33
Self-citationRabbitt, P., Lunn, M., Ibrahim, S., & McInnes, L. (2009) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210802633461 · Cited by 20
Erratum: Top-down suppression deficit underlies working memory impairment in normal aging
Gazzaley, A. H., Cooney, J. W., Rissman, J., & D’Esposito, M. (2005) · Nature Neuroscience · doi:10.1038/nn1205-1791c · Cited by 9
Learning OpenGL ES for iOS: A Hands-on Guide to Modern 3D Graphics Programming
Buck, E. M. (2012) · International Linear Collider · Cited by 3
Erratum: Corrigendum: Orexin, drugs and motivated behaviors
Scammell, T. E., & Saper, C. B. (2005) · Nature Neuroscience · doi:10.1038/nn1205-1791d · Cited by 3
Beginning iPhone Games Development
Bakhirev, P., Cabrera, P. J., Marsh, I., Penberthy, S., Smith, B. B., & Wing, E. (2010) · Apress eBooks · doi:10.1007/978-1-4302-2600-0 · Cited by 2
Flagged first, then by citations
Cited by
Sorted by citations
Lumsden, J., Edwards, E. A., Lawrence, N. S., Coyle, D., & Munafò, M. R. (2016) · JMIR Serious Games · doi:10.2196/games.5888 · Cited by 555
Oteros‐Rozas, E., Martín‐López, B., Fagerholm, N., Bieling, C., & Plieninger, T. (2017) · Ecological Indicators · doi:10.1016/j.ecolind.2017.02.009 · Cited by 402
Conducting perception research over the internet: A tutorial review
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015) · PeerJ · doi:10.7717/peerj.1058 · Cited by 301
Association of neural and emotional impacts of reward prediction errors with major depression
Rutledge, R. B., Moutoussis, M., Smittenaar, P., Zeidman, P., Taylor, T., Hrynkiewicz, L., Lam, J., Skandali, N., Siegel, J. Z., Ousdal, O. T., et al. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.1713 · Cited by 209
Self-citationMachine learning and big data in psychiatry: toward clinical applications
Rutledge, R. B., Chekroud, A. M., & Huys, Q. J. M. (2019) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2019.02.006 · Cited by 206
Self-citation
Metadata
- Authors
- Brown, H. R., Zeidman, P., Smittenaar, P., Adams, R. A., McNab, F., Rutledge, R. B., & Dolan, R. J.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Brown, Harriet R.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-07-15
- Volume
- 9
- Issue
- 7
- Pages
- e100662
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4099129
- OpenAlex
- W2044432453
- Cited by
- 129
- References
- 31
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 7 · 2023: 10 · 2022: 8 · 2021: 16 · 2020: 14 · 2019: 19 · 2018: 13 · 2017: 6 · 2016: 15 · 2015: 11 · 2014: 1
- FWCI
- 16.39
- Type
- journal article
- Language
- en
- Update notices
- correction (2014-10-14)
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, cc-by)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion, cc-by)
- Funders
- Wellcome Trust (091593, 091593/Z/10/, 098362, 098362/Z/12/Z, 091593/Z/10/Z, 101252/Z/13/Z)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- crowdsourcing, smartphones, mobile data collection, cognitive science, large-scale experimentation, data validity, reproducibility, experimental paradigms, mobile application, sample size, phenotypic screening, cost reduction
- MeSH terms
- Adolescent, Adult, Age Factors, Aged, Decision Making, Female, Humans, Male, Memory, Short-Term, Middle Aged, Cognitive Science, Cell Phone, Attentional Blink, Young Adult, Wireless Technology, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
