The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Crowdsourcing for Cognitive Science – The Utility of Smartphones
Open accessAbstract
By 2015, there will be an estimated two billion smartphone users worldwide. This technology presents exciting opportunities for cognitive science as a medium for rapid, large-scale experimentation and data collection. At present, cost and logistics limit most study populations to small samples, restricting the experimental questions that can be addressed. In this study we investigated whether the mass collection of experimental data using smartphone technology is valid, given the variability of data collection outside of a laboratory setting. We presented four classic experimental paradigms as short games, available as a free app and over the first month 20,800 users submitted data. We found that the large sample size vastly outweighed the noise inherent in collecting data outside a controlled laboratory setting, and show that for all four games canonical results were reproduced. For the first time, we provide experimental validation for the use of smartphones for data collection in cognitive science, which can lead to the collection of richer data sets and a significant cost reduction as well as provide an opportunity for efficient phenotypic screening of large populations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A common feature was that task sessions were kept as short as possible to make them more convenient and to increase likelihood of completion [28,32].”
From Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy (2016) · cites this paper as [32]
In the citing paper
“Wii Tests [21], Shapebuilder [37], The Great Brain Experiment [32,39], BAM-COG [28], and Tap the Hedgehog [27] were all found to produce output measures/scores that correlated fairly well with their non–gamelike counterparts”
From Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy (2016) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Temporary suppression of visual processing in an RSVP task: An attentional blink?
Raymond, J. E., Shapiro, K. L., & Arnell, K. M. (1992) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.18.3.849 · Cited by 2,286
Temporary suppression of visual processing in an RSVP task: an attentional blink?
Raymond, J. E., Shapiro, K. L., & Arnell, K. M. (1992) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.18.3.849 · Cited by 1,649
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
On the ability to inhibit simple and choice reaction time responses: A model and a method.
Logan, G. D., Cowan, W. B., & Davis, K. A. (1984) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.10.2.276 · Cited by 1,542
A two-stage model for multiple target detection in rapid serial visual presentation.
Chun, M. M., & Potter, M. C. (1995) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.21.1.109 · Cited by 1,465
Response inhibition in the stop-signal paradigm
Verbruggen, F., & Logan, G. D. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.07.005 · Cited by 1,373
reCAPTCHA: Human-Based Character Recognition via Web Security Measures
von Ahn, L., Maurer, B., McMillen, C. P., Abraham, D., & Blum, M. (2008) · Science · doi:10.1126/science.1160379 · Cited by 1,116
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Top-down suppression deficit underlies working memory impairment in normal aging
Gazzaley, A., Cooney, J. W., Rissman, J., & D'Esposito, M. (2005) · Nature Neuroscience · doi:10.1038/nn1543 · Cited by 934
Human Research and Data Collection via the Internet
Birnbaum, M. H. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.141601 · Cited by 927
Flagged first, then by citations
Cited by
Sorted by citations
Items 101–128 of 128
Ellis, D. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671408.001
Ellis, D. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671408.004
Ellis, D. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671408.002
Ellis, D. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671408.009
Ellis, D. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671408.007
Gamification and Game-Based Assessments
Leutner, F., Akhtar, R., & Chamorro‐Premuzic, T. (2022) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83867-559-220221005
McGregor, T., Purves, K. L., Barry, T. J., Constantinou, E., Craske, M. G., Breen, G. D., Young, K. S., & Eley, T. C. (2021) · doi:10.31219/osf.io/5scyq
Correction: Crowdsourcing for Cognitive Science – The Utility of Smartphones
Staff, T. P. O. (2014) · PLoS ONE · doi:10.1371/journal.pone.0111159
Emotional states and self-confidence independently fluctuate at different time scalesPreprint
da Fonseca, M., Maffei, G., Moreno‐Bote, R., & Hyafil, A. (2021) · doi:10.31234/osf.io/swpcf
Vergani, L., Marton, G., Pizzoli, S. F. M., Monzani, D., Mazzocco, K., & Pravettoni, G. (2018) · doi:10.2196/preprints.10855
Ellis, D. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671408.010
Larger images are better remembered: Generalization to settings outside the labPreprint
Kreichman, O., Masarwa, S., Brook, L., & Gilaie‐Dotan, S. (2023) · doi:10.31234/osf.io/ueas8
Surprising sounds bias risky decision makingPreprint
Feng, G. W., & Rutledge, R. B. (2023) · doi:10.31234/osf.io/gdhzv
Self-citationZech, H., Waltmann, M., Lee, Y., Reichert, M., Bedder, R. L., Rutledge, R. B., Goschke, T., Lenz, B., Bach, P., Kiefer, F., et al. (2023) · doi:10.31234/osf.io/96zac
Self-citation37 Media Psychology Methods: Current and Innovations
Wojdynski, B. W. (2025) · De Gruyter eBooks · doi:10.1515/9783111292144-045
Cheng, A., Konova, A. B., Powers, A. R., Corlett, P., Levy, I., Gu, X., Huys, Q. J. M., Pushkarskaya, H., Fineberg, S. K., Hauser, T. U., et al. (2026) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2026.02.009
Self-citationPositive effects of brief body exercises on mood: An interventional EMA study
Nitzpon, L., Rehder, J., Hohmann, R., Kunst, C., Gross, R., Nigg, C. R., Reichert, M., & Müller, M. (2026) · Neuroscience Applied · doi:10.1016/j.nsa.2026.106999
Caractérisation neuro-computationnelle de l'humeur : causes et conséquences
Carrillo, P. (2024) · theses.fr (ABES) · doi:10.70675/181aec0cz429dz4bccz83f0za479cbef52f7
Zech, H., Waltmann, M., Reichert, D., Arntz, F., Lee, Y., Reichert, M., Bedder, R. L., Rutledge, R. B., Goschke, T., Lenz, B., et al. (2026) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2026.07.019
Self-citationBai, J., Li, L., & 张者玉 (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1888982
Beyond Dissemination: From Critical Science Literacy to Citizen Neuroscience
Raschle, N. M., Federici, E., Hauser, T. U., & Dresler, M. (2026) · European Journal of Neuroscience · doi:10.1111/ejn.70727
Metadata
- Authors
- Brown, H. R., Zeidman, P., Smittenaar, P., Adams, R. A., McNab, F., Rutledge, R. B., & Dolan, R. J.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Brown, Harriet R.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-07-15
- Volume
- 9
- Issue
- 7
- Pages
- e100662
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4099129
- OpenAlex
- W2044432453
- Cited by
- 129
- References
- 31
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 7 · 2023: 10 · 2022: 8 · 2021: 16 · 2020: 14 · 2019: 19 · 2018: 13 · 2017: 6 · 2016: 15 · 2015: 11 · 2014: 1
- FWCI
- 16.39
- Type
- journal article
- Language
- en
- Update notices
- correction (2014-10-14)
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, cc-by)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion, cc-by)
- Funders
- Wellcome Trust (091593, 091593/Z/10/, 098362, 098362/Z/12/Z, 091593/Z/10/Z, 101252/Z/13/Z)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- crowdsourcing, smartphones, mobile data collection, cognitive science, large-scale experimentation, data validity, reproducibility, experimental paradigms, mobile application, sample size, phenotypic screening, cost reduction
- MeSH terms
- Adolescent, Adult, Age Factors, Aged, Decision Making, Female, Humans, Male, Memory, Short-Term, Middle Aged, Cognitive Science, Cell Phone, Attentional Blink, Young Adult, Wireless Technology, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
