Crowdsourcing for Cognitive Science – The Utility of Smartphones
Open accessAbstract
By 2015, there will be an estimated two billion smartphone users worldwide. This technology presents exciting opportunities for cognitive science as a medium for rapid, large-scale experimentation and data collection. At present, cost and logistics limit most study populations to small samples, restricting the experimental questions that can be addressed. In this study we investigated whether the mass collection of experimental data using smartphone technology is valid, given the variability of data collection outside of a laboratory setting. We presented four classic experimental paradigms as short games, available as a free app and over the first month 20,800 users submitted data. We found that the large sample size vastly outweighed the noise inherent in collecting data outside a controlled laboratory setting, and show that for all four games canonical results were reproduced. For the first time, we provide experimental validation for the use of smartphones for data collection in cognitive science, which can lead to the collection of richer data sets and a significant cost reduction as well as provide an opportunity for efficient phenotypic screening of large populations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A common feature was that task sessions were kept as short as possible to make them more convenient and increase likelihood of completion.
“A common feature was that task sessions were kept as short as possible to make them more convenient and to increase likelihood of completion [28,32].”
From Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy (2016) · cites this paper as [32]
Wii Tests, Shapebuilder, The Great Brain Experiment, BAM-COG, and Tap the Hedgehog produced output measures/scores that correlated fairly well with their non-gamelike counterparts.
“Wii Tests [21], Shapebuilder [37], The Great Brain Experiment [32,39], BAM-COG [28], and Tap the Hedgehog [27] were all found to produce output measures/scores that correlated fairly well with their non–gamelike counterparts”
From Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy (2016) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Temporary suppression of visual processing in an RSVP task: An attentional blink?
Raymond, J. E., Shapiro, K. L., & Arnell, K. M. (1992) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.18.3.849 · Cited by 2,276 · see citing works
Temporary suppression of visual processing in an RSVP task: an attentional blink?
Raymond, J. E., Shapiro, K. L., & Arnell, K. M. (1992) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.18.3.849 · Cited by 1,649 · see citing works
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works
On the ability to inhibit simple and choice reaction time responses: A model and a method.
Logan, G. D., Cowan, W. B., & Davis, K. A. (1984) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.10.2.276 · Cited by 1,542 · see citing works
On the ability to inhibit simple and choice reaction time responses: A model and a method.
Logan, G. D., Cowan, W. B., & Davis, K. A. (1984) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.10.2.276 · Cited by 1,537 · see citing works
A two-stage model for multiple target detection in rapid serial visual presentation.
Chun, M. M., & Potter, M. C. (1995) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.21.1.109 · Cited by 1,431 · see citing works
Response inhibition in the stop-signal paradigm
Verbruggen, F., & Logan, G. D. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.07.005 · Cited by 1,373 · see citing works
reCAPTCHA: Human-Based Character Recognition via Web Security Measures
von Ahn, L., Maurer, B., McMillen, C. P., Abraham, D., & Blum, M. (2008) · Science · doi:10.1126/science.1160379 · Cited by 1,116 · see citing works
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · see citing works
Top-down suppression deficit underlies working memory impairment in normal aging
Gazzaley, A., Cooney, J. W., Rissman, J., & D'Esposito, M. (2005) · Nature Neuroscience · doi:10.1038/nn1543 · Cited by 934 · see citing works
A two-stage model for multiple target detection in rapid serial visual presentation.
Chun, M. M., & Potter, M. C. (1995) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.21.1.109 · Cited by 906 · see citing works
Happiness is greater in natural environments
MacKerron, G., & Mourato, S. (2013) · Global Environmental Change · doi:10.1016/j.gloenvcha.2013.03.010 · Cited by 899 · see citing works
Development of inhibitory control across the life span.
Williams, B. R., Ponesse, J. S., Schachar, R. J., Logan, G. D., & Tannock, R. (1999) · Developmental Psychology · doi:10.1037/0012-1649.35.1.205 · Cited by 889 · see citing works
Development of inhibitory control across the life span.
Williams, B. R., Ponesse, J. S., Schachar, R. J., Logan, G. D., & Tannock, R. (1999) · Developmental Psychology · doi:10.1037//0012-1649.35.1.205 · Cited by 880 · see citing works
Human Research and Data Collection via the Internet
Birnbaum, M. H. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.141601 · Cited by 864 · see citing works
Horse-race model simulations of the stop-signal procedure
Band, G. P. H., van der Molen, M. W., & Logan, G. D. (2003) · Acta Psychologica · doi:10.1016/s0001-6918(02)00079-3 · Cited by 687 · see citing works
Algorithm discovery by protein folding game players
Khatib, F., Cooper, S., Tyka, M. D., Xu, K., Makedon, I., Popović, Z., Baker, D. A., & Players, F. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1115898108 · Cited by 583 · see citing works
Sprouse, J. (2010) · Behavior Research Methods · doi:10.3758/s13428-010-0039-7 · Cited by 492 · see citing works
Thinking like a trader selectively reduces individuals' loss aversion
Sokol-Hessner, P., Hsu, M., Curley, N. G., Delgado, M. R., Camerer, C. F., & Phelps, E. A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0806761106 · Cited by 452 · see citing works
Potter, M. C., Chun, M. M., Banks, B. S., & Muckenhoupt, M. (1998) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037//0278-7393.24.4.979 · Cited by 321 · see citing works
A systematic review of cognitive decline in the general elderly population
Park, H. L., O'Connell, J. E., & Thomson, R. G. (2003) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.1023 · Cited by 279 · see citing works
Inhibitory attentional mechanisms and aging.
Kane, M. J., Hasher, L., Stoltzfus, E. R., Zacks, R. T., & al, E. (1994) · Psychology and Aging · doi:10.1037/0882-7974.9.1.103 · Cited by 269 · see citing works
Potter, M. C., Chun, M. M., Banks, B. S., et al. (1998) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.24.4.979 · Cited by 227 · see citing works
Aron, A. R., & Verbruggen, F. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02216.x · Cited by 178 · see citing works
Dufau, S., Duñabeitia, J. A., Moret‐Tatay, C., McGonigal, A., Peeters, D., Alario, F. ‐., Balota, D. A., Brysbaert, M., Carreiras, M., Ferrand, L., et al. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024974 · Cited by 177 · see citing works
Stop and Go: The Neural Basis of Selective Movement Prevention
Coxon, J. P., Stinear, C. M., & Byblow, W. D. (2008) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2009.21081 · Cited by 108 · see citing works
Galaxy Zoo Volunteers Share Pain and Glory of Research
Clery, D. (2011) · Science · doi:10.1126/science.333.6039.173 · Cited by 105 · see citing works
Dissociating distractor-filtering at encoding and during maintenance.
McNab, F., & Dolan, R. J. (2014) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/a0036013 · Cited by 49 · see citing works
Self-citationIndividual but not fragile: Individual differences in task control predict Stroop facilitation
Kalanthroff, E., & Henik, A. (2013) · Consciousness and Cognition · doi:10.1016/j.concog.2013.01.010 · Cited by 38 · see citing works
Smittenaar, P., Prichard, G., FitzGerald, T. H. B., Diedrichsen, J., & Dolan, R. J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086850 · Cited by 31 · see citing works
Self-citationRabbitt, P., Lunn, M., Ibrahim, S., & McInnes, L. (2009) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210802633461 · Cited by 20 · see citing works
Erratum: Top-down suppression deficit underlies working memory impairment in normal aging
Gazzaley, A. H., Cooney, J. W., Rissman, J., & D’Esposito, M. (2005) · Nature Neuroscience · doi:10.1038/nn1205-1791c · Cited by 7 · see citing works
Learning OpenGL ES for iOS: A Hands-on Guide to Modern 3D Graphics Programming
Buck, E. M. (2012) · International Linear Collider · Cited by 3 · see citing works
Erratum: Corrigendum: Orexin, drugs and motivated behaviors
Scammell, T. E., & Saper, C. B. (2005) · Nature Neuroscience · doi:10.1038/nn1205-1791d · Cited by 3 · see citing works
Beginning iPhone Games Development
Bakhirev, P., Cabrera, P. J., Marsh, I., Penberthy, S., Smith, B. B., & Wing, E. (2010) · Apress eBooks · doi:10.1007/978-1-4302-2600-0 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Lumsden, J., Edwards, E. A., Lawrence, N. S., Coyle, D., & Munafò, M. R. (2016) · JMIR Serious Games · doi:10.2196/games.5888 · Cited by 555 · see citing works
Oteros‐Rozas, E., Martín‐López, B., Fagerholm, N., Bieling, C., & Plieninger, T. (2017) · Ecological Indicators · doi:10.1016/j.ecolind.2017.02.009 · Cited by 402 · see citing works
Conducting perception research over the internet: A tutorial review
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015) · PeerJ · doi:10.7717/peerj.1058 · Cited by 301 · see citing works
Association of neural and emotional impacts of reward prediction errors with major depression
Rutledge, R. B., Moutoussis, M., Smittenaar, P., Zeidman, P., Taylor, T., Hrynkiewicz, L., Lam, J., Skandali, N., Siegel, J. Z., Ousdal, O. T., et al. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.1713 · Cited by 209 · see citing works
Self-citationMachine learning and big data in psychiatry: toward clinical applications
Rutledge, R. B., Chekroud, A. M., & Huys, Q. J. M. (2019) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2019.02.006 · Cited by 195 · see citing works
Self-citationHeinz, A., Kiefer, F., Smolka, M. N., Endraß, T., Beste, C., Beck, A., Liu, S., Genauck, A., Romund, L., Banaschewski, T., et al. (2019) · Addiction Biology · doi:10.1111/adb.12866 · Cited by 177 · see citing works
Self-citationThe Psychological and Neural Basis of Loss Aversion
Sokol‐Hessner, P., & Rutledge, R. B. (2018) · Current Directions in Psychological Science · doi:10.1177/0963721418806510 · Cited by 150 · see citing works
Self-citationGuerrero, P., Møller, M. S., Olafsson, A. S., & Snizek, B. (2016) · Urban Planning · doi:10.17645/up.v1i2.609 · Cited by 149 · see citing works
Risk Taking for Potential Reward Decreases across the Lifespan
Rutledge, R. B., Smittenaar, P., Zeidman, P., Brown, H. R., Adams, R. A., Lindenberger, U., Dayan, P., & Dolan, R. J. (2016) · Current Biology · doi:10.1016/j.cub.2016.05.017 · Cited by 146 · see citing works
Self-citationAge-related changes in working memory and the ability to ignore distraction
McNab, F., Zeidman, P., Rutledge, R. B., Smittenaar, P., Brown, H. R., Adams, R. A., & Dolan, R. J. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1504162112 · Cited by 134 · see citing works
Self-citationUsers of the main smartphone operating systems (iOS, Android) differ only little in personality
Götz, F. M., Stieger, S., & Reips, U.-D. (2017) · PLOS ONE · doi:10.1371/journal.pone.0176921 · Cited by 127 · see citing works
Crowdsourcing in biomedicine: challenges and opportunities
Khare, R., Good, B. M., Leaman, R., Su, A. I., & Lu, Z. (2015) · Briefings in Bioinformatics · doi:10.1093/bib/bbv021 · Cited by 115 · see citing works
Using games to understand the mind
Allen, K. R., Brändle, F., Botvinick, M., Fan, J. E., Gershman, S. J., Alison Gopnik, C., Griffiths, T. L., Hartshorne, J. K., Hauser, T. U., Ho, M. K., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01878-9 · Cited by 105 · see citing works
Self-citationA Gamification Framework for Cognitive Assessment and Cognitive Training: Qualitative Study
Khaleghi, A., Aghaei, Z., & Mahdavi, M. A. (2021) · JMIR Serious Games · doi:10.2196/21900 · Cited by 106 · see citing works
Smartphones and the Neuroscience of Mental Health
Gillan, C. M., & Rutledge, R. B. (2021) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-101220-014053 · Cited by 84 · see citing works
Self-citationInformation use by humans during dynamic route choice in virtual crowd evacuations
Bode, N. W. F., Wagoum, A. U. K., & Codling, E. A. (2015) · Royal Society Open Science · doi:10.1098/rsos.140410 · Cited by 80 · see citing works
Proactive and Reactive Response Inhibition across the Lifespan
Smittenaar, P., Rutledge, R. B., Zeidman, P., Adams, R. A., Brown, H. R., Lewis, G., & Dolan, R. J. (2015) · PLoS ONE · doi:10.1371/journal.pone.0140383 · Cited by 72 · see citing works
Self-citationScholl, J., & Klein-Flügge, M. C. (2017) · Behavioural Brain Research · doi:10.1016/j.bbr.2017.09.050 · Cited by 62 · see citing works
Crowdsourcing Language Change with Smartphone Applications
Leemann, A., Kolly, M.-J., Purves, R. S., Britain, D., & Glaser, E. (2016) · PLoS ONE · doi:10.1371/journal.pone.0143060 · Cited by 62 · see citing works
Approach-Induced Biases in Human Information Sampling
Hunt, L. T., Rutledge, R. B., Malalasekera, W. M. N., Kennerley, S. W., & Dolan, R. J. (2016) · PLOS Biology · doi:10.1371/journal.pbio.2000638 · Cited by 60 · see citing works
Self-citationEndogenous fluctuations in the dopaminergic midbrain drive behavioral choice variability
Chew, B., Hauser, T. U., Papoutsi, M., Magerkurth, J., Dolan, R. J., & Rutledge, R. B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900872116 · Cited by 47 · see citing works
Self-citationThe social contingency of momentary subjective well-being
Rutledge, R. B., de Berker, A. O., Espenhahn, S., Dayan, P., & Dolan, R. J. (2016) · Nature Communications · doi:10.1038/ncomms11825 · Cited by 49 · see citing works
Self-citationMobiles and Cognition: The Associations Between Mobile Technology and Cognitive Flexibility
Alexopoulou, A., Batsou, A., & Drigas, A. S. (2020) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v14i03.11233 · Cited by 48 · see citing works
Crowdsourcing visual perception experiments: a case of contrast threshold
Sasaki, K., & Yamada, Y. (2019) · PeerJ · doi:10.7717/peerj.8339 · Cited by 48 · see citing works
Playing a video game is more than mere procrastination
Nordby, K., Løkken, R. A., & Pfuhl, G. (2019) · BMC Psychology · doi:10.1186/s40359-019-0309-9 · Cited by 49 · see citing works
Klindt, D. A., Devaine, M., & Jean Daunizeau, A. (2016) · Cortex · doi:10.1016/j.cortex.2016.09.009 · Cited by 38 · see citing works
Exploring Determinants of Pre-movement Delays in a Virtual Crowd Evacuation Experiment
Bode, N. W. F., & Codling, E. A. (2018) · Fire Technology · doi:10.1007/s10694-018-0744-9 · Cited by 36 · see citing works
Training Cognitive Functions Using Mobile Apps in Breast Cancer Patients: Systematic Review
Vergani, L., Marton, G., Pizzoli, S. F. M., Monzani, D., Mazzocco, K., & Pravettoni, G. (2018) · JMIR mhealth and uhealth · doi:10.2196/10855 · Cited by 34 · see citing works
Spanakis, P., Jones, A., Field, M., & Christiansen, P. (2018) · Substance Use & Misuse · doi:10.1080/10826084.2018.1536716 · Cited by 33 · see citing works
Revisiting the role of computational neuroimaging in the era of integrative neuroscience
Loosen, A. M., Kato, A., & Gu, X. (2024) · Neuropsychopharmacology · doi:10.1038/s41386-024-01946-8 · Cited by 34 · see citing works
The Making and Evaluation of Digital Games Used for the Assessment of Attention: Systematic Review
Wiley, K., Robinson, R., & Mandryk, R. L. (2021) · JMIR Serious Games · doi:10.2196/26449 · Cited by 34 · see citing works
Axelsen, J. L., Meline, J. S. J., Staiano, W., & Kirk, U. (2022) · BMC Psychology · doi:10.1186/s40359-022-00810-y · Cited by 31 · see citing works
Risk taking for potential losses but not gains increases with time of day
Bedder, R. L., Vaghi, M. M. S., Dolan, R. J., & Rutledge, R. B. (2023) · Scientific Reports · doi:10.1038/s41598-023-31738-x · Cited by 27 · see citing works
Self-citationReliable, rapid, and remote measurement of metacognitive bias
Fox, C., McDonogh, A., Donegan, K. R., Teckentrup, V., Crossen, R. J., Hanlon, A., Gallagher, E., Rouault, M., & Gillan, C. (2024) · Scientific Reports · doi:10.1038/s41598-024-64900-0 · Cited by 16 · see citing works
Virtual laboratories: transforming research with AI
Arto Klami, A., Damoulas, T., Engkvist, O., Rinke, P., & Samuel Kaski, A. (2024) · Data-Centric Engineering · doi:10.1017/dce.2024.15 · Cited by 28 · see citing works
Human-Machine Collaboration for Smart Decision Making: Current Trends and Future Opportunities
Geng, B., & Varshney, P. K. (2022) · IEEE International Conference on Collaboration and Internet Computing (CIC) · doi:10.1109/cic56439.2022.00019 · Cited by 25 · see citing works
The Virtuous Cycle of a Data Ecosystem
Voytek, B. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005037 · Cited by 25 · see citing works
Large-Scale Analysis of Auditory Segregation Behavior Crowdsourced via a Smartphone App
Teki, S., Kumar, S., & Griffiths, T. D. (2016) · PLoS ONE · doi:10.1371/journal.pone.0153916 · Cited by 24 · see citing works
Building a New Field of Computational Psychiatry
Moutoussis, M., Eldar, E., & Dolan, R. J. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2016.10.007 · Cited by 23 · see citing works
Self-citationHansen, A. S. (2016) · Journal of Outdoor Recreation and Tourism · doi:10.1016/j.jort.2016.05.001 · Cited by 23 · see citing works
Zech, H., Waltmann, M., Lee, Y., Reichert, M., Bedder, R. L., Rutledge, R. B., Deeken, F., Wenzel, J. G., Wedemeyer, F., Aguilera, A., et al. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-02019-8 · Cited by 21 · see citing works
Self-citationExamining the Use of Game-Based Assessments for Hiring Autistic Job Seekers
Willis, C. M. G., Powell-Rudy, T., Colley, K. L., & Prasad, J. J. (2021) · Journal of Intelligence · doi:10.3390/jintelligence9040053 · Cited by 21 · see citing works
Pedersen, M. K., Díaz, C. M. C., Wang, Q. J., Alba‐Marrugo, M. A., Amidi, A., Basaiawmoit, R. V., Bergenholtz, C., Christiansen, M. H., Gajdacz, M., Hertwig, R., et al. (2023) · Cognitive Science · doi:10.1111/cogs.13308 · Cited by 26 · see citing works
Arithmetic on Your Phone: A Large Scale Investigation of Simple Additions and Multiplications
Zimmerman, F., Shalóm, D. E., Gonzalez, P. A., Garrido, J. M., Heduan, F. Á., Dehaene, S., Sigman, M., & Rieznik, A. (2016) · PLoS ONE · doi:10.1371/journal.pone.0168431 · Cited by 19 · see citing works
Do-It-Yourself Gamified Cognitive Training: Viewpoint
van de Weijer, S. C. F., Kuijf, M. L., de Vries, N. M., Bloem, B. R., & Duits, A. (2019) · JMIR Serious Games · doi:10.2196/12130 · Cited by 19 · see citing works
Explaining distortions in metacognition with an attractor network model of decision uncertainty
Atiya, N. A. A., Huys, Q. J. M., Dolan, R. J., & Fleming, S. M. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009201 · Cited by 17 · see citing works
Self-citationSmartphones within Psychological Science
Ellis, D. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671408 · Cited by 14 · see citing works
Measures of Implicit Cognition for Marketing Research
Teichert, T., Graf, A., Rezaei, S., Wörfel, P., & Duh, H. (2019) · Marketing ZFP · doi:10.15358/0344-1369-2019-3-48 · Cited by 16 · see citing works
Maekawa, T., Anderson, S. J., de Brecht, M., & Yamagishi, N. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195865 · Cited by 14 · see citing works
CrowdXR - Pitfalls and Potentials of Experiments with Remote Participants
Zhao, J., Simpson, M., Sajjadi, P., Wallgrün, J. O., Li, P., Bagher, M. M., Oprean, D., Padilla, L., & Klippel, A. (2021) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar52148.2021.00062 · Cited by 14 · see citing works
Age-dependent Pavlovian biases influence motor decision-making
Chen, X., Rutledge, R. B., Brown, H. R., Dolan, R. J., Bestmann, S., & Galea, J. M. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006304 · Cited by 13 · see citing works
Self-citationWhat smartphone apps may contribute to language evolution research
Morin, O., Winters, J., Müller, T. F., Morisseau, T., Etter, C., & Greenhill, S. J. (2018) · Journal of Language Evolution · doi:10.1093/jole/lzy005 · Cited by 12 · see citing works
Approach-induced biases in human information samplingPreprint
Hunt, L. T., Rutledge, R. B., Malalasekera, W. M. N., Kennerley, S. W., & Dolan, R. J. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/047787 · Cited by 12 · see citing works
Self-citationGundry, D., & Deterding, S. (2018) · International Conference on the Foundations of Digital Games (FDG) · doi:10.1145/3235765.3235803 · Cited by 11 · see citing works
Tünnermann, J., Kristjansson, A. K., Petersen, A., Schubö, A., & Scharlau, I. (2022) · Open Psychology · doi:10.1515/psych-2022-0002 · Cited by 10 · see citing works
Stenner, M., Rutledge, R. B., Zaehle, T., Schmitt, F. C., Kopitzki, K., Kowski, A. B., Voges, J., Heinze, H., & Dolan, R. J. (2015) · Journal of Neurophysiology · doi:10.1152/jn.00260.2015 · Cited by 9 · see citing works
Self-citationMood and implicit confidence independently fluctuate at different time scales
da Fonseca, M., Maffei, G., Moreno‐Bote, R., & Hyafil, A. (2022) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-022-01038-4 · Cited by 10 · see citing works
Lee, R. S. C., Albertella, L., Christensen, E., Suo, C., Segrave, R. A., Brydevall, M., Kirkham, R., Liu, C., Fontenelle, L. F., Chamberlain, S. R., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/44414 · Cited by 8 · see citing works
Resources, challenges and way forward in rare mitochondrial diseases research
Rajput, N. K., Singh, V. K., & Bhardwaj, A. (2015) · F1000Research · doi:10.12688/f1000research.6208.1 · Cited by 8 · see citing works
Thirkettle, M., Lewis, J., Langdridge, D., & Pike, G. E. (2018) · JMIR Serious Games · doi:10.2196/10519 · Cited by 7 · see citing works
Lyons, E., & Zhang, L. (2019) · PLoS ONE · doi:10.1371/journal.pone.0224946 · Cited by 7 · see citing works
On the Impact of COVID-19 on Subjective Digital Media Quality Assessment
Zepernick, H., Pieper, K., Spang, R. P., Engelke, U., Hirth, M., & Naderi, B. (2021) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp53017.2021.9733583 · Cited by 7 · see citing works
Voudouris, K., Slater, B., Cheke, L. G., Schellaert, W., Hernández‐Orallo, J., Halina, M., Patel, M., Alhas, I., Mecattaf, M. G., Burden, J., et al. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02616-3 · Cited by 8 · see citing works
Trading Accuracy for Enjoyment? Data Quality and Player Experience in Data Collection Games
Gundry, D., & Deterding, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502025 · Cited by 6 · see citing works
Exploring the Potentials of Crowdsourcing for Gesture Data Collection
Jung, I.-T., Ahn, S., Seo, J., & Hong, J.-H. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2180235 · Cited by 6 · see citing works
Validation of motion perception of briefly displayed images using a tablet
Linares, D., Marin-Campos, R., Dalmau, J. O., & Compte, A. (2018) · Scientific Reports · doi:10.1038/s41598-018-34466-9 · Cited by 6 · see citing works
The perils of acting rashly: Risk-taking propensity impeding emotion-based learning in entrepreneurs
Seloni, G., Kusrohmaniah, S., & Lufityanto, G. (2023) · Journal of International Entrepreneurship · doi:10.1007/s10843-022-00325-y · Cited by 5 · see citing works
Malamud, J., Gülöksüz, S., van Winkel, R., Delespaul, P., Binder, M. D., Derom, C. A., Thiery, E. W., Jacobs, N., Rutten, B. P. F., & Huys, Q. J. M. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012457 · Cited by 8 · see citing works
Obstacles to Transparency in Privacy Engineering
Brennan-Marquez, K., & Susser, D. (2016) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2016.18 · Cited by 5 · see citing works
The neural dynamics of loss aversion
Nie, S., Wang, M., Li, J., Luo, H., & Zhang, H. (2023) · Imaging Neuroscience · doi:10.1162/imag_a_00047 · Cited by 4 · see citing works
Surprising sounds influence risky decision making
Feng, G. W., & Rutledge, R. B. (2024) · Nature Communications · doi:10.1038/s41467-024-51729-4 · Cited by 4 · see citing works
Self-citationInfluencing and Measuring Behaviour in Crowdsourced Activities
Gould, S. J. J., Cox, A. L., & Brumby, D. P. (2018) · Human-computer interaction series · doi:10.1007/978-3-319-73374-6_7 · Cited by 4 · see citing works
Crowdsourcing Management Education Assessment
Hall, O. P., & Griffy‐Brown, C. (2016) · Journal of international technology and information management · doi:10.58729/1941-6679.1270 · Cited by 4 · see citing works
Santángelo, A. P., & Solovey, G. (2021) · Journal of Cognition · doi:10.5334/joc.200 · Cited by 3 · see citing works
Risk, reward and loss in addictive behavior: a six-year cross-lagged panel study
Kräplin, A., Joshanloo, M., Fröhner, J. H., Baeuchl, C., Bühringer, G., Goschke, T., & Smolka, M. N. (2025) · Scientific Reports · doi:10.1038/s41598-025-17826-0 · Cited by 3 · see citing works
What makes rhythms hard to perform? An investigation using Steve Reich’s Clapping Music
Duffy, S., & Pearce, M. T. (2018) · PLoS ONE · doi:10.1371/journal.pone.0205847 · Cited by 3 · see citing works
Recurrent Auto-Encoding Drift Diffusion ModelPreprint
Vincent, M., & Zénon, A. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/220517 · Cited by 3 · see citing works
Zech, H., Waltmann, M., Lee, Y., Reichert, M., Bedder, R. L., Rutledge, R. B., Deeken, F., Wenzel, J. G., Wedemeyer, F., Aguilera, A., et al. (2022) · doi:10.31234/osf.io/6mhsw · Cited by 3 · see citing works
Self-citationThe DECIDE Framework: Describing Ethical Choices in Digital-Behavioral-Data ExplorationsPreprint
Shaw, H., Brown, O., Hinds, J. M., Nightingale, S. J., Towse, J. N., & Ellis, D. A. (2023) · doi:10.31234/osf.io/c8ert · Cited by 3 · see citing works
Memory sensitivity improves with larger image size during naturalistic encoding
Kreichman, O., Masarwa, S., Brook, L., & Gilaie‐Dotan, S. (2025) · Visual Cognition · doi:10.1080/13506285.2025.2610180 · Cited by 2 · see citing works
He, J. A., Sense, F., & Antshel, K. M. (2015) · The ADHD Report · doi:10.1521/adhd.2015.23.1.1 · Cited by 2 · see citing works
Prisoner of Words: Lessons Learnt from Mobile Gamification of Lab Memory Experiments
Elagroudy, P., Khoshdani, P., Dingler, T., Ward, G. D., Woźniak, P. W., & Schmidt, A. (2021) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3450337.3483502 · Cited by 2 · see citing works
Personality and Individual Differences
Ellis, D. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671408.006 · Cited by 2 · see citing works
Ellis, D. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671408.003 · Cited by 2 · see citing works
Model-Agnostic Fits for Understanding Information Seeking Patterns in Humans
Chatterjee, S., & Shenoy, P. (2021) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v35i1.16160 · Cited by 1 · see citing works
Social Interaction and Interpersonal Relationships
Ellis, D. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671408.005 · Cited by 1 · see citing works
Tünnermann, J., Kristjansson, A. K., Petersen, A., Schubö, A., & Scharlau, I. (2021) · doi:10.31234/osf.io/jybfh · Cited by 1 · see citing works
2016年度日本基礎心理学会第1回フォーラム 基礎心理学の新しい方法を拓く―ビッグデータ・生体センサー・タブレットの活用―
三浦, 佳. (2017) · 基礎心理学研究 · doi:10.14947/psychono.35.22
Productivity and wellbeing in the 21st century workplace : implications of choice
Hanc, M.-L. (2019) · UCL Discovery (University College London)
The role of self-control in pro-environmental behaviour
Oniga, A. (2019) · St Andrews Research Repository (St Andrews Research Repository) · doi:10.17630/10023-19249 · Cited by 1 · see citing works
Hooyman, A., Huentelman, M. J., & Schaefer, S. Y. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.04.14.041384
Santángelo, A. P., & Solovey, G. (2020) · doi:10.31234/osf.io/nuxdg
Digital technology-enabled home health care
van de Weijer, S. C. F. (2021) · doi:10.26481/dis.20210901sw
Axelsen, J. L., Meline, J. S. J., Staiano, W., & Kirk, U. (2021) · Research Square · doi:10.21203/rs.3.rs-881039/v1
Ellis, D. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671408.008
- Authors
- Brown, H. R., Zeidman, P., Smittenaar, P., Adams, R. A., McNab, F., Rutledge, R. B., & Dolan, R. J.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Brown, Harriet R.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-07-15
- Volume
- 9
- Issue
- 7
- Pages
- e100662
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4099129
- OpenAlex
- W2044432453
- Cited by
- 129
- References
- 31
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 7 · 2023: 10 · 2022: 8 · 2021: 16 · 2020: 14 · 2019: 19 · 2018: 13 · 2017: 6 · 2016: 15 · 2015: 11 · 2014: 1
- FWCI
- 16.39
- Type
- journal article
- Language
- en
- Update notices
- correction (2014-10-14)
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, cc-by)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion, cc-by)
- Funders
- Wellcome Trust (091593, 091593/Z/10/, 098362, 098362/Z/12/Z, 091593/Z/10/Z, 101252/Z/13/Z)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- crowdsourcing, smartphones, mobile data collection, cognitive science, large-scale experimentation, data validity, reproducibility, experimental paradigms, mobile application, sample size, phenotypic screening, cost reduction
- MeSH terms
- Adolescent, Adult, Age Factors, Aged, Decision Making, Female, Humans, Male, Memory, Short-Term, Middle Aged, Cognitive Science, Cell Phone, Attentional Blink, Young Adult, Wireless Technology, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
