What can 1 billion trials tell us about visual search?
Open accessAbstract
Mobile technology (e.g., smartphones and tablets) has provided psychologists with a wonderful opportunity: through careful design and implementation, mobile applications can be used to crowd source data collection. By garnering massive amounts of data from a wide variety of individuals, it is possible to explore psychological questions that have, to date, been out of reach. Here we discuss 2 examples of how data from the mobile game Airport Scanner (Kedlin Co., http://www.airportscannergame.com) can be used to address questions about the nature of visual search that pose intractable problems for laboratory-based research. Airport Scanner is a successful mobile game with millions of unique users and billions of individual trials, which allows for examining nuanced visual search questions. The goals of the current Observation Report were to highlight the growing opportunity that mobile technology affords psychological research and to provide an example roadmap of how to successfully collect usable data.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A large number of studies have validated the visual search paradigm to explore visual attention mechanisms in healthy and clinical populations.
“A large number of studies have validated the visual search paradigm to explore visual attention mechanisms in healthy and clinical populations [e.g., (76–79)].”
From The Evolution of Psychological and Behavioral Consequences of Self-Isolation During Lockdown: A Longitudinal Study Across United Kingdom and Italy (2022) · cites this paper as [76]
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
Labeling images with a computer game
von Ahn, L., & Dabbish, L. (2004) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/985692.985733 · Cited by 2,230 · see citing works
Video game training enhances cognitive control in older adults
Anguera, J. A., Boccanfuso, J., Rintoul, J. L., Al-Hashimi, O., Faraji, F., Janowich, J., Kong, E., Larraburo, Y., Rolle, C., Johnston, E., et al. (2013) · Nature · doi:10.1038/nature12486 · Cited by 1,666 · see citing works
Change blindness: Past, present, and future
Simons, D. J., & Rensink, R. A. (2005) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.11.006 · Cited by 1,153 · see citing works
The Interpretation of Reaction Time in Information-Processing Research 1
Pachella, R. G. (2021) · Routledge eBooks · doi:10.4324/9781003176688-2 · Cited by 751 · see citing works
What Can 1 Million Trials Tell Us About Visual Search?
Wolfe, J. M. (1998) · Psychological Science · doi:10.1111/1467-9280.00006 · Cited by 717 · see citing works
Kantowitz, B. H. (2021) · Routledge eBooks · doi:10.4324/9781003176688 · Cited by 555 · see citing works
Visual search: A retrospective
Eckstein, M. P. (2011) · Journal of Vision · doi:10.1167/11.5.14 · Cited by 477 · see citing works
Satisfaction of Search in Diagnostic Radiology
Berbaum, K. S., Franken, E. A., Dorfman, D. D., Rooholamini, S. A., Kathol, M. H., Barloon, T. J., Behlke, F. M., Satō, Y., Lu, C. H., El-Khoury, G. Y., et al. (1990) · Investigative Radiology · doi:10.1097/00004424-199002000-00006 · Cited by 292 · see citing works
Visual Search, Image Organization, and Reader Error in Roentgen Diagnosis
Tuddenham, W. J. (1962) · Radiology · doi:10.1148/78.5.694 · Cited by 206 · see citing works
Mané, A. M., & Donchin, E. (1989) · Acta Psychologica · doi:10.1016/0001-6918(89)90003-6 · Cited by 175 · see citing works
Nakayama, K., & Martini, P. (2010) · Vision Research · doi:10.1016/j.visres.2010.09.003 · Cited by 164 · see citing works
Generalized “satisfaction of search”: Adverse influences on dual-target search accuracy.
Fleck, M. S., Samei, E., & Mitroff, S. R. (2010) · Journal of Experimental Psychology Applied · doi:10.1037/a0018629 · Cited by 127 · see citing works
Time Course of Satisfaction of Search
Berbaum, K. S., Franken, E. A., Dorfman, D. D., Rooholamini, S. A., Coffman, C. E., Cornell, S. H., Cragg, A. H., Galvin, J. R., Honda, H., Kao, S. C.-S., et al. (1991) · Investigative Radiology · doi:10.1097/00004424-199107000-00003 · Cited by 101 · see citing works
ERROR AND VARIATION IN DIAGNOSTIC RADIOLOGY
(1968) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.1968.tb82792.x · Cited by 99 · see citing works
Transfer of skill engendered by complex task training under conditions of variable priority
Boot, W. R., Basak, C., Erickson, K. I., Neider, M. B., Simons, D. J., Fabiani, M., Gratton, G., Voss, M. W., Prakash, R. S., Lee, H., et al. (2010) · Acta Psychologica · doi:10.1016/j.actpsy.2010.09.005 · Cited by 97 · see citing works
A taxonomy of errors in multiple-target visual search
Cain, M. S., Adamo, S. H., & Mitroff, S. R. (2013) · Visual Cognition · doi:10.1080/13506285.2013.843627 · Cited by 91 · see citing works
Mitroff, S. R., & Biggs, A. T. (2013) · Psychological Science · doi:10.1177/0956797613504221 · Cited by 88 · see citing works
Memory for found targets interferes with subsequent performance in multiple-target visual search.
Cain, M. S., & Mitroff, S. R. (2012) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/a0030726 · Cited by 86 · see citing works
Intrinsic motivation and attentional capture from gamelike features in a visual search task
Miranda, A. T., & Palmer, E. M. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0357-7 · Cited by 86 · cited 2× · see citing works
“…Human Perception and Performance © 2014 American Psychological Association 2015, Vol. 41, No. 1, 1–5 0096-1523/15/$12.00 http://dx.doi.org/10.1037/xhp0000012 1 game is only worthwhile if people will play the game, and people are more likely to play if the game is fun (e.g., Miranda & Palmer, 2014).”
BackgroundInfluentialSelf-Induced Attentional Blink
Adamo, S. H., Cain, M. S., & Mitroff, S. R. (2013) · Psychological Science · doi:10.1177/0956797613497970 · Cited by 67 · see citing works
Biggs, A. T., & Mitroff, S. R. (2013) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2013.859715 · Cited by 48 · see citing works
Rare, but obviously there: Effects of target frequency and salience on visual search accuracy
Biggs, A. T., Adamo, S. H., & Mitroff, S. R. (2014) · Acta Psychologica · doi:10.1016/j.actpsy.2014.08.005 · Cited by 34 · see citing works
Satisfaction of search experiments in advanced imaging
Berbaum, K. S. (2012) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.916461 · Cited by 25 · see citing works
Exit Strategies: Visual Search and the Quitting Time Problem
Horowitz, T. S. (2014) · WORLD SCIENTIFIC eBooks · doi:10.1142/9789814590297_0016 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Recent Progress in Virtual Reality Exposure Therapy for Phobias: A Systematic Review
Botella, C., Fernández-Álvarez, J., Guillén, V., García-Palacios, A., & Baños, R. (2017) · Current Psychiatry Reports · doi:10.1007/s11920-017-0788-4 · Cited by 460 · see citing works
Visual Search: How Do We Find What We Are Looking For?
Wolfe, J. M. (2020) · Annual Review of Vision Science · doi:10.1146/annurev-vision-091718-015048 · Cited by 237 · see citing works
Lindner, P., Miloff, A., Hamilton, W., Reuterskiöld, L., Andersson, G., Powers, M. B., & Carlbring, P. (2017) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2017.1280843 · Cited by 187 · see citing works
Oswald, F. L., Behrend, T. S., Putka, D. J., & Sinar, E. F. (2019) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-032117-104553 · Cited by 174 · see citing works
User-Centered Virtual Reality for Promoting Relaxation: An Innovative Approach
Pizzoli, S. F. M., Mazzocco, K., Triberti, S., Monzani, D., Alcañíz, M., & Pravettoni, G. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00479 · Cited by 140 · see citing works
Analyzing Big Data in Psychology: A Split/Analyze/Meta-Analyze Approach
Cheung, M. W. ‐., & Jak, S. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00738 · Cited by 83 · see citing works
Changing What You See by Changing What You Know: The Role of Attention
Lupyan, G. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00553 · Cited by 53 · see citing works
Examining perceptual and conceptual set biases in multiple-target visual search
Biggs, A. T., Adamo, S. H., Dowd, E. W., & Mitroff, S. R. (2015) · Attention Perception & Psychophysics · doi:10.3758/s13414-014-0822-0 · Cited by 45 · see citing works
Adamo, S. H., Gereke, B. J., Shomstein, S., & Schmidt, J. (2021) · Cognitive Research Principles and Implications · doi:10.1186/s41235-021-00318-w · Cited by 39 · see citing works
Targets Need Their Own Personal Space: Effects of Clutter on Multiple-Target Search Accuracy
Adamo, S. H., Cain, M. S., & Mitroff, S. R. (2015) · Perception · doi:10.1177/0301006615594921 · Cited by 31 · see citing works
Adamo, S. H., Cain, M. S., & Mitroff, S. R. (2016) · Vision Research · doi:10.1016/j.visres.2016.10.010 · Cited by 31 · see citing works
Predicting Airport Screening Officers’ Visual Search Competency With a Rapid Assessment
Mitroff, S. R., Ericson, J. M., & Sharpe, B. T. (2017) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720817743886 · Cited by 25 · see citing works
How many decimals? Rounding descriptive and inferential statistics based on measurement precision
Cousineau, D. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102362 · Cited by 23 · see citing works
Biggs, A. T. (2017) · Attention Perception & Psychophysics · doi:10.3758/s13414-017-1300-2 · Cited by 19 · see citing works
Siritzky, E. M., Cox, P., Nadler, S. M., Grady, J. N., Kravitz, D. J., & Mitroff, S. R. (2023) · Cognitive Research Principles and Implications · doi:10.1186/s41235-023-00520-y · Cited by 20 · see citing works
Playing ‘Pong’ Together: Emergent Coordination in a Doubles Interception Task
Benerink, N. H., Zaal, F. T. J. M., Casanova, R., Bonnardel, N., & Bootsma, R. J. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01910 · Cited by 18 · see citing works
The Sustained Influence of an Error on Future Decision-Making
Schiffler, B., Bengtsson, S., & Lundqvist, D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01077 · Cited by 15 · see citing works
Using big data to solve real problems through academic and industry partnerships
Mitroff, S. R., & Sharpe, B. T. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.09.013 · Cited by 15 · see citing works
Tasks determine what is learned in visual statistical learning
Vickery, T. J., Park, S. H., Gupta, J., & Berryhill, M. E. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1405-6 · Cited by 14 · see citing works
Ryu, H., Ju, U., & Wallraven, C. (2021) · PLoS ONE · doi:10.1371/journal.pone.0258441 · Cited by 13 · see citing works
Goold, J. E., & Meng, M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00155 · Cited by 12 · see citing works
Conscientiousness protects visual search performance from the impact of fatigue
Grady, J. N., Cox, P., Nag, S., & Mitroff, S. R. (2022) · Cognitive Research Principles and Implications · doi:10.1186/s41235-022-00410-9 · Cited by 11 · see citing works
Visual Search: You Are Who You Are (+ A Learning Curve)
Ericson, J. M., Kravitz, D. J., & Mitroff, S. R. (2017) · Perception · doi:10.1177/0301006617721091 · Cited by 11 · see citing works
Kravitz, D. J., & Mitroff, S. R. (2023) · Journal for Reproducibility in Neuroscience · doi:10.36850/jrn.2023.e44 · Cited by 11 · see citing works
Feature integration theory in non-humans: Spotlight on the archerfish
Reichenthal, A., Segev, R., & Ben‐Shahar, O. (2020) · Attention Perception & Psychophysics · doi:10.3758/s13414-019-01884-4 · Cited by 10 · see citing works
Benerink, N. H., Zaal, F. T. J. M., Casanova, R., Bonnardel, N., & Bootsma, R. J. (2017) · Human Movement Science · doi:10.1016/j.humov.2017.11.012 · Cited by 9 · see citing works
Learning “What” and “Where” in Visual Search
Makovski, T. (2017) · Japanese Psychological Research · doi:10.1111/jpr.12146 · Cited by 8 · see citing works
Silverman, M. E., Nag, S., Kalishman, A., Cox, P., & Mitroff, S. R. (2022) · Journal of American College Health · doi:10.1080/07448481.2022.2080507 · Cited by 8 · see citing works
Zaninotto, F., Bossi, F., Terry, P., Riccaboni, M., & Galli, G. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.826277 · Cited by 7 · see citing works
Porfido, C. L., Cox, P., Adamo, S. H., & Mitroff, S. R. (2019) · Visual Cognition · doi:10.1080/13506285.2019.1702602 · Cited by 7 · see citing works
Object-substitution masking weakens but does not eliminate shape interactions
Sweeny, T. D., D’Abreu, L. C., Elias, E., & Padama, L. (2017) · Attention Perception & Psychophysics · doi:10.3758/s13414-017-1381-y · Cited by 5 · see citing works
Prospect theory and its implications for adversarial decision-making
Biggs, A. T., & Pettijohn, K. A. (2019) · The Journal of Defense Modeling and Simulation Applications Methodology Technology · doi:10.1177/1548512919840445 · Cited by 5 · see citing works
Practicing Good Laboratory Hygiene, Even in a Pandemic
Kravitz, D. J., Mitroff, S. R., & Bauer, P. J. (2020) · Psychological Science · doi:10.1177/0956797620920547 · Cited by 5 · see citing works
Modeling Target-Distractor Discrimination for Haptic Search in a 3D Environment
Moringen, A., Aswolinkiy, W., Büscher, G., Walck, G., Haschke, R., & Ritter, H. J. (2018) · IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob) · doi:10.1109/biorob.2018.8487705 · Cited by 4 · see citing works
Assessing satisfaction of search in virtual mammograms for experienced and novice searchers
Roque, N., Barufaldi, B., Schmidt, J., Mello-Thoms, C. R., & Lago, M. A. (2023) · Journal of medical imaging · doi:10.1117/1.jmi.10.s1.s11917 · Cited by 3 · see citing works
Technological Interventions for Anxiety Disorders
Baños, R., Vara, M., Mira, A., García-Palacios, A., & Botella, C. (2020) · Reference Module in Neuroscience and Biobehavioral Psychology · doi:10.1016/b978-0-12-818697-8.00008-x · Cited by 2 · see citing works
Kravitz, D. J., Mitroff, S. R., Flintoft, C. C., & Oie, K. S. (2024) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/23727322241274745 · Cited by 2 · see citing works
Prisoner of Words: Lessons Learnt from Mobile Gamification of Lab Memory Experiments
Elagroudy, P., Khoshdani, P., Dingler, T., Ward, G. D., Woźniak, P. W., & Schmidt, A. (2021) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3450337.3483502 · Cited by 2 · see citing works
Nag, S., Yu, A. B., & Mitroff, S. R. (2021) · doi:10.31234/osf.io/37svc · Cited by 1 · see citing works
Conscientiousness protects visual search performance from the impact of fatiguePreprint
Grady, J. N., Cox, P., Nag, S., & Mitroff, S. R. (2022) · doi:10.31234/osf.io/rxcsw · Cited by 1 · see citing works
Perceptual features win again: the role of knowledge of acting with objects in visual search
Anufrieva, A. A., & Gorbunova, E. S. (2025) · Cognitive Processing · doi:10.1007/s10339-025-01289-2
Bainbridge, W. A. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009343602.009
The card game Dobble as a diagnostic tool
Šolcová, A., & Děchtěrenko, F. (2026) · Acta Psychologica · doi:10.1016/j.actpsy.2026.106696
Attentional capture effects are modulated by the number of concurrently maintained search goals
Moore, K. S., & Kershner, A. M. (2026) · Frontiers in Cognition · doi:10.3389/fcogn.2026.1818391
Malykke, S. B., Siqi-Liu, A., Kravitz, D. J., & Mitroff, S. R. (2026) · Attention Perception & Psychophysics · doi:10.3758/s13414-026-03281-0
Weatherford, D. R., Dicke-Bohmann, A. K., Roberson, D., Erickson, W. B., & Thomas, J. (2026) · Psychology Crime and Law · doi:10.1080/1068316x.2026.2701473
- Authors
- Mitroff, S. R., Biggs, A. T., Adamo, S. H., Dowd, E. W., Winkle, J., & Clark, K.
- Venue
- Journal of Experimental Psychology: Human Perception and Performance
- Published
- 2015-02
- Volume
- 41
- Issue
- 1
- Pages
- 1-5
- ISSN
- 0096-1523 (ISSN-L)
- 1939-1277
- OpenAlex
- W2139917426
- Cited by
- 46
- References
- 25
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 1 · 2023: 3 · 2022: 4 · 2021: 4 · 2020: 5 · 2019: 4 · 2018: 1 · 2017: 12 · 2016: 4 · 2015: 2
- FWCI
- 8.93
- Type
- journal article
- Language
- en
- Open access
- green
- eprints.uwe.ac.uk
- Licence
- cc-by
- Locations
- Journal of Experimental Psychology Human Perception & Performance (publishedVersion)
- PubMed (publishedVersion)
- UWE Research Repository (UWE Bristol) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Keywords
- visual search, mobile games, mobile crowdsourcing, smartphone data collection, large-scale data collection
- MeSH terms
- Data Collection, Humans, Psychomotor Performance, Space Perception, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
