Crowdsourcing Thumbnail Captions: Data Collection and Validation
Open accessAbstract
Speech interfaces, such as personal assistants and screen readers, read image captions to users. Typically, however, only one caption is available per image, which may not be adequate for all situations (e.g., browsing large quantities of images). Long captions provide a deeper understanding of an image but require more time to listen to, whereas shorter captions may not allow for such thorough comprehension yet have the advantage of being faster to consume. We explore how to effectively collect both thumbnail captions—succinct image descriptions meant to be consumed quickly—and comprehensive captions—which allow individuals to understand visual content in greater detail. We consider text-based instructions and time-constrained methods to collect descriptions at these two levels of detail and find that a time-constrained method is the most effective for collecting thumbnail captions while preserving caption accuracy. Additionally, we verify that caption authors using this time-constrained method are still able to focus on the most important regions of an image by tracking their eye gaze. We evaluate our collected captions along human-rated axes—correctness, fluency, amount of detail, and mentions of important concepts—and discuss the potential for model-based metrics to perform large-scale automatic evaluations in the future.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Little is known about writing support for non-experts, such as DSO professionals, in figure accessibility.
“While previous research in figure accessibility has mostly focused on readers [2, 32, 66], or authors of papers [31, 71], little is known about the writing support for non-experts , such as DSO professionals.”
From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [2]
The Introduction and Related Work (pages unavailable in our copy) describe research focused on how blind and visually impaired readers consume image descriptions, which supports the sentence’s characterization of some prior work. The paper’s stated gap concerns collecting thumbnail captions with specific levels of detail; it does not establish that little is known about writing support for non-experts such as DSO professionals in figure accessibility. (LLM, conf=0.78)
Experimental — accuracy not yet measuredWriting effective alt text requires both subject literacy and familiarity with accessibility guidelines.
“Writing effective alt text requires both subject literacy and familiarity with accessibility guidelines [2, 73].”
From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [2]
Section 2.2 says creating alt text is challenging, and Section 2.1 discusses WCAG recommendations for descriptions of complex images. Neither passage says effective alt text requires subject literacy or that its writers must be familiar with accessibility guidelines. Searches for subject-matter expertise and guideline knowledge found no such statement in our copy. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373 · see citing works
The attention system of the human brain
Posner, M. I., & Petersen, S. E. (1990) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.13.030190.000325 · Cited by 8,070 · cited 1× · see citing works
“4 Gaze Tracking and Image Understanding Gaze tracking can be used to gain insight into people’s intentions and cognitive processes based on what they are looking at [4, 11, 23, 28, 51]; it has helped researchers measure and understand human behavior and cognitive load in a variety of tasks, ranging from basic scene vi…”
BackgroundEye movements in reading and information processing: 20 years of research
Rayner, K. (1998) · Psychological Bulletin · doi:10.1037/0033-2909.124.3.372 · Cited by 7,979 · see citing works
Computational modelling of visual attention
Itti, L., & Koch, C. (2001) · Nature Reviews Neuroscience · doi:10.1038/35058500 · Cited by 4,733 · see citing works
Speed of processing in the human visual system
Thorpe, S., Fize, D., & Marlot, C. (1996) · Nature · doi:10.1038/381520a0 · Cited by 3,538 · see citing works
Prediction and Entropy of Printed English
Shannon, C. E. (1951) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1951.tb01366.x · Cited by 2,686 · see citing works
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · see citing works
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
Mark, G., Fussell, S. R., Lampe, C., Schraefel, M., Hourcade, J., Appert, C., & Wigdor, D. J. (2017) · International Conference on Human Factors in Computing Systems · doi:10.1145/3025453 · Cited by 2,222 · see citing works
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108 · see citing works
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,633 · see citing works
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · see citing works
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Hodosh, M., Young, P., & Hockenmaier, J. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3994 · Cited by 1,349 · see citing works
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.595 · Cited by 1,232 · see citing works
Proceedings of the 3rd International Conference on Neural Information Processing Systems
Lippmann, R. P., Moody, J. E., & Touretzky, D. S. (1990) · Cited by 1,204 · see citing works
In what ways do eye movements contribute to everyday activities?
Land, M. F., & Hayhoe, M. (2001) · Vision Research · doi:10.1016/s0042-6989(01)00102-x · Cited by 966 · see citing works
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Artetxe, M., & Schwenk, H. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00288 · Cited by 899 · see citing works
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2587640 · Cited by 875 · see citing works
Eye Movements During Scene Viewing
Henderson, J. M., & Hollingworth, A. (1998) · Elsevier eBooks · doi:10.1016/b978-008043361-5/50013-4 · Cited by 373 · see citing works
Gaze-based interaction for semi-automatic photo cropping
Santella, A., Agrawala, M., DeCarlo, D., Salesin, D., & Cohen, M. (2006) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124886 · Cited by 353 · see citing works
Defending Yarbus: Eye movements reveal observers' task
Borji, A., & Itti, L. (2014) · Journal of Vision · doi:10.1167/14.3.29 · Cited by 246 · see citing works
Visual attention toward tourism photographs with text: An eye-tracking study
Li, Q., Huang, Z., & Christianson, K. (2015) · Tourism Management · doi:10.1016/j.tourman.2015.11.017 · Cited by 228 · see citing works
Understanding Blind People's Experiences with Computer-Generated Captions of Social Media Images
MacLeod, H., Bennett, C. L., Morris, M. R., & Cutrell, E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025814 · Cited by 200 · see citing works
Using gaze patterns to predict task intent in collaboration
Huang, C., Andrist, S., Sauppé, A., & Mutlu, B. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01049 · Cited by 168 · see citing works
Self-citationImproving accessibility of the web with a computer game
von Ahn, L., Ginosar, S., Kedia, M., Liu, R., & Blum, M. (2006) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124785 · Cited by 156 · see citing works
Hayhoe, M. (2017) · Annual Review of Vision Science · doi:10.1146/annurev-vision-102016-061437 · Cited by 156 · see citing works
Stangl, A., Morris, M. R., & Gurari, D. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376404 · Cited by 139 · cited 3× · see citing works
“Evaluating detail importance is challenging for human judges in general-purpose captioning tasks such as ours because the relevance of specific details is highly dependent on the task at hand [50, 62].”
BackgroundMethodInfluentialComparing Automatic Evaluation Measures for Image Description
Elliott, D., & Keller, F. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2074 · Cited by 137 · see citing works
Eye movements and the span of the effective stimulus in visual search
Bertera, J. H., & Rayner, K. (2000) · Perception & Psychophysics · doi:10.3758/bf03212109 · Cited by 133 · see citing works
Eye Guidance and Visual Search
Findlay, J. M., & Gilchrist, I. D. (1998) · Elsevier eBooks · doi:10.1016/b978-008043361-5/50014-6 · Cited by 119 · see citing works
Gleason, C., Carrington, P., Cassidy, C., Morris, M., Kitani, K. M., & Bigham, J. (2019) · The Web Conference · doi:10.1145/3308558.3313605 · Cited by 108 · see citing works
Caption Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image Search
Guinness, D., Cutrell, E., & Morris, M. R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174092 · Cited by 108 · see citing works
Jung, C., Mehta, S., Kulkarni, A., Zhao, Y., & Kim, Y.-S. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114846 · Cited by 91 · see citing works
Salisbury, E., Kamar, E. S., & Morris, M. R. (2017) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v5i1.13301 · Cited by 85 · see citing works
Zhong, Y., Lasecki, W. S., Brady, E. L., & Bigham, J. P. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702437 · Cited by 68 · see citing works
Designing Tools for High-Quality Alt Text Authoring
Mack, K., Cutrell, E., Lee, B., & Morris, M. R. (2021) · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3441852.3471207 · Cited by 58 · cited 5× · see citing works
“Prior work has illustrated that access to captions with multiple levels of detail is beneicial to BVI individuals [31, 44, 46]; future work may explore new methods and mechanisms to support the efective collection of captions at additional levels of detailÐincluding both less and more detail than typical alt text.”
BackgroundInfluentialChoi, D., Kwak, D., Cho, M., & Lee, S. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376569 · Cited by 57 · see citing works
Estimation of mental workload using saccadic eye movements in a free-viewing task
Tokuda, S., Obinata, G., Palmer, E. M., & Chaparro, A. (2011) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.2011.6091121 · Cited by 49 · see citing works
Detecting drivers’ cognitive load from saccadic intrusion
Biswas, P., & Prabhakar, G. (2018) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2018.01.017 · Cited by 45 · see citing works
Tachistoscopic simulations of eye fixations on pictures.
Loftus, G. R. (1981) · Journal of Experimental Psychology Human Learning & Memory · doi:10.1037//0278-7393.7.5.369 · Cited by 40 · see citing works
Do eye movements reflect driving manoeuvres?
Lethaus, F., & Rataj, J. (2007) · IET Intelligent Transport Systems · doi:10.1049/iet-its:20060058 · Cited by 40 · see citing works
Image Accessibility for Screen Reader Users: A Systematic Review and a Road Map
Oh, U., Joh, H., & Lee, Y. (2021) · Electronics · doi:10.3390/electronics10080953 · Cited by 29 · see citing works
Subramanian, R., Katti, H., Huang, R. Z., Chua, T., & Kankanhalli, M. (2009) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1631272.1631399 · Cited by 28 · see citing works
Simons, R. N., Gurari, D., & Fleischmann, K. R. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415176 · Cited by 27 · see citing works
Discriminating cognitive processes with eye movements in a decision-making driving task.
Lemonnier, S., Brémond, R., & Baccino, T. (2014) · Journal of Eye Movement Research · doi:10.16910/jemr.7.4.3 · Cited by 26 · see citing works
Future research directions for accessible social media
Gleason, C., Carrington, P., Chilton, L. B., Gorman, B. M., Kacorri, H., Monroy‐Hernández, A., Morris, M. R., Tigwell, G. W., & Wu, S. (2020) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/3412836.3412839 · Cited by 19 · see citing works
Crowdsourcing Thumbnail Captions via Time-Constrained Methods
Aguirre, C. A., Mahmood, A., & Huang, C. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511136 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
1 of the 55 references couldn’t be matched to a known paper.
Research works printed but not linked: 13
- Somnath Arjun, G. S. Rajshekar Reddy, Abhishek Mukhopadhyay, Sanjana Vinod, and Pradipta Biswas. 2021. Evaluat- ing visual variables in a virtual reality environment. In 34th British HCI Conference 34. BCS Learning & Development, London, 11-22.
- Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O'Reilly Media, Inc., Sebastopol, CA.
- Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. 2021. Multimodal datasets: Misogyny, pornography, and malignant stereotypes. arXiv:2110.01963 [cs.CY] · arXiv
- Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya. 2020. Captioning images taken by people who are blind. In European Conference on Computer Vision. Springer, Springer International Publishing, Cham, 417-434.
- Anett Hoppe, David Morris, and Ralph Ewerth. 2021. Evaluation of automated image descriptions for visually impaired students. In International Conference on Artificial Intelligence in Education. Springer, Springer-Verlag, Berlin, 196-201.
- Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2020. ViL- BERTScore: Evaluating image caption using vision-and-language BERT. In Proceedings of the 1st Workshop on Evalu- ation and Comparison of NLP Systems. Association for Computational Linguistics, Online, 34-39.
- Jaewook Lee, Yi-Hao Peng, Jaylin Herskovitz, and Anhong Guo. 2021. Image Explorer: Multi-layered touch explo- ration to make images accessible. In 23rd International ACM SIGACCESS Conference on Computers and Accessibility. Association for Computing Machinery, New York, NY, 1-4.
- Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, 74-81.
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, Processings of Machine Learning Research, online, 8748-8763.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- Elliot Salisbury, Ece Kamar, and Meredith Ringel Morris. 2018. Evaluating and complementing vision-to-language technology for people who are blind with conversational crowdsourcing. In IJCAI. AAAI Press, Stockholm, 5349-5353.
- Ece Takmaz, Sandro Pezzelle, Lisa Beinborn, and Raquel Fernández. 2020. Generating image descriptions via sequential cross-modal alignment guided by human gaze. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, online, 4664-4677.
- Yu Zhong, Walter S. Lasecki, Erin Brady, and Jeffrey P. Bigham. 2015. RegionSpeak: Quick comprehensive spatial descriptions of complex images for blind users. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, 2353-2362.
Cited by
Sorted by citations
Yoshimura, K., & Oishi, H. (2024) · International Software Product Line Conference (SPLC) · doi:10.1145/3646548.3672581 · Cited by 2 · see citing works
AniDream: Generating Skeleton-Guided Anime Avatars from Text Prompts
Yamada, F. M., & Takahashi, H. (2025) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar67309.2025.00111 · Cited by 2 · see citing works
Raees, M., Iwamoto, Y., Papangelis, K., Heard, J., & Tigwell, G. W. (2026) · arXiv · doi:10.48550/arxiv.2602.08937 · Cited by 1 · see citing works
“Overall, the need for alt text is not exclusive to BLV people; others with learning disabilities such as dyslexia also benefit from alt text to process complex information, including figures and charts [74].”
Reddy, V., & Veeranjaneyulu, N. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08560-z
Davis, A. L., Chau, C. W., & Jacobs, M. L. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720058
Mäkipää, J.-P., Mäenpää, T., Dang, D., & Pasanen, T. A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5261681
Predicting Instagram Short‐Video Popularity by Using Interaction and Visual Features
Yang, C., Chen, T., Sun, P., & Chang, C. (2026) · Expert Systems · doi:10.1111/exsy.70282
- Authors
- Aguirre, C., Cao, S., Mahmood, A., & Huang, C.-M.
- Affiliations
- Johns Hopkins University (US)
- Venue
- ACM Transactions on Interactive Intelligent Systems
- Published
- 2023-09-11
- Volume
- 13
- Issue
- 3
- Pages
- 1-28
- ISSN
- 2160-6455 (ISSN-L)
- 2160-6463
- OpenAlex
- W4361297406
- Cited by
- 6
- References
- 55
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 1
- FWCI
- 1.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Interactive Intelligent Systems (publishedVersion, cc-by)
- Funders
- Malone Center for Engineering in Healthcare
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- image captioning, speech interfaces, screen readers, personal assistants, eye gaze tracking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
