Caption Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image Search
The DOI matches its Crossref record; 35 of the 37 references it cites were found in a registry or an index.
Abstract
Accessing images online is often difficult for users with vision impairments. This population relies on text descriptions of images that vary based on website authors' accessibility practices. Where one author might provide a descriptive caption for an image, another might provide no caption for the same image, leading to inconsistent experiences. In this work, we present the Caption Crawler system, which uses reverse image search to find existing captions on the web and make them accessible to a user's screen reader. We report our system's performance on a set of 481 websites from alexa.com's list of most popular sites to estimate caption coverage and latency, and also report blind and sighted users' ratings of our system's output quality. Finally, we conducted a user study with fourteen screen reader users to examine how the system might be used for personal browsing.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Common issues include inaccessible table formats [36], inadequate image descriptions that lack important visual context [19], insufficient navigation aids [29], and confusing or unclear explanations [13,16].”
From From Cluttered to Clear: Improving the Web Accessibility Design for Screen Reader Users in E-commerce With Generative AI (2025) · cites this paper as [19]
Citations
Cites
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Earcons and Icons: Their Structure and Common Design Principles
Blattner, M., Sumikawa, D., & Greenberg, R. (1989) · Human-Computer Interaction · doi:10.1207/s15327051hci0401_1 · Cited by 980
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601 · cited 2×
“Crowdsourcing captioning methods rely on human annotators in order to help describe a photo [4, 8, 9, 27, 34].”
MethodWeb Content Accessibility Guidelines 2.0: A further step towards accessible digital information
Ribera, M., Porras, M., Boldu, M., Termens, M., Sule, A., & Paris, P. (2009) · Program · doi:10.1108/00330330910998048 · Cited by 595
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Tatarowicz, A., White, B., White, S., & Yeh, T. (2010) · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806020 · Cited by 536
Predictors of answer quality in online Q&A sites
Harper, F. M., Raban, D., Rafaeli, S., & Konstan, J. A. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357191 · Cited by 412
Guidelines are only half of the story: accessibility problems encountered by blind users on the web
Power, C., Freire, A., Petrie, H., & Swallow, D. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207736 · Cited by 327 · cited 1×
“…studies of consumer and government websites repeatedly find high levels of non-compliance in providing alt text descriptions for online images, with high proportions of descriptions that are missing completely or of very poor quality ( e.g. , filenames, the word “image”) [ 14, 18, 19, 23, 25].”
Background(2017) · doi:10.1145/2998181.2998364 · Cited by 258
Understanding Blind People's Experiences with Computer-Generated Captions of Social Media Images
MacLeod, H., Bennett, C. L., Morris, M. R., & Cutrell, E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025814 · Cited by 200 · cited 2×
“However, studies of the output of vision-to-language systems indicate that they are still too error prone to be reliable for people who are VI [20, 29].”
BackgroundResultSelf-citationHow Blind People Interact with Visual Content on Social Networking Services
Voykinska, V., Azenkot, S., Wu, S., & Leshed, G. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2820013 · Cited by 174
Research works printed but not linked: 1
- Patil Swati, P., Pawar, B. V.,&Patil Ajay, S. (2013). Search Engine Optimization: A Study. Research Journal of Computer&Information Technology Sciences, 1(1), 10--13.Patil Swati, P., Pawar, B.V. and Patil Ajay, S., 2013. Search Engine Optimization: A Study. Research Journal of Computer&Information Technology Sciences, 1(1), pp.10--13.
Cited by
Captioning Images Taken by People Who Are Blind
Gurari, D., Zhao, Y., Zhang, M., & Bhattacharya, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58520-4_25 · Cited by 175
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Birhane, A., Prabhu, V. U., & Kahembwe, E. (2021) · arXiv · doi:10.48550/arxiv.2110.01963 · Cited by 160
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Zhang, X., de Greef, L., Swearngin, A., White, S., Murray, K., Yu, L., Shan, Q., Nichols, J., Wu, J., Fleizach, C., et al. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445186 · Cited by 144
Stangl, A., Morris, M. R., & Gurari, D. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376404 · Cited by 139
Self-citationBennett, C. L., Gleason, C., Scheuerman, M. K., Bigham, J. P., Guo, A., & To, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445498 · Cited by 128
Metadata
- Authors
- Guinness, D., Cutrell, E., & Morris, M. R.
- Affiliations
- University of Colorado Boulder (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 1-11
- OpenAlex
- W2791213089
- Cited by
- 108
- References
- 37
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 15 · 2023: 13 · 2022: 17 · 2021: 24 · 2020: 13 · 2019: 8 · 2018: 2
- FWCI
- 20.25
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- image captioning, reverse image search, web accessibility, visual impairment, screen readers, blind users, user study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Caption Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image Search. https://app.cochat.ai/papers/10.1145/3173574.3174092
- The paper
- Guinness, D., Cutrell, E., & Morris, M. R. (2018). Caption Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image Search. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3173574.3174092
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.