Caption Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image Search
The DOI matches its Crossref record; 35 of the 37 references it cites were found in a registry or an index.
Abstract
Accessing images online is often difficult for users with vision impairments. This population relies on text descriptions of images that vary based on website authors' accessibility practices. Where one author might provide a descriptive caption for an image, another might provide no caption for the same image, leading to inconsistent experiences. In this work, we present the Caption Crawler system, which uses reverse image search to find existing captions on the web and make them accessible to a user's screen reader. We report our system's performance on a set of 481 websites from alexa.com's list of most popular sites to estimate caption coverage and latency, and also report blind and sighted users' ratings of our system's output quality. Finally, we conducted a user study with fourteen screen reader users to examine how the system might be used for personal browsing.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Common issues include inaccessible table formats [36], inadequate image descriptions that lack important visual context [19], insufficient navigation aids [29], and confusing or unclear explanations [13,16].”
From From Cluttered to Clear: Improving the Web Accessibility Design for Screen Reader Users in E-commerce With Generative AI (2025) · cites this paper as [19]
Citations
Cites
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Earcons and Icons: Their Structure and Common Design Principles
Blattner, M., Sumikawa, D., & Greenberg, R. (1989) · Human-Computer Interaction · doi:10.1207/s15327051hci0401_1 · Cited by 980
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601 · cited 2×
“Crowdsourcing captioning methods rely on human annotators in order to help describe a photo [4, 8, 9, 27, 34].”
MethodWeb Content Accessibility Guidelines 2.0: A further step towards accessible digital information
Ribera, M., Porras, M., Boldu, M., Termens, M., Sule, A., & Paris, P. (2009) · Program · doi:10.1108/00330330910998048 · Cited by 595
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Tatarowicz, A., White, B., White, S., & Yeh, T. (2010) · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806020 · Cited by 536
Predictors of answer quality in online Q&A sites
Harper, F. M., Raban, D., Rafaeli, S., & Konstan, J. A. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357191 · Cited by 412
Guidelines are only half of the story: accessibility problems encountered by blind users on the web
Power, C., Freire, A., Petrie, H., & Swallow, D. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207736 · Cited by 327 · cited 1×
“…studies of consumer and government websites repeatedly find high levels of non-compliance in providing alt text descriptions for online images, with high proportions of descriptions that are missing completely or of very poor quality ( e.g. , filenames, the word “image”) [ 14, 18, 19, 23, 25].”
Background(2017) · doi:10.1145/2998181.2998364 · Cited by 258
Understanding Blind People's Experiences with Computer-Generated Captions of Social Media Images
MacLeod, H., Bennett, C. L., Morris, M. R., & Cutrell, E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025814 · Cited by 200 · cited 2×
“However, studies of the output of vision-to-language systems indicate that they are still too error prone to be reliable for people who are VI [20, 29].”
BackgroundResultSelf-citationHow Blind People Interact with Visual Content on Social Networking Services
Voykinska, V., Azenkot, S., Wu, S., & Leshed, G. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2820013 · Cited by 174
WebInSight: making web images accessible
Bigham, J. P., Kaminsky, R. S., Ladner, R. E., Danielsson, O. M., & Hempton, G. L. (2006) · Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1168987.1169018 · Cited by 161 · cited 4×
“Hybrid approaches often rely on a combination of automatically generated descriptions or tags with human editing [6, 28, 29] to reduce time and financial costs associated with recruiting human crowd workers.”
BackgroundInfluentialImproving accessibility of the web with a computer game
von Ahn, L., Ginosar, S., Kedia, M., Liu, R., & Blum, M. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124785 · Cited by 156
WebinSitu: a comparative analysis of blind and sighted browsing behavior
Bigham, J. P., Cavender, A. C., Brudvik, J. T., Wobbrock, J. O., & Ladner, R. E. (2007) · Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1296843.1296854 · Cited by 153
Olalere, A., & Lazar, J. (2011) · Government Information Quarterly · doi:10.1016/j.giq.2011.02.002 · Cited by 148 · cited 1×
“…studies of consumer and government websites repeatedly find high levels of non-compliance in providing alt text descriptions for online images, with high proportions of descriptions that are missing completely or of very poor quality ( e.g. , filenames, the word “image”) [ 14, 18, 19, 23, 25].”
BackgroundInvestigating the appropriateness of social network question asking as a resource for blind users
Brady, E. L., Zhong, Y., Morris, M. R., & Bigham, J. P. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441915 · Cited by 147
Self-citationVizWiz::LocateIt - enabling blind people to locate objects in their environment
Bigham, J. P., Jayant, C., Miller, A., White, B., & Yeh, T. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543821 · Cited by 135
Morris, M. R., Zolyomi, A., Yao, C., Bahram, S., Bigham, J. P., & Kane, S. K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858116 · Cited by 131
Self-citationSocial accessibility: achieving accessibility through collaborative metadata authoring
Takagi, H., Kawanaka, S., Kobayashi, M., Itoh, T., & Asakawa, C. (2008) · Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1414471.1414507 · Cited by 106
Rich Image Captioning in the Wild
Tran, K., He, X., Zhang, L., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.61 · Cited by 103
Salisbury, E., Kamar, E., & Morris, M. (2017) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v5i1.13301 · Cited by 83
Self-citationTeevan, J. B., Morris, M. R., & Panovich, K. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14159 · Cited by 82
Self-citationGlobal Web Accessibility Analysis of National Government Portals and Ministry Web Sites
Goodwin, M., Susar, D., Nietzio, A., Snaprud, M., & Jensen, C. S. (2011) · Journal of Information Technology & Politics · doi:10.1080/19331681.2010.508011 · Cited by 74 · cited 1×
“…studies of consumer and government websites repeatedly find high levels of non-compliance in providing alt text descriptions for online images, with high proportions of descriptions that are missing completely or of very poor quality ( e.g. , filenames, the word “image”) [ 14, 18, 19, 23, 25].”
BackgroundGauging Receptiveness to Social Microvolunteering
Brady, E., Morris, M. R., & Bigham, J. P. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702329 · Cited by 68
Self-citationThe state of corporate website accessibility
Loiacono, E. T., Romano, N. C., & McCoy, S. (2009) · Communications of the ACM · doi:10.1145/1562164.1562197 · Cited by 65
Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility
Zhang, X., Ross, A. S., Caspi, A., Fogarty, J., & Wobbrock, J. O. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025846 · Cited by 59 · cited 1×
“Zhang et al. presented a proxy-based method for runtime repair of Android applications [39].”
Method(2007) · Proceedings of the Third International Conference on Web Information Systems and Technologies · doi:10.5220/0001274600590066 · Cited by 34
Marie-4: a high-recall, self-improving Web crawler that finds images using captions
Rowe, N. C. (2002) · IEEE Intelligent Systems · doi:10.1109/mis.2002.1024745 · Cited by 31
Increasing web accessibility by automatically judging alternative text quality
Bigham, J. P. (2007) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1216295.1216364 · Cited by 31
AutoCaption: Automatic caption generation for personal photos
Ramnath, K., Baker, S., Vanderwende, L., El-Saban, M., Sinha, S. N., Kannan, A., Hassan, N., Galley, M., Yang, Y., Ramanan, D., et al. (2014) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2014.6835988 · Cited by 27
Diaper, D., & Worman, L. (2004) · Springer eBooks · doi:10.1007/978-1-4471-3754-2_22 · Cited by 26
Vázquez, S. R. (2016) · International Web for All Conference (W4A) · doi:10.1145/2899475.2899484 · Cited by 12
Improving image tags by exploiting web search results
Zhang, X., Li, Z., & Chao, W. (2011) · Multimedia Tools and Applications · doi:10.1007/s11042-011-0863-5 · Cited by 9
Improving accessibility of html documents bygenerating image-tags in a proxy
Keysers, D., Renn, M., & Breuel, T. M. (2007) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1296843.1296896 · Cited by 7
ABA resolution and report on website accessibility.
(2007) · PubMed · Cited by 3
On usage models of content-based image search, filtering, and annotation
Telleen-Lawton, D., Chang, E. Y., Cheng, K.-T. T., & Chang, C. (2006) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.655792 · Cited by 1
Research works printed but not linked: 1
- Patil Swati, P., Pawar, B. V.,&Patil Ajay, S. (2013). Search Engine Optimization: A Study. Research Journal of Computer&Information Technology Sciences, 1(1), 10--13.Patil Swati, P., Pawar, B.V. and Patil Ajay, S., 2013. Search Engine Optimization: A Study. Research Journal of Computer&Information Technology Sciences, 1(1), pp.10--13.
Cited by
Captioning Images Taken by People Who Are Blind
Gurari, D., Zhao, Y., Zhang, M., & Bhattacharya, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58520-4_25 · Cited by 175
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Birhane, A., Prabhu, V. U., & Kahembwe, E. (2021) · arXiv · doi:10.48550/arxiv.2110.01963 · Cited by 160
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Zhang, X., de Greef, L., Swearngin, A., White, S., Murray, K., Yu, L., Shan, Q., Nichols, J., Wu, J., Fleizach, C., et al. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445186 · Cited by 144
Stangl, A., Morris, M. R., & Gurari, D. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376404 · Cited by 139
Self-citationBennett, C. L., Gleason, C., Scheuerman, M. K., Bigham, J. P., Guo, A., & To, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445498 · Cited by 128
Metadata
- Authors
- Guinness, D., Cutrell, E., & Morris, M. R.
- Affiliations
- University of Colorado Boulder (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 1-11
- OpenAlex
- W2791213089
- Cited by
- 108
- References
- 37
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 15 · 2023: 13 · 2022: 17 · 2021: 24 · 2020: 13 · 2019: 8 · 2018: 2
- FWCI
- 20.25
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- image captioning, reverse image search, web accessibility, visual impairment, screen readers, blind users, user study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.