Toward Scalable Social Alt Text: Conversational Crowdsourcing as a Tool for Refining Vision-to-Language Technology for the Blind
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
The access of visually impaired users to imagery in social media is constrained by the availability of suitable alt text. It is unknown how imperfections in emerging tools for automatic caption generation may help or hinder blind users' understanding of social media posts with embedded imagery. In this paper, we study how crowdsourcing can be used both for evaluating the value provided by existing automated approaches and for enabling workflows that provide scalable and useful alt text to blind users. Using real-time crowdsourcing, we designed experiences that varied the depth of interaction of the crowd in assisting visually impaired users at caption interpretation, and measured trade-offs in effectiveness, scalability, and reusability. We show that the shortcomings of existing AI image captioning systems frequently hinder a user's understanding of an image they cannot see to a degree that even clarifying conversations with sighted assistants cannot correct. Our detailed analysis of the set of clarifying conversations collected from our studies led to the design of experiences that can effectively assist users in a scalable way without the need for real-time interaction. They also provide lessons and guidelines that human captioners and the designers of future iterations of AI captioning systems can use to improve labeling of social media imagery for blind users.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Understanding Blind People's Experiences with Computer-Generated Captions of Social Media Images
MacLeod, H., Bennett, C. L., Morris, M. R., & Cutrell, E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025814 · Cited by 200 · cited 9×
“These findings are in line with recent findings that BVI users place too much trust in AI captions (MacLeod et al. 2017), and our results suggest that this trust in the initial caption may mislead people to the extent that they are unable to properly frame questions to improve their understanding.”
BackgroundMethodResultInfluentialSelf-citationGlobal Web Accessibility Analysis of National Government Portals and Ministry Web Sites
Goodwin, M., Susar, D., Nietzio, A., Snaprud, M., & Jensen, C. S. (2011) · Journal of Information Technology & Politics · doi:10.1080/19331681.2010.508011 · Cited by 74 · cited 2×
“) that provides computer access to people who are BVI (Kloots 2016); however, while no official numbers on alt text compliance for Twitter are yet available, alt text compliance and quality on the web in general is low (Bigham et al. 2006; Goodwin et al. 2011; Shi 2006), and this trend is likely to be exacerbated by q…”
Background
Cited by
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364
Self-citationRich Representations of Visual Content for Screen Reader Users
Morris, M. R., Johnson, J., Bennett, C. L., & Cutrell, E. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173633 · Cited by 159
“Further, automated captions do not yet include the amount and types of detail desired by many screen reader users [27].”
BackgroundInfluentialSelf-citationStangl, A., Morris, M. R., & Gurari, D. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376404 · Cited by 139
Self-citationBennett, C. L., Gleason, C., Scheuerman, M. K., Bigham, J. P., Guo, A., & To, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445498 · Cited by 128
“Researchers have studied several solutions to increase the number of described images from crowdsourcing [93] to generating them automatically [40, 123].”
BackgroundMorris, M. R. (2020) · Communications of the ACM · doi:10.1145/3356727 · Cited by 114
Self-citation
Metadata
- Authors
- Salisbury, E., Kamar, E., & Morris, M.
- Venue
- Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
- Published
- 2017-09-21
- Volume
- 5
- Pages
- 147-156
- ISSN
- 2769-1330 (ISSN-L)
- 2769-1349
- OpenAlex
- W2748163013
- Cited by
- 83
- References
- 4
- FWCI
- 10.10
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Toward Scalable Social Alt Text: Conversational Crowdsourcing as a Tool for Refining Vision-to-Language Technology for the Blind. https://app.cochat.ai/papers/10.1609/hcomp.v5i1.13301
- The paper
- Salisbury, E., Kamar, E., & Morris, M. (2017). Toward Scalable Social Alt Text: Conversational Crowdsourcing as a Tool for Refining Vision-to-Language Technology for the Blind. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. https://doi.org/10.1609/hcomp.v5i1.13301
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.