CrowdVerge: Predicting If People Will Agree on the Answer to a Visual Question
Open accessAbstract
Visual question answering systems empower users to ask any question about any image and receive a valid answer. However, existing systems do not yet account for the fact that a visual question can lead to a single answer or multiple different answers. While a crowd often agrees, disagreements do arise for many reasons including that visual questions are ambiguous, subjective, or difficult. We propose a model, CrowdVerge, for automatically predicting from a visual question whether a crowd would agree on one answer. We then propose how to exploit these predictions in a novel application to efficiently collect all valid answers to visual questions. Specifically, we solicit fewer human responses when answer agreement is expected and more human responses otherwise. Experiments on 121,811 visual questions asked by sighted and blind people show that, compared to existing crowdsourcing systems, our system captures the same answer diversity with typically 14-23% less crowd involvement.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Annotation by crowdworkers without disability-specific expertise leads to inconsistent or inaccurate labels in accessibility datasets.
“leading to inconsistent or inaccurate labels.”
From Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter (2026) · cites this paper as [35]
The Answer (Dis)Agreements section says crowdworkers can disagree when a task requires domain expertise or a worker answers inadequately. The Answers section confirms that the paper collected crowdworker answers for VizWiz questions asked by blind users. It does not establish that a lack of disability-specific expertise caused inconsistent or inaccurate labels in accessibility datasets. (LLM, conf=0.84)
Experimental — accuracy not yet measuredAccessibility datasets are often annotated by crowdworkers without disability-specific expertise.
“accessibility datasets are often annotated by crowdworkers without disability-specific expertise, leading to inconsistent or inaccurate labels.”
From Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter (2026) · cites this paper as [35]
The VizWiz section describes visual questions asked by blind users, and the Answer (Dis)Agreements section reports that crowd-worker mistakes or a need for domain expertise can cause disagreement. The paper does not say that accessibility datasets are often annotated by workers lacking disability-specific expertise, or that this lack causes inaccurate labels. (LLM, conf=0.84)
Experimental — accuracy not yet measuredVizWiz had only 9% unanimous agreement among annotators on the answer to associated visual questions.
“only 9% unanimous agreement among annotators on the answer to associated visual questions [35]”
From Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter (2026) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791 · see citing works
2017 IEEE International Conference on Computer Vision (ICCV)
(2017) · doi:10.1109/iccv37128.2017 · Cited by 3,129 · see citing works
Labeling images with a computer game
von Ahn, L., & Dabbish, L. (2004) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/985692.985733 · Cited by 2,230 · see citing works
Get another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F. J., & Ipeirotis, P. G. (2008) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1401890.1401965 · Cited by 1,180 · see citing works
VQA: Visual Question AnsweringPreprint
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00468 · Cited by 968 · see citing works
The multidimensional wisdom of crowds
Welinder, P., Branson, S., Belongie, S., & Perona, P. (2010) · CaltechAUTHORS (California Institute of Technology) · Cited by 734 · see citing works
Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images
Malinowski, M., Rohrbach, M., & Fritz, M. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.9 · Cited by 633 · see citing works
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601 · see citing works
Learning to Compose Neural Networks for Question Answering
Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1181 · Cited by 450 · see citing works
Visual challenges in the everyday lives of blind people
Brady, E. L., Morris, M. R., Zhong, Y., White, S., & Bigham, J. P. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2481291 · Cited by 236 · see citing works
SQUARE: A Benchmark for Research on Computing Crowd Consensus
Sheshadri, A., & Lease, M. (2013) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v1i1.13088 · Cited by 180 · see citing works
Scalable multi-label annotation
Deng, J., Russakovsky, O., Krause, J., Bernstein, M. S., Berg, A., & Fei-Fei, L. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557011 · Cited by 156 · see citing works
Decision-Theoretic Control of Crowd-Sourced Workflows
Dai, P., Mausam, M., & Weld, D. S. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7760 · Cited by 126 · see citing works
Hara, K., Sun, J., Moore, R. J., Jacobs, D. W., & Froehlich, J. E. (2014) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2642918.2647403 · Cited by 108 · see citing works
VizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World
Guo, A., Chen, X. '., Qi, H., White, S., Ghosh, S., Asakawa, C., & Bigham, J. P. (2016) · Proceedings of the 29th Annual Symposium on User Interface Software and Technology · doi:10.1145/2984511.2984518 · Cited by 99 · see citing works
Crowdsourcing subjective fashion advice using VizWiz
Burton, M. A., Brady, E. L., Brewer, R. N., Neylan, C., Bigham, J. P., & Hurst, A. K. (2012) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2384916.2384941 · Cited by 95 · see citing works
Zhang, J., Ma, S., Sameki, M., Sclaroff, S., Betke, M., Lin, Z., Shen, X., Price, B. L., & Měch, R. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299031 · Cited by 86 · see citing works
Laput, G., Lasecki, W. S., Wiese, J., Xiao, R., Bigham, J. P., & Harrison, C. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702416 · Cited by 81 · see citing works
Visual Madlibs: Fill in the blank Image Generation and Question AnsweringPreprint
Yu, L., Park, E., Berg, A. C., & Berg, T. L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00278 · Cited by 75 · see citing works
Combining Crowd and Expert Labels Using Decision Theoretic Active Learning
Nguyen, A., Wallace, B., & Lease, M. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13225 · Cited by 74 · see citing works
Answering visual questions with conversational crowd assistants
Lasecki, W. S., Thiha, P., Zhong, Y., Brady, E. L., & Bigham, J. P. (2013) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2513383.2517033 · Cited by 70 · see citing works
Embracing Error to Enable Rapid Crowdsourcing
Krishna, R., Hata, K., Chen, S. P., Kravitz, J., Shamma, D. A., Fei-Fei, L., & Bernstein, M. S. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858115 · Cited by 70 · see citing works
Gauging Receptiveness to Social Microvolunteering
Brady, E. L., Morris, M. R., & Bigham, J. P. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702329 · Cited by 69 · see citing works
Zhong, Y., Lasecki, W. S., Brady, E. L., & Bigham, J. P. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702437 · Cited by 68 · see citing works
Predicting Sufficient Annotation Strength for Interactive Foreground Segmentation
Jain, S. D., & Grauman, K. L. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.166 · Cited by 67 · see citing works
Self-citationCost-Sensitive Active Visual Category Learning
Vijayanarasimhan, S., & Grauman, K. L. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-010-0372-4 · Cited by 54 · see citing works
Self-citationMultiview Triplet Embedding: Learning Attributes in Multiple Maps
Amid, E., & Ukkonen, A. (2015) · International Conference on Machine Learning · Cited by 48 · see citing works
To Re(label), or Not To Re(label)
Lin, C. H., Mausam, & Weld, D. (2014) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v2i1.13167 · Cited by 45 · see citing works
Tropel: Crowdsourcing Detectors with Minimal Training
Patterson, G., Horn, G. V., Belongie, S., Perona, P., & Hays, J. H. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13224 · Cited by 26 · see citing works
Pull the Plug? Predicting If Computers or Humans Should Segment Images
Gurari, D., Jain, S. D., Betke, M., & Grauman, K. L. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.48 · Cited by 24 · see citing works
Self-citationSarkar, A., Morrison, C. P., Dorn, J. F., Bedi, R., Steinheimer, S., Boisvert, J. L., Burggraaff, J., D’Souza, M., Kontschieder, P., Bulò, S. R., et al. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858199 · Cited by 18 · see citing works
Jas, M., & Parikh, D. (2015) · Cited by 17 · see citing works
Jas, M., & Parikh, D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298889 · Cited by 15 · see citing works
Flagged first, then by citations
1 of the 37 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- K. Hara, J. Sun, R. Moore, D. Jacobs, and J. Froehlich. 2014. Tohme: Detecting Curb Ramps in Google Street View Using Crowdsourcing, Computer Vision, and Machine Learning. In ACM symposium on User interface software and technology (UIST). 189-204.
- G. Laput, W. S. Lasecki, J. Wiese, R. Xiao, J. P. Bigham, and C. Harrison. 2015. Zensors: Adaptive, Rapidly Deployable, Human-Intelligent Sensor Feeds. In ACM Conference on Human Factors in Computing Systems (CHI). 1935-1944.
- M. Malinowski and M. Fritz. 2014. A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input. In Advances in Neural Information Processing Systems (NIPS). 1682-1690.
- Y. Zhong, W. S. Lasecki, E. Brady, and J. P. Bigham. 2015. Regionspeak: Quick comprehensive spatial descriptions of complex images for blind users. In ACM Conference on Human Factors in Computing Systems (CHI). 2353-2362.
Cited by
Sorted by citations
Empowerment in HCI - A Survey and Framework
Schneider, H., Eiband, M., Ullrich, D., & Butz, A. M. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173818 · Cited by 128 · see citing works
A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning
Kaluarachchi, T. I., Reis, A., & Nanayakkara, S. C. (2021) · Sensors · doi:10.3390/s21072514 · Cited by 132 · see citing works
Gurari, D., Li, Q., Lin, C., Zhao, Y., Guo, A., Stangl, A. J., & Bigham, J. P. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00103 · Cited by 103 · see citing works
Self-citationIshmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 100 · see citing works
Information Visualization Evaluation Using Crowdsourcing
Borgo, R., Micallef, L., Bach, B., McGee, F., & Lee, B. (2018) · Computer Graphics Forum · doi:10.1111/cgf.13444 · Cited by 82 · see citing works
Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions
Aroyo, L. M., Dixon, L., Thain, N., Redfield, O., & Rosen, R. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3317083 · Cited by 64 · see citing works
Assessing Image Quality Issues for Real-World Problems
Chiu, T.-Y., Zhao, Y., & Gurari, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00370 · Cited by 63 · see citing works
Self-citationWhy Does a Visual Question Have Different Answers?
Bhattacharya, N., Li, Q., & Gurari, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00437 · Cited by 59 · see citing works
Self-citationResolvable vs. Irresolvable Disagreement
Schaekermann, M., Goh, J., Larson, K. S., & Law, E. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274423 · Cited by 54 · see citing works
Ambiguity-aware AI Assistants for Medical Data Analysis
Schaekermann, M., Beaton, G., Sanoubari, E., Lim, A. S. P., Larson, K. S., & Law, E. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376506 · Cited by 54 · see citing works
Understanding Expert Disagreement in Medical Data Analysis through Structured Adjudication
Schaekermann, M., Beaton, G., Habib, M., Lim, A. S. P., Larson, K. S., & Law, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359178 · Cited by 52 · see citing works
Efficient Elicitation Approaches to Estimate Collective Crowd Answers
Chung, J. J. Y., Song, J. Y., Kutty, S., Hong, S.-S., Kim, J., & Lasecki, W. S. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359164 · Cited by 46 · see citing works
Misfitting With AI: How Blind People Verify and Contest AI Errors
Alharbi, R., Lor, P., Herskovitz, J., Schoenebeck, S., & Brewer, R. N. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675659 · Cited by 40 · see citing works
Privacy Concerns for Visual Assistance Technologies
Stangl, A., Shiroma, K., Davis, N., Xie, B., Fleischmann, K. R., Findlater, L., & Gurari, D. (2022) · ACM Transactions on Accessible Computing · doi:10.1145/3517384 · Cited by 35 · see citing works
Self-citationCrowd-AI Camera Sensing in the Real World
Guo, A., Jain, A., Ghose, S., Laput, G., Harrison, C., & Bigham, J. P. (2018) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3264921 · Cited by 30 · see citing works
Simons, R. N., Gurari, D., & Fleischmann, K. R. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415176 · Cited by 27 · see citing works
Self-citationJiang, L., Jung, C., Phutane, M., Stangl, A. J., & Azenkot, S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642238 · Cited by 26 · see citing works
How Accurate Does It Feel? – Human Perception of Different Types of Classification Mistakes
Papenmeier, A., Kern, D., Hienert, D., Kammerer, Y., & Seifert, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501915 · Cited by 25 · see citing works
Leveraging Peer Communication to Enhance Crowdsourcing
Tang, W., Yin, M., & Ho, C.-J. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313554 · Cited by 21 · see citing works
Vision Skills Needed to Answer Visual Questions
Zeng, X., Wang, Y., Chiu, T.-Y., Bhattacharya, N., & Gurari, D. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415220 · Cited by 21 · see citing works
Self-citationCho, J. W., Kim, D.-J., Choi, J., Jung, Y., & Kweon, I. S. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00175 · Cited by 20 · see citing works
Visual Question Answer Diversity
Yang, C.-J., Grauman, K. L., & Gurari, D. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13341 · Cited by 20 · see citing works
Self-citationVisual Content Considered Private by People Who are Blind
Stangl, A. J., Shiroma, K., Xie, B., Fleischmann, K. R., & Gurari, D. (2020) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3373625.3417014 · Cited by 20 · see citing works
Self-citationFantastic Answers and Where to Find Them: Immersive Question-Directed Visual Attention
Jiang, M. X., Chen, S., Yang, J., & Zhao, Q. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00305 · Cited by 16 · see citing works
Stangl, A., Sadjo, E., Emami-Naeini, P., Wang, Y., Gurari, D., & Findlater, L. (2023) · 20th International Web for All Conference · doi:10.1145/3587281.3587296 · Cited by 15 · see citing works
Self-citationGoldilocks: Consistent Crowdsourced Scalar Annotations with Relative Uncertainty
Chen, Q. Z., Weld, D. S., & Zhang, A. X. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476076 · Cited by 11 · see citing works
VQA Therapy: Exploring Answer Differences by Visually Grounding Answers
Chen, C., Anjum, S., & Gurari, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01405 · Cited by 10 · see citing works
Self-citationKaspar, A., Patterson, G., Kim, C., Aksoy, Y., Matusik, W., & Elgharib, M. A. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173685 · Cited by 9 · see citing works
Luo, M., Sampat, S. K., Tallman, R., Zeng, Y., Vancha, M., Sajja, A., & Baral, C. R. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.240 · Cited by 13 · see citing works
Lemmer, S. J., Guo, A., & Corso, J. J. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584092 · Cited by 6 · see citing works
TA-Student VQA: Multi-Agents Training by Self-Questioning
Xiong, P., & Wu, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01008 · Cited by 6 · see citing works
Tang, X., Li, J., & Wu, S. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790405 · Cited by 4 · see citing works
“Prior work reported over 40% disagreement between crowdworkers and in-house experts on whether a photo contained privacy or quality issues [36], and only 9% unanimous agreement among annotators on the answer to associated visual questions [35].”
Schaekermann, M., Law, E., Larson, K. S., & Lim, A. S. P. (2018) · Cited by 4 · see citing works
Garg, K., Tang, X., Heo, J., Morgan, D. R., Gergle, D., Sudderth, E. B., & Piper, A. M. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791309 · Cited by 3 · see citing works
“Image quality has been identified as a major challenge in both model development [56] and user interactions [152], leading to issues with annotation [17, 54, 113, 135] and poor model performance [152].”
Fully Authentic Visual Question Answering Dataset from Online Communities
Chen, C., Liu, M., Codella, N., Li, Y., Yuan, L., & Gurari, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73195-2_15 · Cited by 3 · see citing works
Self-citationAn Entropy Clustering Approach for Assessing Visual Question Difficulty
Terao, K., Tamaki, T., Raytchev, B., Kaneda, K., & Satoh, S. (2020) · IEEE Access · doi:10.1109/access.2020.3022063 · Cited by 3 · see citing works
EaSe: A Diagnostic Tool for VQA based on Answer Diversity
Jolly, S., Pezzelle, S., & Nabi, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.192 · Cited by 3 · see citing works
BUOCA: Budget-Optimized Crowd Worker AllocationPreprint
Sameki, M., Lai, S., Mays, K. K., Гуо, Л., Ishwar, P., & Betke, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06237 · Cited by 3 · see citing works
Affective question answering on video
Ruwa, N., Mao, Q., Wang, L., & Gou, J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.06.046 · Cited by 2 · see citing works
Venire: A Machine Learning-Guided Panel Review System for Community Content Moderation
Koshy, V., Choi, F., Chiang, Y.-S., Sundaram, H., Chandrasekharan, E., & Karahalios, K. G. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757699 · Cited by 3 · see citing works
Acknowledging Focus Ambiguity in Visual Questions
Chen, C., Tseng, Y., Li, Z., Venkatesh, A., & Gurari, D. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00122 · Cited by 3 · see citing works
Self-citationThe Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQAPreprint
Jolly, S., Pezzelle, S., Klein, T., Dengel, A. R., & Nabi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.04344 · Cited by 1 · see citing works
Luo, M., Sampat, S. K., Tallman, R., Zeng, Y., Vancha, M., Sajja, A., & Baral, C. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.15022 · Cited by 1 · see citing works
Explaining Disagreement in Visual Question Answering Using Eye Tracking
Hindennach, S., Shi, L., & Bulling, A. (2024) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3649902.3656356
Analyzing Difficulties of Visual Questions Using Entropy Values of Answer Predictions
Yadav, M. G., Nennuri, R., Deepthi, N. L., Harshitha, K. S., & Vinay, A. N. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-8666-4_40
Disambiguating Ambiguous Questions using Eye-Gaze in Visual Question Answering
Inadumi, S., Kawano, S., Yuguchi, A., Kawanishi, Y., & Yoshino, K. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.3
Law, E., & Yin, M. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.008
Doing Human-Centered Machine Learning
Fiebrink, R., Gillies, M. F. P., & Ramos, G. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.006
- Authors
- Gurari, D., & Grauman, K.
- Affiliations
- The University of Texas at Austin (US)
- Venue
- Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2017-05-02
- Pages
- 3511-3522
- OpenAlex
- W2610525153
- Cited by
- 45
- References
- 37
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 6 · 2023: 3 · 2022: 2 · 2021: 6 · 2020: 8 · 2019: 8 · 2018: 8
- FWCI
- 6.05
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- Office of Naval Research (YIP N00014-12-1-0754)
- National Science Foundation (IIS-1065390)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- visual question answering, crowdsourcing, blind users, visual ambiguity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
