The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
“Reliance and automation for Human-AI collaborative data labeling conflict resolution,”
Abstract
Human data labeling with multiple labelers and the resulting conflict resolution remains the norm for many enterprise machine learning pipelines. Conflict resolution can be a time-intensive and costly process. Our goal was to study how human-AI collaboration can improve conflict resolution, by enabling users to automate groups of conflict resolution tasks. However, little is known about whether and how people will rely on automation during conflict resolution. Currently, automation commonly uses labelers' majority vote labels for conflict resolution, as the top chosen label by most labelers is often correct. We envisioned a system where an AI would assist in finding cases where the labeler majority vote was wrong and where automation is supported for batches or groups of conflicts. In order to understand whether humans could use labeler and AI information effectively, we investigated how and when users rely on labeler and AI information and on automated group conflict resolution. We ran a study with 144 Mechanical Turk workers. We found that automation increased users' accuracy/time, use of automated conflict resolution was relatively similar regardless of whether the automation was based on labeler or AI selected labels, and providing labeler and AI selected labels may reduce inappropriate reliance on automation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Third, interactions that involved a third agent (Wu et al., 2022;Brachman et al., 2022;Banas et al., 2022).”
From Human-AI collaboration is not very collaborative yet: a taxonomy of interaction patterns in AI-assisted decision making from a systematic review (2025) · cites this paper as Brachman et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Swift Trust and Temporary Groups
Meyerson, D., Weick, K. E., & Kramer, R. M. (1996) · Trust in Organizations: Frontiers of Theory and Research · doi:10.4135/9781452243610.n9 · Cited by 1,982 · cited 1×
“We included the group resolve modal (Figure 2), that allows users to check the labeler choice and/or AI predicted label for all items in a group, based on literature on swift trust and the benefits of being able to verify the results of trust [33, 52].”
BackgroundCer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · arXiv · doi:10.48550/arxiv.1803.11175 · Cited by 1,267
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622 · cited 1×
“Our work fits broadly in to the context of human-AI interaction [2, 70, 74], from which we drew inspiration.”
BackgroundHow Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 4×
“Other work has found that errors can erode trust, especially trust formed quickly [47, 75].”
BackgroundInfluentialBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441
Research works printed but not linked: 59
- 10.1145/1978942.1978966 · DOI
- 10.1145/3290605.3300233 · DOI · Paper page
- Zahra Ashktorab , Michael Desmond , Josh Andres , Michael Muller , Narendra Nath Joshi , Michelle Brachman, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Christine T Wolf, et al. 2021 . AI-Assisted Human Labeling: Batching for Efficiency without Overreliance. Proceedings of the ACM on Human-Computer Interaction, Vol. 5 , CSCW 1 (2021), 1--27. Zahra Ashktorab, Michael Desmond, Josh Andres, Michael Muller, Narendra Nath Joshi, Michelle Brachman, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Christine T Wolf, et al. 2021. AI-Assisted Human Labeling: Batching for Efficiency without Overreliance. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW1 (2021), 1--27.
- 10.1145/3415167 · DOI
- Gagan Bansal , Besmira Nushi , Ece Kamar , Eric Horvitz , and Daniel S Weld . 2020a. Optimizing AI for Teamwork. arXiv preprint arXiv:2004.13102 ( 2020 ). Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. 2020a. Optimizing AI for Teamwork. arXiv preprint arXiv:2004.13102 (2020). · arXiv
- Gagan Bansal , Besmira Nushi , Ece Kamar , Eric Horvitz , and Daniel S Weld . 2021 . Is the Most Accurate AI the Best Teammate ? Optimizing AI for Teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35 . 11405--11414. Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. 2021. Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 11405--11414.
- Matthew Barnfield . 2020. Think twice before jumping on the bandwagon: Clarifying concepts in research on the bandwagon effect. Political studies review , Vol. 18 , 4 ( 2020 ), 553--574. Matthew Barnfield. 2020. Think twice before jumping on the bandwagon: Clarifying concepts in research on the bandwagon effect. Political studies review, Vol. 18, 4 (2020), 553--574.
- Jürgen Bernard , Marco Hutter , Matthias Zeppelzauer , Dieter Fellner , and Michael Sedlmair . 2017. Comparing visual-interactive labeling with active learning: An experimental study . IEEE transactions on visualization and computer graphics, Vol. 24 , 1 ( 2017 ), 298--308. Jürgen Bernard, Marco Hutter, Matthias Zeppelzauer, Dieter Fellner, and Michael Sedlmair. 2017. Comparing visual-interactive labeling with active learning: An experimental study. IEEE transactions on visualization and computer graphics, Vol. 24, 1 (2017), 298--308.
- 10.1145/3366423.3380250 · DOI
- Zana Bucc inca , Maja Barbara Malaya, and Krzysztof Z Gajos . 2021 . To Trust or to Think : Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making . arXiv preprint arXiv:2102.09692 (2021). Zana Bucc inca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. arXiv preprint arXiv:2102.09692 (2021). · arXiv
- 10.3115/1699510.1699548 · DOI · Paper page
- 10.1145/3025453.3026044 · DOI
- 10.1145/3290605.3300460 · DOI
- 10.1145/3359164 · DOI
- 10.1002/meet.14504701328 · DOI · Paper page
- 10.1609/hcomp.v4i1.13270 · DOI
- 10.1109/PACIFICVIS.2017.8031598 · DOI
- 10.1609/hcomp.v6i1.13330 · DOI
- 10.1177/1071181319631264 · DOI
- 10.1080/09298215.2016.1200631 · doi:10.1080/09298215.2016.1200631
- Bhavya Ghai , Q Vera Liao , Yunfeng Zhang , Rachel Bellamy , and Klaus Mueller . 2021 . Explainable Active Learning (XAL) Toward AI Explanations as Interfaces for Machine Teachers . Proceedings of the ACM on Human-Computer Interaction , Vol. 4 , CSCW3 (2021), 1--28. Bhavya Ghai, Q Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2021. Explainable Active Learning (XAL) Toward AI Explanations as Interfaces for Machine Teachers. Proceedings of the ACM on Human-Computer Interaction, Vol. 4, CSCW3 (2021), 1--28.
- 10.1145/1378773.1378804 · DOI
- 10.1109/mis.2013.24 · DOI · Paper page
- 10.1145/2856767.2856811 · DOI
- 10.5555/1564131.1564137 · doi:10.5555/1564131.1564137
- Yan Huang and S Shyam Sundar . 2020. Do We Trust the Crowd? Effects of Crowdsourcing on Perceived Credibility of Online Health Information. Health Communication ( 2020 ), 1--10. Yan Huang and S Shyam Sundar. 2020. Do We Trust the Crowd? Effects of Crowdsourcing on Perceived Credibility of Online Health Information. Health Communication (2020), 1--10.
- Maia Jacobs , Melanie F Pradier , Thomas H McCoy , Roy H Perlis , Finale Doshi-Velez , and Krzysztof Z Gajos . 2021. How machine-learning recommendations influence clinician treatment selections: the example of the antidepressant selection. Translational psychiatry , Vol. 11 , 1 ( 2021 ), 1--9. Maia Jacobs, Melanie F Pradier, Thomas H McCoy, Roy H Perlis, Finale Doshi-Velez, and Krzysztof Z Gajos. 2021. How machine-learning recommendations influence clinician treatment selections: the example of the antidepressant selection. Translational psychiatry, Vol. 11, 1 (2021), 1--9.
- 10.1145/2818048.2820016 · DOI
- Jan-Christoph Klie , Michael Bugert , Beto Boullosa , Richard Eckart de Castilho , and Iryna Gurevych . 2018 . The inception platform: Machine-assisted and knowledge-oriented interactive annotation . In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations. 5--9. Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The inception platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations. 5--9.
- 10.1207/s15506878jobem4903_3 · DOI
- 10.31234/osf.io/nfc45 · DOI · Paper page
- 10.1145/2556288.2557238 · DOI
- Cheng-Yu Lee and Von-Wun Soo . 2006. The conflict detection and resolution in knowledge merging for image annotation. Information processing & management , Vol. 42 , 4 ( 2006 ), 1030--1055. Cheng-Yu Lee and Von-Wun Soo. 2006. The conflict detection and resolution in knowledge merging for image annotation. Information processing & management, Vol. 42, 4 (2006), 1030--1055.
- 10.1518/hfes.46.1.50.30392 · doi:10.1518/hfes.46.1.50.30392 · Paper page
- Poornima Madhavan , Douglas A Wiegmann , and Frank C Lacson . 2006. Automation failures on tasks easily performed by operators undermine trust in automated aids. Human factors , Vol. 48 , 2 ( 2006 ), 241--256. Poornima Madhavan, Douglas A Wiegmann, and Frank C Lacson. 2006. Automation failures on tasks easily performed by operators undermine trust in automated aids. Human factors, Vol. 48, 2 (2006), 241--256.
- 10.1145/3173574.3173922 · DOI
- John M McGuirl and Nadine B Sarter . 2006. Supporting trust calibration and the effective use of decision aids by presenting dynamic system confidence information. Human factors , Vol. 48 , 4 ( 2006 ), 656--665. John M McGuirl and Nadine B Sarter. 2006. Supporting trust calibration and the effective use of decision aids by presenting dynamic system confidence information. Human factors, Vol. 48, 4 (2006), 656--665.
- 10.1016/j.jbusres.2020.09.004 · DOI
- 10.1145/3290607.3313054 · DOI
- 10.1145/3411764.3445402 · DOI · Paper page
- 10.1145/1743384.1743478 · DOI · Paper page
- Vlad L Pop , Alex Shrewsbury , and Francis T Durso . 2015. Individual differences in the calibration of trust in automation. Human factors , Vol. 57 , 4 ( 2015 ), 545--556. Vlad L Pop, Alex Shrewsbury, and Francis T Durso. 2015. Individual differences in the calibration of trust in automation. Human factors, Vol. 57, 4 (2015), 545--556.
- Donghao Ren , Saleema Amershi , Bongshin Lee , Jina Suh , and Jason D Williams . 2016 . Squares: Supporting interactive performance analysis for multiclass classifiers . IEEE transactions on visualization and computer graphics, Vol. 23 , 1 (2016), 61--70. Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, and Jason D Williams. 2016. Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE transactions on visualization and computer graphics, Vol. 23, 1 (2016), 61--70.
- 10.1145/3411764.3445591 · DOI
- 10.1145/3359178 · DOI · Paper page
- 10.1145/3313831.3376506 · DOI
- 10.1145/3274423 · DOI
- 10.1108/IMR-09-2014-0302 · DOI
- Rion Snow , Brendan O'connor , Dan Jurafsky , and Andrew Y Ng . 2008 . Cheap and fast--but is it good? evaluating non-expert annotations for natural language tasks . In Proceedings of the 2008 conference on empirical methods in natural language processing. 254--263 . Rion Snow, Brendan O'connor, Dan Jurafsky, and Andrew Y Ng. 2008. Cheap and fast--but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing. 254--263.
- 10.18653/v1/2020.coling-main.422 · DOI
- 10.1145/3025171.3025208 · DOI
- S Shyam Sundar . 2008. The MAIN model: A heuristic approach to understanding technology effects on credibility . MacArthur Foundation Digital Media and Learning Initiative . S Shyam Sundar. 2008. The MAIN model: A heuristic approach to understanding technology effects on credibility. MacArthur Foundation Digital Media and Learning Initiative.
- 10.5555/1228983.1228994 · doi:10.5555/1228983.1228994
- 10.1145/1358628.1358873 · DOI
- 10.1016/j.chb.2020.106278 · DOI
- Jacob Whitehill , Ting-fan Wu, Jacob Bergsma , Javier Movellan , and Paul Ruvolo . 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems , Vol. 22 ( 2009 ), 2035--2043. Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems, Vol. 22 (2009), 2035--2043.
- 10.1145/2930238.2930290 · DOI
- 10.1145/3173574.3173733 · DOI
- Jing Zhang , Xindong Wu , and Victor S . Sheng . 2016 . Learning from crowdsourced labeled data: a survey. Artificial Intelligence Review ( 2016). https://doi.org/10.1007/s10462-016--9491--9 10.1007/s10462-016--9491--9 Jing Zhang, Xindong Wu, and Victor S. Sheng. 2016. Learning from crowdsourced labeled data: a survey. Artificial Intelligence Review (2016). https://doi.org/10.1007/s10462-016--9491--9 · doi:10.1007/s10462-016--9491--9
Cited by
Gomez, C., Cho, S. M., Ke, S., Huang, C.-M., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 209
Wischnewski, M., Krämer, N., & Müller, E. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581197 · Cited by 146
“[71, 109, 110, 132, 141] and eight which specifcally referred to artifcial intelligence [11, 12, 43, 83, 100, 118, 144, 145].”
BackgroundRaees, M., Khan, V.-J., Lykourentzou, I., & Papangelis, K. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791467 · Cited by 5
“These studies use (experiment) conditions to test the effect of feature-based [60, 153, 156, 164], example-based [17, 23, 108, 146], model-based [8, 33, 55, 114], or general explanations methods [102, 117, 127].”
InfluentialFrom Trust to Appropriate Reliance: Measurement Constructs in Human-AI Decision-Making
Raees, M., & Papangelis, K. (2026) · arXiv · doi:10.48550/arxiv.2604.23896 · Cited by 5
The Impact of LLM Assistance on User Spam Detection
Morales Flores, A., Jhaveri, J., Sharma, A., Rege, S., & Munyaka, I. (2025) · Proceedings of the 2025 Workshop on Human-Centered AI Privacy and Security · doi:10.1145/3733816.3760754 · Cited by 2
Self-citation
Metadata
- Authors
- Brachman, M., Ashktorab, Z., Desmond, M., Duesterwald, E., Dugan, C., Joshi, N. N., Pan, Q., & Sharma, A.
- Affiliations
- IBM Research, Cambridge, MA, USA
- IBM Research, Yorktown Heights, NY, USA
- Adobe, San Jose, CA, USA
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2022-11-07
- Volume
- 6
- Issue
- CSCW2
- Pages
- 1-27
- ISSN
- 2573-0142
- Cited by
- 14
- References
- 21
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: “Reliance and automation for Human-AI collaborative data labeling conflict resolution,”. https://app.cochat.ai/papers/10.1145/3555212
- The paper
- Brachman, M., Ashktorab, Z., Desmond, M., Duesterwald, E., Dugan, C., Joshi, N. N., Pan, Q., & Sharma, A. (2022). “Reliance and automation for Human-AI collaborative data labeling conflict resolution,”. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3555212
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
