Feature-Based Explanations Don't Help People Detect Misclassifications of Online Toxicity
Open accessAbstract
We present an experimental assessment of the impact of feature attribution-style explanations on human performance in predicting the consensus toxicity of social media posts with advice from an unreliable machine learning model. By doing so we add to a small but growing body of literature inspecting the utility of interpretable machine learning in terms of human outcomes. We also evaluate interpretable machine learning for the first time in the important domain of online toxicity, where fully-automated methods have faced criticism as being inadequate as a measure of toxic behavior.We find that, contrary to expectations, explanations have no significant impact on accuracy or agreement with model predictions, through they do change the distribution of subject error somewhat while reducing the cognitive burden of the task for subjects. Our results contribute to the recognition of an intriguing expectation gap in the field of interpretable machine learning between the general excitement the field has engendered and the ambiguous results of recent experimental work, including this study.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many empirical studies on human-AI decision-making did not observe complementary performance.
“However, many empirical studies on human-AI decision-making did not observe complementary performance [6,16,37,52,58,61,68,82,84,87].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Gauriat, C.-M., Pencolé, Y., Ribot, P., et al. (2024) · HAL · doi:10.4230/oasics.dx.2024.27 · Cited by 12,623 · see citing works
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761 · see citing works
A Survey on Automatic Detection of Hate Speech in Text
Fortuna, P., & Nunes, S. (2019) · ACM Computing Surveys · doi:10.1145/3232676 · Cited by 1,241 · see citing works
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040 · see citing works
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · see citing works
The Risk of Racial Bias in Hate Speech Detection
Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1163 · Cited by 817 · see citing works
Rationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777 · see citing works
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · see citing works
Measuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674 · see citing works
ERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561 · see citing works
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545 · see citing works
Attention is not ExplanationPreprint
Jain, S., & Wallace, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10186 · Cited by 468 · see citing works
Hate Speech Dataset from a White Supremacy Forum
de Gibert, O., Pérez, N., Pablos, A. G., & Cuadros, M. (2018) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w18-5102 · Cited by 428 · see citing works
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · see citing works
Developing an online hate classifier for multiple social media platforms
Salminen, J. O., Hopf, M., Chowdhury, S. A., Jung, S., Almerekhi, H., & Jansen, B. J. J. (2020) · Human-centric Computing and Information Sciences · doi:10.1186/s13673-019-0205-6 · Cited by 295 · see citing works
Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012429 · Cited by 284 · see citing works
Challenges in discriminating profanity from hate speech
Malmasi, S., & Zampieri, M. (2017) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2017.1409284 · Cited by 270 · see citing works
"What is relevant in a text document?": An interpretable machine learning approach
Arras, L., Horn, F., Montavon, G., Müller, K., & Samek, W. (2017) · PLoS ONE · doi:10.1371/journal.pone.0181142 · Cited by 268 · see citing works
Faithful and Customizable Explanations of Black Box Models
Lakkaraju, H., Kamar, E. S., Caruana, R., & Leskovec, J. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314229 · Cited by 261 · see citing works
Ex Machina: Personal Attacks Seen at ScalePreprint
Wulczyn, E., Thain, N., & Dixon, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.08914 · Cited by 260 · see citing works
Pathologies of Neural Models Make Interpretations Difficult
Feng, S., Wallace, E., Grissom, A., Iyyer, M., Rodríguez, P., & Boyd-Graber, J. L. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1407 · Cited by 260 · see citing works
Analyzing Labeled Cyberbullying Incidents on the Instagram Social Network
Hosseinmardi, H., Mattson, S. A., Rafiq, R. I., Han, R. Y., Lv, Q., & Mishra, S. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-27433-1_4 · Cited by 257 · see citing works
Deceiving Google's Perspective API Built for Detecting Toxic CommentsPreprint
Hosseini, H., Kannan, S., Zhang, B., & Poovendran, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08138 · Cited by 216 · see citing works
A Large Labeled Corpus for Online Harassment Research
Golbeck, J., Ashktorab, Z., Banjo, R. O., Berlinger, A., Bhagwan, S., Buntain, C., Cheakalos, P., Geller, A. A., Gergory, Q., Gnanasekaran, R. K., et al. (2017) · ACM Web Science Conference (WebSci) · doi:10.1145/3091478.3091509 · Cited by 203 · see citing works
Toxic Talk: How Online Incivility Can Undermine Perceptions of Media
Anderson, A., Yeo, S. K., Brossard, D., Scheufele, D. A., & Xenos, M. A. (2016) · International Journal of Public Opinion Research · doi:10.1093/ijpor/edw022 · Cited by 182 · see citing works
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Nguyen, D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1097 · Cited by 175 · see citing works
Communicating Algorithmic Process in Online Behavioral Advertising
Eslami, M., Krishna Kumaran, S. R., Sandvig, C., & Karahalios, K. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174006 · Cited by 171 · see citing works
Reddit Rules! Characterizing an Ecosystem of Governance
Fiesler, C., Jiang, J., McCann, J., Frye, K., & Brubaker, J. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.15033 · Cited by 165 · see citing works
Crossmod: A Cross-Community Learning-based System to Assist Reddit Moderators
Chandrasekharan, E., Gandhi, C., Mustelier, M. W., & Gilbert, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359276 · Cited by 151 · cited 2× · see citing works
“…et al. 2017; de Gibert et al. 2018; Hosseinmardi et al. 2015), while many papers have been published on the application of machine learning to the detection thereof (Salminen et al. 2020; Pavlopoulos, Malakasiotis, and Androutsopoulos 2017; Chancellor et al. 2017; Chandrasekharan et al. 2019).”
BackgroundHuman-Centred Machine Learning
Gillies, M. F. P., Fiebrink, R. A., Tanaka, A., Garcia, J., Bevilacqua, F. P., Héloir, A., Nunnari, F., Mackay, W. E., Amershi, S., Lee, B., et al. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2856492 · Cited by 138 · see citing works
An Evaluation of the Human-Interpretability of Explanation
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B., Gershman, S., & Doshi-Velez, F. (2019) · arXiv · doi:10.48550/arxiv.1902.00006 · Cited by 125 · see citing works
Deep Learning for User Comment Moderation
Pavlopoulos, J., Malakasiotis, P., & Androutsopoulos, I. (2017) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w17-3004 · Cited by 124 · see citing works
Rethinking Cooperative Rationalization: Introspective Extraction and Complement Control
Yu, M., Chang, S., Zhang, Y., & Jaakkola, T. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1420 · Cited by 121 · see citing works
(2018) · arXiv (Cornell University) · Cited by 113 · see citing works
Like Trainer, Like Bot? Inheritance of Bias in Algorithmic Content Moderation
Binns, R., Veale, M., Kleek, M. G. V., & Shadbolt, N. (2017) · UCL Discovery (University College London) · Cited by 113 · see citing works
Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)
Kumar, R., Ojha, A. K., Zampieri, M., & Malmasi, S. (2018) · Cited by 108 · see citing works
Multimodal Classification of Moderated Online Pro-Eating Disorder Content
Chancellor, S., Kalantidis, Y., Pater, J. A., Choudhury, M. D., & Shamma, D. A. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025985 · Cited by 87 · see citing works
When Online Harassment Is Perceived as Justified
Blackwell, L., Chen, T., Schoenebeck, S. Y., & Lampe, C. A. C. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.15036 · Cited by 80 · see citing works
Proceedings of the First Workshop on Abusive Language Online
(2017) · doi:10.18653/v1/w17-30 · Cited by 63 · see citing works
Finding Good Conversations Online: The Yahoo News Annotated Comments Corpus
Napoles, C., Tetreault, J., Pappu, A., Rosato, E., & Provenzale, B. (2017) · Linguistic Annotation Workshop · doi:10.18653/v1/w17-0802 · Cited by 56 · see citing works
Assessing the Local Interpretability of Machine Learning ModelsPreprint
Slack, D., Friedler, S. A., Scheidegger, C., & Roy, C. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.03501 · Cited by 50 · see citing works
A Human-Grounded Evaluation of SHAP for Alert ProcessingPreprint
Weerts, H., van Ipenburg, W., & Pechenizkiy, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.03324 · Cited by 45 · see citing works
Carton, S., Mei, Q., & Resnick, P. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1386 · Cited by 37 · see citing works
Self-citationLai, V., & Tan, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.07901 · Cited by 35 · see citing works
Like trainer, like bot? Inheritance of bias in algorithmic content moderationPreprint
Binns, R., Veale, M., Kleek, M. G. V., & Shadbolt, N. (2017) · doi:10.31219/osf.io/97u3q · Cited by 31 · see citing works
Dice in the Black Box: User Experiences with an Inscrutable Algorithm
Springer, A., Hollis, V., & Whittaker, S. (2018) · arXiv · doi:10.48550/arxiv.1812.03219 · Cited by 29 · see citing works
Improving Moderation of Online Discussions via Interpretable Neural Models
Švec, A., Pikuliak, M., Šimko, M., & Bieliková, M. (2018) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w18-5108 · Cited by 20 · see citing works
Rethinking Cooperative Rationalization: Introspective Extraction and Complement ControlPreprint
Yu, M., Chang, S., Zhang, Y., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.13294 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 12
- [1]1. Abdul, A.; Vermeulen, J.; Wang, D.; Lim, B. Y.; and Kankanhalli, M. 2018. Trends and Trajectories for Ex- plainable, Accountable and Intelligible Systems: An HCI Research Agenda. In Proceedings of the CHI Conference on Human Factors in Computing Systems, 1-18. Montreal QC, Canada: ACM Press.
- [6]6. Cham: Springer.
- [8]8. Blackwell, L.; Dimond, J.; Schoenebeck, S.; and Lampe, C. 2018b. Classification and Its Consequences for Online Ha- rassment: Design Insights from HeartMob. In Proceedings of the ACM Conference on Computer Supported Coopera- tive Work (CSCW 2018), 1-19.
- [11]11. Chandrasekharan, E.; Samory, M.; Jhaver, S.; Charvat, H.; Bruckman, A.; Lampe, C.; Eisenstein, J.; and Gilbert, E. 2018. The Internet's Hidden Rules: An Empirical Study of Reddit Norm Violations at Micro, Meso, and Macro Scales. Proceedings of the ACM on Human-Computer Interaction - CSCW 2(CSCW):1-25.
- [17]17. Dove, G.; Halskov, K.; Forlizzi, J.; and Zimmerman, J. 2017. UX Design Innovation: Challenges for Working with Ma- chine Learning as a Design Material. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Sys- tems -CHI '17, 278-288. Denver, Colorado, USA: ACM Press.
- [29]29. Jhaver, S.; Birman, I.; Gilbert, E.; and Bruckman, A. 2019. Human-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator. ACM Trans. Comput.-Hum. Interact. 26(5):31:1-31:35.
- [36]36. Lei, T.; Barzilay, R.; and Jaakkola, T. 2016. Rationaliz- ing Neural Predictions. In Proceedings of the 2016 Confer- ence on Empirical Methods in Natural Language Process- ing, 107-117.
- [39]39. Murdoch, W. J.; Singh, C.; Kumbier, K.; Abbasi-Asl, R.; and Yu, B. 2019. Interpretable machine learning: definitions, methods, and applications. arXiv preprint. arXiv:1901.04592. · arXiv
- [42]42. Olteanu, A.; Talamadupula, K.; and Varshney, K. R. 2017. The Limits of Abstract Evaluation Metrics: The Case of Hate Speech Detection. In Proceedings of the 2017 ACM on Web Science Conference -WebSci '17, 405-406.
- [43]43. Troy, New York, USA: ACM Press.
- [53]53. Wang, C. 2018. Interpreting Neural Network Hate Speech Classiers. In Proceedings of the 2nd Workshop on Abusive Language Online, 7.
- [56]56. Wiegreffe, S., and Pinter, Y. 2019. Attention is not not Ex- planation. arXiv preprint. arXiv: 1908.04626. · arXiv
Cited by
Sorted by citations
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · see citing works
Will Humans-in-the-Loop Become Borgs? Merits and Pitfalls of Working with AI
Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16553 · Cited by 322 · see citing works
How transparency modulates trust in artificial intelligence
Zerilli, J., Bhatt, U., & Weller, A. (2022) · Patterns · doi:10.1016/j.patter.2022.100455 · Cited by 257 · see citing works
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204 · see citing works
Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584066 · Cited by 208 · see citing works
Wysocki, O., Davies, J. K., Vigo, M., Armstrong, A. C., Landers, D., Lee, R., & Freitas, A. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2022.103839 · Cited by 161 · see citing works
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 145 · see citing works
“However, many empirical studies on human-AI decision-making do not observe complementary performance [6, 15, 34, 47, 53, 56, 60, 71, 73, 76].”
BackgroundComplementarity in human-AI collaboration: concept, sources, and evidence
Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2025) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2475962 · Cited by 145 · see citing works
Liu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111 · see citing works
Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction
Hemmer, P., Westphal, M., Schemmer, M., Vetter, S., Vössing, M., & Satzger, G. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584052 · Cited by 111 · see citing works
Effects of Explanations in AI-Assisted Decision Making: Principles and Comparisons
Wang, X., & Yin, M. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3519266 · Cited by 82 · see citing works
Chiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581015 · Cited by 60 · see citing works
Selective Explanations: Leveraging Human Input to Align Explainable AI
Lai, V. W.-M., Zhang, Y., Chen, C., Liao, Q. V., & Tan, C. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610206 · Cited by 64 · see citing works
Fok, R., & Weld, D. S. (2024) · AI Magazine · doi:10.1002/aaai.12182 · Cited by 81 · see citing works
Fok, R., & Weld, D. S. (2023) · arXiv · doi:10.48550/arxiv.2305.07722 · Cited by 54 · see citing works
Naveed, S., Stevens, G., & Robin-Kern, D. (2024) · Applied Sciences · doi:10.3390/app142311288 · Cited by 54 · see citing works
Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making
Lu, Z., Wang, D., & Yin, M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653708 · Cited by 40 · see citing works
Wang, X., & Yin, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581366 · Cited by 38 · see citing works
Impact of Model Interpretability and Outcome Feedback on Trust in AI
Ahn, D., Almaatouq, A., Gulabani, M., & Hosanagar, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642780 · Cited by 39 · see citing works
Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making
Guerdan, L., Coston, A., Wu, Z. S., & Holstein, K. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594036 · Cited by 28 · see citing works
Investigating toxicity changes of cross-community redditors from 2 billion posts and comments
Almerekhi, H., Kwak, H., & Jansen, B. J. J. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1059 · Cited by 19 · see citing works
Schemmer, M., Kühl, N., & Satzger, G. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.185 · Cited by 18 · see citing works
PROVOKE: Toxicity trigger detection in conversations from the top 100 subreddits
Almerekhi, H., Kwak, H., Salminen, J. O., & Jansen, B. J. J. (2022) · Data and Information Management · doi:10.1016/j.dim.2022.100019 · Cited by 17 · see citing works
Exploring the impact of social network structures on toxicity in online mental health communities
Akar, E. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108542 · Cited by 15 · see citing works
The Impact of Explanations on Fairness in Human-AI Decision-Making: Protected vs Proxy Features
Goyal, N., Baumler, C., Nguyen, T., & Daumé, H. (2023) · International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645210 · Cited by 19 · see citing works
Ferguson, S. A., Aoyagui, P. A., Rizvi, R., Kim, Y., & Kuzminykh, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687056 · Cited by 15 · see citing works
To Solicit or Not to Solicit? Impact of AI Assistance Delivery Mechanisms on Decision-Making
Nair, P., Gressel, G., Anand, M. N., & Achuthan, K. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2536617 · Cited by 12 · see citing works
ConvEx: A Visual Conversation Exploration System for Discord Moderators
Choi, F., Bajpai, T., Pratipati, S., & Chandrasekharan, E. (2023) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3610053 · Cited by 27 · see citing works
Towards Understanding AI Delegation: The Role of Self-Efficacy and Visual Processing Ability
Westphal, M., Hemmer, P., Vössing, M., Schemmer, M., Vetter, S., & Satzger, G. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3696423 · Cited by 10 · see citing works
Ferguson, S. A., Aoyagui, P. A., & Kuzminykh, A. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585727 · Cited by 9 · see citing works
Human-centred explanations for artificial intelligence systems
Baber, C., Kandola, P., Apperly, I. A., & McCormick, E. (2024) · Ergonomics · doi:10.1080/00140139.2024.2334427 · Cited by 8 · see citing works
Papenkordt, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60606-9_14 · Cited by 8 · see citing works
HamiltonDinggg at SemEval-2021 Task 5: Investigating Toxic Span Detection using RoBERTa Pre-training
Ding, H., & Jurgens, D. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.31 · Cited by 8 · see citing works
Measuring Toxicity Toward Women in Game-Based Communities
Belskie, M., Zhang, H., & Hemminger, B. M. (2023) · Journal of Electronic Gaming and Esports · doi:10.1123/jege.2022-0035 · Cited by 8 · see citing works
Schemmer, M., Kühl, N., & Satzger, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.13827 · Cited by 9 · see citing works
Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labeling
Zhang, D., Chatzimparmpas, A., Kamali, N., & Hullman, J. R. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642446 · Cited by 6 · see citing works
Modeling Users and Online Communities for Abuse Detection: A Position on Ethics and Explainability
Mishra, P., Yannakoudakis, H., & Shutova, E. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.287 · Cited by 6 · see citing works
Joshi, B., Liu, Z., Ramnath, S., Chan, A., Tong, Z., Nie, S., Wang, Q., Choi, Y., & Ren, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.392 · Cited by 8 · see citing works
A Study of Explainability Features to Scrutinize Faceted Filtering Results
Qu, J., Arguello, J., & Wang, Y. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482409 · Cited by 5 · see citing works
A First Look at User-Controlled Moderation on Web3 Social Media: The Case of Memo.cash
Zuo, W., Raman, A., MondragÓN, R. J., & Tyson, G. (2023) · OASIS@HT · doi:10.1145/3599696.3612901 · Cited by 7 · see citing works
Understanding the Effects of Explaining Predictive but Unintuitive Features in Human-XAI Interaction
Qu, J., Arguello, J., & Wang, Y. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732021 · Cited by 4 · see citing works
Human-AI Teaming: Following the IMOI Framework
Kleanthous, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60611-3_27 · Cited by 3 · see citing works
Qu, J., Arguello, J., & Wang, Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658547 · Cited by 3 · see citing works
Model Transparency: Why Do We Care?
Papantonis, I., & Belle, V. (2023) · International Conference on Agents and Artificial Intelligence · doi:10.5220/0011726300003393 · Cited by 2 · see citing works
Qu, J., Arguello, J., & Wang, Y. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578315 · Cited by 2 · see citing works
Clement, J. D., Ren, Y., & Curley, S. P. (2026) · Medical Decision Making · doi:10.1177/0272989x261484927 · Cited by 1 · see citing works
Boosting Benefits, Offsetting Obstacles—The Impact of Explanations on AI Users’ Task Performance
Walter, M. C., Broder, H. R., & Förster, M. (2025) · Lecture notes in information systems and organisation · doi:10.1007/978-3-031-80122-8_7 · Cited by 1 · see citing works
Register, Y., & Spiro, E. S. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19337 · Cited by 1 · see citing works
Willing to Revise? Confidence and Recommendation Adoption in AI-Assisted Image Recognition
Röseler, L., Scholtes, I., Sendhoff, B., & Hannák, A. (2022) · Frontiers in artificial intelligence and applications · doi:10.3233/faia220222
Chuan, C., Sun, R., Tian, S., & Tsai, W. S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4624907
Masciotra, V., & Boudrias, J. (2024) · Ad machina l avenir de l humain au travail · doi:10.1522/radm.no8.1840
AI Behavioral SciencePreprint
Jackson, M. O., Mei, Q., Wang, S., Xie, Y., Yuan, W., Benzell, S. G., Brynjolfsson, E., Evans, J. A., Jabarian, B., Kleinberg, J. M., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5395006 · Cited by 1 · see citing works
Self-citationBoyacı, T., Canyakmaz, C., & de Véricourt, F. (2026) · Management Science · doi:10.1287/mnsc.2024.07450 · Cited by 3 · see citing works
Tuo, B., Miller, T., & Demartini, G. (2026) · ACM Conference on Human-AI Complementarity and Alignment · doi:10.1145/3834580.3838739
- Authors
- Carton, S., Mei, Q., & Resnick, P.
- Affiliations
- University of Michigan (US)
- Venue
- Proceedings of the International AAAI Conference on Web and Social Media
- Published
- 2020-05-26
- Volume
- 14
- Pages
- 95-106
- ISSN
- 2162-3449 (ISSN-L)
- 2334-0770
- OpenAlex
- W3037843601
- Cited by
- 58
- References
- 50
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 14 · 2023: 15 · 2022: 14 · 2021: 7
- FWCI
- 3.44
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the International AAAI Conference on Web and Social Media (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- National Science Foundation (1633370, 1620319, 1717688)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- feature attribution, explainable AI, online toxicity, social media posts, model reliability, cognitive load, model interpretability, human evaluation of explanations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
