"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
To support human decision making with machine learning models, we often need to elucidate patterns embedded in the models that are unsalient, unknown, or counterintuitive to humans. While existing approaches focus on explaining machine predictions with real-time assistance, we explore model-driven tutorials to help humans understand these patterns in a train- ing phase. We consider both tutorials with guidelines from scientific papers, analogous to current practices of science communication, and automatically selected examples from training data with explanations. We use deceptive review detection as a testbed and conduct large-scale, randomized human-subject experiments to examine the effectiveness of such tutorials. We find that tutorials indeed improve human performance, with and without real-time assistance. In particular, although deep learning provides superior predictive performance than simple models, tutorials and explanations from simple models are more useful to humans. Our work suggests future directions for human-centered tutorials and explanations towards a synergy between humans and AI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, many empirical studies on human-AI decision-making did not observe complementary performance [6,16,37,52,58,61,68,82,84,87].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We would like to emphasize that deception detection is a difficult task for humans [42, 43].”
From Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations (2023) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“Many studies have investigated the role of AI in assisting humans in making individual decisions [9,16,36, 37, 50, 51, 55, 58, 69, 83, 84, 89], e.g., predicting whether a person will recidivate in the near future.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Since we deal with textual data, a common technique to display the values is to highlight the respective words according to their computed influence on the AI's decision [42].”
From Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations (2023) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“The implemented AI advisor is based on a Support Vector Machine with an accuracy of 86%, which is a performance that is similar to the performance in related literature [42].”
From Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations (2023) · cites this paper as [42]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Social Media and Fake News in the 2016 Election
Allcott, H., & Gentzkow, M. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.211 · Cited by 6,767 · cited 3×
“While our samples are likely to face the challenges of deception on the Internet and would benefit from enhancements in deception detection, they may not be representative of the general population.”
BackgroundTrust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
The Effect of Word of Mouth on Sales: Online Book Reviews
Chevalier, J. A., & Mayzlin, D. (2006) · Journal of Marketing Research · doi:10.1509/jmkr.43.3.345 · Cited by 6,003 · cited 1×
“, online reviews for making purchase decisions [10, 57, 64, 68]), deceptive information also becomes prevalent [9, 45, 53].”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
Research works printed but not linked: 18
- Diane Farsetta and Daniel Price. 2006. Fake TV news: Widespread and undisclosed. Center for Media and Democracy 6 (2006).
- Song Feng, Ritwik Banerjee, and Yejin Choi. 2012. Syntactic stylometry for deception detection. In Proceedings of ACL (short papers).
- Vanessa Wei Feng and Graeme Hirst. 2013. Detecting deceptive opinions with profile compatibility. In Proceedings of IJCNLP.
- Nir Grinberg, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson, and David Lazer. 2019. Fake news on Twitter during the 2016 US presidential election. Science 363, 6425 (2019), 374--378.
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2019. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 (2019), 93.
- Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016. Examples are not enough, learn to criticize! criticism for interpretability. In Proceedings of NIPS.
- Been Kim, Cynthia Rudin, and Julie A Shah. 2014. The bayesian case model: A generative approach for case-based reasoning and prototype classification. In Proceedings of NIPS.
- Robert M Krauss, Valerie Geller, and Christopher Olson. 1976. Modalities and cues in the detection of deception. In Meeting of the American Psychological Association, Washington, DC.
- 10.1037/1076-898X.6.2.104 · doi:10.1037/1076-898x.6.2.104 · Paper page
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016). · arXiv
- Myle Ott, Claire Cardie, and Jeffrey T Hancock. 2013. Negative deceptive opinion spam. In Proceedings of NAACL.
- Supreme Court of Wisconsin. 2016. State of Wisconsin, Plaintiff-Respondent, v. Eric L. Loomis, Defendant-Appellant. (2016).
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS.
- Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science 359, 6380 (2018), 1146--1151.
- Sandra Wachter Brent Mittelstadt and Chris Russell. 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. · DOI · Paper page
- Qiang Ye, Rob Law, Bin Gu, and Wei Chen. 2011. The influence of user-generated content on traveler behavior: An empirical investigation on the effects of e-word-of-mouth to hotel online bookings. Computers in Human behavior 27, 2 (2011), 634--639.
- Kyung-Hyan Yoo and Ulrike Gretzel. 2009. Comparison of deceptive and truthful travel reviews. Information and communication technologies in tourism 2009 (2009), 37--47.
- Amy X Zhang, Aditya Ranganathan, Sarah Emlen Metz, Scott Appling, Connie Moon Sehat, Norman Gilmore, Nick B Adams, Emmanuel Vincent, Martin Robbins, Ed Bice, Sandro Hawke, David Karger, and An Xiao Mina. 2018. A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles. In Proceedings of WWW (Companion).
Cited by
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“However, models outperformed every team (see Table 1 and Figure 6 in [35]).”
BackgroundMethodResultInfluentialWhen combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“(see Supplementary Tables 5 and 6 for more details).”
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538
Metadata
- Authors
- Lai, V., Liu, H., & Tan, C.
- Affiliations
- University of Colorado Boulder (US)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-13
- OpenAlex
- W3032816739
- Cited by
- 123
- References
- 42
- Citations by year
- 2026: 10 · 2025: 17 · 2024: 30 · 2023: 31 · 2022: 19 · 2021: 13 · 2020: 3
- FWCI
- 6.30
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1849931, 1927322, 1837986)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- human-AI collaboration, explainable machine learning, model interpretability, human decision-making, human subjects research, model complexity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: "Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans. https://app.cochat.ai/papers/10.1145/3313831.3376873
- The paper
- Lai, V., Liu, H., & Tan, C. (2020). "Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3313831.3376873
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.