"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
To support human decision making with machine learning models, we often need to elucidate patterns embedded in the models that are unsalient, unknown, or counterintuitive to humans. While existing approaches focus on explaining machine predictions with real-time assistance, we explore model-driven tutorials to help humans understand these patterns in a train- ing phase. We consider both tutorials with guidelines from scientific papers, analogous to current practices of science communication, and automatically selected examples from training data with explanations. We use deceptive review detection as a testbed and conduct large-scale, randomized human-subject experiments to examine the effectiveness of such tutorials. We find that tutorials indeed improve human performance, with and without real-time assistance. In particular, although deep learning provides superior predictive performance than simple models, tutorials and explanations from simple models are more useful to humans. Our work suggests future directions for human-centered tutorials and explanations towards a synergy between humans and AI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, many empirical studies on human-AI decision-making did not observe complementary performance [6,16,37,52,58,61,68,82,84,87].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We would like to emphasize that deception detection is a difficult task for humans [42, 43].”
From Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations (2023) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“Many studies have investigated the role of AI in assisting humans in making individual decisions [9,16,36, 37, 50, 51, 55, 58, 69, 83, 84, 89], e.g., predicting whether a person will recidivate in the near future.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Since we deal with textual data, a common technique to display the values is to highlight the respective words according to their computed influence on the AI's decision [42].”
From Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations (2023) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“The implemented AI advisor is based on a Support Vector Machine with an accuracy of 86%, which is a performance that is similar to the performance in related literature [42].”
From Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations (2023) · cites this paper as [42]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Social Media and Fake News in the 2016 Election
Allcott, H., & Gentzkow, M. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.211 · Cited by 6,767 · cited 3×
“While our samples are likely to face the challenges of deception on the Internet and would benefit from enhancements in deception detection, they may not be representative of the general population.”
BackgroundTrust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
The Effect of Word of Mouth on Sales: Online Book Reviews
Chevalier, J. A., & Mayzlin, D. (2006) · Journal of Marketing Research · doi:10.1509/jmkr.43.3.345 · Cited by 6,003 · cited 1×
“, online reviews for making purchase decisions [10, 57, 64, 68]), deceptive information also becomes prevalent [9, 45, 53].”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Trusov, M., Bucklin, R. E., & Pauwels, K. (2009) · Journal of Marketing · doi:10.1509/jmkg.73.5.90 · Cited by 2,451
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 2×
“Most relevant to our work is feature attribution that assigns an importance weight to each feature [37, 42, 50, 51].”
BackgroundAccuracy of Deception Judgments
Bond, C. F., & DePaulo, B. M. (2006) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr1003_2 · Cited by 1,953
Jindal, N., & Liu, B. (2008) · Proceedings of the international conference on Web search and web data mining - WSDM '08 · doi:10.1145/1341531.1341560 · Cited by 1,542 · cited 1×
“Computer scientists have been developing machine learning models to identify deception in texts, images, and videos [1, 15, 16, 24, 47, 48, 63, 66].”
BackgroundExplaining machine learning classifiers through diverse counterfactual explanations
Mothilal, R. K., Sharma, A., & Tan, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372850 · Cited by 1,233 · cited 1×
“Other popular approaches include 1) example-based [26, 27, 43, 52, 61], e.”
Self-citationSurvey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Finding Deceptive Opinion Spam by Any Stretch of the Imagination
Ott, M., Choi, Y., Cardie, C., & Hancock, J. T. (2011) · arXiv · doi:10.48550/arxiv.1107.4557 · Cited by 1,008
Rationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777 · cited 2×
“[37] Tao Lei, Regina Barzilay, and Tommi Jaakkola.”
BackgroundZhang, Z., Ye, Q., Law, R., & Li, Y. (2010) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2010.02.002 · Cited by 760
Submodular Function Maximization
Krause, A., & Golovin, D. (2014) · Tractability · doi:10.1017/cbo9781139177801.004 · Cited by 751
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · cited 2×
“In addition to studies mentioned in the introduction, researchers have investigated other desiderata of explanations [5, 8, 17, 18, 32, 49, 65].”
BackgroundKleinberg, J., Ludwig, J., Mullainathan, S., & Obermeyer, Z. (2015) · American Economic Review · doi:10.1257/aer.p20151023 · Cited by 584 · cited 1×
“Notably, many critical policy decisions such as bail decisions resemble the discovering mode more than the emulating mode because policy decisions are usually challenging (to humans) in nature [29].”
BackgroundHuman Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545
Detecting lies and deceit: the psychology of lying and the implications for professional practice
Clifford, B. R. (2001) · Applied Cognitive Psychology · doi:10.1002/acp.743 · Cited by 537
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 3×
“It is important to highlight both strengths and caveats of ML models in the tutorials, echoing recent work on understanding trust [32, 65].”
BackgroundBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 1×
“erature on appropriate reliance with general automation [36,38]. Retaining human agency is particularly important in societally critical domains where consequences can be dire. Finally, Bansal et al. [4] provide feedback during decision making, which can be seen as a form of continuous learning. Our focus is to unders…”
BackgroundThe Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441
Spaced Repetition Promotes Efficient and Effective Learning: Policy Implications for Instruction
Kang, S. H. K. (2016) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/2372732215624708 · Cited by 408
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 1×
“In addition to studies mentioned in the introduction, researchers have investigated other desiderata of explanations [5, 8, 17, 18, 32, 49, 65].”
BackgroundLai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · cited 5×
“o apply in the prediction phase without any real-time assistance. Therefore, our second experiment is designed to understand the effect of tutorials with real-time assistance. Inspired by Lai and Tan [34], we develop a spectrum with varying levels of real-time assistance between full human agency and full automation (…”
BackgroundMethodInfluentialSelf-citationTowards a General Rule for Identifying Deceptive Opinion Spam
Li, J., Ott, M., Cardie, C., & Hovy, E. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1147 · Cited by 362
Getting access to what goes on in people's heads?: reflections on the think-aloud technique
Nielsen, J., Clemmensen, T., & Yssing, C. (2002) · Proceedings of the second Nordic conference on Human-computer interaction · doi:10.1145/572020.572033 · Cited by 315
Estimating the prevalence of deception in online review communities
Ott, M., Cardie, C., & Hancock, J. (2012) · Proceedings of the 21st international conference on World Wide Web · doi:10.1145/2187836.2187864 · Cited by 282
Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275 · cited 3×
“[17] Ben Green and Yiling Chen.”
BackgroundThe dynamics of trust: Comparing humans to automation.
Lewandowsky, S., Mundy, M., & Tan, G. P. A. (2000) · Journal of Experimental Psychology: Applied · doi:10.1037//1076-898x.6.2.104 · Cited by 240
Efficient Search for Diverse Coherent Explanations
Russell, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287569 · Cited by 218 · cited 1×
“Other popular approaches include 1) example-based [26, 27, 43, 52, 61], e.”
Tan, C., Lee, L., & Pang, B. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1017 · Cited by 188
Self-citationOnline Deception: Prevalence, Motivation, and Emotion
Caspi, A., & Gorsky, P. (2006) · CyberPsychology & Behavior · doi:10.1089/cpb.2006.9.54 · Cited by 186
Local Rule-Based Explanations of Black Box Decision Systems
Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D., Turini, F., & Giannotti, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10820 · Cited by 179
Enhancing human learning via spaced repetition optimization
Tabibian, B., Upadhyay, U., De, A., Zarezade, A., Schölkopf, B., & Gomez-Rodriguez, M. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1815156116 · Cited by 178
Distortion as a validation criterion in the identification of suspicious reviews
Wu, G., Greene, D., Smyth, B., & Cunningham, P. (2010) · doi:10.1145/1964858.1964860 · Cited by 163
'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
(2018) · doi:10.31235/osf.io/9wqxr · Cited by 161
Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
Kunkel, J., Donkers, T., Michael, L., Barbu, C.-M., & Ziegler, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300717 · Cited by 156 · cited 3×
“he tutorials failed to establish proper trust in machine predictions. It is important to highlight both strengths and caveats of ML models in the tutorials, echoing recent work on understanding trust [32,65]. A challenge lies in how to bridge the gap between training and generalization in tutorials, i.e., model behavi…”
BackgroundInfluentialVerbal and Nonverbal Clues for Real-life Deception Detection
Pérez‐Rosas, V., Abouelenien, M., Mihalcea, R. F., Xiao, Y., Linton, C., & Burzo, M. G. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1281 · Cited by 92
A Human-Grounded Evaluation of SHAP for Alert ProcessingPreprint
Weerts, H., van Ipenburg, W., & Pechenizkiy, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.03324 · Cited by 91
Deception detection using a multimodal approach
Abouelenien, M., Pérez‐Rosas, V., Mihalcea, R. F., & Burzo, M. G. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2663229 · Cited by 67
Rating Reliability and Bias in News Articles: Does AI Assistance Help Everyone?
Horne, B. D., Nevo, D., O’Donovan, J., Cho, J.-H., & Adalı, S. (2019) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v13i01.3226 · Cited by 67
Prevalence and mitigation of forum spamming
Shin, Y., Gupta, M., & Myers, S. (2011) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infcom.2011.5935048 · Cited by 40
Lai, V., Cai, Z., & Tan, C. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1046 · Cited by 27
Self-citation
Research works printed but not linked: 18
- Diane Farsetta and Daniel Price. 2006. Fake TV news: Widespread and undisclosed. Center for Media and Democracy 6 (2006).
- Song Feng, Ritwik Banerjee, and Yejin Choi. 2012. Syntactic stylometry for deception detection. In Proceedings of ACL (short papers).
- Vanessa Wei Feng and Graeme Hirst. 2013. Detecting deceptive opinions with profile compatibility. In Proceedings of IJCNLP.
- Nir Grinberg, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson, and David Lazer. 2019. Fake news on Twitter during the 2016 US presidential election. Science 363, 6425 (2019), 374--378.
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2019. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 (2019), 93.
- Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016. Examples are not enough, learn to criticize! criticism for interpretability. In Proceedings of NIPS.
- Been Kim, Cynthia Rudin, and Julie A Shah. 2014. The bayesian case model: A generative approach for case-based reasoning and prototype classification. In Proceedings of NIPS.
- Robert M Krauss, Valerie Geller, and Christopher Olson. 1976. Modalities and cues in the detection of deception. In Meeting of the American Psychological Association, Washington, DC.
- 10.1037/1076-898X.6.2.104 · doi:10.1037/1076-898x.6.2.104 · Paper page
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016). · arXiv
- Myle Ott, Claire Cardie, and Jeffrey T Hancock. 2013. Negative deceptive opinion spam. In Proceedings of NAACL.
- Supreme Court of Wisconsin. 2016. State of Wisconsin, Plaintiff-Respondent, v. Eric L. Loomis, Defendant-Appellant. (2016).
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS.
- Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science 359, 6380 (2018), 1146--1151.
- Sandra Wachter Brent Mittelstadt and Chris Russell. 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. · DOI · Paper page
- Qiang Ye, Rob Law, Bin Gu, and Wei Chen. 2011. The influence of user-generated content on traveler behavior: An empirical investigation on the effects of e-word-of-mouth to hotel online bookings. Computers in Human behavior 27, 2 (2011), 634--639.
- Kyung-Hyan Yoo and Ulrike Gretzel. 2009. Comparison of deceptive and truthful travel reviews. Information and communication technologies in tourism 2009 (2009), 37--47.
- Amy X Zhang, Aditya Ranganathan, Sarah Emlen Metz, Scott Appling, Connie Moon Sehat, Norman Gilmore, Nick B Adams, Emmanuel Vincent, Martin Robbins, Ed Bice, Sandro Hawke, David Karger, and An Xiao Mina. 2018. A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles. In Proceedings of WWW (Companion).
Cited by
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“However, models outperformed every team (see Table 1 and Figure 6 in [35]).”
BackgroundMethodResultInfluentialWhen combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“(see Supplementary Tables 5 and 6 for more details).”
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538
Metadata
- Authors
- Lai, V., Liu, H., & Tan, C.
- Affiliations
- University of Colorado Boulder (US)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-13
- OpenAlex
- W3032816739
- Cited by
- 123
- References
- 42
- Citations by year
- 2026: 10 · 2025: 17 · 2024: 30 · 2023: 31 · 2022: 19 · 2021: 13 · 2020: 3
- FWCI
- 6.30
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1849931, 1927322, 1837986)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- human-AI collaboration, explainable machine learning, model interpretability, human decision-making, human subjects research, model complexity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.