CoChat
Sign Up
PreprintCited by 62Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AIUsability and User Interface Design

Rethinking Explainability as a Dialogue: A Practitioner's Perspective

Open access
Lakkaraju, Himabindu, Slack, Dylan, Chen, Yuxin, Tan, Chenhao, Singh, Sameer·2022·arXiv
DOI 10.48550/arxiv.2202.01875arXiv 2202.01875
Open this paper in CoChat — free accountOpen PDF

Abstract

As practitioners increasingly deploy machine learning models in critical domains such as health care, finance, and policy, it becomes vital to ensure that domain experts function effectively alongside these models. Explainability is one way to bridge the gap between human decision-makers and machine learning models. However, most of the existing work on explainability focuses on one-off, static explanations like feature importances or rule lists. These sorts of explanations may not be sufficient for many use cases that require dynamic, continuous discovery from stakeholders. In the literature, few works ask decision-makers about the utility of existing explanations and other desiderata they would like to see in an explanation going forward. In this work, we address this gap and carry out a study where we interview doctors, healthcare professionals, and policymakers about their needs and desires for explanations. Our study indicates that decision-makers would strongly prefer interactive explanations in the form of natural language dialogues. Domain experts wish to treat machine learning models as "another colleague", i.e., one who can be held accountable by asking why they made a particular decision through expressive and accessible natural language interactions. Considering these needs, we outline a set of five principles researchers should follow when designing interactive explanations as a starting place for future work. Further, we show why natural language dialogues satisfy these principles and are a desirable way to build interactive explanations. Next, we provide a design of a dialogue system for explainability and discuss the risks, trade-offs, and research opportunities of building these systems. Overall, we hope our work serves as a starting place for researchers and engineers to design interactive explainability systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2202.01875

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The quality of generated content and its uses may be more difficult to evaluate objectively.

    “but the quality of generated content and its uses may be more difficult to evaluate objectively (e.g., consider how one would objectively evaluate the quality of an LLM-generated email) [87,99].”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [99]

    Section 4.2.2 (p. 11) says there are few metrics for determining the quality of generated explanations, making it hard to decide when to disregard them. This supports the difficulty of evaluating explanations and their use, but the paper does not extend that point to generated content generally or to LLM-generated emails. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 62

Research works printed but not linked: 99

  1. [1][1] A. Abdul, J. Vermeulen, D. Wang, B. Y. Lim, and M. Kankanhalli. Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda , page 1-18. Association for Computing Machinery, New York, NY, USA, 2018. ISBN 9781450356206. URL https://doi.org/10.1145/3173574.3174156 . - · DOI · Paper page
  2. [2][2] A. Abid, A. Abdalla, A. Abid, D. Khan, A. Alfozan, and J. Zou. Gradio: Hassle-free sharing and testing of ml models in the wild. arXiv preprint arXiv:1906.02569 , 2019. - · arXiv
  3. [3][3] J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems , pages 9505-9515, 2018. -
  4. [4][4] S. Agarwal, S. Jabbari, C. Agarwal, S. Upadhyay, S. Wu, and H. Lakkaraju. Towards the unification and robustness of perturbation and gradient based explanations. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 110-119. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/agarwal21c.html . -
  5. [5][5] D. Alvarez-Melis and T. S. Jaakkola. On the robustness of interpretability methods. ICML Workshop on Human Interpretability in Machine Learning , 2018. -
  6. [6][6] C. Anders, P. Pasliev, A.-K. Dombrowski, K.-R. Müller, and P. Kessel. Fairwashing explanations with off-manifold detergent. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 314-323. PMLR, 13-18 Jul 2020. URL http://proceedings.mlr.press/v119/anders20a.html . -
  7. [7][7] E. Angelino, N. Larus-Stone, D. Alabi, M. Seltzer, and C. Rudin. Learning certifiably optimal rule lists for categorical data. arXiv preprint arXiv:1704.01701 , 2017. - · arXiv
  8. [8][8] S. Arora, K. Batra, and S. Singh. Dialogue system: A brief review. ArXiv , abs/1306.4134, 2013. - · arXiv
  9. [9][9] A. Balakrishnan, J. Rao, K. Upasani, M. White, and R. Subba. Constrained decoding for neural NLG from compositional representations in task-oriented dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 831-844, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1080. URL https://aclanthology.org/P19-1080 . - · DOI · Paper page
  10. [10][10] S. Barocas, A. D. Selbst, and M. Raghavan. The hidden assumptions behind counterfactual explanations and principal reasons. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , FAT* '20, page 80-89, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450369367. doi: 10.1145/3351095.3372830. URL https://doi.org/10.1145/3351095.3372830 . - · DOI
  11. [11][11] O. Bastani, C. Kim, and H. Bastani. Interpretability via model extraction. FAT/ML Workshop 2017 , 2017. -
  12. [12][12] V. Bellotti and K. Edwards. Intelligibility and accountability: Human considerations in contextaware systems. Hum.-Comput. Interact. , 16(2):193-212, dec 2001. ISSN 0737-0024. doi: 10.1207/ S15327051HCI16234\_05. URL https://doi.org/10.1207/S15327051HCI16234\_05 . - · DOI
  13. [13][13] A. Bordes, Y.-L. Boureau, and J. Weston. Learning end-to-end goal-oriented dialog. ICLR , 2017. -
  14. [14][14] L. Breiman. Random forests. Machine learning , 45(1):5-32, 2001. -
  15. [15][15] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 18771901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf . -
  16. [16][16] B. Byrne, K. Krishnamoorthi, C. Sankar, A. Neelakantan, B. Goodrich, D. Duckworth, S. Yavuz, A. Dubey, K.-Y. Kim, and A. Cedilnik. Taskmaster-1: Toward a realistic and diverse dialog dataset. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4516-4525, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1459. URL https://aclanthology.org/D19-1459 . - · DOI
  17. [17][17] P. Chalasani, J. Chen, A. R. Chowdhury, X. Wu, and S. Jha. Concise explanations of neural networks using adversarial training. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1383-1391. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/ v119/chalasani20a.html . -
  18. [18][18] C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, and J. K. Su. This looks like that: Deep learning for interpretable image recognition. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/ paper/2019/file/adf7ee2dcf142b0e11888e72b43fcb75-Paper.pdf . -
  19. [19][19] H. Chen, X. Liu, D. Yin, and J. Tang. A survey on dialogue systems: Recent advances and new frontiers. SIGKDD Explor. Newsl. , 19(2):25-35, nov 2017. ISSN 1931-0145. doi: 10.1145/3166054. 3166058. URL https://doi.org/10.1145/3166054.3166058 . - · DOI
  20. [20][20] J. Chen, L. Song, M. J. Wainwright, and M. I. Jordan. L-shapley and c-shapley: Efficient model interpretation for structured data. In International Conference on Learning Representations , 2019. -
  21. [21][21] I. Covert and S.-I. Lee. Improving kernelshap: Practical shapley value estimation using linear regression. In A. Banerjee and K. Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 3457-3465. PMLR, 13-15 Apr 2021. URL https://proceedings.mlr.press/ v130/covert21a.html . -
  22. [22][22] J. Demšar, B. Zupan, G. Leban, and T. Curk. Orange: From experimental machine learning to interactive data mining. In J.-F. Boulicaut, F. Esposito, F. Giannotti, and D. Pedreschi, editors, Knowledge Discovery in Databases: PKDD 2004 , pages 537-539, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg. ISBN 978-3-540-30116-5. -
  23. [23][23] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423 . - · DOI · Paper page
  24. [24][24] B. Dimanov, U. Bhatt, M. Jamnik, and A. Weller. You shouldn't trust me: Learning models which conceal unfairness from multiple explanation methods. In SafeAI@AAAI , 2020. -
  25. [25][25] A.-K. Dombrowski, M. Alber, C. J. Anders, M. Ackermann, K.-R. Müller, and P. Kessel. Explanations can be manipulated and geometry is to blame. arXiv preprint arXiv:1906.07983 , 2019. - · arXiv
  26. [26][26] F. Doshi-Velez and B. Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 , 2017. - · arXiv
  27. [27][27] P. Dourish. Algorithms and their others: Algorithmic culture in context. Big Data & Society , 3 (2):2053951716665128, 2016. doi: 10.1177/2053951716665128. URL https://doi.org/10.1177/ 2053951716665128 . - · DOI · Paper page
  28. [28][28] L. El Asri, H. Schulz, S. Sharma, J. Zumer, J. Harris, E. Fine, R. Mehrotra, and K. Suleman. Frames: a corpus for adding memory to goal-oriented dialogue systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue , pages 207-219, Saarbrücken, Germany, Aug. 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-5526. URL https://aclanthology.org/W17-5526 . - · DOI
  29. [29][29] J. A. Fails and D. R. Olsen. Interactive machine learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces , IUI '03, page 39-45, New York, NY, USA, 2003. Association for Computing Machinery. ISBN 1581135866. doi: 10.1145/604045.604056. URL https://doi.org/10.1145/604045.604056 . - · DOI · Paper page
  30. [30][30] J. H. Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics , pages 1189-1232, 2001. -
  31. [31][31] J. Gao, M. Galley, and L. Li. Neural approaches to conversational AI. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts , pages 2-7, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-5002. URL https://aclanthology.org/P18-5002 . - · DOI · Paper page
  32. [32][32] D. Garreau and U. von Luxburg. Looking deeper into lime. arXiv preprint arXiv:2008.11092 , 2020. - · arXiv
  33. [33][33] A. Ghorbani, A. Abid, and J. Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 3681-3688, 2019. -
  34. [34][34] B. Green and Y. Chen. Disparate interactions: An algorithm-in-the-loop analysis of fairness in risk assessments. In Proceedings of the Conference on Fairness, Accountability, and Transparency , pages 90-99. ACM, 2019. -
  35. [35][35] B. Green and Y. Chen. The principles and limits of algorithm-in-the-loop decision making. Proceedings of the ACM on Human-Computer Interaction , 3(CSCW):50, 2019. -
  36. [36][36] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR) , 51(5):1-42, 2018. -
  37. [37][37] D. Ham, J.-G. Lee, Y. Jang, and K.-E. Kim. End-to-end neural pipeline for goal-oriented dialogue systems using GPT-2. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 583-592, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.54. URL https://aclanthology.org/2020.acl-main.54 . - · DOI
  38. [38][38] H. He and J. D. Choi. Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT. In Proceedings of the 33rd International Florida Artificial Intelligence Research Society Conference , FLAIRS'20, pages 228-233, 2020. URL https://www.flairs-33.info . Best Student Paper Candidate. -
  39. [39][39] M. Henderson. Machine learning for dialog state tracking: A review. 2015. -
  40. [40][40] M. Henderson, B. Thomson, and S. Young. Deep neural network approach for the dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference , pages 467-471, Metz, France, Aug. 2013. Association for Computational Linguistics. URL https://aclanthology. org/W13-4073 . -
  41. [41][41] F. Hohman, A. Head, R. Caruana, R. DeLine, and S. M. Drucker. Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models , page 1-13. Association for Computing Machinery, New York, NY, USA, 2019. ISBN 9781450359702. URL https: //doi.org/10.1145/3290605.3300809 . - · DOI
  42. [42][42] M. Kale and A. Rastogi. Template guided text generation for task oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6505-6520, Online, Nov. 2020. Association for Computational Linguistics. URL https://www. aclweb.org/anthology/2020.emnlp-main.527 . -
  43. [43][43] A.-H. Karimi*, J. von Kügelgen*, B. Schölkopf, and I. Valera. Algorithmic recourse under imperfect causal knowledge: a probabilistic approach. In Advances in Neural Information Processing Systems 33 , Dec. 2020. *equal contribution. -
  44. [44][44] A.-H. Karimi, B. Schölkopf, and I. Valera. Algorithmic recourse: from counterfactual explanations to interventions. In 4th Conference on Fairness, Accountability, and Transparency (ACM FAccT) , Mar. 2021. -
  45. [45][45] H. Kaur, H. Nori, S. Jenkins, R. Caruana, H. Wallach, and J. Wortman Vaughan. Interpreting interpretability: Understanding data scientists' use of interpretability tools for machine learning. In CHI , April 2020. -
  46. [46][46] A. Kim, H. Song, and S. Park. A two-step neural dialog state tracker for task-oriented dialog processing. Computational Intelligence and Neuroscience , 2018, 2018. ISSN 1687-5265. doi: 10.1155/2018/5798684. Funding Information: .is research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (no. NRF-2016R1D1A1B04935678). Funding Information: This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (no. NRF-2016R1D1A1B04935678). Publisher Copyright: Copyright © 2018 A-Yeong Kim et al. - · DOI
  47. [47][47] B. Kim, C. Rudin, and J. Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. arXiv preprint arXiv:1503.01161 , 2015. - · arXiv
  48. [48][48] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, and R. S ayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 2668-2677, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. -
  49. [49][49] I. Lage, E. Chen, J. He, M. Narayanan, B. Kim, S. Gershman, and F. Doshi-Velez. An evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1902.00006 , 2019. - · arXiv
  50. [50][50] V. Lai and C. Tan. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency , pages 29-38, 2019. -
  51. [51][51] V. Lai, J. Z. Cai, and C. Tan. Many faces of feature importance: Comparing built-in and post-hoc feature importance in text classification. In Proceedings of EMNLP , 2019. -
  52. [52][52] V. Lai, H. Liu, and C. Tan. " why is' chicago'deceptive?" towards building model-driven tutorials for humans. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , pages 1-13, 2020. -
  53. [53][53] H. Lakkaraju, S. H. Bach, and J. Leskovec. Interpretable decision sets: A joint framework for description and prediction. In Knowledge Discovery and Data mining (KDD) , 2016. -
  54. [54][54] H. Le, D. Sahoo, C. Liu, N. Chen, and S. C. Hoi. UniConv: A unified conversational neural architecture for multi-domain task-oriented dialogues. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1860-1877, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.146. URL https://aclanthology.org/2020.emnlp-main.146 . - · DOI
  55. [56][56] A. Levine, S. Singla, and S. Feizi. Certifiably robust interpretation in deep learning. arXiv preprint arXiv:1905.12105 , 2019. - · arXiv
  56. [57][57] O. Li, H. Liu, C. Chen, and C. Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In AAAI , 2018. -
  57. [58][58] Q. V. Liao, D. Gruen, and S. Miller. Questioning the ai: informing design practices for explainable ai user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , pages 1-15, 2020. -
  58. [59][59] Z. C. Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue , 16(3):31-57, 2018. -
  59. [60][60] H. Liu, V. Lai, and C. Tan. Understanding the effect of out-of-distribution examples and interactive explanations on human-ai decision making. arXiv preprint arXiv:2101.05303 , 2021. - · arXiv
  60. [61][61] R. Logan, N. F. Liu, M. E. Peters, M. Gardner, and S. Singh. Barack's wife hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 5962-5971, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1598. URL https://aclanthology.org/ P19-1598 . - · DOI · Paper page
  61. [62][62] Y. Lou, R. Caruana, J. Gehrke, and G. Hooker. Accurate intelligible models with pairwise interactions. In KDD , 2013. -
  62. [63][63] S. M. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems , pages 4765-4774, 2017. -
  63. [64][64] J. Ni, T. Young, V. Pandelea, F. Xue, V. A. K. Adiga, and E. Cambria. Recent advances in deep learning based dialogue systems: A systematic survey. ArXiv , abs/2105.04387, 2021. - · arXiv
  64. [65][65] O. Oluwatobi and E. Mueller. DLGNet: A transformer-based model for dialogue response generation. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI , pages 54-62, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.nlp4convai-1.7. URL https://aclanthology.org/2020.nlp4convai-1.7 . - · DOI
  65. [66][66] T. Paek and E. Horvitz. Conversation as action under uncertainty. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence , UAI'00, page 455-464, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607099. -
  66. [67][67] D. Peskov, N. Clarke, J. Krone, B. Fodor, Y. Zhang, A. Youssef, and M. Diab. Multi-domain goaloriented dialogues (MultiDoGO): Strategies toward curating and annotating large scale dialogue data. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4526-4536, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1460. URL https://aclanthology.org/D19-1460 . - · DOI
  67. [68][68] F. Poursabzi-Sangdeh, D. G. Goldstein, J. M. Hofman, J. W. Vaughan, and H. Wallach. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 , 2018. - · arXiv
  68. [69][69] F. Poursabzi-Sangdeh, D. G. Goldstein, J. M. Hofman, J. W. Vaughan, and H. M. Wallach. Manipulating and measuring model interpretability. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , 2021. -
  69. [70][70] R. Poyiadzi, K. Sokol, R. Santos-Rodriguez, T. De Bie, and P. Flach. Face: feasible and actionable counterfactual explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , pages 344-350, 2020. -
  70. [71][71] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020. URL http://jmlr.org/papers/v21/20-074.html . -
  71. [72][72] A. Rastogi, X. Zang, S. Sunkara, R. Gupta, and P. Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. arXiv preprint arXiv:1909.05855 , 2019. - · arXiv
  72. [73][73] M. T. Ribeiro, S. Singh, and C. Guestrin. Why Should I Trust You? explaining the predictions of any classifier. In Knowledge Discovery and Data mining (KDD) , 2016. -
  73. [74][74] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence , 1(5):206, 2019. -
  74. [75][75] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV , 2017. -
  75. [76][76] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations , 2014. -
  76. [77][77] K. Singla, Z. Chen, D. Atkins, and S. Narayanan. Towards end-2-end learning for predicting behavior codes from spoken utterances in psychotherapy conversations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 3797-3803, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.351. URL https://aclanthology.org/2020.acl-main.351 . - · DOI
  77. [78][78] D. Slack, S. A. Friedler, C. E. Scheidegger, and C. D. Roy. Assessing the local interpretability of machine learning models. Workshop on Human-Centric Machine Learning, NeurIPS , 2019. -
  78. [79][79] D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. Conference on Artificial Intelligence, Ethics, and Society (AIES) , 2020. -
  79. [80][80] D. Slack, S. Hilgard, S. Singh, and H. Lakkaraju. Reliable Post hoc Explanations Modeling Uncertainty in Explainability. In Neural Information Processing Systems (NeurIPS) , 2021. -
  80. [81][81] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg. Smoothgrad: removing noise by adding noise. Workshop on Visualization for Deep Learning, ICML , 2017. -
  81. [82][82] M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 3319-3328. JMLR. org, 2017. -
  82. [83][83] S. Tan, R. Caruana, G. Hooker, and Y. Lou. Distill-and-compare: Auditing black-box models using transparent model distillation. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pages 303-310, 2018. -
  83. [84][84] B. Ustun, S. Traca, and C. Rudin. Supersparse linear integer models for interpretable classification. arXiv preprint arXiv:1306.6677 , 2013. - · arXiv
  84. [85][85] B. Ustun, A. Spangher, and Y. Liu. Actionable recourse in linear classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency , FAT* '19, pages 10-19, 2019. ISBN 978-1-4503-6125-5. -
  85. [86][86] A. Van Looveren and J. Klaise. Interpretable Counterfactual Explanations Guided by Prototypes. arXiv , art. arXiv:1907.02584, July 2019. - · arXiv
  86. [88][88] B. C. Wallace, K. Small, C. E. Brodley, J. Lau, and T. A. Trikalinos. Deploying an interactive machine learning system in an evidence-based practice center: Abstrackr. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium , IHI '12, page 819-824, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781450307819. doi: 10.1145/2110363.2110464. URL https://doi.org/10.1145/2110363.2110464 . - · DOI · Paper page
  87. [89][89] J. Wang, J. Tuyls, E. Wallace, and S. Singh. Gradient-based Analysis of NLP Models is Manipulable. In Findings of the Association for Computational Linguistics: EMNLP (EMNLP Findings) , page 247-258, 2020. -
  88. [90][90] W. Wang, J. Zhang, Q. Li, M.-Y. Hwang, C. Zong, and Z. Li. Incremental learning from scratch for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 3710-3720, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1361. URL https://aclanthology.org/ P19-1361 . - · DOI
  89. [91][91] X. Wang and M. Yin. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26rd International Conference on Intelligent User Interfaces , 2021. -
  90. [92][92] X. Wang and C. Yuan. Recent advances on human-computer dialogue. CAAI Transactions on Intelligence Technology , 1(4):303-312, 2016. ISSN 2468-2322. doi: https://doi.org/10.1016/j.trit.2016. 12.004. URL https://www.sciencedirect.com/science/article/pii/S2468232216301081 . - · DOI
  91. [93][93] J. Williams, A. Raux, D. Ramachandran, and A. Black. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference , pages 404-413, Metz, France, Aug. 2013. Association for Computational Linguistics. URL https://aclanthology.org/W13-4065 . -
  92. [94][94] T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev. Spider: A large-scale human-labeled dataset for complex and crossdomain semantic parsing and text-to-sql task. In "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing" , "Brussels, Belgium", 2018. "Association for Computational Linguistics". -
  93. [95][95] T. Yu, R. Zhang, H. Er, S. Li, E. Xue, B. Pang, X. V. Lin, Y. C. Tan, T. Shi, Z. Li, Y. Jiang, M. Yasunaga, S. Shim, T. Chen, A. Fabbri, Z. Li, L. Chen, Y. Zhang, S. Dixit, V. Zhang, C. Xiong, R. Socher, W. Lasecki, and D. Radev. CoSQL: A conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 1962-1979, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1204. URL https://aclanthology.org/D19-1204 . - · DOI
  94. [96][96] M. B. Zafar, M. Donini, D. Slack, C. Archambeau, S. Das, and K. Kenthapadi. On the lack of robust interpretability of neural text classifiers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 3730-3740, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-acl.327. URL https: //aclanthology.org/2021.findings-acl.327 . - · DOI
  95. [97][97] M. Zaib, Q. Z. Sheng, and W. Emma Zhang. A short survey of pre-trained language models for conversational ai-a new age in nlp. In Proceedings of the Australasian Computer Science Week Multiconference , ACSW '20, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450376976. doi: 10.1145/3373017.3373028. URL https://doi.org/10.1145/3373017. 3373028 . - · DOI
  96. [98][98] Y. Zhang, S. Sun, M. Galley, Y.-C. Chen, C. Brockett, X. Gao, J. Gao, J. J. Liu, and B. Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. In arXiv:1911.00536 , November 2019. URL https://www.microsoft.com/en-us/research/publication/ dialogpt-large-scale-generative-pre-training-for-conversational-response-generation/ . - · arXiv
  97. [99][99] Y. Zhang, Q. V. Liao, and R. K. Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. arXiv preprint arXiv:2001.02114 , 2020. - · arXiv
  98. [100][100] V. Zhong, C. Xiong, and R. Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR , abs/1709.00103, 2017. - · arXiv
  99. [101][101] Q. Zhu, K. Huang, Z. Zhang, X. Zhu, and M. Huang. CrossWOZ: A large-scale Chinese cross-domain task-oriented dialogue dataset. Transactions of the Association for Computational Linguistics , 8:281-295, 2020. doi: 10.1162/tacl\_a\_00314. URL https://aclanthology.org/ 2020.tacl-1.19 . · DOI

Authors
Lakkaraju, H., Slack, D., Chen, Y., Tan, C., & Singh, S.
Affiliations
  • Harvard University (US)
  • University of California, Irvine (US)
  • University of Chicago (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2202.01875
arXiv
2202.01875
OpenAlex
W4221160814
Cited by
62
References
0
Citations by year
2026: 4 · 2025: 14 · 2024: 23 · 2023: 16 · 2022: 5
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
explainable machine learning, interactive explanations, natural language dialogue, human-computer dialogue, practitioner perspectives, healthcare decision-makers, policy makers, dialogue system design, stakeholder needs

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review