The DOI matches its Crossref record; 112 of the 115 references it cites were found in a registry or an index; none of the 45 sources its claims cite is checked for support yet.
Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
Open accessAbstract
Understanding how ML models work is a prerequisite for responsibly designing, deploying, and using ML-based systems. With interpretability approaches, ML can now offer explanations for its outputs to aid human understanding. Though these approaches rely on guidelines for how humans explain things to each other, they ultimately solve for improving the artifact—an explanation. In this paper, we propose an alternate framework for interpretability grounded in Weick’s sensemaking theory, which focuses on who the explanation is intended for. Recent work has advocated for the importance of understanding stakeholders’ needs—we build on this by providing concrete properties (e.g., identity, social context, environmental cues, etc.) that shape human understanding. We use an application of sensemaking in organizations as a template for discussing design guidelines for sensible AI, AI that factors in the nuances of human cognition when trying to explain itself.
Overview
CoChat Summary
TL;DR
The paper argues that current interpretability and explainability approaches focus on improving explanations as artifacts and therefore fall short of helping people reliably use ML-based systems. It proposes grounding interpretability in Weick’s sensemaking theory and introduces Sensible AI, a framework using seven sensemaking properties and high-reliability organization principles as design guidelines.
Key findings
- Current interpretability and explainability solutions “will always fall short” because they focus on designing better explanations rather than understanding how humans make sense of complex, sometimes conflicting information; sensemaking subsumes interpretability and includes contextual factors such as identity, social and organizational context, and prior experience.
- The paper translates Weick’s seven sensemaking properties—identity construction, social, retrospective, enactive, ongoing, focused on extracted cues, and plausibility over accuracy—into claims for the human-machine context, e.g., people internalize explanations that support their identity, social context affects explanation effectiveness, providing explanations before reflection negatively affects sensemaking, explanation order affects understanding, interruptions and emotion change what is understood, highlighting different parts of explanations leads to varying understanding, and plausible explanations reduce the search for accurate ones.
- It proposes Sensible AI design guidelines inspired by high-reliability organizations: seamful design, inducing skepticism, adversarial design, and continuous monitoring and feedback, drawing on HRO principles such as preoccupation with failure, reluctance to simplify, sensitivity to operations, commitment to resilience, and deference to expertise.
- No original empirical numbers are reported; cited reviews include 289 core papers and 12,412 citing papers, and 409 papers taxonomized, but these are prior work, not the paper’s own findings.
Methods
- Conceptual/theoretical paper with no empirical study, dataset, or sample; it reviews interpretability and explainability literature and translates Weick’s sensemaking theory from organizational studies to the human-machine context.
- Uses an application of sensemaking in organizations and high-reliability organizations as a template, presenting example vignettes for each sensemaking property (e.g., healthcare diagnosis, content moderation, diabetic retinopathy, predictive policing, hiring, radiology).
- Analysis is argumentative and framework-building: it defines the seven properties, derives claims for human-machine sensemaking, and proposes design guidelines for Sensible AI.
Limitations
- The paper states the example vignettes are crafted based on popular press articles and research papers, are not intended as representative, and do not claim a causal relationship between the property and the example; other reasons for the described system behavior are possible.
- The paper presents “initial design ideas” and a “research agenda”; no empirical validation, user study, deployment, or evaluation of Sensible AI is reported (evident from the text).
- The paper notes there is not yet consensus on how interpretability and explainability differ from an implementation point of view, and it uses “interpretability” for the rest of the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 1 Left: DARPA's conceptualization of Explainable AI, adapted from [35]. Right: Weick's sensemaking properties (1-7) categorized using the high-level Enactment-Selection-Retention organizational model, adapted from [49]. Enactment includes properties about perceiving and acting on environmental changes; Selection, properties related to interpreting what the changes mean; and Retention, properties that describe storing and using prior experiences [56]. Our Sensible AI framework extends the existing definition of interpretability and explainability to include Weick's sensemaking properties.](/paper-figures/0d084958fd6844c697ff6f4fc4d43b5c/sensible-ai-re-imagining-interpretability-and-explainability-using-sensemaking-figure-1.png)
![Fig. 2 Saliency maps for chest radiographs, adapted from [20].](/paper-figures/9b14734865984f4ca91dd08ba4d8c9e3/sensible-ai-re-imagining-interpretability-and-explainability-using-sensemaking-figure-2.png)
Verification
Claims the paper makes
0 Valid · 3 Not valid · 30 Not checked · 26 Own results and interpretations
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If the police officers were also provided an explanation for the model's predictions, the type of explanation and the order in which they are seen (e.g., global vs. local explanation first) changes the enacted environment for the officers.”
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > 3.4.2 Enactment in the Human-Machine Context.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
In the paper
“For example, accurate mental models and deliberative reasoning can help avoid ML practitioners' misuse of, and over-reliance on, interpretability outputs [52].”
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > 2.2 Understanding the "who" in interpretability and explainability
What the sources say
- Not matchedcites [52]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“Explanation interfaces that are interactive and collaborative can improve overall accuracy [108].”
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > 2.2 Understanding the "who" in interpretability and explainability
What the sources say
- Not matchedcites [108]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because sensemaking is sensitive to enacted environments, it is important to remember that any information or explanation about the model is not treated by people as static or isolated.”
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > 3.4.2 Enactment in the Human-Machine Context.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Weick notes the work on socially shared cognition (e.g., [62,91]) which shows that human cognition and social functioning are essential to each other.”
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > 3.2.1 Social Elements of Sensemaking in the
What the sources say
- Not matchedcites [62]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [91]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
22 more claims
Create a free account to see all 59 claims and every source verdict.
- Probably not a substantial claim. 20
- It may state two separate claims as one. 16
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > Abstract
“Weick's sensemaking is more procedural: "placement of items into frameworks, comprehending, redressing surprise, constructing meaning, interacting in pursuit of mutual understanding, and patterning" [118, p.6].”
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > References
“118. K.E. Weick, K.M. Sutcliffe, and D. Obstfeld. 1999. Organizing for high reliability: Processes of Collective Mindfulness. Research in Organizational Behaviour 21 (1999), 81-123.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > 3.2.1 Social Elements of Sensemaking in the
“recall Weick's reference to Mead's work on the individual being "a parliament of selves" [74]”
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > References
“74. George Mandler. 1984. Mind and body: Psychology of emotion and stress. WW Norton & Company Incorporated.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > 3.3.1 Retrospective Sensemaking in the Human-Human Context.
“"as I begin the latter portion of a long word, my utterance of the first part is already in the past" [36, p.44].”
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > References
“36. C. Hartshorne. 1962. Mind as Memory and Creative Love. In Theories of the Mind, Jordan M. Scher (Ed.). Free Press of Glencoe, 440-463.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > 3.5.1 Sensemaking as an Ongoing Activity in the
“Winograd and Flores describe these situations as having the following properties: (1) you cannot avoid acting; (2) you cannot step back and reflect on your actions, i.e., you have to rely on your intuitions; (3) the effects of action cannot be predicted;”
§ Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory > References
“123. Mark Wenman. 2013. Agonistic democracy: Constituent power in the era of globalisation. Cambridge University Press.”
Citations
Cites
Qualitative Data Analysis: An Expanded Sourcebook
(1994) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(05)80231-2 · Cited by 48,420
James, W. (1913) · Henry Holt and Company eBooks · doi:10.1037/11059-000 · Cited by 25,226
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
Quinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Gioia, D. A., Mehra, A., & Weick, K. E. (1996) · The Academy of Management Review · doi:10.2307/259169 · Cited by 15,614
Generalized Additive Models: An Introduction with R
Wood, S. N. (2017) · doi:10.1201/9781315370279 · Cited by 13,852
Telling more than we can know: Verbal reports on mental processes.
Nisbett, R. E., & Wilson, T. D. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.231 · Cited by 11,317
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Koyak, R. A., Hastie, T. J., & Tibshirani, R. J. (1991) · Journal of the American Statistical Association · doi:10.2307/2290538 · Cited by 7,596
Organizing and the process of sensemaking
Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. (2005) · Organization Science · doi:10.1287/orsc.1050.0133 · Cited by 6,943
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Reason, J. (1990) · doi:10.1017/cbo9781139062367 · Cited by 5,108
The phenomenology of the social world
Bittner, E., Schutz, A., Walsh, G., & Lehnert, F. (1968) · American Sociological Review · doi:10.2307/2092457 · Cited by 3,588
Perspectives on socially shared cognition.
(1991) · American Psychological Association eBooks · doi:10.1037/10096-000 · Cited by 3,446
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Studies in the Logic of Explanation
Hempel, C. G., & Oppenheim, P. (1948) · Philosophy of Science · doi:10.1086/286983 · Cited by 3,174
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Asymmetrical effects of positive and negative events – the mobilization minimization hypothesis.
Taylor, S. E. (1991) · Psychological Bulletin · doi:10.1037//0033-2909.110.1.67 · Cited by 2,387
Groups of diverse problem solvers can outperform groups of high-ability problem solvers
Hong, L., & Page, S. E. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0403723101 · Cited by 2,103
Agonistics: Thinking the World Politically
Mouffe, C. (2013) · Medical Entomology and Zoology · Cited by 1,978
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 2×
“, [18, 37, 50, 58, 89, 124]) or post-hoc explanations for blackbox models (e.”
BackgroundDARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360
How to Make Cognitive Illusions Disappear: Beyond “Heuristics and Biases”
Gigerenzer, G. (1991) · European Review of Social Psychology · doi:10.1080/14792779143000033 · Cited by 1,159
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084
The Embodied Mind: Cognitive Science and Human Experience
Dennett, D. C., Varela, F. J., Thompson, E., & Rosch, E. (1993) · The American Journal of Psychology · doi:10.2307/1422869 · Cited by 1,079
Counterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061
Understanding computers and cognition: A new foundation for design
Suchman, L. A. (1987) · Artificial Intelligence · doi:10.1016/0004-3702(87)90024-5 · Cited by 1,055
Sengers, P., Boehner, K., David, S., & Kaye, J. '. (2005) · Proceedings of the 4th decennial conference on Critical computing: between sense and sensibility · doi:10.1145/1094562.1094569 · Cited by 1,030
Some Characteristics of One Type of High Reliability Organization
Roberts, K. H. (1990) · Organization Science · doi:10.1287/orsc.1.2.160 · Cited by 934
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
(2015) · doi:10.1002/9781119175834 · Cited by 864
Advances in Experimental Social Psychology, Vol. 7.
Fink, E. L., & Berkowitz, L. (1976) · Contemporary Sociology · doi:10.2307/2062988 · Cited by 819
A systematic review of algorithm aversion in augmented decision making
Burton, J. W., Stein, M., & Jensen, T. B. (2020) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2155 · Cited by 814
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
Mind and Body: Psychology of Emotion and Stress
Mandler, G. (1984) · Medical Entomology and Zoology · Cited by 766
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 5×
“Human evaluations of interpretability tools show that this confirmatory use of explanations is often the case, even when explanations reveal issues with the underlying model [10, 16, 52].”
BackgroundInfluentialDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743
Culture, Self-Identity, and Work
Erez, M., & Earley, P. C. (1993) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195075809.001.0001 · Cited by 742
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728
The cost structure of sensemaking
Russell, D. M., Stefik, M. J., Pirolli, P., & Card, S. K. (1993) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 · doi:10.1145/169059.169209 · Cited by 726
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687
The structure and function of explanations
Lombrozo, T. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.08.004 · Cited by 639
AI for radiographic COVID-19 detection selects shortcuts over signal
DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00338-7 · Cited by 631
Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614
Executives ‘ Perceptual Filters: What They Notice and How They Make Sense
Milliken, F. J. (2006) · Organizational Realities · doi:10.1093/oso/9780199288519.003.0009 · Cited by 584
Production Rules as a Representation for a Knowledge-Based Consultation Program
Davis, R., Buchanan, B., & Shortliffe, E. (1977) · Artificial Intelligence · doi:10.1016/0004-3702(77)90003-0 · Cited by 569
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564
Self-citationMinding Matters: The Consequences of Mindlessness–Mindfulness
Langer, E. J. (1989) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60307-x · Cited by 560
Social Foundations of Cognition
Levine, J. M., Resnick, L. B., & Higgins, E. T. (1993) · Annual Review of Psychology · doi:10.1146/annurev.ps.44.020193.003101 · Cited by 558
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538 · cited 1×
“, algorithmic aversion [17, 21]), the socio-organizational context [26], have been observed as being critical to understanding AI and ML outputs.”
BackgroundWILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · cited 2×
“When ML-based systems are deployed in real-world settings, changes in data collection and distributional drifts are a given [55].”
BackgroundVeale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 1×
“For example, [42, 45, 70, 113, 126] all highlight the challenges of operating within an organization that either develops or employs an AI-based system.”
BackgroundAlgorithms and their others: Algorithmic culture in context
Dourish, P. (2016) · Big Data & Society · doi:10.1177/2053951716665128 · Cited by 481
Thinking is for doing: Portraits of social cognition from Daguerreotype to laserphoto.
Fiske, S. T. (1992) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.63.6.877 · Cited by 476
Intelligibility and Accountability: Human Considerations in Context-Aware Systems
Bellotti, V., & Edwards, K. (2001) · Human–Computer Interaction · doi:10.1207/s15327051hci16234_05 · Cited by 471
Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436
How the mind explains behavior: folk explanations, meaning, and social interaction
(2005) · Choice Reviews Online · doi:10.5860/choice.42-6170 · Cited by 436
Heuristics and biases: Expertise and task realism in auditing.
Smith, J. F., & Kida, T. (1991) · Psychological Bulletin · doi:10.1037//0033-2909.109.3.472 · Cited by 436
XPLAIN: a system for creating and explaining expert consulting programs
Swartout, W. R. (1983) · Artificial Intelligence · doi:10.1016/s0004-3702(83)80014-9 · Cited by 393
Too much, too little, or just right? Ways explanations impact end users' mental models
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K. (2013) · 2013 IEEE Symposium on Visual Languages and Human Centric Computing · doi:10.1109/vlhcc.2013.6645235 · Cited by 388
Organizational Mindfulness and Mindful Organizing: A Reconciliation and Path Forward
Vogus, T. J., & Sutcliffe, K. M. (2012) · Academy of Management Learning & Education · doi:10.5465/amle.2011.0002c · Cited by 365
Staw, B. M. (1975) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(75)90060-4 · Cited by 358
Lipton, P. (1990) · Royal Institute of Philosophy Supplement · doi:10.1017/s1358246100005130 · Cited by 281
Kocielnik, R., Amershi, S., & Bennett, P. N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300641 · Cited by 277
Explainable AI for Designers: A Human-Centered Perspective on Mixed-Initiative Co-Creation
Zhu, J., Liapis, A., Risi, S., Bidarra, R., & Youngblood, G. M. (2018) · 2018 IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2018.8490433 · Cited by 255
The challenge of crafting intelligible intelligence
Weld, D. S., & Bansal, G. (2019) · Communications of the ACM · doi:10.1145/3282486 · Cited by 253 · cited 1×
“, [12, 25, 32, 82, 83, 121, 126]) with the motivation that by translating ideas from how people explain things to each other, we can design better solutions for how ML models can be explained to people.”
BackgroundGamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models
Hohman, F., Head, A., Caruana, R., DeLine, R. A., & Drucker, S. M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300809 · Cited by 252
Explanation and abductive inference
Lombrozo, T. (2012) · The Oxford Handbook of Thinking and Reasoning · doi:10.1093/oxfordhb/9780199734689.013.0014 · Cited by 242
Passi, S., & Jackson, S. J. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274405 · Cited by 225
Interpretable classification models for recidivism prediction
Zeng, J., Ustun, B., & Rudin, C. (2017) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12227 · Cited by 220
Human factors in model interpretability: Industry practices, challenges, and needs
Hong, S. R., Hullman, J., & Bertini, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392878 · Cited by 219
Miller, T., Howe, P., & Sonenberg, L. (2017) · arXiv · doi:10.48550/arxiv.1712.00547 · Cited by 218
Lomborg, S., & Kapsch, P. H. (2019) · Media Culture & Society · doi:10.1177/0163443719855301 · Cited by 194
Contrastive explanation: a structural-model approach
Miller, T. (2021) · The Knowledge Engineering Review · doi:10.1017/s0269888921000102 · Cited by 186
Interacting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184
Jury Learning: Integrating Dissenting Voices into Machine Learning Models
Gordon, M., Lam, M. S., Park, J.-S., Patel, K., Hancock, J. T., Hashimoto, T., & Bernstein, M. S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502004 · Cited by 180
A causal framework for explaining the predictions of black-box sequence-to-sequence models
Alvarez-Melis, D., & Jaakkola, T. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1042 · Cited by 178
Knorr-Cetina, K. D. (1981) · Cited by 166
Moods and Memories: Elation, depression, and the remembering of the events of one's life
Snyder, M., & White, P. (1982) · Journal of Personality · doi:10.1111/j.1467-6494.1982.tb01020.x · Cited by 153
Human-Centred Machine Learning
Gillies, M. F. P., Fiebrink, R. A., Tanaka, A., Garcia, J., Bevilacqua, F. P., Héloir, A., Nunnari, F., Mackay, W. E., Amershi, S., Lee, B., et al. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2856492 · Cited by 151
Agonistic democracy: Constituent power in the era of globalisation
Patberg, M. (2014) · Contemporary Political Theory · doi:10.1057/cpt.2014.2 · Cited by 146
Some Observations on Mental Models
Norman, D. A. (2014) · Psychology Press eBooks · doi:10.4324/9781315802725-5 · Cited by 139
Berg, W. M., & Leiter, K. (1983) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2069019 · Cited by 134
Mental Models and Causal Explanation: Judgements of Probable Cause and Explanatory Relevance
Hilton, D. (1996) · Thinking & Reasoning · doi:10.1080/135467896394447 · Cited by 125
Perspectives on Socially Shared Cognition.
Cradock, R. M., Resnick, L. Β., Levine, J. M., & Teasley, S. D. (1992) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2075588 · Cited by 125
Button, G. (1981) · Sociology · doi:10.1177/003803858101500325 · Cited by 124
“Duality of Structure” and “Intentionality” in an Ecological Psychology
Shotter, J. (1983) · Journal for the Theory of Social Behaviour · doi:10.1111/j.1468-5914.1983.tb00460.x · Cited by 123
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123
Fragments on the phenomenology of music
Schütz, A., & Kersten, F. (1976) · Music and Man · doi:10.1080/01411897608574487 · Cited by 115
Simple Rules for Complex DecisionsPreprint
Jung, J., Concannon, C., Shroff, R., Goel, S., & Goldstein, D. G. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2919024 · Cited by 109
"Beautiful Seams": Strategic Revelations and Concealments
Inman, S., & Ribes, D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300508 · Cited by 88
To Live in Their Utopia: Why Algorithmic Systems Create Absurd Outcomes
Alkhatib, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445740 · Cited by 81
Attribution in conversational context: Effect of mutual knowledge on explanation‐giving
Slugoski, B. R., Lalljee, M., Lamb, R., & Ginsburg, G. P. (1993) · European Journal of Social Psychology · doi:10.1002/ejsp.2420230302 · Cited by 79
Glynn, M. A., & Watkiss, L. (2020) · Journal of Management Studies · doi:10.1111/joms.12613 · Cited by 78
Towards relatable explainable AI with the perceptual process
Zhang, W., & Lim, B. Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501826 · Cited by 67
Computational Red Teaming: Past, Present and Future
Abbass, H. A., Bender, A., Gaidow, S., & Whitbread, P. (2011) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2010.939578 · Cited by 65
Goal-based explanation evaluation
Leake, D. B. (1991) · Cognitive Science · doi:10.1207/s15516709cog1504_2 · Cited by 58
Kudesia, R. S. (2016) · Oxford University Press eBooks · doi:10.1093/acrefore/9780190236557.013.78 · Cited by 34
Adams, B. N., & Sydie, R. A. (2001) · Sage eBooks · doi:10.4135/9781452233925.n20 · Cited by 30
Who Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?
Makinen, S., Skogstrom, H., Laaksonen, E., & Mikkonen, T. (2021) · 2021 IEEE/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN) · doi:10.1109/wain52551.2021.00024 · Cited by 28
From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence
Alvarez-Melis, D., Kaur, H., Daumé, H., Wallach, H., & Vaughan, J. W. (2021) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v9i1.18938 · Cited by 27
Self-citationFormal and informal Dimensions of Transactions
Ring, P. S., & de Ven, A. H. V. (2000) · Oxford University Press eBooks · doi:10.1093/oso/9780195139761.003.0006 · Cited by 25
The Contributions of Diversity, Accuracy, and Group Size on Collective AccuracyPreprint
Hong, L., & Page, S. E. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3712299 · Cited by 18
A Pragmatic Theory of Explanation
Tuomela, R. (1984) · Springer eBooks · doi:10.1007/978-94-009-6317-7_10 · Cited by 12
Prange, W. W., Jowett, D., & Fogel, B. (2019) · Routledge eBooks · doi:10.4324/9780367274832-12 · Cited by 10
Research works printed but not linked: 16
- Gordon W Allport. 1985. The historical background of social psychology (Vol. 1). The handbook of social psychology(1985).
- 10.18653/v1 · doi:10.18653/v1
- Ellen Berscheid. 1983. Emotion. In Close Relationships, H.H. Kelley, E. Berscheid, A. Christensen, J. Harvey, T. Huston, G. Levinger, E. McClintock, A. Peplau, and D.R. Peterson (Eds.). WH Freeman, 110–168.
- Wilhelm Dilthey and Frederic Jameson. 1972. The rise of hermeneutics. New literary history 3, 2 (1972), 229–244.
- C. Hartshorne. 1962. Mind as Memory and Creative Love. In Theories of the Mind, Jordan M. Scher (Ed.). Free Press of Glencoe, 440–463.
- Will Douglas Heaven. 2020. Predictive policing algorithms are racist. They need to be dismantled. MIT Technology Review 17(2020), 2020.
- Daniel J Isenberg. 1986. The structure and process of understanding: Implications for managerial action. In The Thinking Organization, H.P. Sims Jr. and D.A. Gioia (Eds.). Jossey Bass, San Fransisco, 238–262.
- P Devereaux Jennings and Royston Greenwood. 2003. Constructing the iron cage: Institutional theory and enactment. Debating organization: point-counterpoint in organization studies 195 (2003).
- Jeremy Kahn. 2021. HireVue drops facial monitoring amid A.I. algorithm audit. Fortune (2021).
- George Herbert Mead. 1934. Mind, self and society. Vol. 111. Chicago University of Chicago Press.
- Charles Sanders Peirce. 1878. Illustrations of the Logic of Science: IV The Probability of Induction. Popular Science Monthly 12 (Apr 1878), 705–718.
- Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis, Vol. 5. McLean, VA, USA, 2–4.
- Joseph C Pitt. 1988. Theories of explanation. Oxford University Press.
- Louis R Pondy and Ian I Mitroff. 1979. Beyond open system models of organization. Research in organizational behavior 1, 1 (1979), 3–39.
- Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. In ICCV.
- K.E. Weick, K.M. Sutcliffe, and D. Obstfeld. 1999. Organizing for high reliability: Processes of Collective Mindfulness. Research in Organizational Behaviour 21 (1999), 81–123.
Cited by
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623261 · Cited by 256
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172
Liao, Q. V., Subramonyam, H., Wang, J., & Vaughan, J. W. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580652 · Cited by 121
Harnessing Biomedical Literature to Calibrate Clinicians’ Trust in AI Decision Support Systems
Yang, Q., Hao, Y., Quan, K., Yang, S., Zhao, Y., Kuleshov, V., & Wang, F. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581393 · Cited by 91
Metadata
- Authors
- Kaur, H., Adar, E., Gilbert, E., & Lampe, C.
- Affiliations
- University of Michigan (US)
- University of Michigan, USA
- Venue
- 2022 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 702-714
- OpenAlex
- W4280636581
- Semantic Scholar
- 2ae9a9ec0c6a614d5bc522e64983cded8e6f4fb9
- Cited by
- 92
- References
- 115
- Citations by year
- 2026: 27 · 2025: 23 · 2024: 24 · 2023: 16 · 2022: 1
- FWCI
- 6.71
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Google (PhD Fellowship)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- explainable AI, sensemaking theory, Weick, stakeholder needs, social environment, environmental cues, human cognition, responsible AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
