CoChat
Sign Up
PreprintCited by 2

Roadmap of Designing Cognitive Metrics for Explainable Artificial Intelligence (XAI)

Hsiao, Janet Hui-wen, Ngai, Hilary Hei Ting, Qiu, Luyu, Yang, Yi, Cao, Caleb Chen·2021·arXiv
DOI 10.48550/arxiv.2108.01737arXiv 2108.01737
Open this paper in CoChat — free account

Abstract

More recently, Explainable Artificial Intelligence (XAI) research has shifted to focus on a more pragmatic or naturalistic account of understanding, that is, whether the stakeholders understand the explanation. This point is especially important for research on evaluation methods for XAI systems. Thus, another direction where XAI research can benefit significantly from cognitive science and psychology research is ways to measure understanding of users, responses and attitudes. These measures can be used to quantify explanation quality and as feedback to the XAI system to improve the explanations. The current report aims to propose suitable metrics for evaluating XAI systems from the perspective of the cognitive states and processes of stakeholders. We elaborate on 7 dimensions, i.e., goodness, satisfaction, user understanding, curiosity & engagement, trust & reliance, controllability & interactivity, and learning curve & productivity, together with the recommended subjective and objective psychological measures. We then provide more details about how we can use the recommended measures to evaluate a visual classification XAI system according to the recommended cognitive metrics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2108.01737

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The required auditability property of Trustworthy AI systems demands the development of practical tools that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability, as well as metrics beyond explainability, such as on traceability, data quality and integrity.

    “The required auditability property of Trustworthy AI systems demands the development of practical tools [109] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [110], as well as metrics beyond explainability [111-115], such as on traceability, data quality and integrity.”

    From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [112]

    Table 1 in the System Controllability / Interactivity section proposes XAI metrics beyond explanation goodness, including user trust, understanding, and performance (page unavailable). It does not address metrics for traceability, data quality, or integrity. The claims about practical verification tools and neural-network properties carry separate citations [109] and [110], so they are not attributed to this source. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 114

  1. Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. arXiv preprint arXiv:1810.03292. · arXiv
  2. Alqaraawi, A., Schuessler, M., Weis, P., Costanza, E., Berthouze, N. (2020). Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study. In Proceedings of the 25th International Conference on Intelligent User Interfaces IUI'20, 275-285.
  3. Argyropoulos, S. V., Hicks, J. A., Nash, J. R., Bell, C. J., Rich, A. S., Nutt, D. J., & Bommer, W. H., Johnson, J. L., Rich, G. A., Podsakoff, P. M., & MacKenzie, S. B. (1995). On the interchangeability of objective and subjective measures of employee performance: A meta- analysis. Personnel psychology, 48(3), 587-605.
  4. Ashoori, M., & Weisz, J. D. (2019). In AI we trust? Factors that influence trustworthiness of AI- infused decision-making processes. arXiv preprint arXiv:1912.02675. · arXiv
  5. Back, M. D., Schmukle, S. C., & Egloff, B. (2009). Predicting actual behavior from the explicit and implicit self-concept of personality. Journal of personality and social psychology, 97(3), 533.
  6. Barral, O., Eugster, M. J., Ruotsalo, T., Spape, M. M., Kosunen, I., Ravaja, N., Kaski, S., and Jacucci, G. (2015). Exploring peripheral physiology as a predictor of perceived relevance in information retrieval. In Proceedings of the 20th International Conference on Intelligent User Interfaces, IUI '15, New York, NY, USA. ACM.
  7. Barral O., & Jacucci G. (2014). Applying Physiological Computing Methods to Study Psychological, Affective and Motivational Relevance. In: Jacucci G., Gamberini L., Freeman J., Spagnolli A. (eds) Symbiotic Interaction. Symbiotic 2015. Lecture Notes in Computer Science, vol 8820. Springer, Cham.
  8. Barral, O., Kosunen, I., & Jacucci, G. Influence of reading speed on pupil size as a measure of perceived relevance. In Proceedings of the Joint Workshop on Personalized Information Access (PIA 2014).
  9. Bilgic, M., & Mooney, R.J. (2005). Explaining recommendations: satisfaction vs. promotion. Beyond Personalization Workshop. IUI, 5:153.
  10. Biran, O., & McKeown, K. (2014). Justification narratives for individual classifications. In: Proceedings of the AutoML Workshop at ICML 2014.
  11. Brézillon, P., & Pomerol, J.C. (1997). Joint cognitive systems, cooperative systems and decision support systems: A cooperation in context. In Proceedings of the European Conference on Cognitive Science, Manchester, 129-139.
  12. Chan, C. Y. H., Chan, A. B., Lee, T. M. C., & Hsiao, J. H. (2018). Eye movement patterns in face recognition are associated with cognitive decline in older adults. Psychonomic Bulletin & Review, 25(6), 2200-2207.
  13. Chan, F. H. F., Jackson, T., Hsiao, J. H., Chan, A. B., & Barry, T. J. (2020). The interrelation between interpretation biases, threat expectancies and pain-related attentional processing. European Journal of Pain, 24(10), 1956-1967.
  14. Chan, F. H. F., Suen, H., Hsiao, J. H., Chan, A. B., & Barry, T. J. (2020). Interpretation biases and visual attention in the processing of ambiguous information in chronic pain. European Journal of Pain, 24(7), 1242-1256.
  15. Chuk, T., Chan, A. B., & Hsiao, J. H. (2014). Understanding eye movements in face recognition using hidden Markov models. Journal of Vision, 14(11):8, 1-14.
  16. Chuk, T., Chan, A. B., Shimojo, S., & Hsiao, J. H. (2020). Eye movement analysis with switching hidden Markov models. Behavior Research Methods, 52(3), 1026-1043.
  17. Cole, M.J., Gwizdka, J., Liu, C., Belkin, N.J., & Zhang, X. (2013). Inferring user knowledge level from eye movement patterns. Information Processing & Management, 49(5): 1075-1091.
  18. Collins, R. P., Litman, J. A., & Spielberger, C. D. (2004). The measurement of perceptual curiosity. Personality and Individual Differences 36(5): 1127-1141.
  19. Cook, J. A., Ramsay, C. R., & Fayers, P. (2007). Using the literature to quantify the learning curve: a case study. The International Journal of Technology Assessment in Health Care, 23(2):255-260
  20. Crooker, K., & Near, J. (1998). Happiness and Satisfaction: Measures of Affect and Cognition? Social Indicators Research, 44(2), 195-224. http://www.jstor.org/stable/27522317
  21. Cummins, R. (2000). "How does it work?" versus "What are the laws?" In F. C. Keil and R. A. Wilson (eds.), Explanation and Cognition. Cambridge, MA: MIT Press.
  22. De Graaf, M.M., & Malle, B.F. (2017). How people explain action (and autonomous intelligent systems should too). AAAI Fall Symposium. https://www.aaai.org/ocs/index.php/FSS/FSS17/paper/view/16009
  23. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013). DeCAF: a deep convolutional activation feature for generic visual recognition. Workshop at ICML 2013.
  24. Dumont, R. G., & Troelstrup, R. L. (1980). Exploring relationships between objective and subjective measures of instructional outcomes. Research in Higher Education, 12(1), 37-51.
  25. Eugster, M. J., Ruotsalo, T., Spape, M. M., Kosunen, I., Barral, O., Ravaja, N., Jacucci, G., & Kaski, S. (2014). Predicting term-relevance from brain signals. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '14, New York, NY, USA. ACM, 425-434.
  26. Festinger, L. (1957). A Theory of cognitive dissonance. Stanford, CA: Stanford University Press.
  27. Fong, R.C., & Vedaldi, A. (2017). Interpretable Explanations of Black Boxes by Meaningful Perturbation. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), 3492-3437.
  28. Gallistel, C. R., Fairhurst, S., & Balsam, P. (2004). The learning curve: implications of a quantitative analysis. Proceedings of the National Academy of Sciences, 101(36), 13124-13131.
  29. Gates, L., Kisby, C., & Leake, D. (2019, September). CBR confidence as a basis for confidence in black box systems. In International Conference on Case-Based Reasoning, 95-109. Springer, Cham.
  30. Gilden, D. L., Thornton, T., & Mallon, M. W. (1995). 1/f noise in human cognition. Science, 267(5205), 1837-1839.
  31. Taylor & Francis.
  32. Hase, P. & Bansal, M. (2020). Evaluating Explainable AI: Which Algorithmic Explanations Help Predict Model Behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5540-5552.
  33. Heerink, M., Kröse, B., Evers, V., & Wielinga, B. (2010). Assessing acceptance of assistive social agent technology by older adults: The Almere model. International journal of social robotics, 2(4), 361-375.
  34. Hendricks, L.A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016). Generating Visual Explanations. In: Leibe B., Matas J., Sebe N., Welling M. (eds) Computer Vision -ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, vol 9908. Springer, Cham. https://doi.org/10.1007/978-3-319-46493-0_1 · DOI · Paper page
  35. Hoffman, G., Birnbaum G.,E., Vanunu K., Sass O., & Reis, H.T. (2014). Robot Responsiveness to Human Disclosure Affects Social Impression and Appeal. In Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction (HRI '14), 1-8. New York: ACM Press.
  36. Hoffman, R., Mueller, S., Klein, G., & Litman, J. (2018, February 01). Metrics for explainable AI: Challenges and prospects. https://arxiv.org/abs/1812.04608 · arXiv
  37. Holzinger, A., Carrington, A. & Müller, H. (2020). Measuring the Quality of Explanations: The System Causability Scale (SCS). Künstlich Intelligenz, 34: 193-198. https://doi.org/10.1007/s13218- 020-00636-z · DOI · Paper page
  38. Hoppe, S., Loetscher, T., Morey, S., & Bulling, A. (2015, September). Recognition of curiosity using eye movement analysis. In Adjunct proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2015 ACM international symposium on wearable computers, 185-188.
  39. Horstmann, G., & Herwig, A. (22015). Surprise attracts the eyes and binds the gaze. Psychonomic Bulletin & Review, 22(3): 743-9.
  40. Hoxmeier, J.A., & DiCesare, C. (2000). System Response Time and User Satisfaction: An Experimental Study of Browser-based Applications. In AMCIS 2000 Proceedings, 347.
  41. Hsiao, J. H., Lan, H., Zheng, Y., & Chan, A. B. (2021). Eye Movement analysis with Hidden Markov Models (EMHMM) with co-clustering. Behavior Research Methods. https://doi.org/10.3758/s13428- 021-01541-5 · DOI
  42. Hu, W. L., Akash, K., Jain, N., & Reid, T. (2016). Real-time sensing of trust in human-machine interactions. IFAC-PapersOnLine, 49(32), 48-53.
  43. Huang, C.M., & Bilge, M. (2012). Robot Behavior Toolkit: Generating Effective Social Behaviors for Robots. In Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI '12), 25-32. New York: ACM.
  44. Joshi, S., Koyejo, O., Kim, B., & Ghosh, J. (2018). xGEMs: Generating examplars to explain black- box models. arXiv preprint arXiv:1806.08867. · arXiv
  45. Jung, M. F., Martelaro, N., & Hinds, P. J. (2015). Using robots to moderate team conflict: the case of repairing violations. In Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction, 229-236.
  46. Kapoor, A., W. Burleson, & R.W. Picard. (2007). Automatic prediction of frustration. International Journal of Human-Computer Studies, 65: 724-736.
  47. Kenny, E. M., & Keane, M. T. (2019, July). Twin-systems to explain artificial neural networks using case-based reasoning: Comparative tests of feature-weighting methods in ANN-CBR twins for XAI. In Twenty-Eighth International Joint Conferences on Artificial Intelligence (IJCAI), Macao, 10-16 August 2019, 2708-2715.
  48. Kéri, S., & Kiss, I. (2011). Oxytocin response in a trust game and habituation of arousal. Physiology & behavior, 102(2), 221-224.
  49. Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., & Viegas, F. (2018, July). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning (pp. 2668-2677).
  50. PMLR.
  51. Kim, T.W. (2018). Explainable artificial intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv preprint arXiv: 1810.09598 · arXiv
  52. Knyazev, G.G., Slobodskaya, H.R., & Wilson, G.D. (2002). Psychophysiological correlates of behavioural inhibition and activation. Personality and Individual Differences, 33(4), 647-660.
  53. Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015). Principles of explanatory debugging to personalize interactive machine learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces (pp. 126-137). Atlanta, Georgia. https://doi.org/10.1145/2678025.2701399 · DOI · Paper page
  54. Kulesza, T., Stumpf, S., Burnett, M., Wong, W.-K., Riche, Y., Moore, T., ... & McIntosh, K. (2010). Explanatory debugging: Supporting end-user debugging of machine-learned programs. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (pp. 41- 48) New York: IEEE. [doi: 10.1109/VLHCC.2010. · DOI
  55. Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W. (2013). Too much, too little, or just right ? Ways explanations impact end users' mental models. 2013 IEEE Symposium on Visual Languages and Human Centric Computing, 3-10.
  56. Lamberti, D.M., & Wallace, W.A. (1990). Intelligent interface design: An empirical assessment of knowledge presentation in expert systems. MIS Quarterly, 14, 279-311.
  57. Leake, D.B. (1996). Case-Based Reasoning: Experiences, Lessons and Future Directions. MIT Press, Cambridge.
  58. Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human factors, 46(1), 50-80.
  59. Lee, M.K., Kiesler, S., Forlizzi, J., Srinivasa, S., & Rybski, P. (2010). Gracefully Mitigating Breakdowns in Robotic Services. Proceedings of the 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 203-10.
  60. Levin, D., Caroline H., Paul, N.A., Tao Z., & Adams J.A. (2013). Cognitive Dissonance as a Measure of Reactions to Human-Robot Interaction. Journal of HumanRobot Interaction, 2(3): 1-17.
  61. Li, X.H., Shi, Y., Li, H.Y., Bai, W., Song, Y., Cao, C.C., & Chen, L. (2021). Quantitative Evaluations on Saliency Methods: An Experimental Study. In Xi''an '21: ACM SIGMOD International Conference on Management of Data. https://arxiv.org/pdf/2012.15616.pdf · arXiv
  62. Liao, Q. V., Gruen, D., & Miller, S. (2020). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 463, 1-15. https://doi.org/10.1145/3313831.3376590 · DOI · Paper page
  63. Lipton, P. (2004). What good is an explanation? Explanations: Styles of Explanation in Science. Oxford University Press.
  64. Liu, T., Chuk, T., Yeh, S. L., & Hsiao, J. H. (2016). Transfer of perceptual expertise: The case of simplified and traditional Chinese character recognition. Cognitive Science, 40(8), 1941-1968.
  65. Malle, B. F., Knobe J., O'Laughlin M.J., Pearce, G.E., & Nelson, S.E. (2000). Conceptual Structure and Social Functions of Behavior Explanations: Beyond PersonSituation Attributions. Journal of Personality and Social Psychology, 79 (3): 309-26.
  66. Mandryk, R. L., Inkpen, K.M., & Calvert, T. W. (2006). Using Psychophysiological Techniques to Measure User Experience with Entertainment Technologies. Behaviour and Information Technology. Special Issue on User Experience, 25(2):141-158.
  67. Meyer, W.U., Niepel, M., Rudolph, U, & Schutzwohl, A. (1989). An experimental analysis of surprise. Cognition and Emotion, 5(4): 295-311.
  68. Miller, T., Howe, P., & Sonenberg, L. (2017). Explainable AI: Beware of Inmates Running the Asylum. IJCAI-17 Workshop on Explainable Artificial Intelligence (XAI). Retrieved from http://home.earthlink.net/~dwaha/research/meetings/ijcai17-xai/6.%20(Miller+%20XAI- 17%20extended)%20Explainable%20AI%20-%20Beware%20of%20Inmates%20Running%20the%2 0Asylum.pdf
  69. Moshfeghi, Y., Triantafillou, P., & Pollick, F.E. (2016). Understanding Information Need: an fMRI Study. In SIGIR '16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, 335-344.
  70. Murphy, K.R., & Davidshofer, C.O. (2001). Psychological Testing: Principles and Applications. Upper Saddle River, NJ: Prentice Hall.
  71. Norman, D.A. (1983). Some observations on Mental Models. In Mental Models, Dedre Gentner and Albert L. Stevens (Eds.). Psychology Press, Chapter 1, 7-14.
  72. Nosek, B. A. (2007). Implicit-explicit relations. Current Directions in Psychological Science, 16(2), 65-69.
  73. Páez, A. (2019). The Pragmatic Turn in Explainable Artificial Intelligence (XAI). Minds and Machines, 29: 441-459. https://doi.org/10.1007/s11023-019-09502-w · DOI
  74. Prasada, S. (2017). The scope of formal explanation. Psychon Bull Rev, 24: 1478-1487. https://doi.org/10.3758/s13423-017-1276-x · DOI
  75. Ratcliff, R., & Rouder, J. N. (1998). Modeling response times for two-choice decisions. Psychological science, 9(5), 347-356.
  76. Ravaja, N., Saari, T., Turpeinen, M., Laarni, J., Salminen, M., & Kivikangas, M. (2006). Spatial Presence and Emotions during Video Game Playing: Does It Matter with Whom You Play? Presence Teleoperators & Virtual Environments, 15(4):381-392.
  77. Roediger, H. L. (1990). Implicit memory: Retention without remembering. American psychologist, 45(9), 1043.
  78. Rouse, W.B., & Morris, N.M. (1986). On looking into the black box: Prospects and limits in the search for mental models. Psychological Bulletin, 100(3), 349-363.
  79. Rousseau, D. M., Sitkin, S. B., Burt, R. S., & Camerer, C. (1998). Not so different after all: A cross- discipline view of trust. Academy of management review, 23(3), 393-404.
  80. Rutjes, H., Willemsen, M.C., & Ijsselsteijn, W.A. (2019). Considerations on Explainable AI and Users' Mental Models. In Where is the Human? Bridging the Gap Between AI and HCI: Workshop at CHI'19, May 4-9, 2019, Glasgow, Scotland UK. Association for Computing Machinery, Inc. http://www.martijnwillemsen.nl/recommenderlab/RutjesChHI2019ws.pdf
  81. Rzepka, C., & Berger, B. (2018). User Interaction with AI-enabled Systems: A Systematic Review of IS Research. In Conference: International Conference on Information Systems (ICIS), San Francisco, vol.
  82. Samangouei, P., Saeedi, A., Nakagawa, L., & Silberman, N. (2018). Explaingan: Model explanation via decision boundary crossing transformations. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 666-681).
  83. Schaefer, K. E. (2013). The perception and measurement of human-robot trust. Doctoral dissertation, University of Central Florida Orlando, Florida.
  84. Schraagen, J. M., Elsasser, P., Fricke, H., Hof, M., & Ragalmuto, F. (2020). Trusting the X in XAI: Effects of different types of explanations by a self-driving car on trust, explanation satisfaction and mental models. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64(1), 339-343. https://doi.org/10.1177/1071181320641077 · DOI
  85. Sherman, S. J. (1980). On the self-erasing nature of errors of prediction. Journal of personality and Social Psychology, 39(2), 211.
  86. Shinozawa, K., Naya, F., Yamato, J., & Kogure, K. (2005). Differences in effect of robot and screen agent recommendations on human decision-making. International journal of human-computer studies, 62(2), 267-279.
  87. Sobel, D.M., Yoachim, C.M., Gopnik A., Meltzoff, A.N. & Blumenthal E.J. (2007). The Blicket Within: Preschoolers' Inferences About Insides and Causes. Journal of Cognition and Development, 8(2): 159-182. https://doi.org/10.1080/15248370701202356 · DOI
  88. Srinivasan, R., & Chander, A. (2020). Explanations Perspectives from the Cognitive Sciences -A Survey. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20), 4812-4818.
  89. Stoerig, P., & Cowey, A. (2007). Blindsight. Current Biology, 17(19), R822-R824.
  90. Schmitt, F. F., & Lahroodi, R. (2008). The epistemic value of curiosity. Educational theory, 58(2), 125-148.
  91. Strube, M. J., & Barbour, L. S. (1983). The decision to leave an abusive relationship: Economic dependence and psychological commitment. Journal of Marriage and the Family, 785-793.
  92. Szafir, D., & Mutlu, B. (2012, May). Pay attention! Designing adaptive agents that monitor and improve user engagement. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 11-20).
  93. Taylor, J. E. T., & Taylor, G. W. (2020). Artificial cognition: How experimental psychology can help generate explainable artificial intelligence. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-020-01825-5 · DOI
  94. Thorup, B., Crookes, K.; Chang, P.P.W., Burton, N., Pond, S., Li, T. K., Hsiao, J. H., & Rhodes, G. (2018). Perceptual Experience Shapes our Ability to Categorise Faces by National Origin: A New Other-Race Effect. British Journal of Psychology, 109(3), 583-603
  95. Tso, R. V. Y., Au, T. K., & Hsiao, J. H. (2014). Perceptual expertise: Can sensorimotor experience change holistic processing and left side bias? Psychological Science, 25(9), 1757-1767.
  96. Van der Was, J., Nieuwburg, E., Cremers, A., & Neerincs, M. (2021). Evaluating XAI: A comparison of rule-based and example-based explanations. Artificial Intelligence, 291. https://doi.org/10.1016/j.artint.2020.103404 · DOI · Paper page
  97. Vanhamme, J. (2000). The Link Between Surprise and Satisfaction: An Exploratory Research on how best to Measure Surprise. Journal of Marketing Management, 16(6): 565-582.
  98. Vasilyeva, N.Y., & Lombrozo, T. (2015). Explanations and causal judgments are differentially sensitive to covariation and mechanism information. In D.C. Noelle, R. Dale, A. S. Warlaumont, J. Yoshimi, T. Matlock, C.D. Jennings & P.P. Maglio (Eds.), Proceedings of the 37th Annual Conference of the Cognitive Science Society (p.2475-2480). Austin, TX: Cognitive Science Society.
  99. Vasilyeva, N., Wilkenfeld, D., & Lombrozo, T. (2015). Goals affect the perceived quality of explanations. In Proceedings of the 37th Annual Conference of the Cognitive Science Society, (pp. 2469-2474). Austin, TX: Cognitive Science Society.
  100. Vasilyeva, N., Wilkenfeld, D., & Lombrozo, T. (2017). Contextual utility affects the perceived quality of explanations. Psychometric Bulletin Review, 24:1436-1450.
  101. Wainer, J., Dabbish, L., & Kraut, R. (2011). Should I open this email? Inbox-level cues, curiosity and attention to email. In Proceedings of the SIGCHI Conference inhuman factors in computing systems, 3439-3448.
  102. Walker, C.M., Lombrozo T., Legare, C.H., & Gopnik A. (2014). Explaining prompts children to privilege inductively rich properties. Cognition, 133(2): 343-357. https://doi.org/10.1016/j.cognition.2014.07.008 · DOI
  103. Walker, F., Verwey, W., & Martens, M. (2018, June). Gaze behaviour as a measure of trust in automated vehicles. In Proceedings of the 6th Humanist Conference (June 2018).
  104. Wang, D., Yang, Q., Abdul, A., & Lim, B.Y. (2019). Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI conference on human factors in computing systems, 1-15.
  105. Wilson, S.J. (2003). Correlation of subjective and objective sleep measurements at different stages of the treatment of depression. Psychiatry research, 120(2), 179-190.
  106. Wortham, R.H., Theodorou, A., & Bryson, J.J. (2016). Robot Transparency, Trust and Utility. In AISB Workshop on Principles of Robotics, University of Sheffield.
  107. Wu, P. H., & Wu, H. K. (2020). Constructing a model of engagement in scientific inquiry: Investigating relationships between inquiry-related curiosity, dimensions of engagement, and inquiry abilities. Instructional Science, 48(1), 79-113.
  108. Wu, Y., Liu, Y., Tsai, Y.H.R., Yau, S.T. (2019). Investigating the role of eye movements and physiological signals in search satisfaction prediction using geometric analysis. Journal of the Association for Information Science and Technology, 70(9): 981-999.
  109. Xu, J., & Zhong, B. (2018). Review on portable EEG technology in educational research. Computers in Human Behavior, 81: 340-349.
  110. Yang, E., Dorneich, M.C. (2015). The Effect of Time Delay on Emotion, Arousal, and Satisfaction in Human-Robot Interaction. In Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting, 443-447.
  111. Yannakakis, G.N. (2008). How to Model and Augment Player Satisfaction: A Review. In WOCCI 2008: The 1st Workshop on Child, Computer, and Interaction Chania, Crete, Greece.
  112. Zhang, J., Chan, A. B., Lau, E. Y. Y., & Hsiao, J. H. (2019). Individuals with insomnia misrecognize angry faces as fearful faces while missing the eyes: An eye-tracking study. Sleep, 42(2), 220.
  113. Zhang J., Wang, J., He, L., Li, Z., & Yu, P.S. (2018). Layerwise Perturbation-Based Adversarial Training for Hard Drive Health Degree Prediction. In The 2018 IEEE International Conference on Data Mining (ICDM'18). arXiv:1809.04188. · arXiv
  114. Zheng, Y., Ye, X., & Hsiao, J. H. (2019). Does video content facilitate or impair comprehension of documentaries? The effect of cognitive abilities and eye movement strategy. Proceeding of the 41th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society.

Cited by

Sorted by citations

Authors
Hsiao, J. H.-W., Ngai, H. H. T., Qiu, L., Yang, Y., & Cao, C. C.
Venue
arXiv
Publisher
arXiv
Published
2021
DOI
10.48550/arxiv.2108.01737
arXiv
2108.01737
References
10
Type
Preprint
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review