CoChat
Sign Up
Cited by 117Explainable Artificial Intelligence (XAI)Usability and User Interface DesignHuman-Automation Interaction and Safety

Progressive disclosure

Open access
Springer, Aaron, Whittaker, Steve·2019·Proceedings of the 24th International Conference on Intelligent User Interfaces
DOI 10.1145/3301275.3302322
Open this paper in CoChat — free accountOpen PDF

Abstract

As we increasingly delegate important decisions to intelligent systems, it is essential that users understand how algorithmic decisions are made. Prior work has often taken a technocentric approach to transparency. In contrast, we explore empirical user-centric methods to better understand user reactions to transparent systems. We assess user reactions to transparency in two studies. In Study 1, users anticipated that a more transparent system would perform better, but retracted this evaluation after experience with the system. Qualitative data suggest this arose because transparency is distracting and undermines simple heuristics users form about system operation. Study 2 explored these effects in depth, suggesting that users may benefit from initially simplified feedback that hides potential system errors and assists users in building working heuristics about system operation. We use these findings to motivate new progressive disclosure principles for transparency in intelligent systems.

Integrity signals

Registry

Crossref · 10.1145/3301275.3302322

Cites

90 of 90 references verified0 citation errors · 0 not checkable

Statistics

3 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Claims the paper makes

21 of 46 cited sources checked · 8 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Making accurate predictions about emotions is difficult.

    “Further, making accurate predictions about emotions is also difficult, allowing us to examine a key challenge for intelligent systems: how to deal with system error.”
    • Also stated in: § 2.4 Emotional Analytics · boosted
      “previous research demonstrates difficulty in accurately predicting emotion from text [47,69].”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [47] 47. Shoushan Li, Lei Huang, Rong Wang, and Guodong Zhou. 2015. Sentence-level Emotion Classification with Label and Context Dependence. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 1045-1053. Retrieved September 21, 2018 from http://www.aclweb.org/anthology/P15-1101 source
    • cites [69] 69. Lena Reed, Jiaqi Wu, Shereen Oraby, Pranav Anand, and Marilyn Walker. 2017. Learning Lexico-Functional Patterns for First-Person Affect. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 141-147. Retrieved September 18, 2018 from http://aclweb.org/anthology/P17-2022 source
  • Premisecitation

    Transparency can promote user learning and insight from complex data.

    “It can also promote user learning and insight from complex data, which is important as humans increasingly work with complex inferential systems for analytic purposes [43,71].”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [43] 43. Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of Explanatory Debugging to Personalize Interactive Machine Learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces -IUI '15, 126-137. https://doi.org/10.1145/2678025.2701399 source

      The Related Work section (p. 3) says transparency helps users understand how a learning system operates, and the Conclusion (p. 11) reports improved understanding among users of Explanatory Debugging. The paper does not establish that transparency gives users insight into complex data for analytic purposes; its measured learning concerns the system itself. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [71] 71. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135-1144. https://doi.org/10.1145/2939672.2939778 source
  • Premisecitation

    The way we present transparency may account for contradictory results about transparency's effects.

    “Some research suggests that the way we present transparency may account for these contradictory results [26,42].”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [26] 26. Pedro García Garcia, Enrico Costanza, Jhim Verame, Diana Nowacka, and Sarvapali D. Ramchurn. 2018. Seeing (Movement) is Believing: The Effect of Motion on Perception of Automatic Systems Performance. Human- Computer Interaction 0, 0: 1-51. https://doi.org/10.1080/07370024.2018.1453815 source

      Section 2.2 (page unavailable) says explanations can help users understand smart systems but can also overwhelm or confuse them; it explores animation as another way to present system activity. This supports the idea that presentation matters, but the paper does not say that presentation accounts for contradictory findings about transparency’s effects; it also identifies system uncertainty as a reason effects differ. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [42] 42. René F. Kizilcec. 2016. How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface. 2390- 2395. https://doi.org/10.1145/2858036.2858402 source
  • Findingcitation

    Users anticipated a preference for transparency before using the system.

    “Consistent with our initial expectations and the prior literature on transparency [26,48], users anticipated a preference for transparency before using the system.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [26] 26. Pedro García Garcia, Enrico Costanza, Jhim Verame, Diana Nowacka, and Sarvapali D. Ramchurn. 2018. Seeing (Movement) is Believing: The Effect of Motion on Perception of Automatic Systems Performance. Human- Computer Interaction 0, 0: 1-51. https://doi.org/10.1080/07370024.2018.1453815 source

      Section 2.2 discusses how explanations can help users understand smart systems, but it does not report that users anticipated preferring transparency before use. Section 3.4 describes a questionnaire before the task and evaluations after participants saw the systems; it does not report a pre-use transparency preference. Searches for that preference and its timing found no supporting passage in our copy. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites [48] 48. Brian Y. Lim and Anind K. Dey. 2009. Assessing demand for intelligibility in context-aware applications. In Proceedings of the 11th international conference on Ubiquitous computing, 195-204. Retrieved May 8, 2017 from http://dl.acm.org/citation.cfm?id=1620576 source
  • Premisecitation

    Transparency can enable oversight by system designers.

    “Transparency can also enable oversight by system designers.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [33] 33. Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of Opportunity in Supervised Learning. arXiv:1610.02413 [cs]. Retrieved from http://arxiv.org/abs/1610.02413 source

      The Summary of Contributions (p. 2) describes an interpretable, easily checkable framework, and the Conclusions (p. 20) say it can reveal concerns requiring further scrutiny. That supports the possibility of oversight, but the paper does not specifically say that transparency enables oversight by system designers. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
    • cites [51] 51. Zachary C. Lipton. 2016. The Mythos of Model Interpretability. arXiv:1606.03490 [cs, stat]. Retrieved September 21, 2018 from http://arxiv.org/abs/1606.03490 source
  • May combine two claims. 22
  • Classified as probably not a substantial claim. 23
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    In prior experiments, users judged the same algorithm to be accurate but not perfect; the median user response for accuracy was 'Accurate' (6 on a 7-point Likert scale with the highest rating ('7') being 'Very Accurate').

    “In prior experiments, users judged the same algorithm to be accurate but not perfect; the median user response for accuracy was 'Accurate' (6 on a 7-point Likert scale with the highest rating ('7') being 'Very Accurate') [78].”
    • Also stated in: § 3.1 Machine Learning Model · plain
      “As we noted above, previous users found the system to be accurate to their experience, judging median accuracy to be 6 out of 7 on a scale where 7 is 'Very Accurate'.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [78] 78. Aaron Springer and Steve Whittaker. 2018. What are You Hiding? Algorithmic Transparency and User Perceptions. In 2018 AAAI Spring Symposium Series. source

  • May combine two claims.

    Premisecitation

    The unigram-based regression model is less accurate than state-of-the-art methods using deep neural networks, but provides a straightforward operationalization of transparency that is intuitively understandable while still appearing accurate and convincing to users.

    “While such a model may be less accurate than state of the art methods using deep neural networks [19], it provides a straightforward operationalization of transparency that is intuitively understandable while still appearing accurate and convincing to users [78].”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [19] 19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs]. Retrieved December 27, 2018 from http://arxiv.org/abs/1810.04805 source

      Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites [78] 78. Aaron Springer and Steve Whittaker. 2018. What are You Hiding? Algorithmic Transparency and User Perceptions. In 2018 AAAI Spring Symposium Series. source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The E-meter system and protocol have been successfully deployed in multiple previous experiments.

    “The E-meter system and protocol have been successfully deployed in multiple previous experiments [76,78].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [76] 76. Aaron Springer, Victoria Hollis, and Steve Whittaker. 2017. Dice in the Black Box: User Experiences with an Inscrutable Algorithm. Retrieved April 24, 2017 from https://aaai.org/ocs/index.php/SSS/SSS17/paper/view/1537 2 source

      The E-meter System and Materials and Methodology sections (p. 2) describe the system and procedure used with participants, supporting this paper as an example of a prior deployment. The Discussion (p. 3) calls it a small pilot; this source does not establish that the E-meter system and protocol were deployed in multiple previous experiments. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [78] 78. Aaron Springer and Steve Whittaker. 2018. What are You Hiding? Algorithmic Transparency and User Perceptions. In 2018 AAAI Spring Symposium Series. source

      The Users section (p. 2) reports that 41 users tested the E-meter system, and the E-meter System section (p. 2) describes the protocol. This supports its deployment in a previous experiment, but the paper describes two conditions within that study, not multiple previous experiments using the system and protocol. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Progressive disclosure has a long history in UI research, dating back to the Xerox Star and early word processing systems.

    “Progressive disclosure has a long history in UI research, dating back to the Xerox Star and early word processing systems [14,61,74].”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [14] 14. John M. Carroll and Caroline Carrithers. 1984. Training Wheels in a User Interface. Commun. ACM 27, 8: 800-806. https://doi.org/10.1145/358198.358218 source

      The Training Wheels Interface and Discussion sections describe a 1984 word-processor interface that made basic functions available while withholding advanced ones, and call this approach “staging” the presentation of functions. This supports the early word-processing part of the claim, but our copy does not connect the approach to the Xerox Star. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [61] 61. Lloyd H. Nakatani and John A. Rohrlich. 1983. Soft machines: A philosophy of user-computer interface design. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems -CHI '83, 19-23. https://doi.org/10.1145/800045.801573 source

      The “Operability with Flexibility” section explicitly identifies progressive disclosure as characteristic of the Xerox Star interface. The paper mentions dedicated word processors elsewhere, but does not link early word-processing systems to progressive disclosure (page numbers unavailable). (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [74] 74. David Canfield Smith. Designing the Star User Interface. 21.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Transparency can increase end user control and improve acceptance of complex algorithmic systems.

    “Greater transparency potentially increases end user control and improves acceptance of complex algorithmic systems [43].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [43] 43. Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of Explanatory Debugging to Personalize Interactive Machine Learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces -IUI '15, 126-137. https://doi.org/10.1145/2678025.2701399 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The original concept of progressive disclosure involved hiding advanced interface controls, allowing users to make fewer initial errors and learn the system more effectively.

    “The original concept involved hiding advanced interface controls; allowing users to make fewer initial errors and learn the system more effectively [14].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [14] 14. John M. Carroll and Caroline Carrithers. 1984. Training Wheels in a User Interface. Commun. ACM 27, 8: 800-806. https://doi.org/10.1145/358198.358218 source

      The Training Wheels Interface section says advanced functions were blocked while basic functions remained available; the abstract reports faster learning and better achievement. But advanced menu choices were not hidden: users could select them and receive an unavailable-function message. The paper supports preventing troublesome error states, not necessarily users making fewer initial mistakes. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    This replicates previous work [78] where the median user found the E-meter to be 'Accurate'.

    “This replicates our previous work [78] where the median user found the E-meter to be 'Accurate'.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [78] 78. Aaron Springer and Steve Whittaker. 2018. What are You Hiding? Algorithmic Transparency and User Perceptions. In 2018 AAAI Spring Symposium Series. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Other researchers have examined transparency in the context of deep learning models predicting patient outcomes in the medical realm.

    “For example, other researchers have examined transparency in the context of deep learning models predicting patient outcomes in the medical realm [45].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [45] 45. B. C. Kwon, M. Choi, J. T. Kim, E. Choi, Y. B. Kim, S. Kwon, J. Sun, and J. Choo. 2018. RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records. IEEE Transactions on Visualization and Computer Graphics: 1-1. https://doi.org/10.1109/TVCG.2018.2865027 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Each feature can be visualized and ranked for its contribution to the overall predicted diagnosis risk score.

    “Each feature can be visualized and ranked for its contribution to the overall predicted diagnosis risk score.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [45] 45. B. C. Kwon, M. Choi, J. T. Kim, E. Choi, Y. B. Kim, S. Kwon, J. Sun, and J. Choo. 2018. RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records. IEEE Transactions on Visualization and Computer Graphics: 1-1. https://doi.org/10.1109/TVCG.2018.2865027 source

  • Classified as probably not a substantial claim.

    Premisecitation

    One method used in prior studies is to provide potential system users with hypothetical scenarios describing system operation and eliciting reactions to those systems.

    “One method used in prior studies is to provide potential system users with hypothetical scenarios describing system operation and eliciting reactions to those systems.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [26] 26. Pedro García Garcia, Enrico Costanza, Jhim Verame, Diana Nowacka, and Sarvapali D. Ramchurn. 2018. Seeing (Movement) is Believing: The Effect of Motion on Perception of Automatic Systems Performance. Human- Computer Interaction 0, 0: 1-51. https://doi.org/10.1080/07370024.2018.1453815 source

      Section 3.4 (page unavailable) describes participants viewing system interfaces and evaluating their performance, rather than reacting to hypothetical scenarios. Section 2.2 discusses prior studies of explanations and system intelligibility, but does not describe the claimed scenario-based method. Searches for hypothetical scenarios and vignettes found no passage establishing it. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites [49] 49. Brian Y. Lim and Anind K. Dey. 2011. Investigating intelligibility for uncertain context-aware applications. In Proceedings of the 13th international conference on Ubiquitous computing, 415-424. Retrieved April 25, 2017 from http://dl.acm.org/citation.cfm?id=2030168 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Users can compare outcomes between patients and conduct what-if analyses by modifying patient attributes and seeing how predicted risk changes.

    “In addition, users can compare outcomes between patients and conduct what-if analyses by modifying patient attributes and seeing how predicted risk changes.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [45] 45. B. C. Kwon, M. Choi, J. T. Kim, E. Choi, Y. B. Kim, S. Kwon, J. Sun, and J. Choo. 2018. RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records. IEEE Transactions on Visualization and Computer Graphics: 1-1. https://doi.org/10.1109/TVCG.2018.2865027 source

  • May combine two claims.

    Findingown data

    In Study 1, users anticipated that a more transparent system would perform better, but retracted this evaluation after experience with the system.

    “In Study 1, users anticipated that a more transparent system would perform better, but retracted this evaluation after experience with the system.”
    • Also stated in: § 1.1 Contribution · plain
      “In Study 1, users anticipated that a transparent system would perform better, but retracted this evaluation after experience with the system.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The linear regression model tested at R^2 = 0.25 and mean absolute error of .95 on the target variable (mood) scale of (-3,3).

    “Using a train/test split of 85/15 the linear regression tested at R 2 = 0.25; mean absolute error was .95 on the target variable (mood) scale of (-3,3).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Overall, the median user found the E-meter to be 'Slightly Accurate' and 'Slightly Trustworthy'.

    “Overall, the median user found the E-meter to be 'Slightly Accurate' and 'Slightly Trustworthy'.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Some users continued to prefer the transparent version and were likely to cite the increased insight that it facilitated into the algorithm's underlying operation.

    “As we had originally anticipated, some users continued to prefer the transparent version and they were likely to cite the increased insight that it facilitated into the algorithm's underlying operation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Some users seemed to find highlighting distracting, although cognitive load results do not support this.

    “Some of these users seemed to find highlighting to be distracting, although our cognitive load results do not support this.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both retrospective accuracy and subjective trust distributions were bimodal due to the 7-point Likert item containing an infrequently used neutral item.

    “Both retrospective accuracy and subjective trust distributions were bimodal (due to the 7-point Likert item containing an infrequently used neutral item)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    There was no difference between conditions in retrospective accuracy and subjective trust (p = .24, p=.41).

    “there was no difference between conditions (p = .24,p=.41).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Qualitative user comments supported the anticipated benefits of the transparent system; users cited transparent color-coded feedback giving a clearer sense of system operation and boosting confidence.

    “Qualitative user comments supported these anticipated benefits.”
    • Also stated in: § System Perceived as Moderately Accurate and Trustworthy: · plain
      “users drew attention to the benefits of transparent color-coded feedback giving a clearer sense of how the system was operating and boosting their confidence that the system was operating appropriately.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Some users, like those in Study 1, may want yet more transparency than seeing the major factors influencing the algorithm.

    “However, some users (like those in Study 1) may want yet more transparency.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Results from Study 1 indicate the importance of direct user experience when making system evaluations; users' perceptions of the system were very different following actual usage compared with their projected reactions prior to usage.

    “However, results from Study 1 indicate the importance of direct user experience when making system evaluations; users' perceptions of the system were very different following actual usage compared with their projected reactions prior to usage.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Following prior methodological recommendations in [18], we eliminated 26 respondents based on their responses to open ended questions, leaving us with a sample of 74 users.

    “Following prior methodological recommendations in [18], we eliminated 26 respondents based on their responses to open ended questions, leaving us with a sample of 74 users.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18] 18. Michael A. DeVito, Jeremy Birnholtz, Jeffery T. Hancock, Megan French, and Sunny Liu. 2018. How People Form Folk Theories of Social Media Feeds and What it Means for How We Study Self-Presentation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18, 1-12. https://doi.org/10.1145/3173574.3173694 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The study design was counterbalanced; half the users experienced the non-transparent system version first.

    “The study design was counterbalanced; half the users experienced the non-transparent system version first.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    This study was approved by an Institutional Review Board.

    “This study was approved by an Institutional Review Board.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    Questions and procedure were carefully piloted and had been used before in multiple prior studies.

    “Questions and procedure were carefully piloted and had been used before in multiple prior studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Qualitative data suggest that transparency is distracting and undermines simple heuristics users form about system operation.

    “Qualitative data suggest this arose because transparency is distracting and undermines simple heuristics users form about system operation.”
    • Also stated in: § 1.1 Contribution · hedged
      “Qualitative data suggest this may be because incremental transparency feedback is distracting, potentially undermining simple heuristics users form of system operation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    We used Latin text because we wanted high-level system comparisons that were not based on users' reactions to specific English words.

    “We used Latin text because we wanted high-level system comparisons that were not based on users' reactions to specific English words.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Even if the system is objectively correct when displaying transparency information, it can still cause users to take issue and result in poorer perceptions of that system.

    “Even if the system is objectively correct when displaying transparency information, it can still cause users to take issue and result in poorer perceptions of that system.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Following recommendations from an IRB panel, the PGWBI screening was intended to exclude users who might find the emotional reflection distressing.

    “Following recommendations from an IRB panel, the PGWBI screening was intended to exclude users who might find the emotional reflection distressing.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The absence of real user feedback makes it challenging to operationalize transparency in ways that positively impact users.

    “The absence of real user feedback makes it challenging to operationalize transparency in ways that positively impact users.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationcitation

    In interacting with a system, users first engage in rapid, approximate, System 1 thinking and only engage in deeper, more analytic, System 2 thinking when directly prompted or confronted with clearly anomalous information.

    “As with other areas of reasoning, it may be that in interacting with a system users first engage in rapid, approximate, System 1 thinking and only engage in deeper, more analytic, System 2 thinking when directly prompted or confronted with clearly anomalous information [40,66].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [40] 40. Daniel Kahneman. 2011. Thinking, Fast and Slow. Macmillan.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [66] 66. Richard E. Petty and John T. Cacioppo. 1986. The Elaboration Likelihood Model of Persuasion. Advances in IUI'19, March, 2019, Marina Del Rey, CA USA Aaron Springer and Steve Whittaker Experimental Social Psychology 19: 123-205. https://doi.org/10.1016/S0065-2601(08)60214-2 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    The machine learning model was limited to the 600 most predictive words of mood.

    “While the machine learning model was limited to the 600 most predictive words of mood, that model still seemed to present too many extraneous words that users felt were unimportant.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    Non-transparent feedback can hide low-level errors from users.

    “One possibility is that non-transparent feedback can hide low-level errors from users.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    Both systems use the same underlying machine learning model which uses solely individual word features.

    “Recall that both systems use the same underlying machine learning model which uses solely individual word features.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The deployment of a working algorithm meant that results were obtained for situations where the algorithm generated moderate numbers of errors.

    “Furthermore, our deployment of a working algorithm meant that results were obtained for situations where our algorithm generated moderate numbers of errors-future research should compare contexts where there are different levels of errors [46,47].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Providing transparency on demand removes confusions and inefficiencies arising from spurious, unwanted explanations, and adjusts explanations to users' requirements.

    “Note too that because transparency is provided 'on demand' this removes confusions and inefficiencies arising from spurious, unwanted explanations, and adjusts explanations to the users' requirements.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    While users generated their own data in the system, results were not directly used to inform other aspects of the user's personal behavior, so the costs of system errors were low.

    “Additionally, while users generated their own data in our system, results were not directly used to inform other aspects of the user's personal behavior so the costs of system errors were low.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    There are many other ways to depict how an algorithm operates, including verbal explanations, concrete user exploration, and so forth.

    “There are many other ways to depict how an algorithm operates including verbal explanations, concrete user exploration, and so forth [42,44,50,71].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [42] 42. René F. Kizilcec. 2016. How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface. 2390- 2395. https://doi.org/10.1145/2858036.2858402 source
    • cites [44] 44. Todd Kulesza, Weng-Keen Wong, Simone Stumpf, Stephen Perona, Rachel White, Margaret M. Burnett, Ian Oberst, and Andrew J. Ko. 2008. Fixing the program my computer learned: barriers for end users, challenges for the machine. In Proceedingsc of the 13th international conference on Intelligent user interfaces -IUI '09, 187. https://doi.org/10.1145/1502650.1502678 source
    • cites [50] 50. Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In Proceedings of the 27th international conference on Human factors in computing systems - CHI 09, 2119. https://doi.org/10.1145/1518701.1519023 source
    • cites [71] 71. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135-1144. https://doi.org/10.1145/2939672.2939778 source

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    The current study indicates a promising design approach involving progressive disclosure.

    “The current study indicates a promising design approach involving progressive disclosure which we intend to explore in future work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Progressive disclosure of transparency is not limited to predictions from text.

    “Progressive disclosure of transparency is not limited to predictions from text.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The major contribution of progressive disclosure is avoiding overwhelming users with information, instead slowly increasing transparency as users indicate a willingness to engage meaningfully with it.

    “Again, the major contribution of progressive disclosure is avoiding overwhelming users with information, but instead slowly increasing transparency as users indicate a willingness to engage meaningfully with it.”
    Quote in the paperSubstantial claim
41 of 41

  • The paper relies on this without support.

    Users are the ultimate experts on their own emotions, providing them with ground truth assessments about whether the system is correct.

    “And users are the ultimate experts on their own emotions, providing them with ground truth assessments about whether the system is correct.”
1 of 1

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

  • Inconsistent p-value: t(73)=.910,p = .343

    Minor

    The paper reports p = .343 here; recomputing from the test statistic and its degrees of freedom gives p = .366 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .343 · recomputed p = .366

  • Inconsistent p-value: t(73) = -.05,p = .95

    Minor

    The paper reports p = .95 here; recomputing from the test statistic and its degrees of freedom gives p = .960 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .95 · recomputed p = .960

  • Inconsistent p-value: t(73)=5.452,p = .022

    Minor

    The paper reports p = .022 here; recomputing from the test statistic and its degrees of freedom gives p < .001 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .022 · recomputed p < .001

3 of 3

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upt(73) = 5.452reported p = .022 · recomputed p < .001

    “t(73)=5.452,p = .022”

  • Doesn’t add upt(73) = 0.91reported p = .343 · recomputed p = .366

    “t(73)=.910,p = .343”

  • Doesn’t add upt(73) = -0.05reported p = .95 · recomputed p = .960

    “t(73) = -.05,p = .95”

3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Study 1's reported t statistic and p value conflict

    Major

    For 73 degrees of freedom, a t statistic of 5.452 has a two-sided p value below 0.000001, not 0.022. The reported significance value is therefore inconsistent with the reported test statistic.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 10

  1. Angwin Julia, 2016, ProPublica. Retrieved
  2. Devlin Jacob, 2018, Retrieved
  3. Lakkaraju Himabindu, 2017, Retrieved
  4. Lipton Zachary C., 2016, Retrieved
  5. Miller Tim, 2017, Retrieved
  6. Springer Aaron, 2017, Retrieved
  7. Springer Aaron, 2018, Algorithmic Transparency and User Perceptions. In 2018 AAAI Spring Symposium Series.
  8. Daniel, 2018, Retrieved
  9. Zisook Miriam, 2016, Retrieved
  10. Woebot, 2018, Retrieved

Cited by

Sorted by citations

Related

Authors
Springer, A., & Whittaker, S.
Affiliations
University of California, Santa Cruz (US)
ORCID
Venue
Proceedings of the 24th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2019-03-17
Pages
107-120
DOI
10.1145/3301275.3302322
OpenAlex
W2918066172
Cited by
115
References
87
Citations by year
2026: 19 · 2025: 17 · 2024: 11 · 2023: 17 · 2022: 15 · 2021: 14 · 2020: 16 · 2019: 2 · 2018: 1 · 2017: 1 · 2016: 1 · 2012: 1
FWCI
4.50
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
Keywords
progressive disclosure, transparency, algorithmic decision-making, intelligent systems, system errors, user distraction, user-centered design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review