The Placebo Effect of Artificial Intelligence in Human–Computer Interaction
Open accessThe DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.
Abstract
In medicine, patients can obtain real benefits from a sham treatment. These benefits are known as the placebo effect. We report two experiments (Experiment I: N = 369; Experiment II: N = 100) demonstrating a placebo effect in adaptive interfaces. Participants were asked to solve word puzzles while being supported by no system or an adaptive AI interface. All participants experienced the same word puzzle difficulty and had no support from an AI throughout the experiments. Our results showed that the belief of receiving adaptive AI support increases expectations regarding the participant’s own task performance, sustained after interaction. These expectations were positively correlated to performance, as indicated by the number of solved word puzzles. We integrate our findings into technological acceptance theories and discuss implications for the future assessment of AI-based user interfaces and novel technologies. We argue that system descriptions can elicit placebo effects through user expectations biasing the results of user-centered studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Even non-adaptive systems are deemed adaptive and personalized if introduced as such [67].”
From The AI Ghostwriter Effect: When Users do not Perceive Ownership of AI-Generated Text but Self-Declare as Authors (2024) · cites this paper as [67]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483 · cited 1×
“In user studies, subjective measures for acceptance and usability such as the Technology Acceptance Model (TAM) [15, 16, 101], Unified Theory of Acceptance and Use of Technology (UTAUT) [46], or the NASA Task Load Index (NASA-TLX) [38] for the measurement of workload, are widely used metrics in the…”
A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies
Venkatesh, V., & Davis, F. D. (2000) · Management Science · doi:10.1287/mnsc.46.2.186.11926 · Cited by 23,066 · cited 2×
“In user studies, subjective measures for acceptance and usability such as the Technology Acceptance Model (TAM) [15, 16, 101], Unified Theory of Acceptance and Use of Technology (UTAUT) [46], or the NASA Task Load Index (NASA-TLX) [38] for the measurement of workload, are widely used metrics in the field of HCI [37].”
Venkatesh, V., Thong, J. Y. L., & Xu, X. (2012) · MIS Quarterly · doi:10.2307/41410412 · Cited by 15,756 · cited 9×
“Technology acceptance models such as the UTAUT [104] emphasize the role of subjective evaluation, as participants’ demographics, prior experience, and so on can mitigate both performance expectations and their influence over user behavior.”
InfluentialDevelopment of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · cited 8×
“After each adaptivity, participants filled in a NASA-TLX questionnaire.”
InfluentialConfirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267 · cited 2×
“One could explain this continuance of belief in terms of confirmation bias [76]), which describes the propensity of participants to gather evidence for confirming a hypothesis.”
(1985) · Cited by 6,185
Ekman, P., & Friesen, W. V. (1975) · Cited by 4,604
NASA-Task Load Index (NASA-TLX); 20 Years Later
Hart, S. G. (2006) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120605000909 · Cited by 4,297 · cited 3×
“Following that, they were asked to solve the 20 word puzzles and to fill in a NASA-TLX questionnaire [37, 38].”
The aligned rank transform for nonparametric factorial analyses using only anova procedures
Wobbrock, J. O., Findlater, L., Gergle, D., & Higgins, J. J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978963 · Cited by 2,854 · cited 1×
“We will test user expectations before and after interaction with the system in a rank-aligned repeated-measures Analysis of Variance (ANOVA) [49].”
Beecher, H. K. (1955) · Journal of the American Medical Association · doi:10.1001/jama.1955.02960340022006 · Cited by 1,842 · cited 3×
“Such a placebo can alleviate pain [60] or support the treatment of ailments [3, 48], thereby providing an effective medical treatment without an illness-specific mechanism of effect.”
Oliver, R. L. (1977) · Journal of Applied Psychology · doi:10.1037/0021-9010.62.4.480 · Cited by 1,770 · cited 1×
“This has been shown in survey-based methods [15, 98, 102] and also experimentally [4, 39, 78].”
Bayesian inference for psychology. Part II: Example applications with JASP
Wagenmakers, E.-J., Love, J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Selker, R., Gronau, Q. F., Dropmann, D., Boutin, B., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1323-7 · Cited by 1,715
Bayesian estimation supersedes the t-test
Kruschke, J. K. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0029146 · Cited by 1,530
Is the Placebo Powerless?: An Analysis of Clinical Trials Comparing Placebo with No Treatment
Hróbjartsson, A., & Gøtzsche, P. C. (2001) · New England Journal of Medicine · doi:10.1056/nejm200105243442106 · Cited by 1,503
Thong, J. Y. L., Hong, S.-J., & Tam, K. Y. (2006) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2006.05.001 · Cited by 1,337 · cited 3×
“This has been shown in survey-based methods [15, 98, 103] and also experimentally [4, 39, 78].”
Biological, clinical, and ethical advances of placebo effects
Finniss, D. G., Kaptchuk, T. J., Miller, F., & Benedetti, F. (2010) · The Lancet · doi:10.1016/s0140-6736(09)61706-2 · Cited by 1,272 · cited 5×
“Motivated by the classification of the search results (see Figure 1), we ran two experiments manipulating system descriptions [24] aligning with placebotheory [6, 29, 95] (see Figure 1(c)).”
InfluentialCurrent practice in measuring usability: Challenges to usability studies and research
Hornbæk, K. (2006) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2005.06.002 · Cited by 1,123 · cited 1×
“AI-based adaptive interfaces optimize usability by linking the system design to user input [43], where the overall goal of user interface adaptations is the improvement of the interaction efficiency and perceived usability [43].”
In AI we trust? Perceptions about automated decision-making by artificial intelligence
Araujo, T., Helberger, N., Kruikemeier, S., & de Vreese, C. H. (2020) · AI & SOCIETY · doi:10.1007/s00146-019-00931-w · Cited by 1,038
Machine Learning and Artificial Intelligence: Definitions, Applications, and Future Directions
Helm, J. M., Swiergosz, A. M., Haeberle, H. S., Karnuta, J. M., Schaffer, J. L., Krebs, V. E., Spitzer, A. I., & Ramkumar, P. N. (2020) · Current Reviews in Musculoskeletal Medicine · doi:10.1007/s12178-020-09600-8 · Cited by 710 · cited 1×
“Note that different definitions of AI exist (for a range of potential definitions and conceptualizations, see [40, 63, 79]).”
The placebo response in medicine: minimize, maximize or personalize?
Enck, P., Bingel, U., Schedlowski, M., & Rief, W. (2013) · Nature Reviews Drug Discovery · doi:10.1038/nrd3923 · Cited by 671 · cited 1×
“Note that these characteristics are also relevant in medical and psychological placebo studies [3, 27].”
Boot, W. R., Simons, D. J., Stothart, C., & Stutts, C. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691613491271 · Cited by 670 · cited 4×
“In line with practices in psychology [6], and Medicine [29], we will use the term “placebo-studies” to refer to studies that employ a shamtreatment and compare it to a baseline condition, and “placebo-controlled studies”, studies that compare a sham-treatment against a functional system.”
InfluentialThe Placebo Effect: Dissolving the Expectancy Versus Conditioning Debate.
Stewart-Williams, S., & Podd, J. (2004) · Psychological Bulletin · doi:10.1037/0033-2909.130.2.324 · Cited by 664 · cited 7×
“Drawing from placebo-theory in psychology and medicine [3, 6, 60, 95], we systematically list HCI studies controlling for placebo effects.”
InfluentialAn Aligned Rank Transform Procedure for Multifactor Contrast Tests
Elkin, L. A., Kay, M., Higgins, J. J., & Wobbrock, J. O. (2021) · The 34th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3472749.3474784 · Cited by 664 · cited 1×
“Followup analysis is based on Bonferroni-corrected ART-C contrasts [26] Correlations for Likert-type scales (ordinal) are estimated using Spearman-rank correlations.”
Bayes factor approaches for testing interval null hypotheses.
Morey, R. D., & Rouder, J. N. (2011) · Psychological Methods · doi:10.1037/a0024377 · Cited by 582 · cited 1×
“NULL-hypotheses relevant for our main research questions are evaluated using Bayes factors computed using the BayesFactor package [73] or JASP [105](6).”
23 Ways to Nudge: A Review of Technology-Mediated Nudging in Human-Computer Interaction
Caraban, A., Karapanos, E., Gonçalves, D., & Campos, P. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300733 · Cited by 475
Analyzing facebook privacy settings: user expectations vs. reality
Liu, Y., Gummadi, K. P., Krishnamurthy, B., & Mislove, A. (2011) · Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference · doi:10.1145/2068816.2068823 · Cited by 457
A study of the placebo response
Lasagna, L., Mosteller, F., von Felsinger, J. M., & Beecher, H. K. (1954) · The American Journal of Medicine · doi:10.1016/0002-9343(54)90441-6 · Cited by 433 · cited 2×
“Such a placebo can alleviate pain [60] or support the treatment of ailments [3, 48], thereby providing an effective medical treatment without an illness-specific mechanism of effect.”
Cardiovascular reactivity to stress predicts future blood pressure status
Matthews, K. A., Woodall, K. L., & Allen, M. T. (1993) · Hypertension · doi:10.1161/01.hyp.22.4.479 · Cited by 402 · cited 1×
“Physiology-based Adaptation: Participants were informed about the presence of a physiologybased adaptation of an AI, changing the word puzzle difficulty by analyzing the heart rate [62, 68] and emotions via facial expressions [25] (i.”
‘Powerful Placebo: The Dark Side of the Randomised Controlled Trial’
Kaptchuk, T. J. (1998) · The Lancet · doi:10.1016/s0140-6736(97)10111-8 · Cited by 339 · cited 1×
“Such a placebo can alleviate pain [60] or support the treatment of ailments [3, 48], thereby providing an effective medical treatment without an illness-specific mechanism of effect.”
Accountability and automation bias
SKITKA, L. J., MOSIER, K., & BURDICK, M. D. (2000) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0349 · Cited by 324
Wells, J. D., Campbell, D. E., Valacich, J. S., & Featherman, M. (2010) · Decision Sciences · doi:10.1111/j.1540-5915.2010.00292.x · Cited by 305
Meta-analysis of correlations among usability measures
Hornbæk, K., & Law, E. L.-C. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240722 · Cited by 276 · cited 1×
“Although user satisfaction and effectiveness are correlated [42], it is unclear to which extent user expectations are manipulated.”
Heart rate variability and cognitive processing: The autonomic response to task demands
Luque-Casado, A., Perales, J. C., Cárdenas, D., & Sanabria, D. (2016) · Biological Psychology · doi:10.1016/j.biopsycho.2015.11.013 · Cited by 243 · cited 1×
“The participants were told that changes in heart rate [67, 81] and perceived emotions [84] are correlated to cognitive demand and stress-levels that can be captured using a webcam, and that an AI (i.e., neural network) aggregates those two metrics to calculate the currently perceived cognitive…”
Automation bias in intelligent time critical decision support systems
Cummings, M. L. (2017) · Decision Making in Aviation · doi:10.4324/9781315095080-17 · Cited by 233
Hans Berger: From Psychic Energy to the EEG
Millett, D. (2001) · Perspectives in Biology and Medicine · doi:10.1353/pbm.2001.0070 · Cited by 217 · cited 1×
“…also try to resolve the issue by turning to so-called objective measures [87], where the participant is blind to the measurement rationale employed (e.g., an implicit association test) or physiological measures where the response is hard to modulate deliberately (e.g., electroencephalography [70]).”
Exploring the design space for adaptive graphical user interfaces
Gajos, K. Z., Czerwinski, M., Tan, D. S., & Weld, D. S. (2006) · Proceedings of the working conference on Advanced visual interfaces - AVI '06 · doi:10.1145/1133265.1133306 · Cited by 198 · cited 1×
“Therefore, descriptions of systems can be regarded as a potential placebo, if aside from perceived effectiveness the task performance is susceptible to such manipulations [22, 32].”
AI-generated vs. Human Artworks. A Perception Bias Towards Artificial Intelligence?
Ragot, M., Martin, N., & Cojean, S. (2020) · Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3334480.3382892 · Cited by 189 · cited 1×
“Describing art as being generated by AI prompted less favorable evaluations compared to descriptions of human-made artwork [85].”
Goal Activation, Expectations, and the Placebo Effect.
Geers, A. L., Weiland, P. E., Kosbab, K., Landry, S. J., & Helfer, S. G. (2005) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.89.2.143 · Cited by 185 · cited 1×
“User studies in HCI are confronted with larger obstacles in terms of placebo-control compared to pharmacological trials [34].”
EmotionCheck: leveraging bodily signals and false feedback to regulate our emotions
Costa, J., Adams, A. T., Jung, M. F., Guimbretière, F., & Choudhury, T. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2971648.2971752 · Cited by 168
Mechanisms of Placebo Pain Reduction: An Empirical Investigation
Montgomery, G., & Kirsch, I. (1996) · Psychological Science · doi:10.1111/j.1467-9280.1996.tb00352.x · Cited by 159 · cited 1×
“The only critical determinant of the placebo effect is the patient’s expectation in the placebo’s efficacy, leading to a positive evaluation after treatment [72, 95].”
The Impact of Placebic Explanations on Trust in Intelligent Systems
Eiband, M., Buschek, D., Kremer, A., & Hussmann, H. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3312787 · Cited by 152
How Effective is Targeted Advertising?
Farahat, A., & Bailey, M. C. (2012) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2013.6580780 · Cited by 145
Hudlická, E., & McNeese, M. D. (2002) · User Modeling and User-Adapted Interaction · doi:10.1023/a:1013337427135 · Cited by 140 · cited 1×
“For example, one can adapt letter presentation in an E-reader to alpha oscillations; an indicator of workload in the electroencephalogram [55], reduce the number of interaction possibilities with the number of errors made by the user [45], or adapt the interface to the mood of the user [61].”
Autonomously Generating Hints by Inferring Problem Solving Policies
Piech, C., Sahami, M., Huang, J., & Guibas, L. J. (2015) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2724660.2724668 · Cited by 139 · cited 1×
“[1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 104] We categorize the studies into (a) studies that employed a placebo-control, (b) studies that specifically investigate the existence of a placebo effect, and (c) studies that conclude with a need for placebo-controls.”
Proceedings of the 52nd ACM Technical Symposium on Computer Science Education
Whalley, J., Settle, A., & Luxton-Reilly, A. (2021) · doi:10.1145/3408877 · Cited by 133
Prakash, R. S., Hussain, M. A., & Schirda, B. (2015) · Psychology and Aging · doi:10.1037/a0038544 · Cited by 129 · cited 1×
“The participants were told that changes in heart rate [67, 81] and perceived emotions [84] are correlated to cognitive demand and stress-levels that can be captured using a webcam, and that an AI (i.”
The experimental situation as a psychological problem.
Rosenzweig, S. (1933) · Psychological Review · doi:10.1037/h0074916 · Cited by 126 · cited 1×
“One could also try to resolve the issue by turning to so-called objective measures [87], where the participant is blind to the measurement rationale employed (e.”
It Is Your Turn: Collaborative Ideation With a Co-Creative Robot through Sketch
Lin, Y., Guo, J., Chen, Y., Yao, C., & Ying, F. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376258 · Cited by 125 · cited 1×
“Placebo-Control 4 [35, 71, 97, 111] Placebo Study 12 [18, 20, 22, 23, 28, 82, 100, 107, 110, 110] Acknowledge Placebo Effects as a Confound 11 [1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 105]”
Springer, A., & Whittaker, S. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302322 · Cited by 115
DETECTION OF HUMAN STRESS USING SHORT-TERM ECG AND HRV SIGNALS
Karthikeyan, P., Murugappan, M., & Yaacob, S. (2013) · Journal of Mechanics in Medicine and Biology · doi:10.1142/s0219519413500383 · Cited by 109 · cited 1×
“The participants were told that changes in heart rate [67, 81] and perceived emotions [84] are correlated to cognitive demand and stress-levels that can be captured using a webcam, and that an AI (i.”
Cognition and the Placebo Effect – Dissociating Subjective Perception and Actual Performance
Schwarz, K. A., & Büchel, C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0130492 · Cited by 106 · cited 1×
“For instance, a sham sound treatment [89] or an expectation manipulation of receiving Ritalin, a potent drug that increases focus [66], only increased subjective expectations but not the objective performance metrics themselves.”
Framing the user experience: information biases on website quality judgement
Hartmann, J., Angeli, A. D., & Sutcliffe, A. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357190 · Cited by 102 · cited 1×
“This has been shown in survey-based methods [15, 98, 102] and also experimentally [4, 39, 78].”
BrightBeat: Effortlessly Influencing Breathing for Cultivating Calmness and Focus
Ghandeharioun, A., & Picard, R. W. (2017) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3027063.3053164 · Cited by 101
Measuring and Controlling Social Desirability Bias: Applications in Information Systems Research
Kwak, D.-H., Holtkamp, P., & Kim, S. S. (2019) · Journal of the Association for Information Systems · doi:10.17705/1jais.00537 · Cited by 96
“I’d like an Explanation for That!”Exploring Reactions to Unexpected Autonomous Driving
Wiegand, G., Eiband, M., Haubelt, M., & Hußmann, H. (2020) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/3379503.3403554 · Cited by 92 · cited 1×
“Placebo-Control 4 [35, 71, 97, 111] Placebo Study 12 [18, 20, 22, 23, 28, 82, 100, 107, 110, 110] Acknowledge Placebo Effects as a Confound 11 [1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 105]”
Expectation to receive methylphenidate enhances subjective arousal but not cognitive performance.
Looby, A. R., & Earleywine, M. (2011) · Experimental and Clinical Psychopharmacology · doi:10.1037/a0025252 · Cited by 84
Flaten, M. A., & Blumenthal, T. D. (1999) · Psychopharmacology · doi:10.1007/s002130051038 · Cited by 79
Your Eyes Tell: Leveraging Smooth Pursuit for Assessing Cognitive Workload
Kosch, T., Hassib, M., Woźniak, P. W., Buschek, D., & Alt, F. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174010 · Cited by 79
Self-citationChittaro, L., & Sioni, R. (2014) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2014.01.007 · Cited by 78 · cited 1×
“Second, physiological responses can also be altered by the participants’ expectations [11], hence explaining mixed results when researching implicit physiologically adapting systems [96].”
Measuring and Controlling Social Desirability Bias: Applications in Information Systems Research
Kwak, D., Holtkamp, P., & Kim, S. S. (2019) · Journal of the AIS · doi:10.17005/1.jais.00537 · Cited by 77
Can the true effects of exercise on psychological variables be separated from placebo effects
Ojanen, M. (1994) · International journal of sport psychology · Cited by 70
The Illusion of Control: Placebo Effects of Control Settings
Vaccaro, K., Huang, D., Eslami, M., Sandvig, C., Hamilton, K., & Karahalios, K. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173590 · Cited by 65
An Exploration of the Relation Between Expectations and User Experience
Michalco, J., Simonsen, J. G., & Hornbæk, K. (2015) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2015.1065696 · Cited by 63 · cited 1×
“In a gaming context, Michalco et al. [69] primed users with positive, neutral, or negative game ratings to investigate how knowledge about ratings influences the user’s assessment of the game.”
Judging a bot by its cover: an experiment on expectation setting for personal robots
Paepcke, S., & Takayama, L. A. (2010) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/1734454.1734472 · Cited by 63
The Effect of Ad Blocking on User Engagement with the Web
Miroglio, B. P., Zeber, D., Kaye, J., & Weiss, R. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186162 · Cited by 62
Stop helping me - I'm bored! why assembly assistance needs to be adaptive
Funk, M., Dingler, T., Cooper, J., & Schmidt, A. (2015) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2800835.2807942 · Cited by 58 · cited 1×
“Error-basedAdaptation: Adapting the task difficulty according to the task performance is an established paradigm [31, 93] enabling the user to self-validate the functionality of the task difficulty adaptation.”
Self-citationEffects of Solution Word Attributes on Anagram Difficulty: A Regression Analysis
Gilhooly, K. J., & Johnson, C. E. (1978) · Quarterly Journal of Experimental Psychology · doi:10.1080/14640747808400654 · Cited by 57 · cited 1×
“Each condition contained 40 word puzzles with words that were selected from Gilhooly and Johnson [36].”
de Jong, P. J., Baast, R., Arntz, A., & Merckelbach, H. L. G. J. (1996) · International Journal of Behavioral Medicine · doi:10.1207/s15327558ijbm0301_2 · Cited by 56 · cited 2×
“, pills) or psychological treatment [17, 95].”
Proceedings of the 2015 Federated Conference on Computer Science and Information Systems
(2015) · Annals of Computer Science and Information Systems · doi:10.15439/978-83-60810-66-8 · Cited by 55
AutoManner: An Automated Interface for Making Public Speakers Aware of Their Mannerisms
Tanveer, M. I., Zhao, R., Chen, K., Tiet, Z., & Hoque, M. E. (2016) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/2856767.2856785 · Cited by 53 · cited 1×
“Placebo-Control 4 [35, 71, 97, 111] Placebo Study 12 [18, 20, 22, 23, 28, 82, 100, 107, 110, 110] Acknowledge Placebo Effects as a Confound 11 [1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 105]”
On Being Told How We Feel: How Algorithmic Sensor Feedback Influences Emotion Perception
Hollis, V., Pekurovsky, A., Wu, E., & Whittaker, S. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3264924 · Cited by 52
Traceability and SysML design slices to support safety inspections
Briand, L., Falessi, D., Nejati, S., Sabetzadeh, M., & Yue, T. (2014) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2559978 · Cited by 47
The Placebo Effect in Digital Games: Phantom Perception of Adaptive Artificial Intelligence
Denisova, A., & Cairns, P. A. (2015) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/2793107.2793109 · Cited by 46
Kosch, T., Funk, M., Schmidt, A., & Chuang, L. L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3229093 · Cited by 46
Self-citationComparing Effects of Dynamic Difficulty Adjustment Systems on Video Game Experience
Ang, D., & Mitchell, A. (2017) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3116595.3116623 · Cited by 45 · cited 1×
“[1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 104] We categorize the studies into (a) studies that employed a placebo-control, (b) studies that specifically investigate the existence of a placebo effect, and (c) studies that conclude with a need for placebo-controls.”
Artificial Intelligence for HCI: A Modern Approach
Li, Y., Kumar, R., Lasecki, W. S., & Hilliges, O. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3375147 · Cited by 44
(Over)Trust in Automated Driving: The Sleeping Pill of Tomorrow?
Kundinger, T., Wintersberger, P., & Riener, A. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3312869 · Cited by 42 · cited 1×
“Overtrust refers to the notion that subjective system capabilities exceed objective system capabilities [58].”
VEmotion: Using Driving Context for Indirect Emotion Prediction in Real-Time
Bethge, D., Kosch, T., Große-Puppendahl, T., Chuang, L. L., Kari, M., Jagaciak, A., & Schmidt, A. (2021) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3472749.3474775 · Cited by 41 · cited 1×
“This includes adaptations based on the user’s perceived workload [53–55], emotions [5, 52], and interaction difficulties due to the complex visualization of UI elements [51].”
Self-citationPlayer experience and deceptive expectations of difficulty adaptation in digital games
Denisova, A., & Cairns, P. A. (2018) · Entertainment Computing · doi:10.1016/j.entcom.2018.12.001 · Cited by 39 · cited 1×
“[19, 20] told players that a game adapted the difficulty to their performance, even though no adaptation took place.”
Power-Ups in Digital Games: The Rewarding Effect of Phantom Game Elements on Player Experience
Denisova, A., & Cook, E. (2019) · Proceedings of the Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3311350.3347173 · Cited by 37 · cited 7×
“While the Error-based Adaptation condition presents a control condition of an AI system description in which participants could validate the adaptive system, the Physiology-based Adaptation presents a placebo condition — a description of an AI-based system that has been employed in several HCI studies [18, 20, 100].”
InfluentialUsage of Subjective Scales in Accessibility Research
Trewin, S., Marques, D., & Guerreiro, T. (2015) · Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility - ASSETS '15 · doi:10.1145/2700648.2809867 · Cited by 35 · cited 1×
“Objective measures such as task completion time, error rates, or physiological measures (e.g., heart rate) are less affected by such biases [99].”
Emotions on the Go: Mobile Emotion Assessment in Real-Time using Facial Expressions
Kosch, T., Hassib, M. E., Reutter, R., & Alt, F. (2020) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3399715.3399928 · Cited by 32
Self-citationUser-context for adaptive user interfaces
Shankar, A., Louis, S. J., Dascalu, S. M., Hayes, L. J., & Houmanfar, R. A. (2007) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1216295.1216357 · Cited by 31 · cited 1×
“Further examples include the user’s context [90] and their interaction performance as well as their physiology [56] serving as implicit measurements for user interface adaptations.”
How do Elementary Students Conceptualize Artificial Intelligence?
Ottenbreit-Leftwich, A. T., Glazewski, K. D., Jeon, M., Hmelo‐Silver, C. E., Mott, B., Lee, S., & Lester, J. C. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3439642 · Cited by 31 · cited 1×
“Note that different definitions of AI exist (for a range of potential definitions and conceptualizations, see [40, 63, 79]).”
The user in experimental computer systems research
Dinda, P. A., Memik, G., Dick, R. P., Lin, B., Mallik, A., Gupta, A., & Rossoff, S. (2007) · Workshop on Experimental Computer Science (ExpCS) · doi:10.1145/1281700.1281710 · Cited by 30 · cited 1×
“[1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 104] We categorize the studies into (a) studies that employed a placebo-control, (b) studies that specifically investigate the existence of a placebo effect, and (c) studies that conclude with a need for placebo-controls.”
Cognitive Bias in Usability Testing
Natesan, D., Walker, M., & Clark, S. (2016) · Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care · doi:10.1177/2327857916051015 · Cited by 26 · cited 1×
“They require the participant’s ability to think back and evaluate the interaction with the UI can be susceptible to socially desirable responding (e.g., see [59, 75] or confirmation-bias [76]).”
Left Them 4 Dead: Perception of Humans versus Non-Player Character Teammates in Cooperative Gameplay
Wehbe, R. R., Lank, E., & Nacke, L. E. (2017) · Designing Interactive Systems Conference · doi:10.1145/3064663.3064712 · Cited by 26
Your skin resists: exploring electrodermal activity as workload indicator during manual assembly
Kosch, T., Karolus, J., Ha, H., & Schmidt, A. (2019) · ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) · doi:10.1145/3319499.3328230 · Cited by 25
Self-citationKosch, T., Schmidt, A., Thanheiser, S., & Chuang, L. L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376766 · Cited by 25 · cited 1×
“For example, one can adapt letter presentation in an E-reader to alpha oscillations; an indicator of workload in the electroencephalogram [55], reduce the number of interaction possibilities with the number of errors made by the user [45], or adapt the interface to the mood of the user [61].”
Self-citationRutten, I., & Geerts, D. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376668 · Cited by 25
Chakraborti, T., & Subbarao Kambhampati, B. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314281 · Cited by 23 · cited 1×
“[1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 104] We categorize the studies into (a) studies that employed a placebo-control, (b) studies that specifically investigate the existence of a placebo effect, and (c) studies that conclude with a need for placebo-controls.”
Jeon, M. (2012) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2370216.2370455 · Cited by 22 · cited 1×
“[1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 104] We categorize the studies into (a) studies that employed a placebo-control, (b) studies that specifically investigate the existence of a placebo effect, and (c) studies that conclude with a need for placebo-controls.”
Conceptual Priming for In-game BCI Training
Kosmyna, N., Tarpin-Bernard, F., & Rivet, B. (2015) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2808228 · Cited by 20
Personalized Feedback Emails: A Case Study on Online Introductory Computer Science Courses
Voghoei, S., Tonekaboni, N. H., Yazdansepas, D., Soleymani, S., Farahani, A., & Arabnia, H. R. (2020) · ACM Southeast Regional Conference (ACMSE) · doi:10.1145/3374135.3385274 · Cited by 18 · cited 1×
“Placebo-Control 4 [35, 71, 97, 110] Placebo Study 12 [18, 20, 22, 23, 28, 82, 100, 106, 109, 109] Acknowledge Placebo Effects as a Confound 11 [1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 104]”
Seeing (Movement) is Believing: The Effect of Motion on Perception of Automatic Systems Performance
García García, P., Costanza, E., Verame, J., Nowacka, D., & Ramchurn, S. D. (2021) · Human–Computer Interaction · doi:10.1080/07370024.2018.1453815 · Cited by 17 · cited 6×
“[33] showed that adding an animation that supports a user’s mental model can increase performance ratings of a system, even if it performs objectively in the same manner or even worse than the non-animated system.”
InfluentialSchmorrow, D. D. (2005) · CRC Press eBooks · doi:10.1201/9781482289701-22 · Cited by 15
A Design Platform for Emotion-Aware User Interfaces
Lee, E., Kim, G.-W., Kim, B.-S., & Kang, M.-A. (2014) · ERM4HCI '14 · doi:10.1145/2668056.2668058 · Cited by 15 · cited 1×
“For example, one can adapt letter presentation in an E-reader to alpha oscillations; an indicator of workload in the electroencephalogram [55], reduce the number of interaction possibilities with the number of errors made by the user [45], or adapt the interface to the mood of the user [61].”
Spiel, K., Bertel, S., & Kayali, F. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025721 · Cited by 13 · cited 1×
“Error-based Adaptation: Adapting the task difficulty according to the task performance is an established paradigm [31, 93] enabling the user to self-validate the functionality of the task difficulty adaptation.”
All Talk: How Increasing Interpersonal Communication on Wikis May Not Enhance Productivity
Narayan, S., TeBlunthuis, N., Hale, W. S., Hill, B. M., & Shaw, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359203 · Cited by 11
Should We Give Learners Control Over Item Difficulty?
Papoušek, J., & Pelánek, R. (2017) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3099023.3099080 · Cited by 10 · cited 1×
“Placebo-Control 4 [35, 71, 97, 111] Placebo Study 12 [18, 20, 22, 23, 28, 82, 100, 107, 110, 110] Acknowledge Placebo Effects as a Confound 11 [1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 105]”
Reliability of NIRS-based BCIs: a placebo-controlled replication and reanalysis of brainput
Strait, M. K., Canning, C., & Scheutz, M. J. (2014) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2559206.2578866 · Cited by 8
What Do We Actually Learn from Evaluations in the “Heroic Era” of Visualization? : Position Paper
Correll, M. (2020) · doi:10.1109/beliv51497.2020.00013 · Cited by 8 · cited 1×
“This placebo-control is widely implemented in other scientific fields that evaluate human responses, such as psychological treatment [6], sport science [77], and even visualization research [12].”
Duarte, L., & Carriço, L. M. (2013) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2468356.2468555 · Cited by 6 · cited 2×
“[22, 23] found that players’ heart rates increased in response to sham game elements that allegedly empowered the game character.”
Wilde, D., & Andersen, K. (2010) · Australian Conference on Computer-Human Interaction (OzCHI) · doi:10.1145/1952222.1952262 · Cited by 5 · cited 1×
“Placebo-Control 4 [35, 71, 97, 111] Placebo Study 12 [18, 20, 22, 23, 28, 82, 100, 107, 110, 110] Acknowledge Placebo Effects as a Confound 11 [1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 105]”
Computer games to decrease pain and improve mood and movement
Simmonds, M. J., & Zikos, D. (2014) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/2674396.2674435 · Cited by 5 · cited 1×
“Placebo-Control 4 [35, 71, 97, 110] Placebo Study 12 [18, 20, 22, 23, 28, 82, 100, 106, 109, 109] Acknowledge Placebo Effects as a Confound 11 [1, 7, 10, 21, 47, 64, 74, 83, 86, 91, 94, 104]”
Reiley, D. (2011) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2020408.2020535 · Cited by 3
Duarte, L., & Carriço, L. M. (2012) · doi:10.1145/2367616.2367627 · Cited by 1 · cited 5×
“However, previous placebo research [22, 80, 96, 100] indicated that the sole description of a non-functional adaptive system can positively increase user satisfaction without system functionality.”
Influential
Research works printed but not linked: 5
- T. Bentley. 2000. Biasing web site user evaluations: A study. In Proceedings of the Conference of the Computer Human Interaction Special Interest Group of the Ergonomics Society of Australia. 130–134.
- Aviarini Indrati, Edi Minaji, Sugiharti Binastuti, and Philipus Dwi Raharjo. 2014. Comparation of model unified theory of acceptance and use technology (UTAUT) And technology acceptance model (TAM) for internet adoption of credit union staff. In Proceedings of the 1st International Credit Union Conference on Social Micro.
- Irving Kirsch. 1997. Specifying nonspecifics: Psychological mechanisms of placebo effects. In The Placebo Effect: An Interdisciplinary Exploration Anne Harrington (Ed.). Harvard University Press 166–186.
- 10.2307/41410412 · DOI · Paper page
- 10.1111/j.1540-5915.2010.00292.x · DOI · Paper page
Cited by
Halkiopoulos, C., & Gkintoni, E. (2024) · Electronics · doi:10.3390/electronics13183762 · Cited by 328
Gkintoni, E., Antonopoulou, H., Sortwell, A., & Halkiopoulos, C. (2025) · Brain Sciences · doi:10.3390/brainsci15020203 · Cited by 309
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023) · ACM Computing Surveys · doi:10.1145/3582272 · Cited by 304
Self-citationDraxler, F., Werner, A., Lehmann, F., Hoppe, M., Schmidt, A., Buschek, D., & Welsch, R. (2023) · arXiv · doi:10.48550/arxiv.2303.03283 · Cited by 284
Self-citationPataranutaporn, P., Liu, R., Finn, E., & Maes, P. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00720-7 · Cited by 192
“…also asked to respond to scales from an adapted version of the Unified Theory of Acceptance and Use of Technology (UTAUT) questionnaire and the Task Load Index, which are often used as metrics in the field of human–computer interaction to measure acceptance/usability and workload, respectively 95 .”
Method
Metadata
- Authors
- Kosch, T., Welsch, R., Chuang, L., & Schmidt, A.
- Affiliations
- Utrecht University (NL)
- Ludwig-Maximilians-Universität München (DE)
- Chemnitz University of Technology (DE)
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2022-12-31
- Volume
- 29
- Issue
- 6
- Pages
- 1-32
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4224105048
- Cited by
- 137
- References
- 106
- Citations by year
- 2026: 33 · 2025: 38 · 2024: 30 · 2023: 29 · 2022: 7
- FWCI
- 12.85
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Instant Teaming between Humans and Production Systems
- Saxony State Ministry of Science and Art (SMWK3-7304/35/3-2021/4819)
- European Union’s Horizon 2020 Programme (683008 AMPLIFY)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- placebo effect, artificial intelligence, adaptive user interfaces, user expectations, task performance, technology acceptance, system description, user-centered evaluation, word puzzles
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.