Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
A core tradition of HCI lies in the experimental evaluation of the effects of techniques and interfaces to determine if they are useful for achieving their purpose. However, our individual analyses tend to stand alone, and study results rarely accrue in more precise estimates via meta-analysis: in a literature search, we found only 56 meta-analyses in HCI in the ACM Digital Library, 3 of which were published at CHI (often called the top HCI venue). Yet meta-analysis is the gold standard for demonstrating robust quantitative knowledge. We treat this as a user-centered design problem: the failure to accrue quantitative knowledge is not the users' (i.e. researchers') failure, but a failure to consider those users' needs when designing statistical practice. Using simulation, we compare hypothetical publication worlds following existing frequentist against Bayesian practice. We show that Bayesian analysis yields more precise effects with each new study, facilitating knowledge accrual without traditional meta-analyses. Bayesian practices also allow more principled conclusions from small-n studies of novel techniques. These advantages make Bayesian practices a likely better fit for the culture and incentives of the field. Instead of admonishing ourselves to spend resources on larger studies, we propose using tools that more appropriately analyze small studies and encourage knowledge accrual from one study to the next. We also believe Bayesian methods can be adopted from the bottom up without the need for new incentives for replication or meta-analysis. These techniques offer the potential for a more user- (i.e. researcher-) centered approach to statistical analysis in HCI.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Bayesian statistics has also been proposed as a better alternative to frequentist statistics for HCI research[43].”
From Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2023) · cites this paper as [43]
In the citing paper
“Within HCI, there is an emphasis on novel technology tools and artifacts, not necessarily reliable effect size estimates/causal inferences [47].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [47]
In the citing paper
“As demonstrated elsewhere, careful use of priors within studies of novel systems can be used to help mitigate problems with magnitude effect errors and thus could be a more appropriate statistical method for the generate phase [47].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [47]
Citations
Cites
<tt>emcee</tt>: The MCMC Hammer
Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. (2013) · Publications of the Astronomical Society of the Pacific · doi:10.1086/670067 · Cited by 12,414
Stan : A Probabilistic Programming Language
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.i01 · Cited by 7,754 · cited 1×
“These tools include modeling languages like JAGS [23] and Stan [2] (both with R packages, and Stan includes a Python interface), and Python-specific libraries like emcee [7] and PyMC [22].”
MethodJAGS: A program for analysis of Bayesian graphical models using Gibbs sampling
Plummer, M. (2003) · Cited by 4,720
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“There are many problems with this approach, particularly when a field utilizes small samples to estimate statistical significance, as many of these significant differences are likely due to chance [17].”
Gelman, A. (2006) · Bayesian Analysis · doi:10.1214/06-ba117a · Cited by 4,050 · cited 1×
“Other accessible articles have also been written about practical concerns in Bayesian analysis, including discussions of how to choose priors [9,11].”
BackgroundThe New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
The aligned rank transform for nonparametric factorial analyses using only anova procedures
Wobbrock, J. O., Findlater, L., Gergle, D., & Higgins, J. J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978963 · Cited by 2,854
A weakly informative default prior distribution for logistic and other regression models
Gelman, A., Jakulin, A., Pittau, M. G., & Su, Y.-S. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas191 · Cited by 1,790 · cited 4×
“[9]).”
BackgroundMethodInfluentialPrior distributions for variance parameters in hierarchical modelsPreprint
Gelman, A. (2004) · RePEc: Research Papers in Economics · Cited by 1,632
Bayesian estimation supersedes the t-test
Kruschke, J. K. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0029146 · Cited by 1,530
A WEAKLY INFORMATIVE DEFAULT PRIOR DISTRIBUTION FOR LOGISTIC AND OTHER REGRESSION MODELS
Gelman, A., Jakulin, A., & Grazia, M. (2009) · Cited by 1,352
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343 · cited 3×
“With a frequentist analysis, this increases the probability of what Gelman calls a magnitude error [8]: because the confidence intervals are so wide, the only effects that reach significance are those that overestimate the effect size.”
BackgroundInfluentialPyMC: Bayesian Stochastic Modelling in Python
Patil, A., Huard, D., & Fonnesbeck, C. (2010) · Journal of Statistical Software · doi:10.18637/jss.v035.i04 · Cited by 711 · cited 1×
“These tools include modeling languages like JAGS [23] and Stan [2] (both with R packages, and Stan includes a Python interface), and Python-specific libraries like emcee [7] and PyMC [22].”
MethodUsability evaluation considered harmful (some of the time)
Greenberg, S., & Buxton, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357074 · Cited by 598
Vote-counting methods in research synthesis.
Hedges, L. V., & Olkin, I. (1980) · Psychological Bulletin · doi:10.1037//0033-2909.88.2.359 · Cited by 495 · cited 1×
“The classic strategy for knowledge accrual of a series of NHST studies is literature reviews conducted often in the related work section of a CHI paper that implicitly use the vote-counting method of knowledge accumulation [13].”
BackgroundRobust misinterpretation of confidence intervals
Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.-J. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0572-3 · Cited by 425 · cited 2×
“Unfortunately, this is an inaccurate interpretation and lies at the heart of the argument to shift interpretation towards effect sizes and confidence intervals [3,6] (though 95% confidence intervals are the inverse of p .05, thus still succumbing to the interpretation problem [14]).”
BackgroundFair Statistical Communication in HCI
Dragicevic, P. (2016) · Human–Computer Interaction Series · doi:10.1007/978-3-319-26633-6_13 · Cited by 325 · cited 1×
“Others emphasize deeper changes to statistical practice, such as reducing the focus on p-values in favor of effect size estimates and confidence intervals (the New Statistics [3]; or in HCI, Kaptein & Robertson [18] or Dragicevic [5]), or the abandonment of frequentist null hypothe-”
BackgroundExample-centric programming: integrating web search into the development environment
Brandt, J., Dontcheva, M., Weskamp, M., & Klemmer, S. R. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753402 · Cited by 324
Kruschke, J. K. (2010) · WIREs Cognitive Science · doi:10.1002/wcs.72 · Cited by 317
Meta-analysis of correlations among usability measures
Hornbæk, K., & Law, E. L.-C. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240722 · Cited by 276
Modern Statistical Methods for HCI
(2016) · Human–Computer Interaction Series · doi:10.1007/978-3-319-26633-6 · Cited by 253
Yee, N., Bailenson, J. N., & Rickertsen, K. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240626 · Cited by 242
Self disclosure on computer forms: meta-analysis and implications
Weisband, S., & Kiesler, S. (1996) · Proceedings of the SIGCHI conference on Human factors in computing systems common ground - CHI '96 · doi:10.1145/238386.238387 · Cited by 178 · cited 1×
“Only 3 were published at the venues above [15,25,28].”
BackgroundHoward, G. S., Maxwell, S. E., & Fleming, K. J. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.3.315 · Cited by 141
Gelman, A., & Weakliem, D. L. (2009) · American Scientist · doi:10.1511/2009.79.310 · Cited by 137 · cited 1×
“Due to limited resources and an emphasis on novelty, HCI studies are often conducted with fewer participants than a traditional power analysis would suggest is prudent [10,18].”
BackgroundKosugi, K. E., Maeda, K., & Hirakawa, M. (2017) · The Proceedings of the Annual Convention of the Japanese Psychological Association · doi:10.4992/pacjpa.81.0_tws-011 · Cited by 85
Where Am I? A Meta-Analysis of Experiments on the Effects of Progress Indicators for Web Surveys
Villar, A., Callegaro, M., & Yang, Y. (2013) · Social Science Computer Review · doi:10.1177/0894439313497468 · Cited by 75 · cited 4×
“In their meta-analysis, Villar et al. [24], found that using a slow-to-fast progress indicator decreased the probability that a person would complete the survey.”
MethodInfluentialRethinking statistical analysis methods for CHI
Kaptein, M., & Robertson, J. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208557 · Cited by 70
RepliCHI - CHI should be replicating and validating results more: discuss
Wilson, M. L., Mackay, W., Chi, E., Bernstein, M., Russell, D., & Thimbleby, H. (2011) · CHI '11 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1979742.1979491 · Cited by 56 · cited 2×
“Paralleling so-called replication crises in psychology and medicine, movements such as RepliCHI [26] have called for an increased focus on replication and meta-analysis to effectively accrue quantitative knowledge about the utility and generalizability of our technologies.”
BackgroundMethod
Research works printed but not linked: 3
- Joel Brandt, Mira Dontcheva, Marcos Weskamp, and Scott R. Klemmer. 2010. Example-centric programming. Proceedings of the 28th international conference on Human factors in computing systems CHI '10, ACM Press, 513. Retrieved January 7, 2016 from http://dl.acm.org/citation.cfm?id=1753326.1753402
- Zoltan Dienes. 2011. Bayesian Versus Orthodox Statistics: Which Side Are You On? Perspectives on psychological science?: a journal of the Association for Psychological Science 6, 3: 274-90. Retrieved January 7, 2016 from http://pps.sagepub.com/content/6/3/274.abstract
- 10.1037/0033-2909.88.2.359 · DOI · Paper page
Cited by
An Aligned Rank Transform Procedure for Multifactor Contrast Tests
Elkin, L. A., Kay, M., Higgins, J. J., & Wobbrock, J. O. (2021) · The 34th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3472749.3474784 · Cited by 664
“[25] and Robertson and Kaptein [38] introduced more modern statistical methods, and Wobbrock et al.”
MethodSelf-citationThe Experience Sampling Method on Mobile Devices
van Berkel, N., Ferreira, D., & Kostakos, V. (2018) · ACM Computing Surveys · doi:10.1145/3123988 · Cited by 432
“However, we do recommend researchers to use Bayesian statistics, as this allows for statistical inference on small sample sizes (Kay et al. 2016).”
MethodExplanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
The Effects of Generative AI on Design Fixation and Divergent Thinking
Wadinambiarachchi, S., Kelly, R. M., Pareek, S., Zhou, Q., & Velloso, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642919 · Cited by 320
“We model the random effects of participants and images as being drawn from a normal distribution with mean zero and standard deviation computed from the data through partial pooling.”
Kim, N. J., Belland, B. R., & Walker, A. E. (2018) · Educational Psychology Review · doi:10.1007/s10648-017-9419-1 · Cited by 220
“Another possible reason is that BMA could resolve issues of overweighted effect size caused by smaller studies, which were included in the traditional meta-analysis (Kay et al. 2016; Kicinski 2013; Mengersen et al. 2016).”
MethodInfluential
Metadata
- Authors
- Kay, M., Nelson, G. L., & Hekler, E. B.
- Affiliations
- University of Washington (US)
- University of Washington, Seattle, WA, USA
- Arizona State University (US)
- Venue
- Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2016-05-07
- Pages
- 4521-4532
- OpenAlex
- W2404411786
- Cited by
- 113
- References
- 30
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 11 · 2023: 16 · 2022: 11 · 2021: 10 · 2020: 14 · 2019: 14 · 2018: 7 · 2017: 12 · 2016: 2
- FWCI
- 12.36
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- National Science Foundation (DGE-1256082)
- Robert Wood Johnson Foundation (71995)
- Intel Corporation (Intel Science and Technology Center for Pervasive Computing)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- Bayesian analysis, frequentist statistics, meta-analysis, replication, user-centered design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.