Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
A core tradition of HCI lies in the experimental evaluation of the effects of techniques and interfaces to determine if they are useful for achieving their purpose. However, our individual analyses tend to stand alone, and study results rarely accrue in more precise estimates via meta-analysis: in a literature search, we found only 56 meta-analyses in HCI in the ACM Digital Library, 3 of which were published at CHI (often called the top HCI venue). Yet meta-analysis is the gold standard for demonstrating robust quantitative knowledge. We treat this as a user-centered design problem: the failure to accrue quantitative knowledge is not the users' (i.e. researchers') failure, but a failure to consider those users' needs when designing statistical practice. Using simulation, we compare hypothetical publication worlds following existing frequentist against Bayesian practice. We show that Bayesian analysis yields more precise effects with each new study, facilitating knowledge accrual without traditional meta-analyses. Bayesian practices also allow more principled conclusions from small-n studies of novel techniques. These advantages make Bayesian practices a likely better fit for the culture and incentives of the field. Instead of admonishing ourselves to spend resources on larger studies, we propose using tools that more appropriately analyze small studies and encourage knowledge accrual from one study to the next. We also believe Bayesian methods can be adopted from the bottom up without the need for new incentives for replication or meta-analysis. These techniques offer the potential for a more user- (i.e. researcher-) centered approach to statistical analysis in HCI.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Bayesian statistics has also been proposed as a better alternative to frequentist statistics for HCI research[43].”
From Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2023) · cites this paper as [43]
In the citing paper
“Within HCI, there is an emphasis on novel technology tools and artifacts, not necessarily reliable effect size estimates/causal inferences [47].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [47]
In the citing paper
“As demonstrated elsewhere, careful use of priors within studies of novel systems can be used to help mitigate problems with magnitude effect errors and thus could be a more appropriate statistical method for the generate phase [47].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [47]
Citations
Cites
<tt>emcee</tt>: The MCMC Hammer
Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. (2013) · Publications of the Astronomical Society of the Pacific · doi:10.1086/670067 · Cited by 12,414
Stan : A Probabilistic Programming Language
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.i01 · Cited by 7,754 · cited 1×
“These tools include modeling languages like JAGS [23] and Stan [2] (both with R packages, and Stan includes a Python interface), and Python-specific libraries like emcee [7] and PyMC [22].”
MethodJAGS: A program for analysis of Bayesian graphical models using Gibbs sampling
Plummer, M. (2003) · Cited by 4,720
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“There are many problems with this approach, particularly when a field utilizes small samples to estimate statistical significance, as many of these significant differences are likely due to chance [17].”
Gelman, A. (2006) · Bayesian Analysis · doi:10.1214/06-ba117a · Cited by 4,050 · cited 1×
“Other accessible articles have also been written about practical concerns in Bayesian analysis, including discussions of how to choose priors [9,11].”
BackgroundThe New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
The aligned rank transform for nonparametric factorial analyses using only anova procedures
Wobbrock, J. O., Findlater, L., Gergle, D., & Higgins, J. J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978963 · Cited by 2,854
A weakly informative default prior distribution for logistic and other regression models
Gelman, A., Jakulin, A., Pittau, M. G., & Su, Y.-S. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas191 · Cited by 1,790 · cited 4×
“[9]).”
BackgroundMethodInfluentialPrior distributions for variance parameters in hierarchical modelsPreprint
Gelman, A. (2004) · RePEc: Research Papers in Economics · Cited by 1,629
Bayesian estimation supersedes the t-test
Kruschke, J. K. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0029146 · Cited by 1,530
Research works printed but not linked: 3
- Joel Brandt, Mira Dontcheva, Marcos Weskamp, and Scott R. Klemmer. 2010. Example-centric programming. Proceedings of the 28th international conference on Human factors in computing systems CHI '10, ACM Press, 513. Retrieved January 7, 2016 from http://dl.acm.org/citation.cfm?id=1753326.1753402
- Zoltan Dienes. 2011. Bayesian Versus Orthodox Statistics: Which Side Are You On? Perspectives on psychological science?: a journal of the Association for Psychological Science 6, 3: 274-90. Retrieved January 7, 2016 from http://pps.sagepub.com/content/6/3/274.abstract
- 10.1037/0033-2909.88.2.359 · DOI · Paper page
Cited by
Items 101–138 of 138
Meinhardt, L.-M., Dragic, M., Colley, M., Lukoff, K., & Rukzio, E. (2026) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3831979 · Cited by 1
Dym, B., & Fiesler, C. L. (2019) · International ACM SIGGROUP Conference on Supporting Group Work (GROUP) · doi:10.1145/3323994.3369881 · Cited by 1
Dix, A. J. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3503758 · Cited by 1
“It will also help attendees to make sense of the various alternative approaches presented in recent articles in HCI and wider scientifc literature (e.g. [1, 2, 5–7]).”
BackgroundSuzuki, M., & Yamamoto, Y. (2022) · International Conference on Information Technology for Social Good (GoodIT) · doi:10.1145/3524458.3547222 · Cited by 1
“In addition, a GLMM allows for more accurate analysis of small-scale data than methods using frequentist statistics [7].”
MethodEmpowering domain experts to author valid statistical analyses
Jun, E. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526114.3558530 · Cited by 1
Does Interactive Conditioning Help Users Better Understand the Structure of Probabilistic Models?
Taka, E., Stein, S., & Williamson, J. H. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3231967 · Cited by 1
“Non-experienced researchers who conduct experiments and analyze data do not feel confident to use such methods for the analysis [1], [2] even when they have access to learning and exploration tools [3].”
Dix, A. J. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3574185 · Cited by 1
Malysheva, Y., Kelleher, C. L., & Ericson, B. J. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699562 · Cited by 1
“Inrecent years, researchers have argued that Bayesian meth-ods are better suited than traditional Null Hypothesis Significance Testing (NHST) for several areas of research on human behavior, including Psychology [14], Human-Computer Interaction [11], and Organizational Sciences [15].”
Wang, Y., & Reiterer, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10315 · Cited by 1
“Besides, Bayesian methods also allow more principled conclusions from small-n studies of novel techniques in the field of human-computer interaction [20].”
MethodData quality and quantity in mobile experience sampling
van Berkel, N. (2019) · Minerva Access (University of Melbourne) · Cited by 6
Crowd process design : how to coordinate crowds to solve complex problems
Boer, P. D. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-144556 · Cited by 3
Making HCI statistics better: saving the baby
Dix, A. J. (2017)
Weasel Finder: Highlighting Ambiguous Sentences for Promoting Critical Web Browsing
Saito, F., & Yamamoto, Y. (2020) · Transactions of the Japanese Society for Artificial Intelligence · doi:10.1527/tjsai.36-1_wi2-h
Designing Technologies to Support Patient Engagement in the Hospital
Mishra, S. R. (2019) · ResearchWorks at the University of Washington (University of Washington) · Cited by 1
Dix, A. J. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3445018
“It will also help attendees to make sense of the various alternative approaches presented in recent articles in HCI and wider scientifc literature (e.g. [1, 2, 5–7]).”
BackgroundEvaluating Bayesian Model VisualisationsPreprint
Stein, S., & Williamson, J. H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.03604
“Some common data science practices have been criticised [22] for being a unidirectional, statistician-controlled process: starting with a dataset that is publicly available or has been collected in advance, models are learned and compared and a subset of evaluation data is presented via static…”
BackgroundTransparent Practices for Quantitative Empirical Research (Course proposal)Preprint
Wacharamanotham, C., Yang, F., Pu, X., Sarma, A., & Padilla, L. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/2dnq7
Model-based Evaluation of Recall-based Interaction Techniques
Gori, J., Fruchard, B., & Bailly, G. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642637
Budget, Cost, or Both? An Empirical Exploration of Mechanisms in Quadratic Surveys
Cheng, T.-C., Li, T. W., Karahalios, K. G., & Sundaram, H. (2025) · ACM Collective Intelligence Conference (CI) · doi:10.1145/3715928.3737474 · Cited by 1
Kim, E., Shin, H., & Oh, C. (2025) · Archives of Design Research · doi:10.15187/adr.2025.08.38.3.149
PriorWeaver: Prior Elicitation via Iterative Dataset Construction
Xiao, Y., Ma, S., Oulasvirta, A., & Jun, E. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790349 · Cited by 2
“Our work builds on and contributes to the literature on interactive tools for prior elicitation, perspectives on what makes a good prior, and the role of visualization in belief elicitation.”
Kay, M., Nelson, G., & Hekler, E. B. (2016) · Figshare · doi:10.6084/m9.figshare.2061669
Self-citationTransforming the Reading Experience of Scientific Documents with Polymorphism
Masson, D. (2023) · UWSpace (University of Waterloo) · Cited by 2
Henry, N. F. (2023) · White Rose eTheses Online (University of Leeds, The University of Sheffield, University of York) · Cited by 1
Ulysses in cyberspace: examining the effectiveness of design patterns for digital self-control
Lyngs, U. (2019) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.5287/ora-avq9rbk9n · Cited by 4
RDoFlow: Automatically assessing under-specified statistical analyses in HCI
Grunde-McLaughlin, M., Liu, W., Patil, R., Migineishvili, N., Reif, E., Krishna, R., Weld, D. S., & Heer, J. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789160
“…researchers have proposed several structural strategies for improving experiment reliability, e.g., increasing experimental power [26, 52, 78], reforming publication incentive structures [5, 18, 44, 51, 53, 70, 83, 84], improving statistical education [12, 57, 77], and gamifying open science [27].”
Capra, O. (2020) · theses.fr (ABES) · doi:10.70675/ab102fcbz59bez4219zb08bzc81213bac0fe · Cited by 4
Thompson, R. E., Sweet, R. H., Dameff, C. J., Tully, J. L., & Votipka, D. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791788
Guerra, E., Semmelhack, E. A., Schacht, A., & Prilla, M. (2026) · Mensch und Computer (MuC) · doi:10.1145/3820253.3820271
“If this interval includes the null effect value and the interval bounds are close to this value, this indicates a lack of reliable evidence for a difference between the two conditions [22].”
Welsch, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.00163
EEG and Eye-Tracking Evidence That AI Disclosure Shapes Face EvaluationPreprint
Mitrevska, T., Donat, L., Butz, A. M., Kosch, T., Ali, A. E., & Chiossi, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.10182
Markovitch, B. (2026) · TU/e Research Portal
Markovitch, B., Kamps, J. C. C., Markopoulos, P., & Birk, M. V. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3831284 · Cited by 1
Agentic Path Maps: Exposing an Agent's Traversal Through the Garden of Forking PathsPreprint
Grunde-McLaughlin, M., Usman, H., Mitroshkov, D., Reif, E., Migineishvili, N., Yu, L. C. K., Heer, J., & Weld, D. S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.04183
Exploring the Translation-As-Explanation Folk Wisdom for Program Editing Tasks
Yim, J., Lubin, J. P., Ye, K., Walker, L., Rawn, E., & Chasins, S. (2026) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc71707.2026.11772778
Metadata
- Authors
- Kay, M., Nelson, G. L., & Hekler, E. B.
- Affiliations
- University of Washington (US)
- University of Washington, Seattle, WA, USA
- Arizona State University (US)
- Venue
- Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2016-05-07
- Pages
- 4521-4532
- OpenAlex
- W2404411786
- Cited by
- 113
- References
- 30
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 11 · 2023: 16 · 2022: 11 · 2021: 10 · 2020: 14 · 2019: 14 · 2018: 7 · 2017: 12 · 2016: 2
- FWCI
- 12.36
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- National Science Foundation (DGE-1256082)
- Robert Wood Johnson Foundation (71995)
- Intel Corporation (Intel Science and Technology Center for Pervasive Computing)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- Bayesian analysis, frequentist statistics, meta-analysis, replication, user-centered design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.