Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
A core tradition of HCI lies in the experimental evaluation of the effects of techniques and interfaces to determine if they are useful for achieving their purpose. However, our individual analyses tend to stand alone, and study results rarely accrue in more precise estimates via meta-analysis: in a literature search, we found only 56 meta-analyses in HCI in the ACM Digital Library, 3 of which were published at CHI (often called the top HCI venue). Yet meta-analysis is the gold standard for demonstrating robust quantitative knowledge. We treat this as a user-centered design problem: the failure to accrue quantitative knowledge is not the users' (i.e. researchers') failure, but a failure to consider those users' needs when designing statistical practice. Using simulation, we compare hypothetical publication worlds following existing frequentist against Bayesian practice. We show that Bayesian analysis yields more precise effects with each new study, facilitating knowledge accrual without traditional meta-analyses. Bayesian practices also allow more principled conclusions from small-n studies of novel techniques. These advantages make Bayesian practices a likely better fit for the culture and incentives of the field. Instead of admonishing ourselves to spend resources on larger studies, we propose using tools that more appropriately analyze small studies and encourage knowledge accrual from one study to the next. We also believe Bayesian methods can be adopted from the bottom up without the need for new incentives for replication or meta-analysis. These techniques offer the potential for a more user- (i.e. researcher-) centered approach to statistical analysis in HCI.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Bayesian statistics has also been proposed as a better alternative to frequentist statistics for HCI research[43].”
From Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2023) · cites this paper as [43]
In the citing paper
“Within HCI, there is an emphasis on novel technology tools and artifacts, not necessarily reliable effect size estimates/causal inferences [47].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [47]
In the citing paper
“As demonstrated elsewhere, careful use of priors within studies of novel systems can be used to help mitigate problems with magnitude effect errors and thus could be a more appropriate statistical method for the generate phase [47].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [47]
Citations
Cites
<tt>emcee</tt>: The MCMC Hammer
Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. (2013) · Publications of the Astronomical Society of the Pacific · doi:10.1086/670067 · Cited by 12,414
Stan : A Probabilistic Programming Language
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.i01 · Cited by 7,754 · cited 1×
“These tools include modeling languages like JAGS [23] and Stan [2] (both with R packages, and Stan includes a Python interface), and Python-specific libraries like emcee [7] and PyMC [22].”
MethodJAGS: A program for analysis of Bayesian graphical models using Gibbs sampling
Plummer, M. (2003) · Cited by 4,720
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“There are many problems with this approach, particularly when a field utilizes small samples to estimate statistical significance, as many of these significant differences are likely due to chance [17].”
Gelman, A. (2006) · Bayesian Analysis · doi:10.1214/06-ba117a · Cited by 4,050 · cited 1×
“Other accessible articles have also been written about practical concerns in Bayesian analysis, including discussions of how to choose priors [9,11].”
BackgroundThe New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
The aligned rank transform for nonparametric factorial analyses using only anova procedures
Wobbrock, J. O., Findlater, L., Gergle, D., & Higgins, J. J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978963 · Cited by 2,854
A weakly informative default prior distribution for logistic and other regression models
Gelman, A., Jakulin, A., Pittau, M. G., & Su, Y.-S. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas191 · Cited by 1,790 · cited 4×
“[9]).”
BackgroundMethodInfluentialPrior distributions for variance parameters in hierarchical modelsPreprint
Gelman, A. (2004) · RePEc: Research Papers in Economics · Cited by 1,632
Bayesian estimation supersedes the t-test
Kruschke, J. K. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0029146 · Cited by 1,530
Research works printed but not linked: 3
- Joel Brandt, Mira Dontcheva, Marcos Weskamp, and Scott R. Klemmer. 2010. Example-centric programming. Proceedings of the 28th international conference on Human factors in computing systems CHI '10, ACM Press, 513. Retrieved January 7, 2016 from http://dl.acm.org/citation.cfm?id=1753326.1753402
- Zoltan Dienes. 2011. Bayesian Versus Orthodox Statistics: Which Side Are You On? Perspectives on psychological science?: a journal of the Association for Psychological Science 6, 3: 274-90. Retrieved January 7, 2016 from http://pps.sagepub.com/content/6/3/274.abstract
- 10.1037/0033-2909.88.2.359 · DOI · Paper page
Cited by
An Aligned Rank Transform Procedure for Multifactor Contrast Tests
Elkin, L. A., Kay, M., Higgins, J. J., & Wobbrock, J. O. (2021) · The 34th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3472749.3474784 · Cited by 664
“[25] and Robertson and Kaptein [38] introduced more modern statistical methods, and Wobbrock et al.”
MethodSelf-citationThe Experience Sampling Method on Mobile Devices
van Berkel, N., Ferreira, D., & Kostakos, V. (2018) · ACM Computing Surveys · doi:10.1145/3123988 · Cited by 432
“However, we do recommend researchers to use Bayesian statistics, as this allows for statistical inference on small sample sizes (Kay et al. 2016).”
MethodExplanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
The Effects of Generative AI on Design Fixation and Divergent Thinking
Wadinambiarachchi, S., Kelly, R. M., Pareek, S., Zhou, Q., & Velloso, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642919 · Cited by 320
“We model the random effects of participants and images as being drawn from a normal distribution with mean zero and standard deviation computed from the data through partial pooling.”
Kim, N. J., Belland, B. R., & Walker, A. E. (2018) · Educational Psychology Review · doi:10.1007/s10648-017-9419-1 · Cited by 220
“Another possible reason is that BMA could resolve issues of overweighted effect size caused by smaller studies, which were included in the traditional meta-analysis (Kay et al. 2016; Kicinski 2013; Mengersen et al. 2016).”
MethodInfluential
Metadata
- Authors
- Kay, M., Nelson, G. L., & Hekler, E. B.
- Affiliations
- University of Washington (US)
- University of Washington, Seattle, WA, USA
- Arizona State University (US)
- Venue
- Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2016-05-07
- Pages
- 4521-4532
- OpenAlex
- W2404411786
- Cited by
- 113
- References
- 30
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 11 · 2023: 16 · 2022: 11 · 2021: 10 · 2020: 14 · 2019: 14 · 2018: 7 · 2017: 12 · 2016: 2
- FWCI
- 12.36
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- National Science Foundation (DGE-1256082)
- Robert Wood Johnson Foundation (71995)
- Intel Corporation (Intel Science and Technology Center for Pervasive Computing)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- Bayesian analysis, frequentist statistics, meta-analysis, replication, user-centered design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI. https://app.cochat.ai/papers/10.1145/2858036.2858465
- The paper
- Kay, M., Nelson, G. L., & Hekler, E. B. (2016). Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/2858036.2858465
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.