A Primer for Conducting Experiments in Human–Robot Interaction
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
We provide guidelines for planning, executing, analyzing, and reporting hypothesis-driven experiments in Human–Robot Interaction (HRI). The intended audience are researchers in the field of HRI who are not trained in empirical research but who are interested in conducting rigorous human-participant studies to support their research. Following the chronological order of research activities and grounded in updated research practices in psychological and behavioral sciences, this primer covers recommended methods and common pitfalls for defining research questions, identifying constructs and hypotheses, choosing appropriate study designs, operationalizing constructs as variables, planning and executing studies, sampling, choosing statistical tools for data analysis, and reporting results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Three Approaches to Qualitative Content Analysis
Hsieh, H.-F., & Shannon, S. E. (2005) · Qualitative Health Research · doi:10.1177/1049732305276687 · Cited by 44,601
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
Tausczik, Y. R., & Pennebaker, J. W. (2010) · Journal of Language and Social Psychology · doi:10.1177/0261927x09351676 · Cited by 5,774 · cited 1×
“These tools vary from the well-established Linguistic Inquiry and Word Count (LIWC) program [78] to evolving natural language processing (NLP) and machine learning techniques such as topic modeling and word embedding (see Evans and Aceves [28]; also see Lee and Kolodge [52] for an example of using topic modeling to st…”
BackgroundMacKinnon, D. P., Fairchild, A. J., & Fritz, M. S. (2007) · Annual Review of Psychology · doi:10.1146/annurev.psych.58.110405.085542 · Cited by 5,688 · cited 1×
“Mediation analysis enables researchers to examine the “how” and “why” questions by hypothe-sizing an internal (often psychological) process through which one construct affects another [54].”
Background1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Nature · doi:10.1038/466029a · Cited by 3,441
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · cited 1×
“This practice is called pre-registration [62, 83].”
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Scanning the horizon: towards transparent and reproducible neuroimaging research
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.167 · Cited by 1,410
Cronbach's alpha reliability: Interval estimation, hypothesis testing, and sample size planning
Bonett, D. G., & Wright, T. A. (2015) · Journal of Organizational Behavior · doi:10.1002/job.1960 · Cited by 1,237
Socially intelligent robots: Dimensions of human-robot interaction
Dautenhahn, K. (2007) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2006.2004 · Cited by 1,210 · cited 1×
“This tutorial aims to strike a balance between scope and depth and is therefore complementary to existing publications that discuss methods in HRI on a conceptual level [22] or provide targeted recommendations on specific methodological issues [7, 67].”
BackgroundRevised standards for statistical evidence
Johnson, V. E. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1313476110 · Cited by 808 · cited 2×
“, References [6, 44])—yet this suggestion leads others to be concerned about inflated Type II errors [33].”
BackgroundThe rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · cited 1×
“This progress has been in part a response to a replication crisis that started in the early 2010s and has led to active discussion resulting in major changes in how empirical research is conducted in a variety of empirical disciplines [3, 5, 25, 30, 63, 71].”
BackgroundWhy Are Self-Report and Behavioral Measures Weakly Correlated?
Dang, J., King, K. M., & Inzlicht, M. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.01.007 · Cited by 634
Almost human: Anthropomorphism increases trust resilience in cognitive agents.
de Visser, E. J., Monfort, S. S., McKendrick, R., Smith, M. A. B., McKnight, P. E., Krueger, F., & Parasuraman, R. (2016) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000092 · Cited by 614 · cited 1×
“For instance, you could ask people to rate how trustworthy they perceive the robot to be us-ing a questionnaire (e.g., Reference [24]).”
BackgroundFerguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · cited 1×
“This progress has been in part a response to a replication crisis that started in the early 2010s and has led to active discussion resulting in major changes in how empirical research is conducted in a variety of empirical disciplines [3, 5, 25, 30, 63, 71].”
BackgroundThe Robotic Social Attributes Scale (RoSAS): Development and Validation
Carpinella, C. M., Wyman, A. B., Perez, M. A., & Stroessner, S. J. (2017) · Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2909824.3020208 · Cited by 569
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · cited 2×
“” For many years, however, and more so recently, the use of the term “significant” for findings that cross this somewhat arbitrary p-threshold is pointed out as detrimental to research and reliable findings [47, 59].”
BackgroundExperiments with More Than One Random Factor: Designs, Analytic Models, and Statistical Power
Judd, C. M., Westfall, J., & Kenny, D. A. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033702 · Cited by 466 · cited 1×
“[45] and Singmann and Kellen [72] provide information and software listings for multilevel linear models.”
BackgroundAnderson, S. F., Kelley, K., & Maxwell, S. E. (2017) · Psychological Science · doi:10.1177/0956797617723724 · Cited by 406 · cited 2×
“New practices have been introduced and are increasingly considered as necessary to improve research reproducibility [2, 42, 70].”
BackgroundWhat to believe: Bayesian methods for data analysis
Kruschke, J. K. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.05.001 · Cited by 342 · cited 1×
“Bayesian methods have several advantages over NHST, such as using prior probability distributions to incorporate information from previous studies and the interpretability of posterior probabilities [49, 84].”
MethodPhillips, E., Zhao, X., Ullman, D., & Malle, B. F. (2018) · Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3171221.3171268 · Cited by 233
Self-citationWhy Do They Refuse to Use My Robot?: Reasons for Non-Use Derived from a Long-Term Home Study
de Graaf, M., Ben Allouch, S., & van Dijk, J. (2017) · Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2909824.3020236 · Cited by 169 · cited 1×
“Consider textual analysis as an example, a technique widely used in qualitative research to provide valuable insights into people’s subjective experiences [23, 34, 40].”
Method
Cited by
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
Laban, G., Kappas, A., Morrison, V., & Cross, E. S. (2024) · International Journal of Social Robotics · doi:10.1007/s12369-023-01076-z · Cited by 119
Principles and Guidelines for Evaluating Social Robot Navigation Algorithms
Francis, A., Pérez-D’Arpino, C., Li, C., Xia, F., Alahi, A., Alami, R., Bera, A., Biswas, A., Biswas, J., Chandra, R., et al. (2025) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3700599 · Cited by 107
Leichtmann, B., Nitsch, V., & Mara, M. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.838116 · Cited by 66
Robot-Mediated Inclusive Processes in Groups of Children: From Gaze Aversion to Mutual Smiling Gaze
Tuncer, S., Gillet, S., & Leite, I. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.729146 · Cited by 26
Metadata
- Authors
- Hoffman, G., & Zhao, X.
- Venue
- ACM Transactions on Human-Robot Interaction
- Published
- 2020-10-20
- Volume
- 10
- Issue
- 1
- Pages
- 1-31
- ISSN
- 2573-9522 (ISSN-L)
- OpenAlex
- W3095030303
- Cited by
- 135
- References
- 29
- FWCI
- 16.13
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.