Virtual reality check: Statistical power, reported results, and the validity of research on the psychology of virtual reality and immersive environments
The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Virtual reality (VR) is a popular subject of scientific study across a variety of academic fields. In the present study we evaluate methodological trends in behavioral research on VR with respect to data collection practices, statistical reporting, and data availability. In line with this goal, we conducted a meta-scientific analysis of 61 articles encompassing a total of 1,122 statistical tests and highlight three emergent trends that inform our understanding of past and future studies focused on VR. Conclusions from analysis of the data include a high incidence of errors in statistical reporting, and a general lack of transparency with respect to the availability of study data. Transparency in data analysis, increased statistical power, and more careful reporting of statistical outcomes are suggested to heighten methodological rigor and improve reproducibility in the field of VR research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1×
“Their findings indicated that while 97% of the original 100 studies reported having statistically significant results, only 36% of the replications actually did (Open Science Collaboration, 2015).”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 2×
“The American Statistical Association has even issued a statement on how p -values should be used, in part, because of their widespread misuse (Wasserstein & Lazar, 2016).”
BackgroundDefining virtual reality: Dimensions determining telepresence
Steuer, J. (1992) · Journal of Communication · doi:10.1111/j.1460-2466.1992.tb00812.x · Cited by 5,459 · cited 3×
“Krueger focused on “artificial reality” as immersive spaces using projectors, goggles, and reality gloves (Steuer, 1992).”
BackgroundStatistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
A head-mounted three dimensional display
Sutherland, I. E. (1968) · Proceedings of the December 9-11, 1968, fall joint computer conference, part I on - AFIPS '68 (Fall, part I) · doi:10.1145/1476589.1476686 · Cited by 2,065
Practical significance: a concept whose time has come
Kirk, R. E. (1996) · Educational and Psychological Measurement · doi:10.1177/0013164496056005002 · Cited by 1,778 · cited 2×
“Association, Science Directorate, 1999) in part because doing so can assist researchers in determining the practical significance of their work, even in the light of non-significant findings (Kirk, 1996).”
BackgroundP-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · cited 5×
“As discussed previously, inclusion rules were created based on recommendations by (Simonsohn et al., 2014a).”
BackgroundMethodResultInfluentialAn Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1×
“…results (as is the case with VR work) is likely to foster practices that lead to publication bias, such as repeatedly attempting to obtain positive results to support their favored hypotheses (Greenwald, 1975) or believing that null results are less likely to be published (Coursol & Wagner, 1986).”
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
p -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717
Immersive virtual environment technology as a basic research tool in psychology
Loomis, J. M., Blascovich, J. J., & Beall, A. C. (1999) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03200735 · Cited by 717 · cited 1×
“Notable early scholars include Frank Biocca, a communication researcher who was one of the first to study VR as a medium (Biocca, 1992; Lanier & Biocca, 1992), and Jack Loomis, who helped introduce VR and presence to psychology (Loomis, Blascovich, & Beall, 1999).”
BackgroundThe poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510
Virtual reality in the treatment of spider phobia: a controlled study
Garcia-Palacios, A., Hoffman, H., Carlin, A., Furness, T. A., & Botella, C. (2002) · Behaviour Research and Therapy · doi:10.1016/s0005-7967(01)00068-7 · Cited by 490
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000104 · Cited by 343 · cited 2×
“Moderate p -hacking can usually get a researcher to a significance of .05, while more ambitious p -hacking to, say the p .01 level, is often difficult, obvious, and unsuccessful (Simonsohn, Simmons, & Nelson, 2015).”
BackgroundMethodKlinger, E., Bouchard, S., Légeron, P., Roy, S., Lauer, F., Chemin, I., & Nugues, P. (2005) · CyberPsychology & Behavior · doi:10.1089/cpb.2005.8.76 · Cited by 327
Virtual reality in mental health: A review of the literature
Gregg, L., & Tarrier, N. (2007) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-007-0173-4 · Cited by 304 · cited 1×
“Similarly, a meta-analysis by Page and Coxon (2016) looking at Virtual Reality Exposure Therapy (VRET) analyzed 71 articles, and a keyword search by Gregg and Tarrier (2007) to explore the used of VR for mental health returned 3,036 articles, 50 of which met their primary inclusion criteria.”
MethodStatistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science
Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114876 · Cited by 69
Virtual Reality Exposure Therapy for Anxiety Disorders: Small Samples and No Controls?
Page, S., & Coxon, M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00326 · Cited by 46 · cited 1×
“Similarly, a meta-analysis by Page and Coxon (2016) looking at Virtual Reality Exposure Therapy (VRET) analyzed 71 articles, and a keyword search by Gregg and Tarrier (2007) to explore the used of VR for mental health returned 3,036 articles, 50 of which met their primary inclusion criteria.”
Method
Cited by
Dincelli, E., & Yayla, A. (2022) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2022.101717 · Cited by 490
Zhang, M., Ding, H., & Zhang, Y. (2022) · doi:10.20944/preprints202202.0056.v1 · Cited by 211
(2022) · doi:10.3390/bs12050138 · Cited by 211
Dermody, G., Whitehead, L., Wilson, G., & Glass, C. (2020) · Journal of Medical Internet Research · doi:10.2196/17331 · Cited by 209
Metadata
- Authors
- Lanier, M., Waddell, T. F., Elson, M., Tamul, D. J., Ivory, J. D., & Przybylski, A.
- Venue
- Computers in Human Behavior
- Publisher
- Elsevier BV
- Published
- 2019-11
- Volume
- 100
- Pages
- 70-78
- ISSN
- 0747-5632 (ISSN-L)
- OpenAlex
- W2954972306
- Cited by
- 75
- References
- 24
- FWCI
- 7.41
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.