Testing the significance of a correlation with nonnormal data: Comparison of Pearson, Spearman, transformation, and resampling approaches.
The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Abstract
It is well known that when data are nonnormally distributed, a test of the significance of Pearson's r may inflate Type I error rates and reduce power. Statistics textbooks and the simulation literature provide several alternatives to Pearson's correlation. However, the relative performance of these alternatives has been unclear. Two simulation studies were conducted to compare 12 methods, including Pearson, Spearman's rank-order, transformation, and resampling approaches. With most sample sizes (n ≥ 20), Type I and Type II error rates were minimized by transforming the data to a normal shape prior to assessing the Pearson correlation. Among transformation approaches, a general purpose rank-based inverse normal transformation (i.e., transformation to rankit scores) was most beneficial. However, when samples were both small (n ≤ 10) and extremely nonnormal, the permutation test often outperformed other alternatives, including various bootstrap tests.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Deviations from normality usually do not bias regression coefficients (Ramsey & Schafer, 2013; Williams et al., 2013) or impair hypothesis testing (no inflated type I error rate, e.g., Bishara & Hittner, 2012; Ives, 2015; Puth et al., 2014; Ramsey & Schafer, 2013; Szöcs & Schäfer, 2015; Warton et al., 2016) even at relatively small sample sizes.”
From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Bishara & Hittner, 2012
In the citing paper
“We focus on fairly drastic non-normality because only little bias can be expected from minor violations (Bishara & Hittner, 2012; Glass et al., 1972; Hack, 1958; Puth et al., 2014).”
From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Bishara & Hittner, 2012
In the citing paper
“it may be most advisable to apply a rank-based inverse normal (RIN) transformation (aka rankit scores, Bliss, 1967) to the data, which can approximately normalize most distributional shapes and which effectively minimizes type I errors and maximizes statistical power (Bishara & Hittner, 2012; Puth et al., 2014).”
From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Bishara & Hittner, 2012
In the citing paper
“All these deviations are generally thought to be of little concern (e.g., Bishara & Hittner, 2012), even if they are far from fitting to the bell-shaped curve, such as binomial data (Cochran, 1950).”
From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Bishara & Hittner, 2012
Citations
Cites
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 80,032
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences
Cohen, P., Cohen, P., West, S. G., & Aiken, L. S. (2014) · Psychology Press eBooks · doi:10.4324/9781410606266 · Cited by 20,737
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602
An Analysis of Transformations
Box, G. E. P., & Cox, D. R. (1964) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1964.tb00553.x · Cited by 15,052
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
(2007) · Cited by 10,092
Educational Research: Competencies for Analysis and Application
Mills, G. E., & Airasian, P. W. (1995) · Medical Entomology and Zoology · Cited by 5,395
Fundamentals of Biostatistics.
(1969) · Annals of Internal Medicine · doi:10.7326/0003-4819-70-4-873 · Cited by 4,575
Cited by
de Winter, J. C. F., Gosling, S. D., & Potter, J. (2016) · Psychological Methods · doi:10.1037/met0000079 · Cited by 1,201
Transparency and replicability in qualitative research: the case of interviews with elite informants
Aguinis, H., & Solarino, A. M. (2019) · Strategic Management Journal · doi:10.1002/smj.3015 · Cited by 791
Violating the normality assumption may be the lesser of two evils
Knief, U., & Forstmeier, W. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01587-5 · Cited by 654
“…regression coefficients (Williams, Grajales & Kurkiewicz 2013) and usually do not impair hypothesis testing (no inflated type I error rate, e.g. Bishara & Hittner 2012; Puth, Neuhauser & Ruxton 2014; Ives 2015; Szöcs & Schäfer 2015; Warton et al. 2016) even at relatively small sample sizes,…”
BackgroundMethodResultWong, Y. J., Ho, M.-H. R., Wang, S.-Y., & Miller, I. S. K. (2017) · Journal of Counseling Psychology · doi:10.1037/cou0000176 · Cited by 507
Effective use of Pearson's product–moment correlation coefficient
Puth, M.-T., Neuhäuser, M., & Ruxton, G. D. (2014) · Animal Behaviour · doi:10.1016/j.anbehav.2014.05.003 · Cited by 446
Metadata
- Authors
- Bishara, A. J., & Hittner, J. B.
- Affiliations
- College of Charleston (US)
- Venue
- Psychological Methods
- Published
- 2012-09
- Volume
- 17
- Issue
- 3
- Pages
- 399-417
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2111495793
- Cited by
- 766
- References
- 98
- Citations by year
- 2026: 80 · 2025: 113 · 2024: 88 · 2023: 77 · 2022: 86 · 2021: 53 · 2020: 64 · 2019: 52 · 2018: 34 · 2017: 40 · 2016: 23 · 2015: 25 · 2014: 23 · 2013: 5 · 2012: 3
- FWCI
- 20.74
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Keywords
- non-normal data, Pearson correlation, data transformation, permutation tests, bootstrap tests, type I error, statistical power, small sample size
- MeSH terms
- Data Interpretation, Statistical, Monte Carlo Method, Statistics as Topic, Statistical Distributions, Linear Models, Nonlinear Dynamics, Sample Size, Statistics, Nonparametric
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Testing the significance of a correlation with nonnormal data: Comparison of Pearson, Spearman, transformation, and resampling approaches. https://app.cochat.ai/papers/10.1037/a0028087
- The paper
- Bishara, A. J., & Hittner, J. B. (2012). Testing the significance of a correlation with nonnormal data: Comparison of Pearson, Spearman, transformation, and resampling approaches. Psychological Methods. https://doi.org/10.1037/a0028087
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.