Testing the significance of a correlation with nonnormal data: Comparison of Pearson, Spearman, transformation, and resampling approaches.
The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Abstract
It is well known that when data are nonnormally distributed, a test of the significance of Pearson's r may inflate Type I error rates and reduce power. Statistics textbooks and the simulation literature provide several alternatives to Pearson's correlation. However, the relative performance of these alternatives has been unclear. Two simulation studies were conducted to compare 12 methods, including Pearson, Spearman's rank-order, transformation, and resampling approaches. With most sample sizes (n ≥ 20), Type I and Type II error rates were minimized by transforming the data to a normal shape prior to assessing the Pearson correlation. Among transformation approaches, a general purpose rank-based inverse normal transformation (i.e., transformation to rankit scores) was most beneficial. However, when samples were both small (n ≤ 10) and extremely nonnormal, the permutation test often outperformed other alternatives, including various bootstrap tests.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Deviations from normality usually do not bias regression coefficients (Ramsey & Schafer, 2013; Williams et al., 2013) or impair hypothesis testing (no inflated type I error rate, e.g., Bishara & Hittner, 2012; Ives, 2015; Puth et al., 2014; Ramsey & Schafer, 2013; Szöcs & Schäfer, 2015; Warton et al., 2016) even at relatively small sample sizes.”
From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Bishara & Hittner, 2012
In the citing paper
“We focus on fairly drastic non-normality because only little bias can be expected from minor violations (Bishara & Hittner, 2012; Glass et al., 1972; Hack, 1958; Puth et al., 2014).”
From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Bishara & Hittner, 2012
In the citing paper
“it may be most advisable to apply a rank-based inverse normal (RIN) transformation (aka rankit scores, Bliss, 1967) to the data, which can approximately normalize most distributional shapes and which effectively minimizes type I errors and maximizes statistical power (Bishara & Hittner, 2012; Puth et al., 2014).”
From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Bishara & Hittner, 2012
In the citing paper
“All these deviations are generally thought to be of little concern (e.g., Bishara & Hittner, 2012), even if they are far from fitting to the bell-shaped curve, such as binomial data (Cochran, 1950).”
From Violating the normality assumption may be the lesser of two evils (2021) · cites this paper as Bishara & Hittner, 2012
Citations
Cites
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 80,032
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences
Cohen, P., Cohen, P., West, S. G., & Aiken, L. S. (2014) · Psychology Press eBooks · doi:10.4324/9781410606266 · Cited by 20,737
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602
An Analysis of Transformations
Box, G. E. P., & Cox, D. R. (1964) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1964.tb00553.x · Cited by 15,052
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
(2007) · Cited by 10,092
Educational Research: Competencies for Analysis and Application
Mills, G. E., & Airasian, P. W. (1995) · Medical Entomology and Zoology · Cited by 5,395
Fundamentals of Biostatistics.
(1969) · Annals of Internal Medicine · doi:10.7326/0003-4819-70-4-873 · Cited by 4,575
Randomization, Bootstrap and Monte Carlo Methods in Biology
Manly, B. F. J. (2018) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315273075 · Cited by 4,194
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
Statistical Tables for Biological, Agricultural and Medical Research.
Reed, L. J., Fisher, R. A., & Yates, F. (1944) · Journal of the American Statistical Association · doi:10.2307/2280633 · Cited by 2,998
Biostatistics: A Foundation for Analysis in the Health Sciences.
Rosenfeld, B., & Daniel, W. W. (1984) · Journal of the American Statistical Association · doi:10.2307/2288760 · Cited by 2,834
(2000) · Cited by 2,832
Statistical methods for health care research
Munro, B. H. (1986) · Medical Entomology and Zoology · Cited by 2,265
Tukey, J. W. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704711 · Cited by 1,860
A new family of power transformations to improve normality or symmetry
Yeo, I.-K. (2000) · Biometrika · doi:10.1093/biomet/87.4.954 · Cited by 1,702
The unicorn, the normal curve, and other improbable creatures.
Micceri, T. (1989) · Psychological Bulletin · doi:10.1037//0033-2909.105.1.156 · Cited by 1,485
Design and Analysis of Experiments
(2007) · Modern Engineering Statistics · doi:10.1002/9780470187548.ch12 · Cited by 1,424
Applied Statistics: From Bivariate Through Multivariate Techniques
Warner, R. M. (2007) · Cited by 1,325
Statistical Estimates and Transformed Beta-Variables.
Minton, P. D., & Blom, G. (1960) · The American Mathematical Monthly · doi:10.2307/2308690 · Cited by 1,166
Darlington, R. B. (1990) · Cited by 1,111
Bullmore, E. T., Suckling, J., Overmeyer, S., Rabe-Hesketh, S., Taylor, E., & Brammer, M. J. (1999) · IEEE Transactions on Medical Imaging · doi:10.1109/42.750253 · Cited by 1,088
Pagano, M., Gauvreau, K., & Mattie, H. (2022) · Chapman & Hall/CRC eBooks · doi:10.1201/9780429340512 · Cited by 926
The Probability Plot Correlation Coefficient Test for Normality
Filliben, J. J. (1975) · Technometrics · doi:10.1080/00401706.1975.10489279 · Cited by 919
Improving your data transformations: Applying the Box-Cox transformation
Osborne, J. W. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/qbpc-gk17 · Cited by 878
Nelder, J. A., & Bliss, C. I. (1971) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2343879 · Cited by 758
Significance Tests Which May be Applied to Samples from Any Populations
Pitman, E. J. G. (1937) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.2307/2984124 · Cited by 729
Statistics for people who (think they) hate statistics
(2009) · Choice Reviews Online · doi:10.5860/choice.46-6862 · Cited by 710
Permutation, Parametric and Bootstrap Tests of Hypotheses
(2005) · Springer Series in Statistics · doi:10.1007/b138696 · Cited by 707
Permutation methods: a distance function approach
(2002) · Choice Reviews Online · doi:10.5860/choice.39-3427 · Cited by 611
An alternative method for significance testing of waveform difference potentials
BLAIR, R. C., & KARNISKI, W. (1993) · Psychophysiology · doi:10.1111/j.1469-8986.1993.tb02075.x · Cited by 597
Notes on the use of data transformations
Osborne, J. W. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/4vng-5608 · Cited by 577
Detecting Trends Using Spearman's Rank Correlation Coefficient
Gauthier, T. (2001) · Environmental Forensics · doi:10.1006/enfo.2001.0061 · Cited by 513
Algorithm AS 89: The Upper Tail Probabilities of Spearman's Rho
Best, D. J., & Roberts, D. E. (1975) · Applied Statistics · doi:10.2307/2347111 · Cited by 502
Hays, W. L. (1988) · Cited by 462
So Many Correlated Tests, So Little Time! Rapid Adjustment of P Values for Multiple Correlated Tests
Conneely, K. N., & Boehnke, M. (2007) · The American Journal of Human Genetics · doi:10.1086/522036 · Cited by 446
A Monte Carlo Evaluation of Tests for Comparing Dependent Correlations
Hittner, J. B., May, K., & Silver, N. C. (2003) · The Journal of General Psychology · doi:10.1080/00221300309601282 · Cited by 425
Self-citationThe SAGE Handbook of Quantitative Methods in Psychology
Millsap, R., & Maydeu-Olivares, A. (2009) · SAGE Publications Ltd eBooks · doi:10.4135/9780857020994 · Cited by 363
Logan, G. D. (1992) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037//0278-7393.18.5.883 · Cited by 341
Triola, M. F. (2012) · Andalas University Repository (Andalas University) · Cited by 331
Comparison of ANOVA alternatives under variance heterogeneity and specific noncentrality structures.
Tomarken, A. J., & Serlin, R. C. (1986) · Psychological Bulletin · doi:10.1037//0033-2909.99.1.90 · Cited by 326
Rank-Based Inverse Normal Transformations are Increasingly Used, But are They Merited?
Beasley, T. M., Erickson, S., & Allison, D. B. (2009) · Behavior Genetics · doi:10.1007/s10519-009-9281-0 · Cited by 323
Fundamentals of Behavioral Statistics
Kushner, H. W., Runyon, R. P., & Haber, A. (1976) · Teaching Sociology · doi:10.2307/1317197 · Cited by 322
Allison, D. B., Neale, M. C., Zannolli, R., Schork, N. J., Amos, C. I., & Blangero, J. (1999) · The American Journal of Human Genetics · doi:10.1086/302487 · Cited by 297
Statiscal tables : for biological, agricultural and medical research
Fischer, R. A., & Yates, F. (1938) · Oliver and Boyd eBooks · Cited by 225
Exponential Data Transformations
Manly, B. F. J. (1976) · The Statistician · doi:10.2307/2988129 · Cited by 208
Permutation, Parametric, and Bootstrap Tests of Hypotheses
Annis, D. H. (2005) · Journal of the American Statistical Association · doi:10.1198/jasa.2005.s48 · Cited by 195
Effect of violation of normality on the t test of the correlation coefficient.
Edgell, S. E., & Noon, S. M. (1984) · Psychological Bulletin · doi:10.1037//0033-2909.95.3.576 · Cited by 192
Some properties of the bivariate normal distribution considered in the form of a contingency table
LANCASTER, H. O. (1957) · Biometrika · doi:10.1093/biomet/44.1-2.289 · Cited by 184
Effect of the violation of assumptions upon significance levels of the Pearson r.
Havlicek, L. L., & Peterson, N. L. (1977) · Psychological Bulletin · doi:10.1037//0033-2909.84.2.373 · Cited by 175
Statistical tables for biological, agricultural aad medical research.
Fisher, R. A., & Yates, F. (1938) · Cited by 164
Impact of Rank-Based Normalizing Transformations on the Accuracy of Test Scores
Soloman, S. R., & Sawilowsky, S. S. (2009) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1257034080 · Cited by 161
Good, P. (2006) · Cited by 141
Headrick, T. C. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(02)00072-5 · Cited by 128
The art test of interaction: a robust and powerful rank test of interaction in factorial models
Salter, K. C., & Fawcett, R. F. (1993) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610919308813085 · Cited by 126
Statistics for people who (think they) hate statistics, 3rd ed.
Salkind, N. J. (2008) · Cited by 120
Permutation test is not distribution-free: Testing H₀: ρ = 0.
Hayes, A. F. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.184 · Cited by 119
The Practice of Business Statistics
(2004) · Technometrics · doi:10.1198/tech.2004.s217 · Cited by 117
Bias in Estimation and Hypothesis Testing of Correlation
Zimmerman, D. W., Zumbo, B. D., Williams, R. H., & Zimmerman, D. W. (2003) · Cited by 107
Simulating Multivariate Nonnormal Data Using an Iterative Algorithm
Ruscio, J., & Kaczetow, W. (2008) · Multivariate Behavioral Research · doi:10.1080/00273170802285693 · Cited by 107
Simulating Correlated Multivariate Nonnormal Distributions: Extending the Fleishman Power Method
Headrick, T. C., & Sawilowsky, S. S. (1999) · Psychometrika · doi:10.1007/bf02294537 · Cited by 99
A Common Situation Conducive to Bizarre Distribution Shapes
Bradley, J. V. (1977) · The American Statistician · doi:10.1080/00031305.1977.10479224 · Cited by 98
Power and Robustness in Product-Moment Correlation
Fowler, R. L. (1987) · Applied Psychological Measurement · doi:10.1177/014662168701100407 · Cited by 93
Randomization tests for multiple-baseline designs.
Wampold, B. E., & Worsham, N. L. (1986) · Behavioral Assessment · Cited by 93
Limitations of the rank transform statistic in tests for interactions
Blair, R. C., Sawilowsky, S. S., & Higgins, J. J. (1987) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610918708812642 · Cited by 89
A Monte-Carlo Study of Asymptotically Robust Tests for Correlation Coefficients
Duncan, G. T., & Layard, M. W. J. (1973) · Biometrika · doi:10.1093/biomet/60.3.551 · Cited by 86
Similar processes despite divergent behavior in two commonly used measures of risky decision making
Bishara, A. J., Pleskac, T. J., Fridberg, D. J., Yechiam, E., Lucas, J., Busemeyer, J. R., Finn, P., & Stout, J. C. (2009) · Journal of Behavioral Decision Making · doi:10.1002/bdm.641 · Cited by 83
Self-citationComprehending Behavioral Statistics
Hurlburt, R. T. (1993) · Medical Entomology and Zoology · Cited by 82
Essentials of statistics for business and economics
Anderson, D. R., Sweeney, D. J., & Williams, T. A. (1984) · International Linear Collider · Cited by 72
Estimating correlation coefficients: Bootstrap and parametric approaches.
Rasmussen, J. L. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.101.1.136 · Cited by 71
Some Distributions of Sample Means
Brown, G. W., & Tukey, J. W. (1946) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731017 · Cited by 70
Estimating the correlation coefficient: The bootstrap approach.
Lunneborg, C. E. (1985) · Psychological Bulletin · doi:10.1037/0033-2909.98.1.209 · Cited by 67
The Effect of Skew on the Magnitude of Product-Moment Correlations
Dunlap, W. P., Burke, M. J., & Greer, T. F. (1995) · The Journal of General Psychology · doi:10.1080/00221309.1995.9921248 · Cited by 58
Some advantages of permutation tests.
May, R. B., & Hunter, M. A. (1993) · Canadian Psychology/Psychologie canadienne · doi:10.1037/h0078862 · Cited by 56
Bootstrapping correlation coefficients using univariate and bivariate sampling.
Lee, W., & Rodgers, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.1.91 · Cited by 55
Borkowf, C. B. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(01)00081-0 · Cited by 53
Data transformation, Type I error rate and power
Rasmussen, J. L. (1989) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1989.tb00910.x · Cited by 51
Wang, K., & Huang, J. (2002) · The American Journal of Human Genetics · doi:10.1086/338659 · Cited by 50
Keller-Mcnulty, A., & Higgins, J. J. (1987) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610918708812575 · Cited by 49
The future of psychology practice and science.
Bray, J. H. (2010) · American Psychologist · doi:10.1037/a0020273 · Cited by 47
Bootstrapping to test for nonzero population correlation coefficients using univariate sampling.
Beasley, W. H., DeShea, L., Toothaker, L. E., Mendoza, J. L., Bard, D., & Rodgers, J. L. (2007) · Psychological Methods · doi:10.1037/1082-989x.12.4.414 · Cited by 45
Dealing with Nonnormal Data: Parametric Analysis of Transformed Data vs Nonparametric Analysis
Rasmussen, J. L., & Dunlap, W. P. (1991) · Educational and Psychological Measurement · doi:10.1177/001316449105100402 · Cited by 44
Strube, M. (1988) · Psychological Bulletin · doi:10.1037/0033-2909.104.2.290 · Cited by 43
A Note on Homogeneity of Variance of Scores and Ranks
Zimmerman, D. W. (1996) · The Journal of Experimental Education · doi:10.1080/00220973.1996.10806603 · Cited by 34
The Effects of Violating the Normality Assumption Underlying r
Zeller, R. A., & Levine, Z. H. (1974) · Sociological Methods & Research · doi:10.1177/004912417400200406 · Cited by 34
Headrick, T. C., & Sawilowsky, S. S. (2000) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986025004417 · Cited by 28
Cervone, D. (1985) · Cognitive Therapy and Research · doi:10.1007/bf01173085 · Cited by 25
Significance Testing of Correlation Using Scores, Ranks, and Modified Ranks
Zimmerman, D. W., & Zumbo, B. D. (1993) · Educational and Psychological Measurement · doi:10.1177/0013164493053004003 · Cited by 23
Choosing a Two-Sample Location Test
Penfield, D. A. (1994) · The Journal of Experimental Education · doi:10.1080/00220973.1994.9944139 · Cited by 22
IEEE Transactions on Systems, Man & Cybernetics
Sage, A. P. (2004) · Wiley eBooks · doi:10.1002/0471667196.ess1208 · Cited by 19
Co-Ordinate Transformations to Normality and the Power of Normal Tests for Independence
Kowalski, C. J., & Tarter, M. E. (1969) · Biometrika · doi:10.1093/biomet/56.1.139 · Cited by 17
Calkins, D. S. (2017) · Revista Interamericana de Psicología/Interamerican Journal of Psychology · Cited by 16
A CASE AGAINST CONTINUING USE OF THE SPEARMAN FORMULA FOR RANK-ORDER CORRELATION
Roberts, D. M., & Kunst, R. E. (1990) · Psychological Reports · doi:10.2466/pr0.1990.66.1.339 · Cited by 14
Zimmerman, D. W. (2011) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 9
Understanding Statistics: A Research Perspective
McGrath, R. E. (1996) · Cited by 8
Statistics for the behavioral sciences: A short course and student manual.
Gravetter, F. J., & Wallnau, L. B. (2004) · Cited by 8
(2006) · Allyn & Bacon, Inc. eBooks
Cited by
de Winter, J. C. F., Gosling, S. D., & Potter, J. (2016) · Psychological Methods · doi:10.1037/met0000079 · Cited by 1,201
Transparency and replicability in qualitative research: the case of interviews with elite informants
Aguinis, H., & Solarino, A. M. (2019) · Strategic Management Journal · doi:10.1002/smj.3015 · Cited by 791
Violating the normality assumption may be the lesser of two evils
Knief, U., & Forstmeier, W. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01587-5 · Cited by 654
“…regression coefficients (Williams, Grajales & Kurkiewicz 2013) and usually do not impair hypothesis testing (no inflated type I error rate, e.g. Bishara & Hittner 2012; Puth, Neuhauser & Ruxton 2014; Ives 2015; Szöcs & Schäfer 2015; Warton et al. 2016) even at relatively small sample sizes,…”
BackgroundMethodResultWong, Y. J., Ho, M.-H. R., Wang, S.-Y., & Miller, I. S. K. (2017) · Journal of Counseling Psychology · doi:10.1037/cou0000176 · Cited by 507
Effective use of Pearson's product–moment correlation coefficient
Puth, M.-T., Neuhäuser, M., & Ruxton, G. D. (2014) · Animal Behaviour · doi:10.1016/j.anbehav.2014.05.003 · Cited by 446
Metadata
- Authors
- Bishara, A. J., & Hittner, J. B.
- Affiliations
- College of Charleston (US)
- Venue
- Psychological Methods
- Published
- 2012-09
- Volume
- 17
- Issue
- 3
- Pages
- 399-417
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2111495793
- Cited by
- 766
- References
- 98
- Citations by year
- 2026: 80 · 2025: 113 · 2024: 88 · 2023: 77 · 2022: 86 · 2021: 53 · 2020: 64 · 2019: 52 · 2018: 34 · 2017: 40 · 2016: 23 · 2015: 25 · 2014: 23 · 2013: 5 · 2012: 3
- FWCI
- 20.74
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Keywords
- non-normal data, Pearson correlation, data transformation, permutation tests, bootstrap tests, type I error, statistical power, small sample size
- MeSH terms
- Data Interpretation, Statistical, Monte Carlo Method, Statistics as Topic, Statistical Distributions, Linear Models, Nonlinear Dynamics, Sample Size, Statistics, Nonparametric
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.