Detecting and Correcting the Lies That Data Tell
Abstract
Because of the way in which data are typically analyzed and interpreted, they frequently lie to researchers, leading to conclusions that are not only false but more complex than the underlying reality. The several examples of this presented in this article illustrate the possibility that although data may appear to indicate complex phenomena at the surface structure level, the phenomena may be quite simple at the deep structure level, suggesting the possibility of applying Occam’s razor to achieve the scientific goal of parsimony. The approaches to data analysis described in this article may also lead to a solution to the serious problem of construct proliferation in psychology by demonstrating that many constructs are redundant with other existing constructs. The major obstacles to these outcomes are researchers' continued reliance on the use of statistical significance testing in data analysis and interpretation and the failure to correct for the distorting effects of sampling error, measurement error, and other artifacts. Some of these problems have been addressed by the now widespread use of meta-analysis, but examination of the meta-analyses appearing in Psychological Bulletin from 1978 to 2006 shows that most employ a statistically inappropriate model for meta-analysis (the fixed effects model) and that 90% do not correct for the biasing effects of measurement error. Hence, there is still a long way to go in the improvement of data analysis and interpretation methods.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In typically developing children, working memory training yields no generalized far-transfer effect.
“Therefore, given the circumstances, it is possible to apply Occam's razor (Schmidt, 2010) and conclude that WM training does not produce any generalized (far-transfer) effect in TD children.”
From Working memory training in typically developing children: A multilevel meta-analysis (2020) · cites this paper as Schmidt, 2010
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,385 · see citing works
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542 · see citing works
Self-citationThe Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
On the practice of dichotomization of quantitative variables.
MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.19 · Cited by 3,538 · see citing works
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2011) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,177 · see citing works
Hunter, J., & Schmidt, F. (2004) · doi:10.4135/9781412985031 · Cited by 3,023 · see citing works
Self-citationStatistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791 · see citing works
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings.
Raudenbush, S. W., Hunter, J. E., & Schmidt, F. L. (1991) · Journal of the American Statistical Association · doi:10.2307/2289738 · Cited by 2,363 · see citing works
Self-citationValidity and utility of alternative predictors of job performance.
Hunter, J. E., & Hunter, R. F. (1984) · Psychological Bulletin · doi:10.1037/0033-2909.96.1.72 · Cited by 2,107 · see citing works
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
Organizational behavior and human decision processes
Armstrong, J. S. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90125-2 · Cited by 1,766 · see citing works
Cohen, J. (1983) · Applied Psychological Measurement · doi:10.1177/014662168300700301 · Cited by 1,564 · cited 1× · see citing works
“This practice not only loses information but also lowers correlations and creates more variability in findings across studies (Cohen, 1983; Hunter & Schmidt, 1990a, 2004; MacCallum, Zhang, Preacher, & Rucker, 2002).”
BackgroundInference by Eye: Confidence Intervals and How to Read Pictures of Data.
Cumming, G., & Finch, S. (2005) · American Psychologist · doi:10.1037/0003-066x.60.2.170 · Cited by 1,449 · cited 1× · see citing works
“CIs are just as objective as significance tests, and they provide far more information (Borenstein, 1994; Cumming & Finch, 2005).”
BackgroundMeta-Analysis: Cumulating Research Findings across Studies
Feldman, J. M., Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1984) · Academy of Management Review · doi:10.2307/258247 · Cited by 1,199 · see citing works
Self-citationSchmidt, F. L. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.115 · Cited by 1,031 · see citing works
Self-citationThe case against statistical significance testing
Carver, R. (1978) · Harvard Educational Review · doi:10.17763/haer.48.3.t490261645281841 · Cited by 931 · see citing works
Meta-Analysis: Cumulating Research Findings across Studies
Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1986) · Educational Researcher · doi:10.2307/1175263 · Cited by 792 · see citing works
Self-citationDevelopment of a general solution to the problem of validity generalization.
Schmidt, F. L., & Hunter, J. E. (1977) · Journal of Applied Psychology · doi:10.1037/0021-9010.62.5.529 · Cited by 769 · see citing works
Self-citationSchmidt, F. L., Hunter, J. E., & Outerbridge, A. N. (1986) · Journal of Applied Psychology · doi:10.1037/0021-9010.71.3.432 · Cited by 758 · see citing works
Self-citationDo studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748 · see citing works
Schmidt, F. L., Oh, I., & Hayes, T. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711007x255327 · Cited by 713 · see citing works
Self-citationMeasurement error in psychological research: Lessons from 26 research scenarios.
Schmidt, F. L., & Hunter, J. E. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.199 · Cited by 567 · see citing works
Self-citationWhat do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564 · cited 4× · see citing works
“In individual studies, researchers should use point estimates and CIs, not significance tests (Loftus, 1996; Schmidt, 1992; Thompson, 2002).”
BackgroundMethodInfluentialSelf-citationStandards for Reporting on Empirical Social Science Research in AERA Publications
(2006) · Educational Researcher · doi:10.3102/0013189x035006033 · Cited by 548 · see citing works
Thompson, B. (2002) · Educational Researcher · doi:10.3102/0013189x031003025 · Cited by 546 · see citing works
What If There Were No Significance Tests?
(2016) · Routledge eBooks · doi:10.4324/9781315629049 · Cited by 543 · see citing works
Statistical Inference: A Commentary for the Social and Behavioural Sciences
Oakes, M. W. (1986) · Medical Entomology and Zoology · Cited by 392 · see citing works
Effect sizes, confidence intervals, and confidence intervals for effect sizes
Thompson, B. (2007) · Psychology in the Schools · doi:10.1002/pits.20234 · Cited by 392 · see citing works
Dichotomization of continuous variables: The implications for meta-analysis.
Hunter, J. E., & Schmidt, F. L. (1990) · Journal of Applied Psychology · doi:10.1037/0021-9010.75.3.334 · Cited by 376 · see citing works
Self-citationPsychology Will Be a Much Better Science When We Change the Way We Analyze Data
Loftus, G. R. (1996) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep11512376 · Cited by 362 · see citing works
Implications of direct and indirect range restriction for meta-analysis methods and findings.
Hunter, J. E., Schmidt, F. L., & Le, H. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.3.594 · Cited by 315 · see citing works
Self-citationTheory testing and measurement error
Schmidt, F. L., & Hunter, J. E. (1999) · Intelligence · doi:10.1016/s0160-2896(99)00024-0 · Cited by 254 · see citing works
Self-citationSalgado, J. F., Anderson, N. R., Moscoso, S., Bertua, C., & Fruyt, F. D. (2003) · Personnel Psychology · doi:10.1111/j.1744-6570.2003.tb00751.x · Cited by 238 · see citing works
Schmidt, F. L. (1971) · Educational and Psychological Measurement · doi:10.1177/001316447103100310 · Cited by 228 · see citing works
Self-citationStatistical power in criterion-related validation studies.
Schmidt, F. L., Hunter, J. E., & Urry, V. W. (1976) · Journal of Applied Psychology · doi:10.1037//0021-9010.61.4.473 · Cited by 209 · see citing works
Self-citationStatistical power in criterion-related validation studies.
Schmidt, F. L., Hunter, J. E., & Urry, V. W. (1976) · Journal of Applied Psychology · doi:10.1037/0021-9010.61.4.473 · Cited by 196 · see citing works
Self-citationSCHMIDT, F. L., SHAFFER, J. A., & OH, I. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00132.x · Cited by 190 · see citing works
Self-citationValidity Generalization: A Critical Review
Strube, M., & Murphy, K. R. (2004) · Academy of Management Review · doi:10.2307/20159040 · Cited by 152 · see citing works
Schmidt, F. M. (2007) · Organizational Research Methods · doi:10.1177/1094428107303161 · Cited by 109 · see citing works
Self-citationLe, H., Schmidt, F. L., & Putka, D. J. (2007) · Organizational Research Methods · doi:10.1177/1094428107302900 · Cited by 102 · see citing works
Self-citationSchmidt, F. L., Oh, I., & Le, H. (2006) · Personnel Psychology · doi:10.1111/j.1744-6570.2006.00065.x · Cited by 99 · see citing works
Self-citationRacial differences in validity of employment tests: Reality or illusion?
Schmidt, F. L., Berner, J. G., & Hunter, J. E. (1973) · Journal of Applied Psychology · doi:10.1037/h0035408 · Cited by 97 · see citing works
Self-citationThe case for confidence intervals in controlled clinical trials
Borenstein, M. (1994) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)90036-1 · Cited by 84 · see citing works
The harm done by tests of significance
Hauer, E. (2004) · Accident Analysis & Prevention · doi:10.1016/s0001-4575(03)00036-8 · Cited by 70 · see citing works
Correcting for indirect range restriction in meta-analysis: Testing a new meta-analytic procedure.
Le, H. Q., & Schmidt, F. L. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.4.416 · Cited by 53 · see citing works
Self-citationA WITHIN SETTING EMPIRICAL TEST OF THE SITUATIONAL SPECIFICITY HYPOTHESIS IN PERSONNEL SELECTION
Schmidt, F. L., & Hunter, J. E. (1984) · Personnel Psychology · doi:10.1111/j.1744-6570.1984.tb01453.x · Cited by 49 · see citing works
Self-citationOh, I., Schmidt, F. L., Shaffer, J. A., & Le, H. (2008) · Academy of Management Learning and Education · doi:10.5465/amle.2008.35882196 · Cited by 46 · see citing works
Self-citationReflections on methods of statistical inference in research on the effect of safety countermeasures
Hauer, E. (1983) · Accident Analysis & Prevention · doi:10.1016/0001-4575(83)90052-0 · Cited by 25 · see citing works
Schmidt, F. L., OCASIO, B. P., Hillery, J. M., et al. (1985) · Personnel Psychology · doi:10.1111/j.1744-6570.1985.tb00557.x · Cited by 24 · see citing works
Self-citationThe Reliability of Differences Between Linear Regression Weights in Applied Differential Psychology
Schmidt, F. L. (1972) · Educational and Psychological Measurement · doi:10.1177/001316447203200403 · Cited by 19 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Relative Deprivation: A Theoretical and Meta-Analytic Review
Smith, H. J., Pettigrew, T. F., Pippin, G. M., & Bialosiewicz, S. (2012) · Personality and Social Psychology Review · doi:10.1177/1088868311430825 · Cited by 1,255 · see citing works
Reaction time variability in ADHD: A meta-analytic review of 319 studies
Kofler, M. J., Rapport, M. D., Sarver, D. E., Raiker, J. S., Orban, S. A., Friedman, L. M., & Kolomeyer, E. G. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.06.001 · Cited by 648 · see citing works
A meta-analytic review of authentic and transformational leadership: A test for redundancy
Banks, G. C., McCauley, K. D., Gardner, W. L., & Guler, C. E. (2016) · The Leadership Quarterly · doi:10.1016/j.leaqua.2016.02.006 · Cited by 681 · see citing works
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455 · see citing works
Leader autonomy support in the workplace: A meta-analytic review
Slemp, G. R., Kern, M. L., Patrick, K. J., & Ryan, R. M. (2018) · Motivation and Emotion · doi:10.1007/s11031-018-9698-y · Cited by 407 · see citing works
Shaffer, J. A., DeGeest, D., & Li, A. (2016) · Organizational Research Methods · doi:10.1177/1094428115598239 · Cited by 393 · see citing works
Evaluating vigilance deficits in ADHD: A meta-analysis of CPT performance.
Huang-Pollock, C. L., Karalunas, S. L., Tam, H., & Moore, A. N. (2012) · Journal of Abnormal Psychology · doi:10.1037/a0027205 · Cited by 359 · see citing works
Lee, A., Lyubovnikova, J., Tian, A. W., & Knight, C. (2020) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12265 · Cited by 347 · see citing works
The reality and evolutionary significance of human psychological sex differences
Archer, J. (2019) · Biological Reviews · doi:10.1111/brv.12507 · Cited by 291 · see citing works
Construct redundancy in leader behaviors: A review and agenda for the future
Banks, G. C., Gooty, J., Ross, R. L., Williams, C. E., & Harrington, N. T. (2018) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.12.005 · Cited by 275 · see citing works
Cognitive and academic benefits of music training with children: A multilevel meta-analysis
Sala, G., & Gobet, F. (2020) · Memory & Cognition · doi:10.3758/s13421-020-01060-2 · Cited by 235 · see citing works
A meta-analysis of autonomous and controlled forms of teacher motivation
Slemp, G. R., Field, J. G., & Cho, A. S. H. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2020.103459 · Cited by 179 · see citing works
Methods for second order meta-analysis and illustrative applications
Schmidt, F. L., & Oh, I.-S. (2013) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2013.03.002 · Cited by 151 · see citing works
Self-citationObtaining Unbiased Results in Meta-Analysis: The Importance of Correcting for Statistical Artifacts
Wiernik, B. M., & Dahlke, J. A. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919885611 · Cited by 135 · see citing works
Working memory training in typically developing children: A multilevel meta-analysis
Sala, G., & Gobet, F. (2020) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01681-y · Cited by 131 · see citing works
“By contrast, true variance, also referred to as true heterogeneity, is supposed to be accounted for by one or more moderating variables (Schmidt, 2010).”
BackgroundSchmidt, F. L. (2017) · Archives of Scientific Psychology · doi:10.1037/arc0000033 · Cited by 45 · see citing works
Self-citationSingle-case experimental research designs.
Kazdin, A. E. (2015) · Methodological issues and strategies in clinical research (4th ed.). · doi:10.1037/14805-029 · Cited by 39 · see citing works
- Authors
- Schmidt, F.
- Affiliations
- University of Iowa (US)
- Corresponding author
- Schmidt, Frank
- ORCID
- Schmidt, Frank
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2010-05
- Volume
- 5
- Issue
- 3
- Pages
- 233-242
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2160573644
- Cited by
- 163
- References
- 55
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 3 · 2022: 6 · 2021: 37 · 2020: 14 · 2019: 9 · 2018: 9 · 2017: 11 · 2016: 7 · 2015: 8 · 2014: 4 · 2013: 13 · 2012: 13
- FWCI
- 4.24
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Funders
- American Educational Research Association
- Association for Psychological Science
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- data analysis pitfalls, Occam's razor, parsimony, construct redundancy, construct proliferation, statistical significance testing, sampling error, measurement error, fixed effects model, measurement error correction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
