CoChat
Sign Up
Cited by 874Psychometric Methodologies and TestingPersonality Traits and PsychologyPsychological Testing and Assessment

Internal consistency, retest reliability, and their implications for personality scale validity

Open access
McCrae, Robert R., Kurtz, John E., Yamagata, Shinji, Terracciano, Antonio·2011·Personality and Social Psychology Review
DOI 10.1177/1088868310366253PMC PMC2927808
Open this paper in CoChat — free account

Abstract

The authors examined data ( N = 34,108) on the differential reliability and validity of facet scales from the NEO Inventories. They evaluated the extent to which (a) psychometric properties of facet scales are generalizable across ages, cultures, and methods of measurement, and, (b) validity criteria are associated with different forms of reliability. Composite estimates of facet scale stability, heritability, and cross-observer validity were broadly generalizable. Two estimates of retest reliability were independent predictors of the three validity criteria; none of three estimates of internal consistency was. Available evidence suggests the same pattern of results for other personality inventories. Internal consistency of scales can be useful as a check on data quality but appears to be of limited utility for evaluating the potential validity of developed scales, and it should not be used as a substitute for retest reliability. Further research on the nature and determinants of retest reliability is needed.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/1088868310366253

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Summer Health Professions Education Program Career Barriers Survey demonstrated sufficient internal consistency, with Cronbach α of 0.81 preprogram and 0.82 postprogram for motivator items, and 0.87 preprogram and 0.91 postprogram for barrier items.

    “The SCBS demonstrated sufficient internal consistency,29 with a Cronbach α for motivator items of 0.81 for preprogram and 0.82 for postprogram; the Cronbach α for barriers was 0.87 and 0.91 for postprogram barriers.”

    From Perceived Success After Participation in the Summer Health Professions Education Program (2024) · cites this paper as [29]

    In “Coefficient Alpha and Retest Reliability in Contemporary Research” (page unavailable), the paper identifies Cronbach’s alpha as a measure of internal consistency, supporting its use to describe the survey results. It does not study the SCBS or report the stated preprogram and postprogram values, and it cautions against treating alpha alone as sufficient evidence of reliability. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 63

  1. Allik J Laidra K Realo A Pullmann H Personality development from 12 to 18 years of age: Changes in mean levels and structures of traits European Journal of Personality 2004 18 445 462
  2. Ando J Suzuki A Yamagata S Kijima N Maekawa H Ono Y Genetic and environmental structure of Cloninger's temperament and character dimensions Journal of Personality Disorders 2004 18 379 393 15342324
  3. Ashton SG Goldberg LR In response to Jackson's challenge: The comparative validity of personality scales constructed by the external (empirical) strategy and scales developed intuitively by experts, novices, and laymen Journal of Research in Personality 1973 7 1 20
  4. Bleidorn W Kandler C Riemann R Angleitner A Patterns and sources of adult personality development: Growth curve analyses of the NEO-PI-R scales in a longitudinal twin study Journal of Personality and Social Psychology 2009 97 142 155 19586245
  5. Borkenau P Liebler A Observable attributes as manifestations and cues of personality and intelligence Journal of Personality 1995 63 1 25
  6. Bouchard TJ Jr. McGue M Hur Y-M Horn JM A genetic and environmental analysis of the California Psychological Inventory using adult twins reared apart and together European Journal of Personality 1998 12 307 320
  7. Carter JA Herbst JH Stoller KB King VL Kidorf MS Costa PT Jr. Short-term stability of NEO-PI-R personality trait scores in opioid-dependent outpatients Psychology of Addictive Behaviors 2001 15 255 260 11563805
  8. Cheung F Cheung SF Zhang J Leung K Leong F Yeh K-H Relevance of Openness as a personality dimension in Chinese culture: Aspects of its cultural relevance Journal of Cross-Cultural Psychology 2008 39 81 108
  9. Chmielewski M Watson D What is being assessed and why it matters: The impact of transient error on trait research Journal of Personality and Social Psychology 2009 97 186 202 19586248
  10. Churchill GA Jr. Peter JP Research design effects on the reliability of rating scales: A meta-analysis Journal of Marketing Research 1984 21 360 375
  11. Conley JJ The hierarchy of consistency: A review and model of longitudinal findings on adult individual differences in intelligence, personality, and self-opinion Personality and Individual Differences 1984 5 11 26
  12. Conn SR Rieke ML 16PF Fifth Edition technical manual 1994 Institute for Personality and Ability Testing Champaign, IL
  13. Costa PT Jr. McCrae RR Storandt M Siegler IC Elias MF Objective personality assessment The clinical psychology of aging 1978 119 143 Plenum Press New York
  14. Costa PT Jr. McCrae RR Personality in adulthood: A six-year longitudinal study of self-reports and spouse ratings on the NEO Personality Inventory Journal of Personality and Social Psychology 1988 54 853 863 3379583
  15. Dai X-Y Wu Y-Q A study on NEO-PI-R used in 16-20 years old people [in Chinese] Chinese Journal of Clinical Psychology 2005 13 14 18
  16. Fekken GC Holden RR Jackson DN Guthrie GM An evaluation of the validity of the Personality Research Form with Filipino university students International Journal of Psychology 1987 22 399 407
  17. Gottesman II Genetic variance in adaptive personality traits Journal of Child Psychology and Psychiatry 1966 7 199 208 23276006
  18. Gough HG California Psychological Inventory administrator's guide 1987 Consulting Psychologists Press Palo Alto, CA
  19. Gough HG Heilbrun AB Jr. Adjective Check List manual 1983 Consulting Psychologists Press Palo Alto, CA
  20. Grucza RA Goldberg LR The comparative validity of 11 modern personality inventories: Behavioral acts, informant reports, and clinical indicators Journal of Personality Assessment 2007 89 167 187 17764394
  21. Guadiano BA Is symptomatic improvement in clinical trials of cognitive-behavior therapy for psychosis clinically significant? Journal of Psychiatric Practice 2006 12 11 23 16432441
  22. Heise DR Separating reliability and stability in test-retest correlation American Sociological Review 1969 34 93 101
  23. Helson R Moane G Personality change in women from college to midlife Journal of Personality and Social Psychology 1987 53 176 186 3612488
  24. Horn JM Plomin R Rosenman R Heritability of personality traits in adult male twins Behavior Genetics 1976 6 17 30 943159
  25. Jackson DN Personality Research Form manual 1984 Research Psychologists Press Port Huron, MI 3rd. ed.
  26. John OP Robins RW Determinants of interjudge agreement on personality traits: The Big Five domains, observability, evaluativeness, and the unique perspective of the self Journal of Personality 1993 61 521 551 8151500
  27. Johnson AM Vernon PA Harris JA Jang KL A behavior genetic investigation of the relationship between leadership and personality Twin Research 2004 7 27 32 15053851
  28. Kurtz JE Lee PA Sherker JL Internal and temporal reliability estimates for informant ratings of personality using the NEO-PI-R and IAS Assessment 1999 6 103 113 10335016
  29. Kurtz JE Parrish CL Semantic response consistency and protocol validity in structured personality assessment: The case of the NEO-PI-R Journal of Personality Assessment 2001 76 315 332 11393463
  30. Löckenhoff CE Terracciano A Bienvenu OJ Particiu NS Nestadt G McCrae RR Ethnicity, education, and the temporal stability of personality traits in the East Baltimore Epidemiologic Catchment Area study Journal of Research in Personality 2008 42 577 598 19122849 PMC2585805
  31. Loevinger J The attenuation paradox in test theory Psychological Bulletin 1954 51 493 504 13204488
  32. Markon KE Krueger RF Watson D Delineating the structure of normal and abnormal personality: An integrative hierarchical approach Journal of Personality and Social Psychology 2005 88 139 157 15631580 PMC2242353
  33. McCrae RR Consensual validation of personality traits: Evidence from self-reports and ratings Journal of Personality and Social Psychology 1982 43 293 303
  34. McCrae RR Costa PT Jr. Professional manual for the NEO Inventories: NEO-PI-3, NEO-PI-R, and NEO-FFI-3 Psychological Assessment Resources Odessa, FL in press
  35. McCrae RR Costa PT Jr. Lima MP Simões A Ostendorf F Angleitner A Age differences in personality across the adult life span: Parallels in five cultures Developmental Psychology 1999 35 466 477 10082017
  36. McCrae RR Costa PT Jr. Martin TA The NEO-PI-3: A more readable Revised NEO Personality Inventory Journal of Personality Assessment 2005 84 261 270 15907162
  37. McCrae RR Costa PT Jr. Martin TA Oryol VE Rukavishnikov AA Senin IG Consensual validation of personality traits across cultures Journal of Research in Personality 2004 38 179 201
  38. McCrae RR Martin TA Costa PT Jr. Age trends and age norms for the NEO Personality Inventory-3 in adolescents and adults Assessment 2005 12 363 373 16244117
  39. McCrae RR Martin TA Hřebíčková M Urbánek T Boomsma DI Willemsen G Costa PT Jr. Personality trait similarity between spouses in four cultures Journal of Personality 2008 76 1137 1163 18665894 PMC2626346
  40. McCrae RR Stone SV Fagan PJ Costa PT Jr. Identifying causes of disagreement between self-reports and spouse ratings of personality Journal of Personality 1998 66 285 313 9615420
  41. McCrae RR Terracciano A 78 Members of the Personality Profiles of Cultures Project Universal features of personality traits from the observer's perspective: Data from 50 cultures Journal of Personality and Social Psychology 2005a 88 547 561 15740445
  42. McCrae RR Terracciano A 79 Members of the Personality Profiles of Cultures Project Personality profiles of cultures: Aggregate personality traits Journal of Personality and Social Psychology 2005b 89 407 425 16248722
  43. McCrae RR Yik MSM Trapnell PD Bond MH Paulhus DL Interpreting personality profiles across cultures: Bilingual, acculturation, and peer rating studies of Chinese undergraduates Journal of Personality and Social Psychology 1998 74 1041 1055 9569658
  44. Nichols RC Personality change and college American Educational Research Journal 1967 4 173 190
  45. Nunnally JC Bernstein I Psychometric theory 1994 McGraw-Hill New York 3rd ed.
  46. Peter JP Churchill GA Relationships among research design choices and psychometric properties of rating scales: A meta-analysis Journal of Marketing Research 1986 23 1 10
  47. Piedmont RL Cracking the plaster cast: Big Five personality change during intensive outpatient counseling Journal of Research in Personality 2001 35 500 520
  48. Pilia G Chen W-M Scuteri A Orrú M Albai G Deo M Heritability of cardiovascular and personality traits in 6,148 Sardinians PLoS Genetics 2006 2 1207 1223 PMC1557782 16934002
  49. Riemann R Angleitner A Strelau J Genetic and environmental influences on personality: A study of twins reared together using the self- and peer report NEO-FFI scales Journal of Personality 1997 65 449 475
  50. Roberts BW DelVecchio WF The rank-order consistency of personality traits from childhood to old age: A quantitative review of longitudinal studies Psychological Bulletin 2000 126 3 25 10668348
  51. Rushton JP Fulker DW Neale MC Nias DKB Eysenck HJ Altruism and aggression: The heritability of individual differences Journal of Personality and Social Psychology 1986 50 1192 1198 3723334
  52. Scarr S The origins of individual differences in adjective check list scores Journal of Consulting Psychology 1966 30 354 357 5948459
  53. Schinka JA Borum R Readability of normal personality inventories Journal of Personality Assessment 1994 62 95 101
  54. Schmidt FL Le H Ilies R Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs Psychological Methods 2003 8 206 224 12924815
  55. Sijtsma K On the use, the misuse, and the very limited usefulness of Cronbach's alpha Psychometrika 2009 74 107 120 20037639 PMC2792363
  56. Snell MN Mallinckrodt B Hill RD Lambert MJ Predicting counseling center clients' response to counseling: A 1-year follow-up Journal of Counseling Psychology 2001 48 463 473
  57. Streiner DL Starting at the beginning: An introduction to coefficient alpha and internal consistency Journal of Personality Assessment 2003 80 99 103 12584072
  58. Sung MS Kim JH Yang E Abrams KY Lyoo IK Reliability and validity of the Korean version of the Temperament and Character Inventory Comprehensive Psychiatry 2002 43 236 243 11994844
  59. Terracciano A Costa PT Jr. McCrae RR Personality plasticity after age 30 Personality and Social Psychology Bulletin 2006 32 999 1009 16861305 PMC2680603
  60. Vernon PA Jang KL Harris JA McCarthy JM Environmental predictors of personality differences: A twin and sibling study Journal of Personality and Social Psychology 1997 72 177 183 9008379
  61. Vul E Pashler H Measuring the crowd within: Probabilistic representations within individuals Psychological Science 2008 19 645 647 18727777
  62. Watson D Stability versus change, dependability versus error: Issues in the assessment of personality over time Journal of Research in Personality 2004 38 319 350
  63. Yamagata S Ando J Ostendorf F Angleitner A Riemann R Spinath F Cross-cultural differences in heritability of personality traits: Using behavioral genetics to study culture September 2006 Paper presented at the 4th CEFOM/21 International Symposium Tokyo

Cited by

Sorted by citations

Authors
McCrae, R. R., Kurtz, J. E., Yamagata, S., & Terracciano, A.
Affiliations
  • National Institutes of Health (US)
  • Villanova University (US)
  • Keio University (JP)
Corresponding author
McCrae, Robert R.
ORCID
Venue
Personality and Social Psychology Review
Publisher
SAGE Publications
Published
2010-04-30
Volume
15
Issue
1
Pages
28-50
ISSN
  • 1088-8683 (ISSN-L)
  • 1532-7957
DOI
10.1177/1088868310366253
PMC
PMC2927808
OpenAlex
W2111341078
Cited by
874
References
6
Citations by year
2026: 92 · 2025: 68 · 2024: 56 · 2023: 55 · 2022: 59 · 2021: 53 · 2020: 91 · 2019: 54 · 2018: 54 · 2017: 57 · 2016: 82 · 2015: 43 · 2014: 40 · 2013: 29 · 2012: 23
FWCI
29.18
Type
journal article
Language
en
Open access
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
  • Personality and Social Psychology Review (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • National Institutes of Health (Z99 AG999999)
  • National Institute on Aging (ZIAAG000183)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Social Sciences › Psychology › Clinical Psychology › Personality Traits and Psychology
  • Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
Keywords
internal consistency, test-retest reliability, heritability, personality tests
MeSH terms
Age Factors, Character, Cross-Cultural Comparison, Humans, Personality Inventory, Psychometrics, Research, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review