The DOI matches its Crossref record.
Generalizability theory: a primer
Abstract
Concepts in Generalizability Theory Statistical Model Underlying Generalizability Theory Generalizability Studies with Crossed Facets Generalizability Studies with Nested Facets Generalizability Studies with Fixed Facets Decision Studies Measurement Error and Generalizability Coefficients Generalizability and Decision Studies with the Same Design Generalizability and Decision Studies with Different Designs Summary and Next Steps
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Method Variance in Organizational Research: Truth or Urban Legend?
Spector, P. E. (2006) · Organizational Research Methods · doi:10.1177/1094428105284955 · Cited by 3,445
Validating the interpretations and uses of test scores
Kane, M. T. (2013) · Journal of Educational Measurement · doi:10.1111/jedm.12000 · Cited by 2,110
Teachers’ Mathematical Knowledge, Cognitive Activation in the Classroom, and Student Progress
Baumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., Klusmann, U., Krauss, S., Neubrand, M., & Tsai, Y.-M. (2010) · American Educational Research Journal · doi:10.3102/0002831209345157 · Cited by 2,100
When to use agreement versus reliability measures
de Vet, H. C. W., Terwee, C. B., Knol, D. L., & Bouter, L. M. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.10.015 · Cited by 1,786
Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application
Cook, D. A., & Beckman, T. J. (2006) · The American Journal of Medicine · doi:10.1016/j.amjmed.2005.10.036 · Cited by 1,745
The Measurement of Place Attachment: Validity and Generalizability of a Psychometric Approach
Williams, D. R., & Vaske, J. J. (2003) · Forest Science · doi:10.1093/forestscience/49.6.830 · Cited by 1,725
My Current Thoughts on Coefficient Alpha and Successor Procedures
Cronbach, L. J., & Shavelson, R. J. (2004) · Educational and Psychological Measurement · doi:10.1177/0013164404266386 · Cited by 1,606
Measurement reliability and agreement in psychiatry
Shrout, P. E. (1998) · Statistical Methods in Medical Research · doi:10.1177/096228029800700306 · Cited by 1,216
(2011) · Amsterdam UMC (VU Amsterdam) - Institutional Repository · Cited by 1,209
Beyond Dichotomies: Competence Viewed as a Continuum
Blömeke, S., Gustafsson, J.-E., & Shavelson, R. J. (2015) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000194 · Cited by 1,205
Gadermann, A., Guhn, M., & Zumbo, B. D. (2020) · PubMed · doi:10.7275/n560-j767 · Cited by 1,200
Understanding Internal Consistency Reliability Estimates: A Conceptual Primer on Coefficient Alpha
Henson, R. K. (2001) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.2002.12069034 · Cited by 1,058
Stemler, S. E. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/96jp-xz07 · Cited by 992
The Handbook of Research Synthesis
Ziegel, E. R., Cooper, H., & Hedges, L. (1995) · Technometrics · doi:10.2307/1269651 · Cited by 927
Silvia, P. J., Winterstein, B. P., Willse, J. T., Barona, C. M., Cram, J. T., Hess, K. I., Martinez, J. L., & Richard, C. A. (2008) · Psychology of Aesthetics, Creativity, and the Arts · doi:10.1037/1931-3896.2.2.68 · Cited by 912
The Problem of Units and the Circumstance for POMP
Cohen, P., Cohen, J., Aiken, L. S., & West, S. G. (1999) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3403_2 · Cited by 900
Smallest real difference, a link between reproducibility and responsiveness
Beckerman, H., Roebroeck, M. E., Lankhorst, G. J., Becher, J. G., Bezemer, P. D., & Verbeek, A. L. M. (2001) · Quality of Life Research · doi:10.1023/a:1013138911638 · Cited by 857
Cranford, J. A., Shrout, P. E., Iida, M., Rafaeli, E., Yip, T., & Bolger, N. (2006) · Personality and Social Psychology Bulletin · doi:10.1177/0146167206287721 · Cited by 856
Hambleton, R. K., & Jones, R. W. (1993) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.1993.tb00543.x · Cited by 731
GERHART, B., WRIGHT, P. M., MC MAHAN, G. C., & SNELL, S. A. (2000) · Personnel Psychology · doi:10.1111/j.1744-6570.2000.tb02418.x · Cited by 679
Eurasian Journal of Educational Research
(2021) · Eurasian Journal of Educational Research · doi:10.14689/ejer · Cited by 638
Measuring business model innovation: conceptualization, scale development, and proof of performance
Clauss, T. (2017) · R&D Management · doi:10.1111/radm.12186 · Cited by 605
Kleickmann, T., Richter, D., Kunter, M., Elsner, J., Besser, M., Krauss, S., & Baumert, J. (2013) · Journal of Teacher Education · doi:10.1177/0022487112460398 · Cited by 538
Pedagogical content knowledge and content knowledge of secondary mathematics teachers.
Krauss, S., Brunner, M., Kunter, M., Baumert, J., Blum, W., Neubrand, M., & Jordan, A. (2008) · Journal of Educational Psychology · doi:10.1037/0022-0663.100.3.716 · Cited by 491
Bloom, H. S., Hill, C. J., Black, A. R., & Lipsey, M. W. (2008) · Journal of Research on Educational Effectiveness · doi:10.1080/19345740802400072 · Cited by 477
Brand origin recognition accuracy: its antecedents and consumers’ cognitive limitations
Samiee, S., Shimp, T. A., & Sharma, S. (2005) · Journal of International Business Studies · doi:10.1057/palgrave.jibs.8400145 · Cited by 461
Generalizability studies of the Global Assessment of Functioning–Split version
Pedersen, G., Hagtvet, K. A., & Karterud, S. (2007) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2006.03.008 · Cited by 446
Principles of Situation Research: Towards a Better Understanding of Psychological Situations
Rauthmann, J. F., Sherman, R. A., & Funder, D. C. (2015) · European Journal of Personality · doi:10.1002/per.1994 · Cited by 415
Quantitative and qualitative methods:
Sechrest, L., & Sidani, S. (1995) · Evaluation and Program Planning · doi:10.1016/0149-7189(94)00051-x · Cited by 391
Theory-Testing, Generalization, and the Problem of External Validity
Lucas, J. W. (2003) · Sociological Theory · doi:10.1111/1467-9558.00187 · Cited by 372
Learning to read and write genre‐specific text: Roles of authentic experience and explicit teaching
PURCELL‐GATES, V., DUKE, N. K., & MARTINEAU, J. A. (2007) · Reading Research Quarterly · doi:10.1598/rrq.42.1.1 · Cited by 359
Psychometrics Is Datametrics: The Test Is Not Reliable
Thompson, B., & Vacha-Haase, T. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970448 · Cited by 357
Sampling Variability of Performance Assessments
Shavelson, R. J., Baxter, G. P., & Gao, X. (1993) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1993.tb00424.x · Cited by 338
Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012) · Educational Researcher · doi:10.3102/0013189x12437203 · Cited by 331
Review papers : Design and analysis of reliability studies
Dunn, G. (1992) · Statistical Methods in Medical Research · doi:10.1177/096228029200100202 · Cited by 325
Rater bias in psychological research: When is it a problem and what can we do about it?
Hoyt, W. T. (2000) · Psychological Methods · doi:10.1037//1082-989x.5.1.64 · Cited by 322
An Overview on Assessing Agreement with Continuous Measurements
Barnhart, H. X., Haber, M. J., & Lin, L. I. (2007) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543400701376480 · Cited by 319
On the cultural validity of science assessments
Solano‐Flores, G., & Nelson‐Barber, S. (2001) · Journal of Research in Science Teaching · doi:10.1002/tea.1018 · Cited by 314
One lesson is all you need? Stability of instructional quality across lessons
Praetorius, A.-K., Pauli, C., Reusser, K., Rakoczy, K., & Klieme, E. (2014) · Learning and Instruction · doi:10.1016/j.learninstruc.2013.12.002 · Cited by 313
Ostroff, C. L., Kinicki, A. J., & Clark, M. A. (2002) · Journal of Applied Psychology · doi:10.1037//0021-9010.87.2.355 · Cited by 309
Validating Measures of Performance
Kane, M., Crooks, T., & Cohen, A. (1999) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.1999.tb00010.x · Cited by 299
A More Nuanced View of Reliability: Specificity in the Trait Hierarchy
McCrae, R. R. (2015) · Personality and Social Psychology Review · doi:10.1177/1088868314541857 · Cited by 292
Hönekopp, J. (2006) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.32.2.199 · Cited by 290
Equity Issues in Collaborative Group Assessment: Group Composition and Performance
Webb, N. M., Nemer, K. M., Chizhik, A. W., & Sugrue, B. (1998) · American Educational Research Journal · doi:10.3102/00028312035004607 · Cited by 284
Correlation, causation, motivation, and second language acquisition.
Gardner, R. C. (2000) · Canadian Psychology / Psychologie canadienne · doi:10.1037/h0086854 · Cited by 276
4 Reliability Coefficients and Generalizability Theory
Webb, N. M., Shavelson, R. J., & Haertel, E. H. (2006) · Handbook of Statistics · doi:10.1016/s0169-7161(06)26004-8 · Cited by 271
From Evidence to Action: A Seamless Process in Formative Assessment?
Heritage, M., Kim, J., Vendlinski, T., & Herman, J. (2009) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.2009.00151.x · Cited by 263
Solano-Flores, G., & Trumbull, E. (2003) · Educational Researcher · doi:10.3102/0013189x032002003 · Cited by 261
Standardized Achievement Tests and English Language Learners: Psychometrics Issues
Abedi, J. (2002) · Educational Assessment · doi:10.1207/s15326977ea0803_02 · Cited by 258
Reliability Methods: A Note on the Frequency of Use of Various Types
Hogan, T. P., Benjamin, A., & Brezinski, K. L. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970691 · Cited by 258
Krauss, S., Baumert, J., & Blum, W. (2008) · ZDM · doi:10.1007/s11858-008-0141-9 · Cited by 253
Automatic Detection of Acute Bacterial Pneumonia from Chest X-ray Reports
Fiszman, M., Chapman, W. W., Aronsky, D., Evans, R. S., & Haug, P. J. (2000) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2000.0070593 · Cited by 250
An Argument Approach to Observation Protocol Validity
Bell, C. A., Gitomer, D. H., McCaffrey, D. F., Hamre, B. K., Pianta, R. C., & Qi, Y. (2012) · Educational Assessment · doi:10.1080/10627197.2012.715014 · Cited by 236
Metaphorically speaking: cognitive abilities and the production of figurative language
Beaty, R. E., & Silvia, P. J. (2013) · Memory & Cognition · doi:10.3758/s13421-012-0258-5 · Cited by 216
Baker, T. B., Gustafson, D. H., & Shah, D. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2925 · Cited by 221
“Moderation analyses, when done with pre-existing datasets, would dramatically reduce the time spent in obtaining “generalizability data” [46], allowing researchers to discover facets across which findings can be generalized: In which persons and contexts does an intervention work well? These questions lie at the heart…”
BackgroundYin, P., & Fan, X. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970466 · Cited by 197
Expanding test–retest designs to include developmental time-lag components.
McArdle, J. J., & Woodcock, R. W. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.4.403 · Cited by 157
Decaluwé, V., & Braet, C. (2004) · Behaviour Research and Therapy · doi:10.1016/j.brat.2003.07.008 · Cited by 152
Data Analysis in Social Psychology: Recent and Recurring Issues
Judd, C. M., & Kenny, D. A. (2010) · Handbook of Social Psychology · doi:10.1002/9780470561119.socpsy001004 · Cited by 151
Ing, M., Webb, N. M., Franke, M. L., Turrou, A. C., Wong, J., Shin, N., & Fernandez, C. H. (2015) · Educational Studies in Mathematics · doi:10.1007/s10649-015-9625-z · Cited by 136
Measuring Effective Teacher-Student Interactions From a Student Perspective: A Multi-Level Analysis
Downer, J. T., Stuhlman, M., Schweig, J., Martínez, J. F., & Ruzek, E. (2015) · The Journal of Early Adolescence · doi:10.1177/0272431614564059 · Cited by 111
Poncy, B. C., Skinner, C. H., & Axtell, P. K. (2005) · Journal of Psychoeducational Assessment · doi:10.1177/073428290502300403 · Cited by 105
“This is similar to how the Spearman-Brown prophecy formula informs about the length of a test to make relative decisions (Shavelson & Webb, 1991).”
BackgroundMethodInfluentialHerlihy, C., Karger, E., Pollard, C., Hill, H. C., Kraft, M. A., Williams, M., & Howard, S. (2014) · Teachers College Record: The Voice of Scholarship in Education · doi:10.1177/016146811411600108 · Cited by 103
“In many cases (e.g., Bell et al., 2012; Hill, Charalambous, & Kraft, 2012; Shavelson & Webb, 1991), observational data suggests significant score instability due to raters and lessons.”
BackgroundTruong, Q. C., Krägeloh, C. U., Siegert, R. J., Landon, J., & Medvedev, O. N. (2020) · Mindfulness · doi:10.1007/s12671-020-01324-7 · Cited by 65
Bias in participant ratings of psychotherapy process: An initial generalizability study.
Hoyt, W. T. (2002) · Journal of Counseling Psychology · doi:10.1037/0022-0167.49.1.35 · Cited by 38
Li, J., & Huang, J. (2022) · Assessing Writing · doi:10.1016/j.asw.2021.100604 · Cited by 31
The Effects of Method Variance Within Measures
Tepper, B. J., & Tepper, K. (1993) · The Journal of Psychology · doi:10.1080/00223980.1993.9915563 · Cited by 30
Using Classroom Observations in the Evaluation of Special Education Teachers
Jones, N. D., Bell, C. A., Brownell, M., Qi, Y., Peyton, D., Pua, D., Fowler, M., & Holtzman, S. (2022) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737211068523 · Cited by 21
Clayson, P. E., Mcdonald, J. B., Park, B., Holbrook, A., Baldwin, S. A., Riesel, A., & Larson, M. J. (2025) · Psychophysiology · doi:10.1111/psyp.14496 · Cited by 21
Holman, P. A., Ruud, T., & Grepperud, S. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-162 · Cited by 17
“The degree of agreement in priority setting was analyzed by using intraclass correlation analysis (ICC) [26] and generalizability theory [27,28].”
BackgroundMethodCharalambous, C. Y., Kyriakides, E., Tsangaridou, N., & Kyriakides, L. (2017) · School Effectiveness and School Improvement · doi:10.1080/09243453.2017.1311929 · Cited by 11
Corkum, P., Andreou, P., Schachar, R., Tannock, R., & Cunningham, C. (2007) · Educational and Psychological Measurement · doi:10.1177/0013164406292038 · Cited by 7
Metadata
- Venue
- Choice Reviews Online
- Publisher
- American Library Association
- Published
- 1992-03-01
- Volume
- 29
- Issue
- 07
- Pages
- 29-4192-29-4192
- ISSN
- 0009-4978 (ISSN-L)
- 1523-8253
- OpenAlex
- W1958015814
- Cited by
- 1,298
- References
- 0
- Type
- book review
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.