‘I’m pickin' up good regressions': the governance of generalisability analyses
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Context Investigators applying generalisability theory to educational research and evaluation have sometimes done so poorly. The main difficulties have related to: inadequate or non‐random sampling of effects, dealing with naturalistic data, and interpreting and presenting variance components. Methods This paper addresses these areas of difficulty, and articulates an informal consensus amongst medical educators from Europe, Australia and the USA, who are familiar with generalisability theory. Results We make the following recommendations. Ensure that all relevant factors are sampled, and that the sampling meets the theory's assumption that the conditions represent a random and representative sample of the factor's ‘universe’. Research evaluations will require large samples of each factor if they are to generalise adequately. Where feasible, conduct 2 separate studies (pilot and evaluation, or Generalisability and Decision studies). For unbalanced data, use either ur genova , or 1 of the procedures minimum norm quadratic unbiased estimator, ( minque ), maximum likelihood ( ml ) or restricted maximum likelihood ( reml ) in spss or sas if the data are too complex. State which mathematical procedure was used and the degrees of freedom (d.f.) of the effect estimates. If the procedure does not report d.f., re‐analyse with type III sum of squares anova ( anova ss III) and report these d.f. Describe and justify the regression model used. Present the raw variance components. Describe the effects that they represent in plain, non‐statistical language. If standard error of measurement (SEM) or Reliability coefficients are presented, give the equations used to calculate them. Make sure that the method of reporting reliability (precision or discrimination) is appropriate to the purpose of the assessment. This will usually demand a precision indicator such as SEM. Consider a graphical presentation to combine precision and discrimination.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reliability: on the reproducibility of assessment data
Downing, S. M. (2004) · Medical Education · doi:10.1111/j.1365-2929.2004.01932.x · Cited by 721
Generalisability: a key to unlock professional assessment
Crossley, J., Davies, H., Humphris, G., & Jolly, B. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01320.x · Cited by 169
Self-citation
Cited by
Implementing workplace‐based assessment across the medical specialties in the United Kingdom
Wilkinson, J. R., Crossley, J. G. M., Wragg, A., Mills, P., Cowan, G., & Wade, W. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03010.x · Cited by 272
Self-citationRoberts, C., Walton, M., Rothnie, I., Crossley, J., Lyon, P., Kumar, K., & Tiller, D. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03018.x · Cited by 121
Self-citationRoberts, C., Rothnie, I., Zoanetti, N., & Crossley, J. (2010) · Medical Education · doi:10.1111/j.1365-2923.2010.03689.x · Cited by 59
Self-citationUijtdehaage, S., Doyle, L. “., & Parker, N. (2011) · Academic Medicine · doi:10.1097/acm.0b013e3182223ab7 · Cited by 57
Roberts, C., Clark, T., Burgess, A., Frommer, M., Grant, M., & Mossman, K. (2014) · BMC Medical Education · doi:10.1186/1472-6920-14-169 · Cited by 49
“However, the confidence interval crosses three quartiles giving less than 95% confidence that a candidate at the bottom of the top quartile might behave better than a candidate at the top of the 3 quartile [31].”
BackgroundMethodSelf-citation
Metadata
- Authors
- Crossley, J., Russell, J., Jolly, B., Ricketts, C., Roberts, C., Schuwirth, L., & Norcini, J.
- Venue
- Medical Education
- Publisher
- Wiley
- Published
- 2007-09-28
- Volume
- 41
- Issue
- 10
- Pages
- 926-934
- ISSN
- 0308-0110 (ISSN-L)
- 1365-2923
- OpenAlex
- W2093691848
- Cited by
- 77
- References
- 2
- FWCI
- 9.54
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ‘I’m pickin' up good regressions': the governance of generalisability analyses. https://app.cochat.ai/papers/10.1111/j.1365-2923.2007.02843.x
- The paper
- Crossley, J., Russell, J., Jolly, B., Ricketts, C., Roberts, C., Schuwirth, L., & Norcini, J. (2007). ‘I’m pickin' up good regressions': the governance of generalisability analyses. Medical Education. https://doi.org/10.1111/j.1365-2923.2007.02843.x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.