Implications of Small Samples for Generalization: Adjustments and Rules of Thumb
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Background: Policy makers and researchers are frequently interested in understanding how effective a particular intervention may be for a specific population. One approach is to assess the degree of similarity between the sample in an experiment and the population. Another approach is to combine information from the experiment and the population to estimate the population average treatment effect (PATE). Method: Several methods for assessing the similarity between a sample and population currently exist as well as methods estimating the PATE. In this article, we investigate properties of six of these methods and statistics in the small sample sizes common in education research (i.e., 10–70 sites), evaluating the utility of rules of thumb developed from observational studies in the generalization case. Result: In small random samples, large differences between the sample and population can arise simply by chance and many of the statistics commonly used in generalization are a function of both sample size and the number of covariates being compared. The rules of thumb developed in observational studies (which are commonly applied in generalization) are much too conservative given the small sample sizes found in generalization. Conclusion: This article implies that sharp inferences to large populations from small experiments are difficult even with probability sampling. Features of random samples should be kept in mind when evaluating the extent to which results from experiments conducted on nonrandom samples might generalize.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 2×
“This is the case when the experimental sample is randomly drawn from the population, when Pr(Z 1⁄4 1) 1⁄4 1, or if the sample selection and treatment effect heterogeneity are independent (Imai, King, & Stuart, 2008; Rubin, 1974).”
BackgroundMatching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478 · cited 2×
“Like the ASMD for the individual covariates, the logit SMD is commonly used as a measure of assessment (i.e., balance) in observational studies (see Stuart, 2010).”
MethodLogistic Regression in Rare Events Data
King, G., & Zeng, L. (2001) · Political Analysis · doi:10.1093/oxfordjournals.pan.a004868 · Cited by 4,316 · cited 1×
“Various solutions have been proposed for reducing this bias, including the use of rare-events logistic regression (RE-), a profile-likelihood method proposed by King and Zeng (2001). In this paper, we compare results from RE- and standard- logistic regression to determine if and when these small sample corrections mat…”
MethodReducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082
Constructing Inverse Probability Weights for Marginal Structural Models
Cole, S. R., & Hernan, M. A. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn164 · Cited by 2,796 · cited 1×
“This approach can be seen as an extension of the Horvitz–Thompson estimator (Cole & Hernan, 2008).”
BackgroundUsing Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 2×
“With large samples, the benchmark then is a rule of thumb indicating bounds within which regression adjustments will perform well (Rubin, 2001).”
BackgroundMethodInfluentialThe importance of covariate selection in controlling for selection bias in observational studies.
Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010) · Psychological Methods · doi:10.1037/a0018719 · Cited by 490
2014 Rossi Award Lecture: Beyond Internal Validity
Orr, L. L. (2015) · Evaluation Review · doi:10.1177/0193841x15573659 · Cited by 14
Cited by
A Review of Generalizability and Transportability
Degtiar, I., & Rose, S. (2023) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042522-103837 · Cited by 295
Donovan, B. M., Stuhlsatz, M. A. M., Edelson, D. C., & Buck Bracey, Z. E. (2019) · Science Education · doi:10.1002/sce.21502 · Cited by 60
Lee, R. Y., Kross, E. K., Downey, L., Paul, S. R., Heywood, J., Nielsen, E. L., Okimoto, K., Brumback, L. C., Merel, S. E., Engelberg, R. A., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.5088 · Cited by 44
Ageism and Attitudes Toward Older Adults in Arab Culture: A Systematic Integrative Review
Younes, R. S., & Abboud Mzawak, M. (2026) · Social Sciences · doi:10.3390/socsci15020080 · Cited by 7
Younes, R. S., & Rajha, F. (2025) · NPG Neurologie - Psychiatrie - Gériatrie · doi:10.1016/j.npg.2024.11.002 · Cited by 5
Metadata
- Authors
- Tipton, E., Hallberg, K., Hedges, L. V., & Chan, W.
- Venue
- Evaluation Review
- Publisher
- SAGE Publications
- Published
- 2016-07-08
- Volume
- 41
- Issue
- 5
- Pages
- 472-505
- ISSN
- 0145-4692 (ISSN-L)
- 0193-841X
- 1552-3926
- OpenAlex
- W2463925622
- Cited by
- 140
- References
- 8
- FWCI
- 4.87
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Implications of Small Samples for Generalization: Adjustments and Rules of Thumb. https://app.cochat.ai/papers/10.1177/0193841x16655665
- The paper
- Tipton, E., Hallberg, K., Hedges, L. V., & Chan, W. (2017). Implications of Small Samples for Generalization: Adjustments and Rules of Thumb. Evaluation Review. https://doi.org/10.1177/0193841x16655665
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.