Selecting an Appropriate Caliper Can Be Essential for Achieving Good Balance With Propensity Score Matching
Open accessThe DOI matches its Crossref record; 23 of the 25 references it cites were found in a registry or an index.
Abstract
Matching on the propensity score is widely used to estimate the effect of an exposure in observational studies. However, the quality of the matches can be affected by decisions made during the matching process, particularly the order in which subjects are selected for matching and the maximum permitted difference between matched subjects (the "caliper"). This study used simulations to explore the effects of these decisions on both the imbalance of covariates and the closeness of matching, while allowing the numbers of potential matches and strengths of association between the confounding variable and the exposure to vary. It was found that, without a caliper, substantial bias was possible, particularly with a relatively small reservoir of potential matches and strong confounder-exposure association. Use of the recommended caliper reduced the bias considerably, but bias remained if subjects were selected by increasing or decreasing propensity score. A tighter caliper led to greatly reduced bias and closer matches, although some subjects could not be matched. This study suggests that a narrow caliper can improve the performance of propensity score matching. In situations where it is impossible to find appropriate matches for all exposed subjects, it is better to select subjects in order of the best available matches, rather than increasing or decreasing the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most of the literature suggests that the calliper method outperforms other methods such as nearest neighbour with and without replacement and optimal matching [55] and gives better results if it is difficult to find the matches from the control group [56].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241 · cited 4×
“It can be difficult to find appropriate matches when trying to match on several variables, but Rosenbaum and Rubin (7) showed that matching on the propensity score (the conditional probability of exposure given a set of covariates) could produce samples with the same distribution of covariates in exposed and unexposed…”
BackgroundInfluentialSOME PRACTICAL GUIDANCE FOR THE IMPLEMENTATION OF PROPENSITY SCORE MATCHING
Caliendo, M., & Kopeinig, S. (2008) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.2007.00527.x · Cited by 6,238 · cited 1×
“2 standard deviations), more recent papers on the practicalities of matching have not given recommendations for setting a caliper (12, 13).”
BackgroundMatching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478 · cited 2×
“2 standard deviations), more recent papers on the practicalities of matching have not given recommendations for setting a caliper (12, 13).”
BackgroundPropensity Score-Matching Methods for Nonexperimental Causal Studies
Dehejia, R. H., & Wahba, S. (2002) · Review of Economics and Statistics · doi:10.1162/003465302317331982 · Cited by 4,846
Rosenbaum, P. R., & Rubin, D. B. (1985) · The American Statistician · doi:10.2307/2683903 · Cited by 4,774 · cited 6×
“It can be difficult to find appropriate matches when trying to match on several variables, but Rosenbaum and Rubin (7) showed that matching on the propensity score (the conditional probability of exposure given a set of covariates) could produce samples with the same distribution of covariates in exposed and unexposed…”
BackgroundMethodInfluentialAustin, P. C. (2011) · Pharmaceutical Statistics · doi:10.1002/pst.433 · Cited by 3,949 · cited 5×
“In practice, a wide variety of calipers is used (1) and, with the exception of Austin (11) (who recommended reducing the caliper from 0.25 standard deviations to 0.2 standard deviations), more recent papers on the practicalities of matching have not given recommendations for setting a caliper (12,…”
BackgroundMethodInfluentialA critical appraisal of propensity‐score matching in the medical literature between 1996 and 2003
Austin, P. C. (2008) · Statistics in Medicine · doi:10.1002/sim.3150 · Cited by 1,249
Matching Using Estimated Propensity Scores: Relating Theory to Practice
Rubin, D. B., & Thomas, N. (1996) · Biometrics · doi:10.2307/2533160 · Cited by 1,074
Matching Using Estimated Propensity Scores: Relating Theory to Practice
Thomas, N. (2006) · Matched Sampling for Causal Effects · doi:10.1017/cbo9780511810725.024 · Cited by 1,018 · cited 1×
“In order to be able to find suitable matches for all exposed subjects, the number of controls available needs to be greater than the number of exposed subjects; the ratio typically lies in the range of 2–20, although it may be higher (8).”
BackgroundControlling Bias in Observational Studies: A Review
Cochran, W. G. (2006) · Matched Sampling for Causal Effects · doi:10.1017/cbo9780511810725.005 · Cited by 958 · cited 5×
“It can be difficult to find appropriate matches when trying to match on several variables, but Rosenbaum and Rubin (7) showed that matching on the propensity score (the conditional probability of exposure given a set of covariates) could produce samples with the same distribution of covariates in exposed and unexposed…”
MethodInfluentialReducing bias in a propensity score matched-pair sample using greedy matching techniques
Parsons, L. (2001) · Medical Entomology and Zoology · Cited by 928
Matching to remove bias in observational studies
Rubin, D. B. (1973) · Biometrics · doi:10.2307/2529684 · Cited by 737
Optimal Matching for Observational Studies
Rosenbaum, P. R. (1989) · Journal of the American Statistical Association · doi:10.1080/01621459.1989.10478868 · Cited by 715 · cited 1×
“There are optimalmatchingmethods that will breakmatches if doing so can result in a better overallmatched sample, and it has been shown that there are circumstances in which greedy matching will find fewer acceptable matches than optimal matching (17).”
BackgroundMatching to Remove Bias in Observational Studies
(2006) · Matched Sampling for Causal Effects · doi:10.1017/cbo9780511810725.007 · Cited by 600 · cited 2×
“Although it has been suggested that trying to match exposed subjects in descending order of propensity score will lead to the best possible matches (14), a number of other suggestions as to the order in which matches are selected have also been made (5, 15).”
BackgroundMethodThe Use of Matched Sampling and Regression Adjustment to Remove Bias in Observational Studies
(2006) · Matched Sampling for Causal Effects · doi:10.1017/cbo9780511810725.008 · Cited by 442
The Use of Matched Sampling and Regression Adjustment to Remove Bias in Observational Studies
Rubin, D. B. (1973) · Biometrics · doi:10.2307/2529685 · Cited by 442
Experimental Designs in Sociological Research.
Kempthorne, O., & Chapin, F. S. (1948) · Journal of the American Statistical Association · doi:10.2307/2280283 · Cited by 114
Greenwood, E. (1945) · Columbia University Press eBooks · doi:10.7312/gree91078 · Cited by 58
Experimental Sociology: A Study in Method.
Kendall, P. L., & Greenwood, E. (1945) · American Sociological Review · doi:10.2307/2085603 · Cited by 50 · cited 2×
“2014;179(2):226–235 American Journal of Epidemiology © The Author 2013.”
BackgroundMethodCaliper pair-matching on a continuous variable in case-control studies
Raynor, W. J. (1983) · Communication in Statistics- Theory and Methods · doi:10.1080/03610928308828546 · Cited by 30 · cited 3×
“Furthermore, although Raynor (10) considered how the strength of the association between the propensity score and outcome affected the choice of caliper, neither author considered how the appropriate caliper may depend on the difficulty of finding matches, as this article does.”
BackgroundExperimental Sociology: A Study in Method
Mueller, F. H., & Greenwood, E. (1945) · The American Catholic Sociological Review · doi:10.2307/3707536 · Cited by 28
Experimental Designs in Sociological Research
Furfey, P. H., & Chapin, F. S. (1948) · The American Catholic Sociological Review · doi:10.2307/3706430 · Cited by 17
(1950) · doi:10.1002/1097-0142(1950)3:1<32::aid-cncr2820030106>3.0.co;2-3
Cited by
Brown, J. W. L., Coles, A., Horakova, D., Havrdova, E., Izquierdo, G., Prat, A., Girard, M., Duquette, P., Trojano, M., Lugaresi, A., et al. (2019) · JAMA · doi:10.1001/jama.2018.20588 · Cited by 546
Yao, X. I., Wang, X., Speicher, P. J., Hwang, E. S., Cheng, P., Harpole, D. H., Berry, M. F., Schrag, D., & Pang, H. H. (2017) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djw323 · Cited by 386
Bramante, C. T., Ingraham, N. E., Murray, T. A., Marmor, S., Hovertsen, S., Gronski, J., McNeil, C., Feng, R., Guzman, G., Abdelwahab, N., et al. (2021) · The Lancet Healthy Longevity · doi:10.1016/s2666-7568(20)30033-7 · Cited by 291
Ali, M. S., Groenwold, R. H. H., Belitser, S. V., Pestman, W. R., Hoes, A. W., Roes, K. C. B., Boer, A. D., & Klungel, O. H. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.08.011 · Cited by 222
“It is important to note that exclusion of unmatched patients from the analysis not only affect the precision of the effect estimate but also have consequences on the generalizability of results [5,6].”
BackgroundInfluentialStatistical matching for conservation science
Schleicher, J., Eklund, J., D. Barnes, M., Geldmann, J., Oldekop, J. A., & Jones, J. P. G. (2020) · Conservation Biology · doi:10.1111/cobi.13448 · Cited by 229
Metadata
- Authors
- Lunt, M.
- Affiliations
- University of Manchester (GB)
- Corresponding author
- Lunt, Mark
- ORCID
- Lunt, Mark
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2013-10-10
- Volume
- 179
- Issue
- 2
- Pages
- 226-235
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- PMC
- PMC3873103
- OpenAlex
- W2154872949
- Cited by
- 358
- References
- 25
- Citations by year
- 2026: 30 · 2025: 50 · 2024: 21 · 2023: 35 · 2022: 31 · 2021: 39 · 2020: 36 · 2019: 39 · 2018: 24 · 2017: 16 · 2016: 15 · 2015: 16 · 2014: 6
- FWCI
- 10.88
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- American Journal of Epidemiology (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion)
- Funders
- Versus Arthritis (/17552)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- propensity score matching, covariate imbalance, matching bias, confounding, observational studies
- MeSH terms
- Epidemiologic Methods, Humans, Bias, Propensity Score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.