CoChat
Sign Up
Cited by 3,949Advanced Causal Inference TechniquesStatistical Methods in Clinical TrialsStatistical Methods in Epidemiology

Optimal caliper widths for propensity‐score matching when estimating differences in means and differences in proportions in observational studies

Open access
Austin, Peter C.·2011·Pharmaceutical Statistics
DOI 10.1002/pst.433PMC PMC3120982
Open this paper in CoChat — free accountOpen PDF

Abstract

In a study comparing the effects of two treatments, the propensity score is the probability of assignment to one treatment conditional on a subject's measured baseline covariates. Propensity‐score matching is increasingly being used to estimate the effects of exposures using observational data. In the most common implementation of propensity‐score matching, pairs of treated and untreated subjects are formed whose propensity scores differ by at most a pre‐specified amount (the caliper width). There has been a little research into the optimal caliper width. We conducted an extensive series of Monte Carlo simulations to determine the optimal caliper width for estimating differences in means (for continuous outcomes) and risk differences (for binary outcomes). When estimating differences in means or risk differences, we recommend that researchers match on the logit of the propensity score using calipers of width equal to 0.2 of the standard deviation of the logit of the propensity score. When at least some of the covariates were continuous, then either this value, or one close to it, minimized the mean square error of the resultant estimated treatment effect. It also eliminated at least 98% of the bias in the crude estimator, and it resulted in confidence intervals with approximately the correct coverage rates. Furthermore, the empirical type I error rate was approximately correct. When all of the covariates were binary, then the choice of caliper width had a much smaller impact on the performance of estimation of risk differences and differences in means. Copyright © 2010 John Wiley & Sons, Ltd.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/pst.433

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Propensity score matching achieved adequate covariate balance between the Pragmatic and Value-based groups.

    “The effect size estimates for differences on primary need and baseline PHQ-9 scores after matching were small, and the average logit of the propensity score for the groups indicated that adequate covariate balance was achieved (Table 1).48”

    From Using Machine Learning to Match Clients and Therapy Providers: Evaluating Clinical Quality and Cost of Care (2025) · cites this paper as [48]

    Section 3.2 describes matching on the logit of the propensity score, but does not assess balance between the Pragmatic and Value-based groups. The Discussion says the study did not focus on the relationship between caliper width and balance of measured baseline covariates. Our copy therefore cannot verify the cited sentence’s group-specific balance finding. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 20

  1. [2]Rosenbaum PR Rubin DB Reducing bias in observational studies using subclassification on the propensity score Journal of the American Statistical Association 1984 79 516 524
  2. [3]Austin PC Mamdani MM A comparison of propensity score methods: a case-study estimating the effectiveness of post-AMI statin use Statistics in Medicine 2006 25 2084 2106 16220490
  3. [5]Austin PC A critical appraisal of propensity score matching in the medical literature from 1996 to 2003 Statistics in Medicine 2008 27 2037 2049 18038446
  4. [6]Austin PC Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: a systematic review and suggestions for improvement Journal of Thoracic and Cardiovascular Surgery 2007 134 1128 1135 17976439
  5. [7]Austin PC A report card on propensity-score matching in the cardiology literature from 2004 to 2006: results of a systematic review Circulation: Cardiovascular Quality and Outcomes 2008 1 62 67 20031790
  6. [8]Imbens GW Nonparametric estimation of average treatment effects under exogeneity: a review The Review of Economics and Statistics 2004 86 4 29
  7. [9]Austin PC Type I error rates, coverage of confidence intervals, and variance estimation in propensity-score matched analyses The International Journal of Biostatistics 2009 5 1 Article 13. DOI: 10.2202/1557-4679.1146 PMC2949360 20949126 · DOI
  8. [10]Stürmer T Joshi M Glynn RJ Avorn J Rothman KJ Schneeweiss S A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods Journal of Clinical Epidemiology 2006 59 437 447 16632131 PMC1448214
  9. [11]Schechtman E Odds ratio, relative risk, absolute risk reduction, and the number needed to treat–which of these should we use? Value in Health 2002 5 431 436 12201860
  10. [13]Jaeschke R Guyatt G Shannon H Walter S Cook D Heddle N Basis statistics for clinicians 3: assessing the effects of treatment: measures of association Canadian Medical Association Journal 1995 152 351 357 7828099 PMC1337533
  11. [14]Sinclair JC Bracken MB Clinically useful measures of effect in binary analyses of randomized trials Journal of Clinical Epidemiology 1994 47 881 889 7730891
  12. [15]Austin PC Grootendorst P Normand SLT Anderson GM Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study Statistics in Medicine 2007 26 754 768 16783757
  13. [16]Austin PC The performance of different propensity score methods for estimating marginal odds ratios Statistics in Medicine. 2007 26 3078 3094 17187347
  14. [17]Greenland S Interpretation and choice of effect measures in epidemiologic analyses American Journal of Epidemiology 1987 125 761 768 3551588
  15. [19]Austin PC The performance of different propensity score methods for estimating difference in proportions (risk differences or absolute risk reductions) in observational studies Statistics in Medicine 2010 DOI: 10.1002/sim.3854 PMC3068290 20108233 · DOI
  16. [22]Cochran WG Rubin DB Controlling bias in observational studies: a review Sankhya: The Indian Journal of Statistics 1973 35 416 466
  17. [23]Agresti A Min Y Effects and non-effects of paired identical observations in comparing proportions with binary matched-pairs data Statistics in Medicine 2004 23 65 75 14695640
  18. [24]Lee DS Austin PC Rouleau JL Liu PP Naimark D Tu JV Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model Journal of the American Medical Association 2003 290 2581 2587 14625335
  19. [27]Austin PC Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulations Biometrical Journal 2009 51 171 184 DOI: 10.1002/bimj.200810488 19197955 · DOI · Paper page
  20. [29]Austin PC The relative ability of different propensity-score methods to balance measured covariates between treated and untreated subjects in observational studies Medical Decision Making 2009 29 661 677 19684288

Cited by

Sorted by citations

Authors
Austin, P. C.
Affiliations
University of Toronto (CA)
Corresponding author
Austin, Peter C.
ORCID
Austin, Peter C.
Venue
Pharmaceutical Statistics
Publisher
Wiley
Published
2011-03
Volume
10
Issue
2
Pages
150-161
ISSN
  • 1539-1604 (ISSN-L)
  • 1539-1612
DOI
10.1002/pst.433
PMC
PMC3120982
OpenAlex
W2111635289
Cited by
3,949
References
2
Citations by year
2026: 421 · 2025: 454 · 2024: 379 · 2023: 402 · 2022: 387 · 2021: 379 · 2020: 356 · 2019: 279 · 2018: 223 · 2017: 188 · 2016: 179 · 2015: 117 · 2014: 91 · 2013: 51 · 2012: 24
FWCI
25.10
Type
journal article
Language
en
Open access
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor, 28 d)
Locations
  • Pharmaceutical Statistics (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • Institute for Clinical Evaluative Sciences
  • Ontario Ministry of Health and Long-Term Care
  • Canadian Institutes of Health Research (MOP86508)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
propensity score matching, treatment effect estimation, risk difference, mean squared error, bias, confidence interval coverage, type I error, observational studies, matched pairs
MeSH terms
Aged, Aged, 80 and over, Female, Humans, Male, Monte Carlo Method, Risk, Confidence Intervals, Propensity Score

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review