The DOI matches its Crossref record; 75 of the 77 references it cites were found in a registry or an index.
Good Research Practices for Comparative Effectiveness Research: Analytic Methods to Improve Causal Inference from Nonrandomized Studies of Treatment Effects Using Secondary Data Sources: The ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—Part III
Open accessAbstract
Objectives Most contemporary epidemiologic studies require complex analytical methods to adjust for bias and confounding. New methods are constantly being developed, and older more established methods are yet appropriate. Careful application of statistical analysis techniques can improve causal inference of comparative treatment effects from nonrandomized studies using secondary databases. A Task Force was formed to offer a review of the more recent developments in statistical control of confounding. Methods The Task Force was commissioned and a chair was selected by the ISPOR Board of Directors in October 2007. This Report, the third in this issue of the journal, addressed methods to improve causal inference of treatment effects for nonrandomized studies. Results The Task Force Report recommends general analytic techniques and specific best practices where consensus is reached including: use of stratification analysis before multivariable modeling, multivariable regression including model performance and diagnostic testing, propensity scoring, instrumental variable, and structural modeling techniques including marginal structural models, where appropriate for secondary data. Sensitivity analyses and discussion of extent of residual confounding are discussed. Conclusions Valid findings of causal therapeutic benefits can be produced from nonrandomized studies using an array of state-of-the-art analytic techniques. Improving the quality and uniformity of these studies will impr…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Adjustment for time-varying confounding (i.e., confounders that simultaneously act as intermediate steps in the pathway between intervention and outcome) require special methods such as marginal structural analysis or g-estimation [38,54].”
From State-Transition Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3 (2012) · cites this paper as [54]
Citations
Cites
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Sample Selection Bias as a Specification Error
Heckman, J. J. (1979) · Econometrica · doi:10.2307/1912352 · Cited by 28,779
(2001) · RePEc: Research Papers in Economics · Cited by 27,577
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Hosmer, D. W., & Lemeshow, S. (2000) · doi:10.1002/0471722146 · Cited by 20,302
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270
Applied Regression Analysis and Other Multivariable Methods
Walker, E., Kleinbaum, D. G., Kupper, L. L., & Muller, K. E. (1989) · Technometrics · doi:10.2307/1270375 · Cited by 8,538
Instrumental Variables Regression with Weak Instruments
Staiger, D., & Stock, J. H. (1997) · Econometrica · doi:10.2307/2171753 · Cited by 7,081
Everitt, B. S. (2001) · Psychology Press eBooks · doi:10.4324/9781410605542-11 · Cited by 6,877
Rosenbaum, P. R., & Rubin, D. B. (1985) · The American Statistician · doi:10.2307/2683903 · Cited by 6,472
Marginal Structural Models and Causal Inference in Epidemiology
Robins, J. M., Hernán, M. Á., & Brumback, B. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00011 · Cited by 5,581
Instrumental Variables Regression with Weak Instruments
Staiger, D., & Stock, J. (1994) · doi:10.3386/t0151 · Cited by 5,302
(1976) · Annals of Economic and Social Measurement · Cited by 5,035
Bound, J., Jaeger, D. A., & Baker, R. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476536 · Cited by 3,898
Causal Diagrams for Epidemiologic Research
Greenland, S., Pearl, J., & Robins, J. M. (1999) · Epidemiology · doi:10.1097/00001648-199901000-00008 · Cited by 3,728
Estimating Causal Effects from Large Data Sets Using Propensity Scores
Rubin, D. V. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-8_part_2-199710151-00064 · Cited by 3,104
Air Pollution and Hospital Admissions for Heart Disease in Eight U.S. Counties
Schwartz, J. D. (1999) · Epidemiology · doi:10.1097/00001648-199901000-00005 · Cited by 3,045
Applied Survival Analysis: Regression Modeling of Time-to-Event Data
Hosmer, D. W., Lemeshow, S. A., & May, S. J. (2008) · Cited by 3,030
Epidemiologic Research: Principles and Quantitative Methods.
Mayer, L. S., Kleinbaum, D. G., Kupper, L. L., & Morgenstern, H. (1984) · Journal of the American Statistical Association · doi:10.2307/2288427 · Cited by 2,905
Causal diagrams for empirical research
PEARL, J. (1995) · Biometrika · doi:10.1093/biomet/82.4.669 · Cited by 2,355
Variable Selection for Propensity Score Models
Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj149 · Cited by 2,336
Multiple Imputation for Nonresponse in Surveys.
Lynn, P. J., & Rubin, D. B. (1988) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348774 · Cited by 2,330
Rao, S. J. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s263 · Cited by 1,967
Epidemiologic Research: Principles and Quantitative Methods.
Pocock, S., Kleinbaum, D. G., Kupper, L. L., & Morgenstern, H. (1983) · Biometrics · doi:10.2307/2531133 · Cited by 1,893
Maternal Pesticide Exposure from Multiple Sources and Selected Congenital Anomalies
Shaw, G. M., Wasserman, C. R., O’Malley, C. D., Nelson, V. E., & Jackson, R. J. (1999) · Epidemiology · doi:10.1097/00001648-199901000-00011 · Cited by 1,783
Hernán, M. Á., Brumback, B., & Robins, J. M. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00012 · Cited by 1,772
Applied Regression Analysis and Other Multivariable Methods.
Bhattacharyya, H. T., Kleinbaum, D. G., & Kupper, L. L. (1979) · Journal of the American Statistical Association · doi:10.2307/2287012 · Cited by 1,767
Modern Epidemiology 3rd edition
Rothman, K. J., Lash, T. L., & Greenland, S. (1986) · Cited by 1,736
On discrimination and the decomposition of wage differentials
Oaxaca, R. L., & Ransom, M. R. (1994) · Journal of Econometrics · doi:10.1016/0304-4076(94)90074-4 · Cited by 1,663
A critical appraisal of propensity‐score matching in the medical literature between 1996 and 2003
Austin, P. C. (2007) · Statistics in Medicine · doi:10.1002/sim.3150 · Cited by 1,249
Avoiding Invalid Instruments and Coping with Weak Instruments
Murray, M. P. (2006) · The Journal of Economic Perspectives · doi:10.1257/jep.20.4.111 · Cited by 1,094
Risk Adjustment for Measuring Healthcare Outcomes
Iezzoni, L. I. (1994) · Cited by 950
Applied regression analysis and other multivariable methods, 3rd ed.
Kleinbaum, D. G., Kupper, L. L., Muller, K. E., & Nizam, A. (1998) · Cited by 871
D'Agostino, R. B., Lee, M., Belanger, A. J., Cupples, L. A., Anderson, K., & Kannel, W. B. (1990) · Statistics in Medicine · doi:10.1002/sim.4780091214 · Cited by 798
Analysis of Observational Studies in the Presence of Treatment Selection Bias
Stukel, T. A., Fisher, E. S., Wennberg, D. E., Alter, D. A., Gottlieb, D. J., & Vermeulen, M. J. (2007) · JAMA · doi:10.1001/jama.297.3.278 · Cited by 789
How should cost data in pragmatic randomised trials be analysed?
Thompson, S. G. (2000) · BMJ · doi:10.1136/bmj.320.7243.1197 · Cited by 772
Schneeweiß, S. (2006) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1200 · Cited by 741
METHODS FOR ANALYZING HEALTH CARE UTILIZATION AND COSTS
Diehr, P. K., Yanez, D., Ash, A. S., Hornbrook, M. C., & Lin, D. Y. (1999) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.20.1.125 · Cited by 663
Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates
Rubin, D. B., & Thomas, N. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10474233 · Cited by 568
Statistical Models in Epidemiology, the Environment, and Clinical Trials
Halloran, M. E., & Berry, D. A. (2000) · The IMA volumes in mathematics and its applications · doi:10.1007/978-1-4612-1284-3 · Cited by 559
Marginal Structural Models versus Structural nested Models as Tools for Causal inference
Robins, J. M. (2000) · The IMA volumes in mathematics and its applications · doi:10.1007/978-1-4612-1284-3_2 · Cited by 557
Indications for propensity scores and review of their use in pharmacoepidemiology
Glynn, R. J., Schneeweiss, S., & Stürmer, T. (2006) · Basic & Clinical Pharmacology & Toxicology · doi:10.1111/j.1742-7843.2006.pto_293.x · Cited by 541
Risk adjustment for measuring health care outcomes
Allison, J. J. (1996) · Annals of Epidemiology · doi:10.1016/s1047-2797(96)90011-3 · Cited by 537
Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates
Thomas, N. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725.025 · Cited by 508
lezzoni, L. I. (1997) · JAMA · doi:10.1001/jama.1997.03550190064046 · Cited by 500
Shah, B. R., Laupacis, A., Hux, J. E., & Austin, P. C. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.016 · Cited by 493
Analysis of Longitudinal Data.
Diggle, P. J., Liang, K.-Y., & Zeger, S. L. (1997) · Biometrics · doi:10.2307/2533983 · Cited by 448
Motheral, B. R., Brooks, J. M., Clark, M. A., Crown, W. H., Davey, P. G., Hutchins, D., Martin, B. C., & Stang, P. (2003) · Value in Health · doi:10.1046/j.1524-4733.2003.00242.x · Cited by 411
Self-citationBerger, M. L., Mamdani, M., Atkins, D. C., & Johnson, M. L. (2009) · Value in Health · doi:10.1111/j.1524-4733.2009.00600.x · Cited by 408
Self-citationA New Specification Test for the Validity of Instrumental Variables
Hahn, J., & Hausman, J. A. (2002) · Econometrica · doi:10.1111/1468-0262.00272 · Cited by 391
Too Much Ado about Propensity Score Models? Comparing Methods of Propensity Score Matching
Başer, O. (2006) · Value in Health · doi:10.1111/j.1524-4733.2006.00130.x · Cited by 371
Assessment and Control for Confounding by Indication in Observational Studies
Psaty, B. M., Koepsell, T. D., Lin, D., Weiss, N. S., Siscovick, D. S., Rosendaal, F. R., Pahor, M., & Furberg, C. D. (1999) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.1999.tb01603.x · Cited by 363
Data, Design, and Background Knowledge in Etiologic Inference
Robins, J. M. (2001) · Epidemiology · doi:10.1097/00001648-200105000-00011 · Cited by 358
Cox, E., Martin, B. C., Van Staa, T., Garbe, E., Siebert, U., & Johnson, M. L. (2009) · Value in Health · doi:10.1111/j.1524-4733.2009.00601.x · Cited by 307
Self-citationSemi-Automated Sensitivity Analysis to Assess Systematic Errors in Observational Data
Lash, T. L., & Fink, A. K. (2003) · Epidemiology · doi:10.1097/01.ede.0000071419.41011.cf · Cited by 250
Stürmer, T., Schneeweiß, S., Avorn, J., & Glynn, R. J. (2005) · American Journal of Epidemiology · doi:10.1093/aje/kwi192 · Cited by 230
Use of Health Services by Previously Uninsured Medicare Beneficiaries
McWilliams, J. M., Meara, E. R., Zaslavsky, A. M., & Ayanian, J. Z. (2007) · New England Journal of Medicine · doi:10.1056/nejmsa067712 · Cited by 223
Schmid, C. H., Lau, J., McIntosh, M. W., & Cappelleri, J. C. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980915)17:17 · Cited by 211
Self-citationDevelopments in Post-marketing Comparative Effectiveness Research
Schneeweiß, S. (2007) · Clinical Pharmacology & Therapeutics · doi:10.1038/sj.clpt.6100249 · Cited by 200
Basu, A., Heckman, J. J., Navarro, S., & Urzúa, S. (2007) · Health Economics · doi:10.1002/hec.1291 · Cited by 195
Retrospectives: Who Invented Instrumental Variable Regression?
Stock, J. H., & Trebbi, F. (2003) · The Journal of Economic Perspectives · doi:10.1257/089533003769204416 · Cited by 171
Bayesian and classical approaches to instrumental variable regression
Kleibergen, F. R., & Zivot, E. W. (2003) · Journal of Econometrics · doi:10.1016/s0304-4076(02)00219-1 · Cited by 153
An Application of Model-Fitting Procedures for Marginal Structural Models
Mortimer, K. M., Neugebauer, R. S., van der Laan, M. J., & Tager, I. B. (2005) · American Journal of Epidemiology · doi:10.1093/aje/kwi208 · Cited by 108
Cook, N. R., Cole, S. R., & Hennekens, C. H. (2002) · American Journal of Epidemiology · doi:10.1093/aje/155.11.1045 · Cited by 99
Use of Propensity Score Technique to Account for Exposure-Related Covariates
Seeger, J. D., Kurth, T., & Walker, A. M. (2007) · Medical Care · doi:10.1097/mlr.0b013e318074ce79 · Cited by 89
Hadley, J., Polsky, D. E., Mandelblatt, J. S., Mitchell, J. M., Weeks, J. C., Wang, Q., & Hwang, Y. (2003) · Health Economics · doi:10.1002/hec.710 · Cited by 56
Byrne, S., Fisher, S., Fortune, P., Lawn, C., & Wieteska, S. (2008) · Wiley eBooks · doi:10.1002/9781444300833.ch3 · Cited by 46
Brooks, J. M., & Chrischilles, E. A. (2007) · Medical Care · doi:10.1097/mlr.0b013e318070c069 · Cited by 43
Antidepressant Selection and Use and Healthcare Expenditures
Crown, W. H., Hylan, T. R., & Meneades, L. M. (1998) · PharmacoEconomics · doi:10.2165/00019053-199813040-00006 · Cited by 33
Self-citationPropensity score analysis in observational studies: outcomes after abdominal aortic aneurysm repair
Johnson, M. L., Bush, R. L., Collins, T. C., Lin, P. H., Liles, D. R., Henderson, W. G., Khuri, S. F., & Petersen, L. A. (2006) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2006.03.009 · Cited by 32
Self-citationMajor Teaching Hospitals Defying Darwin
Iezzoni, L. I. (1997) · JAMA · doi:10.1001/jama.1997.03550060096044 · Cited by 13
Fu, A. Z., Liu, G. G., Christensen, D., & Hansen, R. (2007) · Value in Health · doi:10.1111/j.1524-4733.2006.00159.x · Cited by 11
Kuykendall, D. H. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00140-l · Cited by 11
D’Agostino (1998) · doi:10.1002/(sici)1097-0258(19981015)17:19
Research works printed but not linked: 2
- Berger, 2009, Good research practices for comparative effectiveness research: defining, reporting and interpreting non-randomized studies of treatment effects using secondary data sources. ISPOR TF Report 2009—Part I, Value Health
- Montaquila JM., Ponikowski CH. An evaluation of alternative imputation methods. Proceedings of the Section on Survey Research Methods, American Statistical Association, 1995.
Cited by
Sorted by citations
State-Transition Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3
Siebert, U., Alagoz, O., Bayoumi, A. M., Jahn, B., Owens, D. K., Cohen, D. J., & Kuntz, K. M. (2012) · Value in Health · doi:10.1016/j.jval.2012.06.014 · Cited by 495
Self-citationBerger, M. L., Sox, H. C., Willke, R. J., Brixner, D. L., Eichler, H., Goettsch, W., Madigan, D., Makady, A., Schneeweiß, S., Tarricone, R., et al. (2017) · Value in Health · doi:10.1016/j.jval.2017.08.3019 · Cited by 464
Berger, M. L., Mamdani, M., Atkins, D. C., & Johnson, M. L. (2009) · Value in Health · doi:10.1111/j.1524-4733.2009.00600.x · Cited by 408
Self-citationKishimoto, T., Hagi, K., Kurokawa, S., Kane, J. M., & Correll, C. U. (2021) · The Lancet Psychiatry · doi:10.1016/s2215-0366(21)00039-0 · Cited by 369
State-Transition Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force–3
Siebert, U., Alagoz, O., Bayoumi, A. M., Jahn, B., Owens, D. K., Cohen, D. J., & Kuntz, K. M. (2012) · Medical Decision Making · doi:10.1177/0272989x12455463 · Cited by 348
Self-citation
Metadata
- Authors
- Johnson, M. L., Crown, W., Martin, B. C., Dormuth, C. R., & Siebert, U.
- Affiliations
- University of Houston (US)
- University of Arkansas for Medical Sciences (US)
- University of British Columbia (CA)
- Harvard University (US)
- Corresponding author
- Johnson, Michael L.
- Venue
- Value in Health
- Publisher
- Elsevier BV
- Published
- 2009-11
- Volume
- 12
- Issue
- 8
- Pages
- 1062-1073
- ISSN
- 1098-3015 (ISSN-L)
- OpenAlex
- W2153012086
- Cited by
- 285
- References
- 77
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 10 · 2023: 10 · 2022: 18 · 2021: 14 · 2020: 25 · 2019: 19 · 2018: 11 · 2017: 30 · 2016: 30 · 2015: 17 · 2014: 19 · 2013: 24 · 2012: 16
- FWCI
- 13.87
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Value in Health (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- comparative effectiveness research, causal inference, non-randomized studies, secondary data sources, confounding adjustment, propensity scoring, instrumental variables, marginal structural models, structural equation modeling, stratified analysis, multivariable regression, residual confounding, sensitivity analysis, good scientific practice
- MeSH terms
- Data Interpretation, Statistical, Decision Making, Epidemiologic Methods, Health Policy, Humans, Regression Analysis, Retrospective Studies, ROC Curve, Selection Bias, Causality, Confounding Factors, Epidemiologic, Multivariate Analysis, Databases, Factual, Outcome Assessment, Health Care, Economics, Pharmaceutical, Internationality, Comparative Effectiveness Research, Drug-Related Side Effects and Adverse Reactions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review