Per-Protocol Analyses of Pragmatic Trials
The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Pragmatic trials are designed to address real-world questions about care options. This article addresses issues that may arise from per-protocol and intention-to-treat analyses of such trials, outlines alternative analytic approaches, and provides guidance on how to choose among them.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We developed a revised risk-of-bias assessment tool to address these issues, incorporate advances in assessment of risk of bias used in other recently developed tools, 9 10 and integrate recent developments in estimation of intervention effects from randomised trials. 11”
From RoB 2: a revised tool for assessing risk of bias in randomised trials (2019) · cites this paper as [11]
In the citing paper
“For trials comparing interventions that are sustained over time, appropriate methods also require measurement of and adjustment for the values of prognostic factors, both before and after randomisation, that predict deviations from intervention. 11”
From RoB 2: a revised tool for assessing risk of bias in randomised trials (2019) · cites this paper as [11]
In the citing paper
“Each effect might be of interest. 11”
From RoB 2: a revised tool for assessing risk of bias in randomised trials (2019) · cites this paper as [11]
In the citing paper
“RoB 2 also incorporates recent developments in estimation of intervention effects from randomised trials: we distinguish bias in the effect of assignment to interventions from bias in the effect of adhering to intervention as specified in the trial protocol.”
From RoB 2: a revised tool for assessing risk of bias in randomised trials (2019) · cites this paper as [11]
Citations
Cites
Marginal Structural Models and Causal Inference in Epidemiology
Robins, J. M., Hernán, M. Á., & Brumback, B. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00011 · Cited by 5,581
Self-citationRobins, J. (1986) · Mathematical Modelling · doi:10.1016/0270-0255(86)90088-6 · Cited by 2,451
Self-citationRosenbaum, P. R. (2002) · Springer Series in Statistics · doi:10.1007/978-1-4757-3692-2 · Cited by 2,394
Doubly robust estimation in missing data and causal inference models
Bang, H., & Robins, J. M. (2005) · Biometrics · doi:10.1111/j.1541-0420.2005.00377.x · Cited by 1,998
Self-citationIntention-to-treat concept: A review
Gupta, S. (2011) · Perspectives in Clinical Research · doi:10.4103/2229-3485.83221 · Cited by 1,921
The prevention and treatment of missing data in clinical trials
Little, R. J., D'Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., Frangakis, C., Hogan, J. W., Molenberghs, G., Murphy, S. A., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmsr1203730 · Cited by 1,884
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
An introduction to instrumental variables for epidemiologists
Greenland, S. (2000) · International Journal of Epidemiology · doi:10.1093/ije/29.4.722 · Cited by 1,156
Instruments for Causal Inference: An Epidemiologist's Dream?
Hernán, M. A., & Robins, J. M. (2006) · Epidemiology · doi:10.1097/01.ede.0000222409.00878.37 · Cited by 961
Self-citationRobins, J. M., & Finkelstein, D. M. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00779.x · Cited by 877
Self-citation
Cited by
Items 101–200 of 661
Translating questions to estimands in randomized clinical trials with intercurrent events
Stensrud, M. J., & Dukes, O. (2022) · Statistics in Medicine · doi:10.1002/sim.9398 · Cited by 44
“For example, formal (causal) definitions of per protocol estimands [16], which arguably are useful in a range of practical settings such as pragmatic trials [15, 17], often require specification of sequential treatment regimes: the per protocol estimands evaluate the effect of taking treatment, as described in the pro…”
BackgroundColl‐Planas, L., Carbó, A., Jansson, A. H., Dostálová, V., Bártová, A., Rautiainen, L. J., Kolster, A., Masó-Aguado, M., Briones‐Buixassa, L., Alias, S. B., et al. (2024) · BMC Public Health · doi:10.1186/s12889-023-17547-x · Cited by 36
“In addition to the ITT analysis, a causal inference-based per-protocol analysis will be performed to assess the effect of compliance on the outcomes of loneliness and quality of life using a structural nested model with g-estimation [40, 41].”
MethodXia, C., He, L., Xu, Y., Guo, G., Yu, X., Wang, W., Dai, S., Dai, C., Zhu, Y., Jiang, K., et al. (2024) · Gastroenterology · doi:10.1053/j.gastro.2024.11.025 · Cited by 41
Blood Pressure Targets for Adults with Vasodilatory Shock — An Individual Patient Data Meta-Analysis
Angriman, F., Momenzade, N., Adhikari, N. K. J., Mouncey, P., Asfar, P., Yarnell, C. J., Ong, S. W. X., Pinto, R. L., Doidge, J., Shankar‐Hari, M., et al. (2024) · NEJM Evidence · doi:10.1056/evidoa2400359 · Cited by 37
Jones, M. D., Hansford, H. J., Bastianon, A., Gibbs, M. T., Gilanyi, Y. L., Foster, N. E., Dean, S. G., Ogilvie, R., Hayden, J. A., & Wood, L. (2025) · Journal of physiotherapy · doi:10.1016/j.jphys.2025.03.004 · Cited by 43
Principles and procedures for data and safety monitoring in pragmatic clinical trials
Simon, G. E., Shortreed, S. M., Rossom, R. C., Penfold, R. B., Sperl‐Hillen, J. M., & O’Connor, P. J. (2019) · Trials · doi:10.1186/s13063-019-3869-3 · Cited by 35
Seal, K. H., Becker, W., Murphy, J. L., Purcell, N., Denneson, L. M., Morasco, B., Martin, A. M., Reddy, K., Iseghem, T. V., Krebs, E., et al. (2020) · Pain medicine (Malden, Mass.) · doi:10.1093/pm/pnaa366 · Cited by 31
Litt, J., Coll‐Planas, L., Sachs, A. L., Rochau, U., Jansson, A. H., Dostálová, V., Daher, C., Beacom, A. M., Bachinski, K., García, G., et al. (2024) · Cities & Health · doi:10.1080/23748834.2023.2300207 · Cited by 42
“In addition to the intention-to-treat analysis, a causal inference-based per-protocol analysis will be performed as a sensitivity analysis using a structural nested model with g-estimation (Hernán and Robins 2017, Latimer et al .”
MethodBoyne, D. J., Cheung, W. Y., Hilsden, R. J., Sajobi, T. T., Batra, A., Friedenreich, C. M., & Brenner, D. R. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.3587 · Cited by 37
Martínez‐González, M. Á., Montero, P., Ruiz‐Canela, M., Toledo, E., Estruch, R., Gómez‐Gracia, E., Li, J., Ros, E., Arós, F., Hernáez, Á., et al. (2023) · Cardiovascular Diabetology · doi:10.1186/s12933-023-01994-2 · Cited by 39
“Current recommendations for pragmatic trials encourage the assessment of actual compliance [22].”
BackgroundKelshiker, M. A., Bächtiger, P., Petri, C. F., Nakhare, S., Mansell, J., Chhatwal, K., Alrumayh, A., Zaman, J., Shah, M., Young, H., et al. (2026) · The Lancet · doi:10.1016/s0140-6736(25)02156-7 · Cited by 37
Statistical Appraisal of 6 Recent Clinical Trials in Cardiology
Pocock, S., & Collier, T. J. (2019) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2019.03.484 · Cited by 32
Dochow-Sondershaus, S., & Neumeyer, S. (2020) · Research in Social Stratification and Mobility · doi:10.1016/j.rssm.2020.100579 · Cited by 38
Materne, E., Choi, H. K., Zhou, B., Costenbader, K. H., Zhang, Y., & Jorge, A. M. (2023) · Arthritis & Rheumatology · doi:10.1002/art.42620 · Cited by 31
“In the per protocol analysis, we adjusted for adherence to assigned treatment group with censoring and inverse probability of treatment weighting, which represents the inverse of the probability of having received the treatment that was received for each subject at a given time (15,22,23).”
MethodNaik, H., Cooke, E. M., Boulter, T., Dyer, R., Bone, J. N., Tsai, M., Cristobal, J., Mckay, R. J., Song, X., & Nacul, L. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-085272 · Cited by 32
“108 All analyses of primary and secondary outcomes will be pre-specified in detail in a Statistical Analysis Plan and signed off on by all investigators prior to data analysis.”
BackgroundChallenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data
Hulme, W. J., Williamson, E., Horne, E. M. F., Green, A. C. A., McDonald, H. I., Walker, A. J., Curtis, H. J., Morton, C. E., MacKenna, B., Croker, R., et al. (2023) · Annals of Internal Medicine · doi:10.7326/m21-4269 · Cited by 45
“This can be done by censoring persons in the no-vaccination group at the time they are vaccinated (7).”
BackgroundSelf-citationThe attributable estimand: A new approach to account for intercurrent events
Darken, P. F., Nyberg, J., Ballal, S., & Wright, D. N. M. (2020) · Pharmaceutical Statistics · doi:10.1002/pst.2019 · Cited by 30
“However, as has been noted by others, this is only the case if the clinical experience (and treatment switching) of patients in the trial is representative of what would happen in clinical practice.(6) Also in regulatory decision-making, the question of interest is whether the new treatment should be licensed.”
BackgroundMottl‐Santiago, J., Dukhovny, D., Cabral, H., Rodrigues, D., Spencer, L., Valle, E. A., & Feinberg, E. (2023) · Health Equity · doi:10.1089/heq.2022.0200 · Cited by 31
“34 Our per-protocol analysis attempts to understand how participants would benefit from the intervention with full adherence to the randomized treatment.”
BackgroundResultInfluentialMontelukast Use and the Risk of Neuropsychiatric Adverse Events in Children
Wintzell, V., Brenner, P., Halldner, L., Rhedin, S., Gong, T., & Almqvist, C. (2025) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2024.5429 · Cited by 31
Cancer outcomes research—a European challenge: measures of the cancer burden
Kalager, M., Adami, H., Lagergren, P., Steindorf, K., & Dickman, P. W. (2021) · Molecular Oncology · doi:10.1002/1878-0261.13012 · Cited by 31
Barnhart, K. T., Hansen, K. R., Stephenson, M. D., Usadi, R., Steiner, A. Z., Cedars, M. I., Jungheim, E. S., Hoeger, K. M., Krawetz, S. A., Mills, B. B., et al. (2021) · JAMA · doi:10.1001/jama.2021.10767 · Cited by 28
Lauffenburger, J. C., Lewey, J., Jan, S., Makanji, S. S., Ferro, C. A., Krumme, A. A., Lee, J. K., Ghazinouri, R., Haff, N., & Choudhry, N. K. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0657 · Cited by 32
Waterhouse, M. A., English, D. R., Armstrong, B. K., Baxter, C., Romero, B. D., Ebeling, P. R., Härtel, G., Kimlin, M. G., McLeod, D. S. A., O'Connell, R. L., et al. (2019) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2019.100333 · Cited by 30
“We are therefore able to perform a perprotocol analysis that will appropriately adjust for post-randomization confounding and selection bias [30,31].”
MethodJúlvez, J., Gignac, F., Fernández‐Barrés, S., Romaguera, D., Sala‐Vila, A., Ranzani, O., Persavento, C., Delgado, A., Carol, A., Torrent, J., et al. (2021) · Frontiers in Pediatrics · doi:10.3389/fped.2021.593847 · Cited by 35
“, single days of reporting eating walnuts (≥ 100 days during 6 months) (34).”
BackgroundKutz, A., Koch, D., Haubitz, S., Conca, A., Baechli, C., Regez, K., Gregoriano, C., Ebrahimi, F., Bassetti, S., Eckstein, J., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.33667 · Cited by 30
The need for clinical judgement in the application of evidence-based medicine
Spence, J. D. (2019) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2019-111300 · Cited by 28
“Hernan and Robins discussed this issue, 46 saying that ITT ‘may not be directly relevant for guiding decisions in clinical settings with different adherence patterns’.”
Estimating causal effects of internet interventions in the context of nonadherence
Hesser, H. (2020) · Internet Interventions · doi:10.1016/j.invent.2020.100346 · Cited by 29
“On the other hand, the validity of such methods requires data on all preassignment and postassignment prognostic factors predictive of adherence which may not be available in a trial and these “modern” perprotocol analyses have not systematically been compared with classic instrument variables methods (Hernán and Robi…”
BackgroundMethodInfluentialSchlenk, R. F., & Käyser, S. (2018) · Recent results in cancer research/Recent Results in Cancer Research · doi:10.1007/978-3-319-91439-8_10 · Cited by 28
Briggs, A. H., Daniele, B., Dick, K. J., Evans, T. R. J., Galle, P. R., Hubner, R. A., López, C., Siebert, U., & Tremblay, G. (2020) · British Journal of Cancer · doi:10.1038/s41416-020-0817-7 · Cited by 28
Guidelines for estimating causal effects in pragmatic randomized trialsPreprint
Murray, E. J., Swanson, S. A., & Hernán, M. Á. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.06030 · Cited by 29
“A common misconception is that trials compare treatments; rather, they compare treatment strategies (13,14).”
BackgroundMethodInfluentialSelf-citationChristensen, R. H., Wedell‐Neergaard, A., Lehrskov, L. L., Legaard, G. E., Dorph, E., Nymand, S. B., Ball, M., Zacho, M., Christensen, R., Ellingsgaard, H., et al. (2018) · Trials · doi:10.1186/s13063-018-2637-0 · Cited by 27
“enced by lack of adherence, unlike the intention-to-treat (ITT) principle (with use of “conservative imputation”) [29].”
Müller, G., Pfinder, M., Schmahl, C. G., Bohus, M., & Lyssenko, L. (2019) · BMC Public Health · doi:10.1186/s12889-019-7585-4 · Cited by 31
Brown, L., Sharma, M., Leggett, S. S., Sung, J. H., Bennett, R., & Azevedo, M. (2020) · Health Promotion Perspectives · doi:10.34172/hpp.2020.41 · Cited by 25
Karagozoglu, K. H., Vissink, A., Forouzanfar, T., de Visscher, J. G. A. M., Maarse, F., Brand, H. S., van de Ven, P. M., & Jager, D. H. J. (2020) · Rheumatology · doi:10.1093/rheumatology/keaa284 · Cited by 25
Ponsonby, A. (2021) · International Journal of Epidemiology · doi:10.1093/ije/dyaa174 · Cited by 29
“If a causal question requires using a causal estimand analogous to a per-protocol effect in a randomized controlled trial, then post-randomization confounding (arising after exposure allocation) needs to be accounted for.(44,46) In life course investigations, prenatal and postnatal disease determinants arise.”
MethodGürdeniz, G., Uusitupa, M. I. J., Hermansen, K., Savolainen, M. J., Schwab, U., Kolehmainen, M., Brader, L., Cloetens, L., Herzig, K., Hukkanen, J., et al. (2021) · Clinical Nutrition · doi:10.1016/j.clnu.2021.12.031 · Cited by 29
Caniglia, E. C., Robins, J. M., Cain, L. E., Sabin, C. A., Logan, R. W., Abgrall, S., Mugavero, M. J., Hernández–Díaz, S., Meyer, L., Seng, R., et al. (2019) · Statistics in Medicine · doi:10.1002/sim.8120 · Cited by 33
Self-citationBykov, K., He, M., Franklin, J. M., Garry, E. M., Seeger, J. D., & Patorno, E. (2019) · Diabetes Obesity and Metabolism · doi:10.1111/dom.13766 · Cited by 26
Interval-cohort designs and bias in the estimation of per-protocol effects: a simulation study
Young, J. G., Vatsa, R., Murray, E. J., & Hernán, M. Á. (2019) · Trials · doi:10.1186/s13063-019-3577-z · Cited by 25
Self-citationHayes, K. N., Zhang, T., Kim, D. H., Daiello, L. A., Lee, Y., Kiel, D. P., Berry, S. D., & Zullo, A. R. (2023) · Journal of the American Heart Association · doi:10.1161/jaha.122.029865 · Cited by 26
A Comparison of Peripherally Inserted Central Catheter Materials
Ullman, A. J., August, D. L., Kleidon, T. M., Walker, R., Marsh, N. M., Bulmer, A. C., Pearch, B. J., Runnegar, N. J., Leema, J., Lee‐Archer, P. F., et al. (2025) · New England Journal of Medicine · doi:10.1056/nejmoa2406815 · Cited by 24
Management of high-risk acute pulmonary embolism: an emulated target trial analysis
Stadlbauer, A., Verbelen, T., Binzenhöfer, L., Goslar, T., Supady, A., Spieth, P. M., Noč, M., Verstraete, A., Hoffmann, S., Schomaker, M., et al. (2025) · Intensive Care Medicine · doi:10.1007/s00134-025-07805-4 · Cited by 27
“The causal contrasts of interests were both the mortality risk differences and risk ratios between the four treat-ment arms, quantified through both pre-registered ITT and (non-naïve, adjusted) PP analyses [22].”
Revisiting the X:BOT Naltrexone Clinical Trial Using a Comprehensive Survival Analysis
Ajazi, E., Dasgupta, N., Marshall, S. W., Monaco, J. H., Howard, A. G., Preisser, J. S., & Schwartz, T. A. (2021) · Journal of Addiction Medicine · doi:10.1097/adm.0000000000000931 · Cited by 26
“Thus, time-independent comparative effectiveness assessment for those 2 outcomes reported in the original paper(4) could potentially have been affected by the differences in induction and drop-out between the 2 arms.(8) Furthermore, the ITT and PP time-to-event analyses assumed a constant treatment effect over time fo…”
BackgroundZhao, S. S., Lyu, H., Solomon, D. H., & Yoshida, K. (2020) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2020-217200 · Cited by 29
Sochacki, K. R., Mather, R. C. “., Nwachukwu, B. U., Dong, D., Nho, S. J., Cote, M. P., & Harris, J. D. (2020) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2020.05.001 · Cited by 28
Geersing, G., Hendriksen, J. M. T., Zuithoff, N. P. A., Roes, K. C. B., Oudega, R., Takada, T., Schutgens, R. E. G., & Moons, K. G. M. (2020) · PLoS Medicine · doi:10.1371/journal.pmed.1003142 · Cited by 24
“Albeit clinically an intuitive approach, other more advanced statistical approaches exist, such as inverse probability weighting [16].”
BackgroundChoi, H. K., Zhang, Y., & Dalbeth, N. (2022) · Nature Reviews Rheumatology · doi:10.1038/s41584-022-00804-5 · Cited by 24
Optimizing Outcomes in Heart Failure: 2022 and Beyond
Jankowska, E. A., Andersson, T., Kaiser‐Albers, C., Bozkurt, B., Chioncel, O. D., Coats, A. J. S., Hill, L., Koehler, F., Lund, L. H., McDonagh, T. A., et al. (2023) · ESC Heart Failure · doi:10.1002/ehf2.14363 · Cited by 29
“Moreover, causal per-protocol analysis adjusting for non-adherence in pragmatic trials 37 can provide important evidence on the full potential of treatments and should therefore be used to inform patients, providers, and payers.”
BackgroundO’Brien, K. S., Feng, R., Sieber, F. E., Marcantonio, E. R., Tierney, A. M., Magaziner, J. S., Carson, J. L., Dillane, D., Sessler, D. I., Menio, D. A., et al. (2023) · Alzheimer s & Dementia · doi:10.1002/alz.13132 · Cited by 25
Reporting non-adherence in cluster randomised trials: A systematic review
Agbla, S. C., & Diaz‐Ordaz, K. (2018) · Clinical Trials · doi:10.1177/1740774518761666 · Cited by 31
“If a set of baseline or post-randomisation variables available is believed to be sufficient to adjust for the confounding, a ‘modified’ per protocol analysis may be valid.(10,11) 10.”
BackgroundMargolis, K. L., Crain, A. L., Bergdall, A. R., Beran, M., ANDERSON, J. P., Solberg, L. I., O’Connor, P. J., Sperl‐Hillen, J. M., Pawloski, P. A., Ziegenfuss, J. Y., et al. (2020) · Contemporary Clinical Trials · doi:10.1016/j.cct.2020.105939 · Cited by 23
Neugebauer, R. S., Schroeder, E. B., Reynolds, K., Schmittdiel, J. A., Loes, L., Dyer, W., DESAI, J. R., Vazquez‐Benitez, G., Ho, P. M., Anderson, J. P., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.18554 · Cited by 22
Madenci, A. L., Wanis, K. N., Cooper, Z. R., Haneuse, S., Subramanian, S. V., Albert Hofman, D., & Hernán, M. Á. (2021) · American Journal of Epidemiology · doi:10.1093/aje/kwab170 · Cited by 22
Self-citationChen, T., Chen, T., Zhang, Y., Wu, K., & Zou, Y. (2024) · Stroke and Vascular Neurology · doi:10.1136/svn-2023-002785 · Cited by 24
van Summeren, J., Holtman, G. A., Kollen, B. J., van Leeuwen, Y., van Ulsen-Rust, A. H. C., Tabbers, M. M., Dekker, J. H., & Berger, M. Y. (2019) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2019.09.048 · Cited by 27
Predicting counterfactual risks under hypothetical treatment strategies: an application to HIV
Dickerman, B. A., Dahabreh, I. J., Cantos, K., Logan, R. W., Lodi, S., Rentsch, C. T., Justice, A. C., & Hernán, M. Á. (2022) · European Journal of Epidemiology · doi:10.1007/s10654-022-00855-8 · Cited by 25
Self-citationKuehne, F., Arvandi, M., Hess, L. M., Faries, D. E., Gothe, R. M., Gothe, H., Beyrer, J. K., Zeimet, A. G., Stojkov, I., Mühlberger, N., et al. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.10.005 · Cited by 28
“These are (1) CA-125 value (absolute value), (2) CA-125 value (change from nadir over time), (3) time since progression, and (4) artificial censoring or death.”
BackgroundMethodEffects of statins and aspirin on HCC risk in alcohol-related cirrhosis: nationwide emulated trials
Kraglund, F., Christensen, D. H., Eiset, A. H., Villadsen, G. E., West, J., & Jepsen, P. (2023) · Hepatology Communications · doi:10.1097/hc9.0000000000000013 · Cited by 24
“Furthermore, a per-protocol analysis of an RCT is effectively an observational study confounded by differential adherence to protocol.[10] Emulating a target trial is a causal inference school of thought, wherein the aim is to imitate an RCT as closely as possible using observational data.”
BackgroundKayentao, K., Ghosh, R., Guindo, L., Whidden, C., Treleaven, E., Chiu, C., Lassala, D., Traoré, M. B., Beckerman, J., Diakité, D., et al. (2023) · BMJ Global Health · doi:10.1136/bmjgh-2022-011071 · Cited by 25
“28 Protocol adherence in the intervention arm was defined as any woman in the household reported having received two or more home visits by the CHW in the month prior to the survey.”
Matthews, A., Dahebreh, I. J., MacDonald, C., Lindahl, B., Hofmann, R., Erlinge, D., Yndigegn, T., Berglund, A., Jernberg, T., & Hernán, M. Á. (2024) · European Journal of Epidemiology · doi:10.1007/s10654-024-01119-3 · Cited by 20
“A more comparable contrast is the per protocol effect [25], which we plan to estimate once trial data become available (as part of a fully harmonized analysis, including the handling of competing events).”
BackgroundSelf-citationMethods in causal inference. Part 1: causal diagrams and confounding
Bulbulia, J. A. (2024) · Evolutionary Human Sciences · doi:10.1017/ehs.2024.35 · Cited by 29
Trick, W. E., Sokalski, S. J., Johnson, S., Bunnell, K. L., Levato, J., Ray, M. J., & Weinstein, R. A. (2018) · Infection Control and Hospital Epidemiology · doi:10.1017/ice.2018.76 · Cited by 20
“We defined per-protocol probiotic administration in the following 2 ways: (1) complete adherence to protocol (ie, on-time administration and no missed days) and (2) ontime administration of the first dose and receipt of ≥80% of inpatient doses.(21) We did not monitor postdischarge probiotic receipt.”
MethodRegulatory-grade clinical trial design using real-world data
Levenson, M. S. (2020) · Clinical Trials · doi:10.1177/1740774520905576 · Cited by 23
Randomization versus Real-World Evidence. Reply.
Collins, R., Bowman, L. J., & Landray, M. (2020) · PubMed · doi:10.1056/nejmc2020020 · Cited by 12
Duffecy, J., Grekin, R., Long, J. D., Mills, J. A., & O'Hara, M. (2022) · Journal of Affective Disorders · doi:10.1016/j.jad.2022.05.123 · Cited by 19
Grace periods in comparative effectiveness studies of sustained treatments
Wanis, K. N., Sarvet, A. L., Wen, L., Block, J. P., Rifas‐Shiman, S. L., Robins, J. M., & Young, J. G. (2024) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1093/jrsssa/qnae002 · Cited by 32
Self-citationMoving beyond the prevalent exposure design for causal inference in dementia research
Cribb, L., Moreno‐Betancur, M., Wu, Z., Wolfe, R. S., Pase, M. P., & Ryan, J. L. (2025) · The Lancet Healthy Longevity · doi:10.1016/j.lanhl.2024.100675 · Cited by 21
Optimising trial designs to identify appropriate antibiotic treatment durations
Pouwels, K. B., Mo, Y., Butler, C., Cooper, B. S., Wordsworth, S., Walker, A. S., & Robotham, J. V. (2019) · BMC Medicine · doi:10.1186/s12916-019-1348-z · Cited by 21
Innes, A. Q., Thomson, G., Cotter, M., King, J. A., Vollaard, N. B. J., & Kelly, B. M. (2019) · BMC Public Health · doi:10.1186/s12889-019-8061-x · Cited by 23
Webster‐Clark, M., Lund, J. L., Stürmer, T., Poole, C., Simpson, R. J., & Edwards, J. K. (2020) · Epidemiology · doi:10.1097/ede.0000000000001230 · Cited by 23
One Drop App With an Activity Tracker for Adults With Type 1 Diabetes: Randomized Controlled Trial
Osborn, C. Y., Hirsch, A., Sears, L. E., Heyman, M., Raymond, J. K., Huddleston, B., & Dachis, J. (2020) · JMIR mhealth and uhealth · doi:10.2196/16745 · Cited by 27
“In pragmatic trials, the appropriate reporting of both results can aid with scientific and clinical interpretation [32].”
Inferences About Drug Safety in Phase III Trials in Oncology: Examples From Advanced Prostate Cancer
Drago, J. Z., Gönen, M., Thanarajasingam, G., Sacks, C. A., Morris, M. J., Kantoff, P. W., & Stopsack, K. H. (2020) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djaa134 · Cited by 19
“Patient characteristics Age, median (range), y 74 (52-97) 73 (53-92) 75 (50-92) 70 (42-92) 68 (43-90) 71 (42-93) 69 (41-89) Excellent performance status, ECOG-PS 0, % 77.”
BackgroundMethods for pragmatic randomized clinical trials of pain therapies: IMMPACT statement
Hohenschurz‐Schmidt, D., Cherkin, D. C., Rice, A. S. C., Dworkin, R. H., Turk, D. C., McDermott, M. P., Bair, M. J., DeBar, L., Edwards, R. R., Evans, S. R., et al. (2024) · Pain · doi:10.1097/j.pain.0000000000003249 · Cited by 25
The effects of the Tripartite Intervention on students’ attitudes towards disability
Freer, J. R. R. (2021) · Journal of Research in Special Educational Needs · doi:10.1111/1471-3802.12512 · Cited by 18
He, S., Choong, E. K. M., Duangthip, D., Chu, C. H., & Lo, E. C. M. (2023) · International Journal of Paediatric Dentistry · doi:10.1111/ipd.13055 · Cited by 22
Angrist, J. D., & Hull, P. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311556120 · Cited by 26
Portela, G., Carson, J. L., Swanson, S. A., Alexander, J. H., Hébert, P. C., Goodman, S. G., Steg, P. G., Bertolet, M. H., Strom, J. B., Fergusson, D. A., et al. (2024) · Annals of Internal Medicine · doi:10.7326/m24-0571 · Cited by 20
“Because adherence is not randomized, we would adjust for selection bias introduced by censoring for nonadherence by assigning each patient a nonstabilized inverse probability (IP) weight at every time point at which they remain adherent (8, 25 – 29).”
Bartels, C. M., Ramly, E., Johnson, H. M., Lauver, D., Panyard, D. J., Li, Z., Sampene, E., Lewicki, K., & McBride, P. E. (2018) · Arthritis Care & Research · doi:10.1002/acr.23612 · Cited by 18
“A priori, we planned an as-treated analysis (including only patients in whom the protocol was initiated as indicated by remeasurement) for this pragmatic design (38), although we also performed intenttotreat analysis.”
MethodA constructive critique of the draft ICH E9 Addendum
Scharfstein, D. O. (2019) · Clinical Trials · doi:10.1177/1740774519853566 · Cited by 18
Key considerations in the design of real-world studies
Fang, Y., He, W., Wang, H., & Wu, M. (2020) · Contemporary Clinical Trials · doi:10.1016/j.cct.2020.106091 · Cited by 18
Holt, J. M., Cusatis, R. N., Winn, A. N., Asan, O., Spanbauer, C., Williams, J. S., Flynn, K. E., Somai, M. M., Talsma, A., Laud, P. W., et al. (2021) · Journal of General Internal Medicine · doi:10.1007/s11606-020-06583-7 · Cited by 22
Fusion designs and estimators for treatment effects
Breskin, A., Cole, S. R., Edwards, J. K., Brookmeyer, R., Eron, J. J., & Adimora, A. A. (2021) · Statistics in Medicine · doi:10.1002/sim.8963 · Cited by 22
“First and foremost, the identification results rely on a set of untestable conditions described in Equations (7) and (8).”
BackgroundDong, Y., Xu, Y., Ding, C., Yu, Z., Yu, Z., Xia, X., Chen, Y., & Jiang, X. (2021) · Cardiovascular Diagnosis and Therapy · doi:10.21037/cdt-21-386 · Cited by 21
“An intention-to-treat analysis would make the intervention less effective if patients assigned to the intervention group were lost to follow-up after randomization (31).”
BackgroundChiu, Y., Yland, J. J., Rinaudo, P. F., Hsu, J., McGrath, S., Hernández–Díaz, S., & Hernán, M. Á. (2022) · Fertility and Sterility · doi:10.1016/j.fertnstert.2022.02.003 · Cited by 22
Self-citationWang, L., Norman, I., Xiao, T., Li, Y., Li, X., Liu, T., Wang, J., Zeng, L., Zhong, Z., Jian, C., et al. (2024) · European Journal of Psychotraumatology · doi:10.1080/20008066.2023.2299195 · Cited by 17
“The pre-proto-col analysis, in which the group comparison includes only those who completed the originally allocated treat-ment, was also conducted (Hernán & Robins, 2017).”
MethodKamel, A. M., & Farag, M. A. (2022) · Journal of Medicinal Food · doi:10.1089/jmf.2022.0025 · Cited by 17
“36 The per-protocol analysis was used as it provides an estimate of the true efficacy of those that completed the treatment as planned.”
Kühne, F., Schomaker, M., Stojkov, I., Jahn, B., Conrads‐Frank, A., Siebert, S., Sroczynski, G., Puntscher, S., Schmid, D., Schnell‐Inderst, P., et al. (2022) · German medical science · doi:10.3205/000314 · Cited by 40
“For more detailed and complex information, the corresponding text books and methodologic papers should be consulted [10], [15], [16], [17], [18], [19], [20], [23], [32], [39], [40], [46], [47], [49], [50], [53], [54], [55], [62], [63], [64], [66], [73], [108], [109], [143], [144].”
BackgroundInfluentialScheuer, H., Engstrom, A., Thomas, P., Moodliar, R., Moloney, K., Walen, M. L., Johnson, P., Seo, S., Vaziri, N., Martinez, A., et al. (2020) · Contemporary Clinical Trials · doi:10.1016/j.cct.2020.105970 · Cited by 18
Lu, H., Cole, S. R., Westreich, D. J., Hudgens, M. G., Adimora, A. A., Althoff, K. N., Silverberg, M. J., Buchacz, K. A., Li, J., Edwards, J. K., et al. (2020) · Clinical Infectious Diseases · doi:10.1093/cid/ciaa1037 · Cited by 18
“We also estimated the per-protocol effect of initiating and remaining on an InSTI-based regimen, compared with initiating and remaining on the EFV-based regimen [24, 25].”
MethodSølling, A. S., Christensen, D. H., Darvalics, B., Harsløf, T., Thomsen, R. W., & Langdahl, B. L. (2021) · Osteoporosis International · doi:10.1007/s00198-020-05745-x · Cited by 20
“As information on BMD and FRAX score is not available, we cannot correct for this potential bias in our analyses, neither at index date nor as postbaseline predictors of adherence to the baseline exposure status in the as-treated analyses [47].”
BackgroundZhong, Y., Brooks, M. M., Kennedy, E. H., Bodnar, L. M., & Naimi, A. I. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.43414 · Cited by 20
Pike, J. R., Huang, A., Reed, N. S., Arnold, M. L., Chisolm, T. H., Couper, D. J., Deal, J. A., Glynn, N. W., Goman, A. M., Hayden, K. M., et al. (2025) · Alzheimer s & Dementia · doi:10.1002/alz.70156 · Cited by 19
“64,65 The propensity model included baseline measures of hearing loss, recruitment source, field site, age, sex, education, the presence of one or more APOE ε 4 alleles, cigarette use, alcohol use, global cognition, executive function, memory, language, and predicted risk of cognitive decline.”
Dodd, M., Fielding, K., Carpenter, J. R., Thompson, J. A., & Elbourne, D. R. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-052656 · Cited by 16
Wintzell, V., Svanström, H., & Pasternak, B. (2022) · Epidemiology · doi:10.1097/ede.0000000000001521 · Cited by 19
Massol, J., Simon‐Tillaux, N., Tohme, J., Hariri, G., Dureau, P., Duceau, B., Belin, L., Hajage, D., Rycke, Y. D., Charfeddine, A., et al. (2023) · Critical Care · doi:10.1186/s13054-023-04328-6 · Cited by 22
“In the setting of a pragmatic trial, we first estimated the observational analog of a per-protocol effect [26].”
BackgroundPritchard, L., Thompson‐Hodgetts, S., McKillop, A. B., Rosychuk, R. J., Mrklas, K., Zwaigenbaum, L., Zwicker, J. D., Andersen, J. C., King, G. A., & Firouzeh, P. (2022) · BMC Pediatrics · doi:10.1186/s12887-022-03381-4 · Cited by 17
“Main analyses will be based on as per protocol analysis, as recommended for pragmatic trials [48], and performed by an analyst blinded to group assignment using R [49].”
MethodXuan, J., Mt‐Isa, S., Latimer, N. R., Bell‐Gorrod, H., Malbecq, W., Vandormael, K., Yorke-Edwards, V., & White, I. R. (2024) · Statistical Methods in Medical Research · doi:10.1177/09622802241289559 · Cited by 17
“In addition, in non-inferiority (NI) trials, even without the estimand explicitly defined, the common belief that ITT is anti-conservative (i.e. prone to reject the null) leads to the frequent use of PP. 21,22 But anti-conservatism cannot be evaluated without the estimand being clearly defined so the simple choice of PP…”
InfluentialFernández, A. I., Bermejo, J., Yotti, R., Martínez‐González, M. Á., Mira, Á., Gophna, U., Karlsson, R., Al‐Daccak, R., de Miguel, I. M., Gutiérrez, E., et al. (2021) · Trials · doi:10.1186/s13063-021-05746-z · Cited by 16
“In addition, a per-protocol analysis will be performed following recent directions for analyses of pragmatic trials to control for confounding (42) after excluding from the intervention arm those patients in whom the food frequency questionnaires and biological markers indicate low compliance to the prescribed MedDiet…”
Chang, R. W., Pimentel, N., Tucker, L.-Y. S., Rothenberg, K. A., Avins, A. L., Flint, A. C., Faruqi, R. M., Nguyen‐Huynh, M. N., & Neugebauer, R. S. (2023) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2023.06.024 · Cited by 16
Bonekamp, N. E., Cruijsen, E., Visseren, F. L. J., van der Schouw, Y. T., Geleijnse, J. M., & Koopal, C. (2023) · Clinical Nutrition · doi:10.1016/j.clnu.2023.06.033 · Cited by 15
Metadata
- Authors
- Hernán, M. A., & Robins, J. M.
- Affiliations
- Harvard University (US)
- ORCID
- Robins, James M.
- Venue
- New England Journal of Medicine
- Publisher
- Massachusetts Medical Society
- Published
- 2017-10-05
- Volume
- 377
- Issue
- 14
- Pages
- 1391-1398
- ISSN
- 0028-4793 (ISSN-L)
- 1533-4406
- OpenAlex
- W2764336450
- Cited by
- 660
- References
- 32
- Citations by year
- 2026: 98 · 2025: 89 · 2024: 95 · 2023: 74 · 2022: 82 · 2021: 79 · 2020: 73 · 2019: 39 · 2018: 26 · 2017: 3
- FWCI
- 26.81
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- New England Journal of Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- pragmatic trials, per-protocol analysis, intention-to-treat analysis, real-world evidence, treatment effect estimation
- MeSH terms
- Clinical Protocols, Data Interpretation, Statistical, Humans, Patient Compliance, Research Design, Intention to Treat Analysis, Pragmatic Clinical Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.