Effect Estimates in Randomized Trials and Observational Studies: Comparing Apples With Apples
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Effect estimates from randomized trials and observational studies might not be directly comparable because of differences in study design, other than randomization, and in data analysis. We propose a 3-step procedure to facilitate meaningful comparisons of effect estimates from randomized trials and observational studies: 1) harmonization of the study protocols (eligibility criteria, treatment strategies, outcome, start and end of follow-up, causal contrast) so that the studies target the same causal effect, 2) harmonization of the data analysis to estimate the causal effect, and 3) sensitivity analyses to investigate the impact of discrepancies that could not be accounted for in the harmonization process. To illustrate our approach, we compared estimates of the effect of immediate with deferred initiation of antiretroviral therapy in individuals positive for the human immunodeficiency virus from the Strategic Timing of Antiretroviral Therapy (START) randomized trial and the observational HIV-CAUSAL Collaboration.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prevention of HIV-1 infection with early antiretroviral therapy.
Cohen, M. S., Chen, Y. Q., McCauley, M., Gamble, T., Hosseinipour, M. C., Kumarasamy, N., Hakim, J. G., Kumwenda, J., Grinsztejn, B., Pilotto, J. H. S., et al. (2011) · New England Journal of Medicine · doi:10.1056/nejmoa1105243 · Cited by 6,873
Sensitivity analysis in observational research: Introducing the E-value
VanderWeele, T. J., & Ding, P. (2017) · Annals of Internal Medicine · doi:10.7326/m16-2607 · Cited by 6,350 · cited 1×
“Future work can extend our general framework to incorporate quantitative assessments of the impact of imperfect harmonization on the randomized-observational discrepancies (46, 47).”
BackgroundRandomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563
Robins, J. (1986) · Mathematical Modelling · doi:10.1016/0270-0255(86)90088-6 · Cited by 2,451
A comparison of observational studies and randomized, controlled trials.
Benson, K., & Hartz, A. J. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422506 · Cited by 2,347 · cited 2×
“In contrast with previous comparisons of observational studies based on meta-analysis (4-8) and within study comparison (45), our approach requires the reanalysis of two two existing studies and will often result in an imperfect harmonization.”
ResultHernán, M. A., Sauer, B. C., Hernández-Díaz, S., Platt, R., & Shrier, I. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.014 · Cited by 1,097 · cited 1×
“For example, the randomized-observational discrepancy for the effect of postmenopausal estrogen plus progestin therapy on coronary heart disease was largely explained by selection bias (because the followup in the observational study started some time after initiation of therapy) whereas unmeasured confounding seemed…”
BackgroundSelf-citationPer-Protocol Analyses of Pragmatic Trials
Hernán, M. A., & Robins, J. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmsm1605385 · Cited by 660 · cited 2×
“As another example, randomized trials tend to use intention-to-treat estimates that quantify the effect of being assigned to treatment, regardless of whether treatment is actually received, whereas many observational studies quantify the effect of the treatment that was actually received (3).”
BackgroundSelf-citationAnglemyer, A., Horvath, H. T., & Bero, L. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub2 · Cited by 534
Beyond the intention-to-treat in comparative effectiveness research
Hernán, M. A., & Hernández-Díaz, S. (2012) · Clinical Trials · doi:10.1177/1740774511420743 · Cited by 528 · cited 1×
“per-protocol effect: the effect that would have been observed under perfect adherence to the protocol (29).”
BackgroundSelf-citationProblems of stopping trials early
Guyatt, G. H., Briel, M., Glasziou, P., Bassler, D., & Montori, V. M. (2012) · BMJ · doi:10.1136/bmj.e3863 · Cited by 115 · cited 1×
“18 tuberculosis), and (v) overestimation of the beneficial effect of immediate treatment initiation in the randomized trial due to early stopping after an interim analysis indicating a benefit (42, 43).”
Background
Cited by
Tocilizumab in patients with severe COVID-19: a retrospective cohort study
Guaraldi, G., Meschiari, M., Cozzi-Lepri, A., Milic, J., Tonelli, R., Menozzi, M., Franceschini, E., Cuomo, G., Orlando, G., Borghi, V., et al. (2020) · The Lancet Rheumatology · doi:10.1016/s2665-9913(20)30173-9 · Cited by 939
Wang, S. V., Schneeweiss, S., Initiative, R.-D., Franklin, J. M., Desai, R. J., Feldman, W., Garry, E. M., Glynn, R. J., Lin, K. J., Paik, J., et al. (2023) · JAMA · doi:10.1001/jama.2023.4221 · Cited by 428
Methods of Public Health Research — Strengthening Causal Inference from Observational Data
Hernán, M. A. (2021) · New England Journal of Medicine · doi:10.1056/nejmp2113319 · Cited by 295
Self-citationGray, C. M., Grimson, F., Layton, D., Pocock, S., & Kim, J. (2020) · Drug Safety · doi:10.1007/s40264-020-00944-1 · Cited by 65
Metadata
- Authors
- Lodi, S., Phillips, A., Lundgren, J., Logan, R., Sharma, S., Cole, S. R., Babiker, A., Law, M., Chu, H., Byrne, D., Horban, A., Sterne, J. A. C., Porter, K., Sabin, C., Costagliola, D., Abgrall, S., Gill, J., Touloumi, G., Pacheco, A. G., van Sighem, A., Reiss, P., Bucher, H. C., Montoliu Giménez, A., Jarrin, I., Wittkop, L., Meyer, L., Perez-Hoyos, S., Justice, A., Neaton, J. D., Hernán, M. A., & INSIGHT START Study Group and the HIV-CAUSAL Collaboration
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-05-07
- Volume
- 188
- Issue
- 8
- Pages
- 1569-1577
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- OpenAlex
- W2939670107
- Cited by
- 112
- References
- 11
- FWCI
- 13.17
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Effect Estimates in Randomized Trials and Observational Studies: Comparing Apples With Apples. https://app.cochat.ai/papers/10.1093/aje/kwz100
- The paper
- Lodi, S., Phillips, A., Lundgren, J., Logan, R., Sharma, S., Cole, S. R., Babiker, A., Law, M., Chu, H., Byrne, D., Horban, A., Sterne, J. A. C., Porter, K., Sabin, C., Costagliola, D., Abgrall, S., Gill, J., Touloumi, G., Pacheco, A. G., . . . INSIGHT START Study Group and the HIV-CAUSAL Collaboration. (2019). Effect Estimates in Randomized Trials and Observational Studies: Comparing Apples With Apples. American Journal of Epidemiology. https://doi.org/10.1093/aje/kwz100
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.