The Registration of Observational Studies—When Metaphors Go Bad
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Should epidemiologists be required to pre-register their studies and hypotheses? Such a proposal was made at a recent workshop. Lest you think this is just an odd idea that will go away, 2 major clinical journals have already endorsed it.1,2 The workshop report3 itself has not been made public before now. We have been given permission by the workshop conveners (the European Center for Ecotoxicology and Toxicology of Chemicals) to post the report on our Web site (https://links.lww.com/EDE/A415). With this report as a starting point, we have invited five epidemiologists (including one who participated in the workshop) to respond. Their commentaries follow this editorial.4–8 The workshop report states that journal editors play “a critical role in promoting the process by ‘selling’ this to potential authors or readers.” As editors, we won't be selling this proposal—but we do have some thoughts about it. There are conceivable benefits from the registration of protocols and hypotheses of observational studies. For one, the information that a hypothesis has previously been explored (even if the results were never published) might help other investigators decide whether the hypothesis is worth further study. Another possible benefit is that researchers working on publicly-accessible data could be informed that others were doing similar analyses.9 Researchers whose results might damage the business interests of their employers would be less able to hide those results (an argument that led to the registration of clinical trials). That said, we do not agree that registration would provide the benefits claimed by the workshop. Those purported benefits include a reduction in “publication bias,” improved research “transparency” and fulfillment of “ethical obligations.” We address these points in turn. Reduction in “Publication Bias” There is a common perception that negative studies fall into publication limbo (or are never written up), whereas positive studies (including false-positives) get a free pass to publication. As supporting evidence, the workshop report refers repeatedly to “observational studies [that] are not published, either because they are not submitted for publication or because they were rejected by journals.” Such studies exist—but are they evidence of a problem? If there is any validity to the self-critical judgment of researchers and the process of peer reviews of their peers, then we should expect some portion of study results to remain unpublished. Some findings are fatally flawed, while others are of little importance. The mere existence of unpublished analyses is not an argument that unpublished analyses are as valid as those published. Furthermore, the timing of a research hypothesis (whether before or after data collection) is irrelevant to its validity. In their commentaries, Charlie Poole7 and Tim Lash5 both discuss the intellectual history of this mistaken perception. A more empirically-based concern is that hypotheses stated in advance may be less prone to false-positive findings. Put another way, findings supported by previous evidence are more likely to be proved valid than findings having no previous evidence. If this is true, we shouldn't be surprised. Prior evidence should not determine whether results are published. Improved “Transparency” “Transparency” is an unassailable scientific virtue. But what does transparency mean in this context? Does transparency require the prior registration of the “universe” of observational data from all sources?3,p2 Unlike clinical trials, which are rigidly designed to address narrow questions, observational studies exploit a wide range of methods and data sources to address evolving public health questions. It would be a daunting task to identify every relevant population registry and health-care-records system—and every possible potentially important hypothesis—let alone register them. Furthermore, the potential “transparency” of such registered information could easily be clouded by the complexity of assembling the information. The workshop report downplays this problem, saying that the bureaucracy for universal registration would be “very minor.”3,p17 This is a disingenuous statement, given the scope of what registration might entail. For example, the report suggests that researchers be obliged to register “any amendments” to their protocols or hypotheses over time.3,p1 The report also suggests that the STROBE guidelines10 for reporting epidemiologic results (these guidelines themselves comprising 31 pages of dense print) could form the basis for the kind of information investigators should register.3,p9 What are the chances that the registration of observational studies would be a “very minor” encumbrance? Judging from the cancerous growth of bureaucracies to protect human subjects in observational studies, we are not optimistic. The best place for transparency may not be at the onset of a study, but at its culmination. Data become “results” when
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Gatekeeping functions may increase this burden, and have been criticized as the bureaucratization of science (107).”
From Issues with data and analyses: Errors, underlying themes, and potential solutions (2018) · cites this paper as [107]
Citations
Cites
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., & Egger, M. (2007) · Epidemiology · doi:10.1097/ede.0b013e3181577511 · Cited by 2,779
Registration of observational studies
Loder, E., Groves, T., & MacAuley, D. (2010) · BMJ · doi:10.1136/bmj.c950 · Cited by 131
Should protocols for observational research be registered?
Lancet, T. (2010) · The Lancet · doi:10.1016/s0140-6736(10)60148-1 · Cited by 72
Epidemiology, Data Sharing, and the Challenge of Scientific Replication
Hernán, M. Á., & Wilcox, A. J. (2009) · Epidemiology · doi:10.1097/ede.0b013e318196784a · Cited by 35
Preregistration of Epidemiologic Studies
Vandenbroucke, J. P. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181e942b8 · Cited by 32
Lash, T. L. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181e9bba6 · Cited by 26
Meta-analysis Protocol Registration
Takkouche, B., & Norman, G. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181e9bbbd · Cited by 20
A Vision of Accessible Epidemiology
Poole, C. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181e9be3f · Cited by 19
To Register or Not To Register
Samet, J. M. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181e9be54 · Cited by 15
Editors, T. (2002) · Epidemiology · doi:10.1097/00001648-200209000-00001 · Cited by 3
Self-citation
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Registries for Evaluating Patient Outcomes: A User’s Guide
(2020) · doi:10.23970/ahrqepcregistries4 · Cited by 960
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Designing deep learning studies in cancer diagnostics
Kleppe, A., Skrede, O.-J., De Raedt, S., Liestøl, K., Kerr, D. J., & Danielsen, H. E. (2021) · Nature Reviews Cancer · doi:10.1038/s41568-020-00327-9 · Cited by 316
Issues with data and analyses: Errors, underlying themes, and potential solutions
Brown, A. W., Kaiser, K. A., & Allison, D. B. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708279115 · Cited by 157
“Gatekeeping functions may increase this burden, and have been criticized as the bureaucrati-zation of science (107).”
BackgroundPeat, G., Riley, R. D., Croft, P., Morley, K. I., Kyzas, P. A., Moons, K. G. M., Perel, P., Steyerberg, E. W., Schroter, S., Altman, D. G., et al. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001671 · Cited by 152
The Production of Knowledge: Enhancing Progress in Social Science
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 241
Making Prospective Registration of Observational Research a Reality
Dal-Ré, R., Ioannidis, J. P., Bracken, M. B., Buffler, P. A., Chan, A.-W., Franco, E. L., La Vecchia, C., & Weiderpass, E. (2014) · Science Translational Medicine · doi:10.1126/scitranslmed.3007513 · Cited by 123
Hörnell, A., Berg, C., Forsum, E., Larsson, C. L., Sonestedt, E., Åkesson, A., Lachat, C. K., Hawwash, D., Kolsteren, P. W., Byrnes, G. B., et al. (2017) · Advances in Nutrition · doi:10.3945/an.117.015941 · Cited by 84
Improving Transparency in Observational Social Science Research: A Pre-Analysis Plan Approach
Burlig, F. (2016) · Economics Letters · doi:10.1016/j.econlet.2018.03.036 · Cited by 79
Debray, T. P. A., Collins, G. S., Riley, R. D., Snell, K. I. E., Van Calster, B., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071058 · Cited by 57
DEBATE-statistical analysis plans for observational studies
Hiemstra, B., Keus, F., Wetterslev, J., Gluud, C. N., & van der Horst, I. C. C. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0879-5 · Cited by 54
Impacts and Determinants of Health Levels in Low-Income Countries
Pascaline Dupas, Y., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22235 · Cited by 46
Lash, T. L., & Vandenbroucke, J. P. (2012) · Epidemiology · doi:10.1097/ede.0b013e318245c05b · Cited by 43
Toward Open and Reproducible Epidemiology
Mathur, M. B., & Fox, M. P. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwad007 · Cited by 56
Should protocols for observational research be registered?
Rushton, L. (2011) · Occupational and Environmental Medicine · doi:10.1136/oem.2010.056846 · Cited by 36
Transparency and disclosure, neutrality and balance: shared values or just shared words?
Greenland, S. (2012) · Journal of Epidemiology and Community Health · doi:10.1136/jech-2011-200459 · Cited by 35
Improving the transparency and reliability of observational studies through registration
Naudet, F., Patel, C. J., DeVito, N., Goff, G. L., Cristea, I. A., Braillon, A., & Hoffmann, S. (2024) · BMJ · doi:10.1136/bmj-2023-076123 · Cited by 37
Pre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 34
Infrequent and incomplete registration of test accuracy studies: analysis of recent study reports
Korevaar, D. A., Bossuyt, P. M. M., & Hooft, L. (2014) · BMJ Open · doi:10.1136/bmjopen-2013-004596 · Cited by 27
Preregistration of Epidemiology Protocols: A Commentary in Support
Bracken, M. B. (2011) · Epidemiology · doi:10.1097/ede.0b013e318207fc7c · Cited by 26
Registration of health and medical research
Cashin, A. G., Richards, G. C., DeVito, N., Mellor, D. T., & Lee, H. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2021-111836 · Cited by 31
Registration of Observational Studies: perspectives from an industry‐based epidemiology group
Chavers, S., Fife, D., Wacholtz, M. C., Stang, P., & Berlin, J. A. (2011) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.2221 · Cited by 20
Making Research Data Accessible
Kapiszewski, D., & Karcher, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.008 · Cited by 16
PEER-REVIEWED JOURNAL EDITORS’ VIEWS ON REAL-WORLD EVIDENCE
Oehrlein, E. M., Graff, J. S., Perfetto, E. M., Mullins, C. D., Dubois, R. W., Anyanwu, C., & Onukwugha, E. (2018) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462317004408 · Cited by 14
Improving Transparency in Observational Social Science Research: A Pre-Analysis Plan ApproachPreprint
Burlig, F. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2731920 · Cited by 11
“There is an active debate in the sciences about preregistration of observational work (see Epidemiology (2010), BMJ (2010), The Lancet (2010), PLOS Medicine (2014) and Dal-Ré et al. (2014)).”
BackgroundImproving Concordance in Environmental Epidemiology: A Three-Part Proposal
LaKind, J. S., Goodman, M. G., Makris, S. L., & Mattison, D. R. (2015) · Journal of Toxicology and Environmental Health Part B · doi:10.1080/10937404.2015.1051612 · Cited by 10
Moons, K. G. M., Altman, D. G., Reitsma, J. B., loannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2022) · Digital Diagnostics · doi:10.17816/dd110794 · Cited by 10
Lash, T. L. (2021) · Epidemiology · doi:10.1097/ede.0000000000001424 · Cited by 10
Malmsiø, D., Norlén, S., Jespersen, C., Neesgaard, V. E., Song, Z., Chan, A., & Hróbjartsson, A. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111686 · Cited by 6
Improving transparency in observational social science research: A pre-analysis plan approachPreprint
Burlig, F. (2017) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/qemkz · Cited by 9
Malmsiø, D., Frost, A. D., & Hróbjartsson, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.12.012 · Cited by 5
Transparency in Infectious Disease Research: Meta-research Survey of Specialty Journals
Zavalis, E. A., Contopoulos‐Ioannidis, D. G., & Ioannidis, J. P. A. (2023) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiad130 · Cited by 7
Reliability of Inference: Analogs of Replication in Qualitative Research
Fairfield, T., & Charman, A. E. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.012 · Cited by 6
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.001 · Cited by 6
Stopsack, K. H., Mucci, L. A., Tworoger, S. S., Kang, J. H., Eliassen, A. H., Willett, W. C., & Stampfer, M. J. (2023) · Epidemiology · doi:10.1097/ede.0000000000001599 · Cited by 6
Correspondence Between Results and Aims of Funding Support in EPIDEMIOLOGY Articles
Lash, T. L., Kaufman, J. S., & Hernán, M. Á. (2017) · Epidemiology · doi:10.1097/ede.0000000000000767 · Cited by 6
Replication for Quantitative Research
Freese, J., & Peterson, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.010 · Cited by 4
Improving Confidence in Observational Studies
Onukwugha, E. (2013) · PharmacoEconomics · doi:10.1007/s40273-012-0019-0 · Cited by 4
Transparency and Reproducibility: Conceptualizing the Problem
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.006 · Cited by 4
Measurement Replication in Qualitative and Quantitative Studies
Reiter, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.011 · Cited by 4
Transparency and Reproducibility: Potential Solutions
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.007 · Cited by 3
Gerring, J., Karcher, S., & Apfeld, B. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.015 · Cited by 3
What’s Wrong with Replicating the Old Boys’ Networks?
Teele, D. L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.016 · Cited by 2
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.013 · Cited by 2
Lieberman, E. S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.003 · Cited by 3
Christensen, G., & Miguel, E. A. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0002 · Cited by 2
Moons, K. G. M., Altman, D. G., Reitsma, J. B., loannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2023) · Вопросы современной педиатрии · doi:10.15690/vsp.v22i2.2557 · Cited by 1
Improving Reproducibility in Epidemiology
Dunbar, M. B., & Held, L. (2024) · Springer eBooks · doi:10.1007/978-1-4614-6625-3_73-1 · Cited by 9
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.018 · Cited by 2
Gerring, J., & Cojocaru, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.005 · Cited by 1
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.014 · Cited by 2
Gross, N., & Robertson, C. T. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.017 · Cited by 2
Liao, T. F. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.004
Elman, C., Gerring, J., & Mahoney, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.019
Metadata
- Authors
- Editors, T. M. A. F. C. D.
- Corresponding author
- The Monaldi Archives for Chest Disease Editors
- Venue
- Epidemiology
- Published
- 2010-09
- Volume
- 21
- Issue
- 5
- Pages
- 607-609
- ISSN
- 1044-3983 (ISSN-L)
- OpenAlex
- W4254262775
- Cited by
- 56
- References
- 10
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 5 · 2022: 3 · 2021: 3 · 2020: 21 · 2019: 1 · 2018: 4 · 2017: 3 · 2016: 2 · 2015: 2 · 2014: 3 · 2013: 1 · 2012: 2
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- DigitalGeorgetown (Georgetown University Library) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- preregistration, observational studies, publication bias, research transparency, research ethics, unpublished research, journal editorial policy, clinical trial registration, research funding conflicts of interest, peer review, research reproducibility, research ethics training, research ethics oversight, research ethics committees
- MeSH terms
- Ethics, Medical, Humans, Registries, Epidemiologic Studies, Publication Bias, Disclosure, Editorial Policies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.