CoChat

The Registration of Observational Studies—When Metaphors Go Bad

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

Abstract

Should epidemiologists be required to pre-register their studies and hypotheses? Such a proposal was made at a recent workshop. Lest you think this is just an odd idea that will go away, 2 major clinical journals have already endorsed it.1,2 The workshop report3 itself has not been made public before now. We have been given permission by the workshop conveners (the European Center for Ecotoxicology and Toxicology of Chemicals) to post the report on our Web site (https://links.lww.com/EDE/A415). With this report as a starting point, we have invited five epidemiologists (including one who participated in the workshop) to respond. Their commentaries follow this editorial.4–8 The workshop report states that journal editors play “a critical role in promoting the process by ‘selling’ this to potential authors or readers.” As editors, we won't be selling this proposal—but we do have some thoughts about it. There are conceivable benefits from the registration of protocols and hypotheses of observational studies. For one, the information that a hypothesis has previously been explored (even if the results were never published) might help other investigators decide whether the hypothesis is worth further study. Another possible benefit is that researchers working on publicly-accessible data could be informed that others were doing similar analyses.9 Researchers whose results might damage the business interests of their employers would be less able to hide those results (an argument that led to the registration of clinical trials). That said, we do not agree that registration would provide the benefits claimed by the workshop. Those purported benefits include a reduction in “publication bias,” improved research “transparency” and fulfillment of “ethical obligations.” We address these points in turn. Reduction in “Publication Bias” There is a common perception that negative studies fall into publication limbo (or are never written up), whereas positive studies (including false-positives) get a free pass to publication. As supporting evidence, the workshop report refers repeatedly to “observational studies [that] are not published, either because they are not submitted for publication or because they were rejected by journals.” Such studies exist—but are they evidence of a problem? If there is any validity to the self-critical judgment of researchers and the process of peer reviews of their peers, then we should expect some portion of study results to remain unpublished. Some findings are fatally flawed, while others are of little importance. The mere existence of unpublished analyses is not an argument that unpublished analyses are as valid as those published. Furthermore, the timing of a research hypothesis (whether before or after data collection) is irrelevant to its validity. In their commentaries, Charlie Poole7 and Tim Lash5 both discuss the intellectual history of this mistaken perception. A more empirically-based concern is that hypotheses stated in advance may be less prone to false-positive findings. Put another way, findings supported by previous evidence are more likely to be proved valid than findings having no previous evidence. If this is true, we shouldn't be surprised. Prior evidence should not determine whether results are published. Improved “Transparency” “Transparency” is an unassailable scientific virtue. But what does transparency mean in this context? Does transparency require the prior registration of the “universe” of observational data from all sources?3,p2 Unlike clinical trials, which are rigidly designed to address narrow questions, observational studies exploit a wide range of methods and data sources to address evolving public health questions. It would be a daunting task to identify every relevant population registry and health-care-records system—and every possible potentially important hypothesis—let alone register them. Furthermore, the potential “transparency” of such registered information could easily be clouded by the complexity of assembling the information. The workshop report downplays this problem, saying that the bureaucracy for universal registration would be “very minor.”3,p17 This is a disingenuous statement, given the scope of what registration might entail. For example, the report suggests that researchers be obliged to register “any amendments” to their protocols or hypotheses over time.3,p1 The report also suggests that the STROBE guidelines10 for reporting epidemiologic results (these guidelines themselves comprising 31 pages of dense print) could form the basis for the kind of information investigators should register.3,p9 What are the chances that the registration of observational studies would be a “very minor” encumbrance? Judging from the cancerous growth of bureaucracies to protect human subjects in observational studies, we are not optimistic. The best place for transparency may not be at the onset of a study, but at its culmination. Data become “results” when

Overview

Integrity signals

Registry

Crossref · 10.1097/ede.0b013e3181eafbcf

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 10 references

Cited by

Metadata

Authors
Editors, T. M. A. F. C. D.
Corresponding author
The Monaldi Archives for Chest Disease Editors
Venue
Epidemiology
Published
2010-09
Volume
21
Issue
5
Pages
607-609
ISSN
1044-3983 (ISSN-L)
OpenAlex
W4254262775
Cited by
56
References
10
Citations by year
2025: 1 · 2024: 2 · 2023: 5 · 2022: 3 · 2021: 3 · 2020: 21 · 2019: 1 · 2018: 4 · 2017: 3 · 2016: 2 · 2015: 2 · 2014: 3 · 2013: 1 · 2012: 2
Type
journal article
Language
en
Open access
closed
Locations
  • Epidemiology (publishedVersion)
  • PubMed (publishedVersion)
  • DigitalGeorgetown (Georgetown University Library) (submittedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
preregistration, observational studies, publication bias, research transparency, research ethics, unpublished research, journal editorial policy, clinical trial registration, research funding conflicts of interest, peer review, research reproducibility, research ethics training, research ethics oversight, research ethics committees
MeSH terms
Ethics, Medical, Humans, Registries, Epidemiologic Studies, Publication Bias, Disclosure, Editorial Policies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.