CoChat

Commentary: Developing best-practice guidelines for the reporting of E-values

Open access
VanderWeele, Tyler J, Mathur, Maya B · 2020 · International Journal of Epidemiology
Cited by 111

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Abstract

We would like to thank Blum et al.1 for their interesting analysis of the current reporting practices around the use of the E-value to evaluate sensitivity to unmeasured confounding.2,3 As with nearly any quantitative tool, the E-value is potentially subject to misuse,4,5 examples of which are indeed documented in their paper. This arguably also points to the need for the development of best-practice guidelines for the reporting and interpretation of E-values. We will here offer some reflections on potential reporting guidelines. Blum et al.1 sampled a set of control papers from the same journals as those that reported E-values. Of these 69 papers, 52 (75.3%) apparently had no discussion whatsoever of unmeasured confounding. Unmeasured confounding is a major threat in most observational studies. That this threat is left both undiscussed and unquantified so frequently is troubling. Some form of sensitivity analysis or critical assessment of potential uncontrolled confounding is needed to address this problem. The E-value was developed as a particularly straightforward approach to do so,3,4,6 but there are of course other tools available.7,8 However, some approach should be employed. As noted in our paper3 and by Blum et al.1, the E-value is not context-free. The E-value needs to be evaluated in light of the measured confounders, the outcome, the exposure and the potentially known unmeasured confounders.2–4,9 Whenever possible, it would be good to report specific variables that are thought to be potential unmeasured confounders. Blum et al.1 report that, of the 87 articles in their sample, 34 (39%) named specific variables that could be confounders and were unaccounted for. Such reporting should be improved. There are settings in which all known risk factors for an outcome are controlled for and, in such circumstances, it is not possible to specifically name a potential unmeasured confounder, but it is unlikely that this constitutes the remaining 61% of cases. Even in such settings wherein no specific unmeasured confounder can be named, it can be worthwhile calculating E-values or performing some other sensitivity analysis, as unknown unmeasured confounders can still be a threat. However, as a general principle, it would be good if all papers either stated that control was made for all known confounders or alternatively discussed which important unmeasured confounders might still have biased the analysis. Blum et al.1 rightly emphasize the need to interpret the magnitude of the required confounding associations. In our papers,2–4 we have not provided ‘cut-offs’ for what were large or small E-values. That will be relative to the outcome and exposure under consideration.2–4,9 It will also be relative to what measured confounders have been adjusted for.2–4,9 If adjustment has been made for numerous measured confounders related to the unmeasured variable, then the residual confounding associations are likely to be small. In contrast, if there are multiple unmeasured confounding variables, then it is possible for the residual confounding associations to be very large and, in such cases, not even a large E-value would provide much evidence for causation.2–4,9 However, if there are several (e.g., three or four) distinct important known unmeasured confounders, then a reasonable effect estimate likely cannot be obtained to begin with (these would not be the right data with which to attempt to address the research question). The E-value approach, and sensitivity analysis more generally, will be most helpful when there is a single known unmeasured confounder, or when adjustment has been made for all known measured confounders but, of course, with the possibility still of an unknown unmeasured confounder. With a single known or unknown unmeasured confounder, it is still important to have some sense as to how large the confounding associations may be. Previous studies that have measured the variable in question (if there is a known unmeasured confounder) can be helpful in this regard, as would be the field-wide systematic umbrella reviews that Blum et al. support. One challenge with such approaches is that covariate associations with outcomes can vary with what other covariates have been adjusted for, and can also vary across populations. Another approach to trying to gain some sense as to what constitutes a small or large E-value in a given context would be to report associations of each of the measured covariates with the outcome. This could be done in an online supplementary table, with some comment given in the main text. In outcome-wide settings in which numerous outcomes,10,11 and thus numerous E-values, are assessed, a single table perhaps reporting the three largest covariate–outcome associations (properly inverted for protective associations) for each outcome could once again be helpful. In principle, a similar approach might also be used for the exposure, by providing the largest exposure–covariate relative risk

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/ije/dyaa094

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 111

Metadata

Authors
VanderWeele, T. J., & Mathur, M. B.
Venue
International Journal of Epidemiology
Published
2020-08-02
Volume
49
Issue
5
Pages
1495-1497
ISSN
  • 0300-5771 (ISSN-L)
  • 1464-3685
OpenAlex
W3047310998
Cited by
111
References
3
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.