Invited Commentary: Beware the Test-Negative Design
Open accessAbstract
In this issue of the Journal, Sullivan et al. (Am J Epidemiol. 2016;184(5):345-353) carefully examine the theoretical justification for use of the test-negative design, a common observational study design, in assessing the effectiveness of influenza vaccination. Using modern causal inference methods (in particular, directed acyclic graphs), they describe different threats to the validity of inferences drawn about the effect of vaccination from test-negative design studies. These threats include confounding, selection bias, and measurement error in either the exposure or the outcome. While confounding and measurement error are common in observational studies, the potential for selection bias inherent in the test-negative design brings into question the validity of inferences drawn from such studies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Controlling for both healthcare seeking behaviour and the propensity to be tested blocks the biasing path, albeit incompletely, under the limitation of measuring healthcare seeking behaviour adequately.
“However, controlling for both healthcare seeking behaviour and the propensity to be tested (inherent to the design) blocks the biasing path, albeit incompletely, under the limitation of measuring healthcare seeking behaviour adequately (fig 3E).42”
Generalisability of the test negative design is limited, because of the association in other populations who did not choose to be tested.
“But alongside its advantages, generalisability of the test negative design is limited, because of the association in other populations who did not choose to be tested.42”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Confounding and Collapsibility in Causal Inference
Greenland, S., Pearl, J., & Robins, J. M. (1999) · Statistical Science · doi:10.1214/ss/1009211805 · Cited by 933 · cited 1× · see citing works
“However, the odds ratio is not collapsible, such that the conditional casual odds ratio will not in general equal the marginal causal odds ratio (4).”
The test-negative design for estimating influenza vaccine effectiveness
Jackson, M. L., & Nelson, J. C. (2013) · Vaccine · doi:10.1016/j.vaccine.2013.02.053 · Cited by 510 · see citing works
Sullivan, S. G., Tchetgen, E. J. T., & Cowling, B. J. (2016) · American Journal of Epidemiology · doi:10.1093/aje/kww064 · Cited by 316 · see citing works
Stampf, S., Graf, E., Schmoor, C., & Schumacher, M. C. (2010) · Statistics in Medicine · doi:10.1002/sim.3811 · Cited by 40 · see citing works
Forbes, A. B., & Shortreed, S. M. (2008) · Statistics in Medicine · doi:10.1002/sim.3362 · Cited by 22 · see citing works
Graf, E., & Schumacher, M. C. (2008) · Statistics in Medicine · doi:10.1002/sim.3271 · Cited by 19 · see citing works
A comparison of marginal odds ratio estimators
Loux, T. M., Drake, C., & Smith‐Gagen, J. (2014) · Statistical Methods in Medical Research · doi:10.1177/0962280214541995 · Cited by 14 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Graf E, Schumacher M. Comments on 'The performance of different propensity score methods for estimating marginal odds ratios' [letter]. Stat Med. 2008;27(19):3915-3917.
Cited by
Sorted by citations
Gazit, S., Saciuk, Y., Perez, G., Peretz, A., Pitzer, V. E., & Patalon, T. (2022) · BMJ · doi:10.1136/bmj-2022-071113 · Cited by 87 · see citing works
- Authors
- Westreich, D., & Hudgens, M. G.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Corresponding author
- Westreich, Daniel
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-08-31
- Volume
- 184
- Issue
- 5
- Pages
- 354-356
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- PMC
- PMC5013886
- OpenAlex
- W2509529180
- Cited by
- 65
- References
- 7
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 11 · 2023: 9 · 2022: 8 · 2021: 5 · 2020: 8 · 2019: 1 · 2018: 7 · 2017: 2 · 2016: 3
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Epidemiology (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Funders
- National Institutes of Health
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Health Sciences › Medicine › Epidemiology › Influenza Virus Research Studies
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- test-negative design, influenza vaccination, causal inference, directed acyclic graphs, selection bias, confounding, measurement error, observational study validity, vaccine effectiveness
- MeSH terms
- Humans, Research Design, Selection Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
