Data dredging, bias, or confounding: They can all get you into the BMJ and the Friday papers
Open accessAbstract
On 4 October 2002, women who were moderate drinkers received good news: their risk of breast cancer was not raised, according to a report in the Lancet that was widely covered by the British media.1 The bad news was that smoking at an early age was now implicated as a risk factor for breast cancer. However, after they had enjoyed guilt-free drinks (without cigarettes) for only a few days, on 13 November the message was reversed: alcohol did increase the risk of breast cancer after all, but smoking was declared innocent.2 The press release proclaimed “Alcohol, tobacco and breast cancer: the definitive answer.” A reader was driven to complain in the letters page of the Guardian (14 November 2002): “So let me get this right—alcohol's no good anymore, and if you smoked within five years of getting your periods, that's bad news too. Oh no, that was a couple of weeks ago; smoking's okay now … Do things stop being bad for us if we just forget about them for a bit, do you think?” This is a familiar story—so much so that in Bristol we set our medical students the exercise of examining the “health scare of the week” that appears each Friday, generally from a study reported in the BMJ or Lancet .w1 The widespread perception that epidemiological studies generate conflicting and often meaningless findingsw2 has received support from recent randomised controlled trials, which have failed to confirm even apparently robust findings from observational epidemiological studies. The most topical of these relates to hormone replacement therapy. In 1991 a meta-analysis of epidemiological results relating the use of hormone replacement therapy to the risk of coronary heart disease concluded that it halved the risk, and that the evidence was statistically robust (relative risk 0.50; 95% …
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There are examples in epidemiology where associations conforming to causality criteria have turned out to be confounded when RCTs have been conducted.
“there are examples in epidemiology where associations conforming to these criteria have turned out to be confounded when RCTs have been conducted (62).”
From Association Between Cannabis and Psychosis: Epidemiologic Evidence (2016) · cites this paper as [62]
In “Observational studies propose, RCTs dispose” (page unavailable), the paper gives examples of apparently robust epidemiological findings, including hormone replacement therapy and β-carotene, that randomized trials failed to confirm. Its discussion of confounding explains the hormone-therapy discrepancy, but the paper does not establish that these associations met the specific causality criteria referred to as “these criteria” in the citing sentence. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe strengths and limitations of methods used to conduct research with routinely collected data may be contentious.
“Because the strengths and limitations of methods used to conduct research with routinely collected data may be contentious, a clear description of a study’s objectives is essential [23,24].”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [24]
Associations reported in epidemiological studies can be produced by other forms of bias.
“or by other forms of bias.”
From Strengthening the Reporting of Observational Studies in Epidemiology using Mendelian Randomisation (STROBE-MR): explanation and elaboration. (2021) · cites this paper as [1]
Associations reported in epidemiological studies are often not reliable estimates of causal effects.
“However, such associations reported in epidemiological studies are often not reliable estimates of causal effects”
From Strengthening the Reporting of Observational Studies in Epidemiology using Mendelian Randomisation (STROBE-MR): explanation and elaboration. (2021) · cites this paper as [1]
Associations reported in epidemiological studies can be produced by confounding.
“and can be produced by confounding (that is, by another factor that affects both the outcome and exposure)1 2 3”
From Strengthening the Reporting of Observational Studies in Epidemiology using Mendelian Randomisation (STROBE-MR): explanation and elaboration. (2021) · cites this paper as [1]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Davey Smith, G., & Ebrahim, S. (2003) · International Journal of Epidemiology · doi:10.1093/ije/dyg070 · Cited by 5,882 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Bowden, J., Davey Smith, G., & Burgess, S. (2015) · International Journal of Epidemiology · doi:10.1093/ije/dyv080 · Cited by 9,816 · see citing works
Davey Smith, G., & Ebrahim, S. (2003) · International Journal of Epidemiology · doi:10.1093/ije/dyg070 · Cited by 5,882 · see citing works
Self-citationBenchimol, E. I., Smeeth, L., Guttmann, A., Harron, K., Moher, D., Petersen, I., Sørensen, H. T., von Elm, E., Langan, S. M., & RECORD Working Committee (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001885 · Cited by 5,749 · see citing works
Mendelian randomization: genetic anchors for causal inference in epidemiological studies
Davey Smith, G., & Hemani, G. (2014) · Human Molecular Genetics · doi:10.1093/hmg/ddu328 · Cited by 4,677 · see citing works
Hormones and endocrine-disrupting chemicals: low-dose effects and nonmonotonic dose responses
Vandenberg, L. N., Colborn, T., Hayes, T. B., Heindel, J. J., Jacobs, D. R., Lee, D.-H., Shioda, T., Soto, A. M., vom Saal, F. S., Welshons, W. V., et al. (2012) · Endocrine Reviews · doi:10.1210/er.2011-1050 · Cited by 3,209 · see citing works
Orienting the causal relationship between imprecisely measured traits using GWAS summary data.
Hemani, G., Tilling, K., & Davey Smith, G. (2017) · PLoS Genetics · doi:10.1371/journal.pgen.1007081 · Cited by 2,607 · see citing works
Cannabis use and risk of psychotic or affective mental health outcomes: a systematic review.
Moore, T. H., Zammit, S., Lingford-Hughes, A., Barnes, T. R., Jones, P. B., Burke, M., & Lewis, G. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61162-3 · Cited by 2,224 · see citing works
Understanding and misunderstanding randomized controlled trials
Deaton, A., & Cartwright, N. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.12.005 · Cited by 1,841 · see citing works
Krieger, N., Smith, K., Naishadham, D., Hartman, C., & Barbeau, E. M. (2005) · Social Science & Medicine · doi:10.1016/j.socscimed.2005.03.006 · Cited by 1,704 · see citing works
Skrivankova, V. W., Richmond, R. C., Woolf, B. A. R., Davies, N. M., Swanson, S. A., VanderWeele, T. J., Timpson, N. J., Higgins, J. P. T., Dimou, N., Langenberg, C., et al. (2021) · BMJ · doi:10.1136/bmj.n2233 · Cited by 1,648 · see citing works
Embodiment: a conceptual glossary for epidemiology
Krieger, N. (2005) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2004.024562 · Cited by 964 · see citing works
Mendelian Randomization: Concepts and Scope
Richmond, R. C., & Davey Smith, G. (2022) · Cold Spring Harbor Perspectives in Medicine · doi:10.1101/cshperspect.a040501 · Cited by 841 · see citing works
Macleod, J., Oakes, R., Copello, A., Crome, I., Egger, M., Hickman, M., Oppenkowski, T., Stokes-Lampard, H., & Smith, G. D. (2004) · The Lancet · doi:10.1016/s0140-6736(04)16200-4 · Cited by 639 · see citing works
Self-citationSmith, G. D., Lawlor, D. A., Harbord, R., Timpson, N., Day, I., & Ebrahim, S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040352 · Cited by 569 · see citing works
Self-citationPrimary Prevention of Cardiovascular Mortality and Events With Statin Treatments
Mills, E. J., Rachlis, B., Wu, P., Devereaux, P. J., Arora, P., & Perri, D. (2008) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2008.08.039 · Cited by 423 · see citing works
Kramer, M. S., Matush, L., Vanilovich, I., Platt, R. W., Bogdanovich, N., Sevkovskaya, Z., Dzikovich, I., Shishko, G., Collet, J.-P., Martin, R. M., et al. (2007) · The American Journal of Clinical Nutrition · doi:10.1093/ajcn/86.5.1717 · Cited by 458 · see citing works
GWAS identifies 14 loci for device-measured physical activity and sleep duration
Doherty, A., Smith-Byrne, K., Ferreira, T., Holmes, M. V., Holmes, C., Pulit, S. L., & Lindgren, C. M. (2018) · Nature Communications · doi:10.1038/s41467-018-07743-4 · Cited by 408 · see citing works
Association Between Cannabis and Psychosis: Epidemiologic Evidence
Gage, S. H., Hickman, M., & Zammit, S. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.08.001 · Cited by 379 · see citing works
Kers, J. G., & Saccenti, E. (2022) · Frontiers in Microbiology · doi:10.3389/fmicb.2021.796025 · Cited by 359 · see citing works
The False-positive to False-negative Ratio in Epidemiologic Studies
Ioannidis, J. P. A., Tarone, R., & McLaughlin, J. K. (2011) · Epidemiology · doi:10.1097/ede.0b013e31821b506e · Cited by 321 · see citing works
Issues in the reporting of epidemiological studies: a survey of recent practice.
Pocock, S. J., Collier, T. J., Dandreo, K. J., de Stavola, B. L., Goldman, M. B., Kalish, L. A., Kasten, L. E., & McCormack, V. A. (2004) · BMJ · doi:10.1136/bmj.38250.571088.55 · Cited by 313 · see citing works
Fletcher, A., Jamal, F., Moore, G., Evans, R. E., Murphy, S., & Bonell, C. (2016) · Evaluation · doi:10.1177/1356389016652743 · Cited by 294 · see citing works
Publication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266 · see citing works
Causal Inference in Developmental Origins of Health and Disease (DOHaD) Research
Gage, S. H., Munafò, M. R., & Davey Smith, G. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033352 · Cited by 173 · see citing works
Willis, B. H., & Riley, R. D. (2017) · Statistics in Medicine · doi:10.1002/sim.7372 · Cited by 171 · see citing works
Martin, R. M., Patel, R., Kramer, M. S., Guthrie, L., Vilchuck, K., Bogdanovich, N., Sergeichick, N., Gusina, N., Foo, Y., Palmer, T., et al. (2013) · JAMA · doi:10.1001/jama.2013.167 · Cited by 168 · see citing works
Self-citationGray, R., Mukherjee, R. A. S., & Rutter, M. (2009) · Addiction · doi:10.1111/j.1360-0443.2008.02441.x · Cited by 104 · see citing works
When Will Mendelian Randomization Become Relevant for Clinical Practice and Public Health?
Davey Smith, G., Paternoster, L., & Relton, C. (2017) · JAMA · doi:10.1001/jama.2016.21189 · Cited by 87 · see citing works
Believability of relative risks and odds ratios in abstracts: Cross sectional study.
Gøtzsche, P. C. (2006) · BMJ · doi:10.1136/bmj.38895.410451.79 · Cited by 85 · see citing works
Wu, B.-S., Zhang, Y.-R., Li, H.-Q., Kuo, K., Chen, S.-D., Dong, Q., Liu, Y., & Yu, J.-T. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01599-x · Cited by 69 · see citing works
Trivasse, H., Webb, T. L., & Waller, G. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101887 · Cited by 51 · see citing works
Zheng, J., Wheeler, E., Pietzner, M., Andlauer, T. F. M., Yau, M. S., Hartley, A. E., Brumpton, B. M., Rasheed, H., Kemp, J. P., Frysz, M., et al. (2023) · Arthritis & Rheumatology · doi:10.1002/art.42538 · Cited by 44 · see citing works
Using Mendelian randomisation to infer causality in depression and anxiety research
Gage, S. H., Smith, G. D., Zammit, S., Hickman, M., & Munafò, M. R. (2013) · Depression and Anxiety · doi:10.1002/da.22150 · Cited by 32 · see citing works
Self-citation
- Authors
- Smith, G. D., & Ebrahim, S.
- Affiliations
- University of Bristol (GB)
- Corresponding author
- Smith, George Davey
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2002-12-21
- Volume
- 325
- Issue
- 7378
- Pages
- 1437-1438
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- PMC
- PMC1124898
- OpenAlex
- W2099817335
- Cited by
- 434
- References
- 2
- Citations by year
- 2026: 13 · 2025: 20 · 2024: 37 · 2023: 87 · 2022: 28 · 2021: 23 · 2020: 21 · 2019: 16 · 2018: 13 · 2017: 14 · 2016: 17 · 2015: 17 · 2014: 11 · 2013: 10 · 2012: 10
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- www.bmj.com/company/legal-information/terms-conditions/legal-information/tdm-licencepolicy (tdm)
- Locations
- BMJ (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- p-hacking, confounding, bias, breast cancer risk, alcohol consumption, smoking, health scares, hormone replacement therapy, coronary artery disease, media coverage of research
- MeSH terms
- Data Collection, Data Interpretation, Statistical, Life Style, Periodicals as Topic, Bias, Confounding Factors, Epidemiologic, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
