Don't Let the Truth Get in the Way of a Good Story: An Illustration of Citation Bias in Epidemiologic Research
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
The production of scientific knowledge is susceptible to bias at every stage of the process, from what questions are asked by the investigator, to which method is chosen to gather data, to which analyses are conducted (e.g., “P-hacking,” wherein the method of statistical analysis and the degrees of freedom are manipulated until they yield statistically significant results) (1). Even after completion of a study, authors sometimes choose not to submit their work for publication because they are not satisfied with the results (i.e., the “file drawer” problem) (1), or they encounter difficulties with getting results published because of reviewer or editorial bias (“publication bias”) (2–4). Although prepublication biases have been well described in epidemiologic textbooks, postpublication biases, such as selective citation, have been less well documented. “Citation bias” occurs when scientists selectively cite papers based upon risk estimates that conform to their preconceived notions (5). When researchers have a bias in favor of “X causing Y,” they are more likely to cite papers that found evidence to support their view. Conversely, when researchers harbor a bias against a hypothesized association, they may selectively cite papers that report null findings. Here, we use research on job strain and the risk of coronary heart disease to examine factors that influence citations in peer-reviewed literature. In addition to the risk estimate for job strain relative to no job strain in each study, we take into account the impact factor of the publishing journal, which is an indicator of its prestige. We used the most recent meta-analysis of job strain and incident coronary heart disease to identify relevant studies for this analysis (6). According to the total evidence from this meta-analysis (26 studies), employees who experienced job strain had 1.34 (95% confidence interval (CI): 1.18, 1.51) times greater disease risk than did those free of job strain (6). To allow an adequate period of time for citations to accumulate, we focused on papers published at least 10 years ago, which yielded a total of 7 cohort studies (Table 1) (7–13). For each study, we obtained relative risk estimates for the job strain–heart disease association, counted citations in the Scopus (Elsevier, Amsterdam, the Netherlands) and Web-of-Science (Thompson Reuters, New York, New York) databases, and obtained the impact factor of the publishing journal from Web-of-Science Journal Citations 2013. In addition, we obtained an indicator of the scientific quality of each study from a review (14) in which the authors had based their evaluation on 8 criteria (e.g., the characteristics of the study population, validity of the exposure measurement and outcome ascertainment, and comprehensiveness of adjustments for confounding factors). A higher score indicated higher quality (range, 0–12) (14). Number of Citations, Effect Size, Journal Impact Factor, and Scientific Quality for Cohort Studies on Job Strain and Coronary Heart Disease Published From 1989 to 2004 Abbreviation: CI, confidence interval. a Citations as of January 25, 2014. b Web of Science journal impact factor for 2013. c Quality score (range, 0–12) was obtained from a previous review (14). The quality score for Kuper et al. (12) is missing because it was not included in that review. Number of Citations, Effect Size, Journal Impact Factor, and Scientific Quality for Cohort Studies on Job Strain and Coronary Heart Disease Published From 1989 to 2004 Abbreviation: CI, confidence interval. a Citations as of January 25, 2014. b Web of Science journal impact factor for 2013. c Quality score (range, 0–12) was obtained from a previous review (14). The quality score for Kuper et al. (12) is missing because it was not included in that review. We computed the associations of effect size and journal impact factor with the number of citations using general linear models (procedure glm in Stata, version 11.2; StataCorp LP, College Station, Texas). Both analyses were adjusted for citation-time (the time period during which the study could have been cited). The total number of citations for the 7 studies was 1,382 in Scopus and 1,305 in Web-of-Science. In the most-cited paper, job strain was reported to double the disease risk (Table 1) (11). In contrast, the least-cited paper reported a null finding (10). There was a general trend toward higher frequencies of citations for papers that reported a higher risk ratio (7–13). After controlling for citation-time, we found that each 10% increase in reported excess risk was associated with 16.5 (95% CI: 7.9, 25.0) additional citations in Scopus (P = 0.001, adjusted R2 = 0.69) and 12.4 (95% CI: 1.1, 23.7) additional citations in Web-of-Science (P = 0.03, adjusted R2 = 0.31). Moreover, a higher journal impact factor was associated with a higher citation frequency, although this was imprecisely estimated, as evidenced by the wide confidence intervals (in Scopus, per eac
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Selective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · cited 2×
“…process, from what questions are asked by the investigator, to which method is chosen to gather data, to which analyses are conducted (e.g., “ P-hacking, ” wherein the method of statistical analysis and the degrees of freedom are manipulated until they yield statistically signi fi cant results) (1).”
BackgroundMethod(2012) · doi:10.1016/s0140-6736(12)60994-5 · Cited by 1,001
Self-citationCallaham, M. (2002) · JAMA · doi:10.1001/jama.287.21.2847 · Cited by 644 · cited 2×
“(2) found that the strength of methodology and study design did not predict the frequency of citations during a 3.”
BackgroundEaker, E. D., Pinsky, J., & Castelli, W. P. (1992) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116381 · Cited by 386
Work-Related Psychosocial Factors and the Development of Ischemic Heart Disease: A Systematic Review
Eller, N. H., Netterstrøm, B., Gyntelberg, F., Kristensen, T. S., Nielsen, F., Steptoe, A., & Theorell, T. (2009) · Cardiology in Review · doi:10.1097/crd.0b013e318198c8e9 · Cited by 323 · cited 4×
“Study quality was determined based on a score obtained from an independent review (14); unfortunately, this score was missing for one of the target papers (12).”
BackgroundMethodInfluentialCitation bias favoring statistically significant studies was present in medical research
Jannot, A.-S., Agoritsas, T., Gayet-Ageron, A., & Perneger, T. V. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.09.015 · Cited by 193
Cited by
Work Stress as a Risk Factor for Cardiovascular Disease
Kivimäki, M., & Kawachi, I. (2015) · Current Cardiology Reports · doi:10.1007/s11886-015-0630-8 · Cited by 518
Self-citationWhen Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298
Scientific citations favor positive results: a systematic review and meta-analysis.
Duyx, B., Urlings, M. J. E., Swaen, G. M. H., Bouter, L. M., & Zeegers, M. P. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.06.002 · Cited by 197
Why do we think we know what we know? A metaknowledge analysis of the salt controversy
Trinquart, L., Johns, D. M., & Galea, S. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv184 · Cited by 77
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
Affirmative citation bias in scientific myth debunking: A three-in-one case study
Letrud, K., & Hernes, S. (2019) · PLOS ONE · doi:10.1371/journal.pone.0222213 · Cited by 37
Metadata
- Authors
- Kivimaki, M., Batty, G. D., Kawachi, I., Virtanen, M., Singh-Manoux, A., & Brunner, E. J.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-07-02
- Volume
- 180
- Issue
- 4
- Pages
- 446-448
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- OpenAlex
- W2106032541
- Cited by
- 23
- References
- 7
- FWCI
- 4.16
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Don't Let the Truth Get in the Way of a Good Story: An Illustration of Citation Bias in Epidemiologic Research. https://app.cochat.ai/papers/10.1093/aje/kwu164
- The paper
- Kivimaki, M., Batty, G. D., Kawachi, I., Virtanen, M., Singh-Manoux, A., & Brunner, E. J. (2014). Don't Let the Truth Get in the Way of a Good Story: An Illustration of Citation Bias in Epidemiologic Research. American Journal of Epidemiology. https://doi.org/10.1093/aje/kwu164
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.