CoChat

Don't Let the Truth Get in the Way of a Good Story: An Illustration of Citation Bias in Epidemiologic Research

Open access
Kivimaki, M., Batty, G. D., Kawachi, I., Virtanen, M., Singh-Manoux, A., Brunner, E. J. · 2014 · American Journal of Epidemiology
Cited by 23

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Abstract

The production of scientific knowledge is susceptible to bias at every stage of the process, from what questions are asked by the investigator, to which method is chosen to gather data, to which analyses are conducted (e.g., “P-hacking,” wherein the method of statistical analysis and the degrees of freedom are manipulated until they yield statistically significant results) (1). Even after completion of a study, authors sometimes choose not to submit their work for publication because they are not satisfied with the results (i.e., the “file drawer” problem) (1), or they encounter difficulties with getting results published because of reviewer or editorial bias (“publication bias”) (2–4). Although prepublication biases have been well described in epidemiologic textbooks, postpublication biases, such as selective citation, have been less well documented. “Citation bias” occurs when scientists selectively cite papers based upon risk estimates that conform to their preconceived notions (5). When researchers have a bias in favor of “X causing Y,” they are more likely to cite papers that found evidence to support their view. Conversely, when researchers harbor a bias against a hypothesized association, they may selectively cite papers that report null findings. Here, we use research on job strain and the risk of coronary heart disease to examine factors that influence citations in peer-reviewed literature. In addition to the risk estimate for job strain relative to no job strain in each study, we take into account the impact factor of the publishing journal, which is an indicator of its prestige. We used the most recent meta-analysis of job strain and incident coronary heart disease to identify relevant studies for this analysis (6). According to the total evidence from this meta-analysis (26 studies), employees who experienced job strain had 1.34 (95% confidence interval (CI): 1.18, 1.51) times greater disease risk than did those free of job strain (6). To allow an adequate period of time for citations to accumulate, we focused on papers published at least 10 years ago, which yielded a total of 7 cohort studies (Table 1) (7–13). For each study, we obtained relative risk estimates for the job strain–heart disease association, counted citations in the Scopus (Elsevier, Amsterdam, the Netherlands) and Web-of-Science (Thompson Reuters, New York, New York) databases, and obtained the impact factor of the publishing journal from Web-of-Science Journal Citations 2013. In addition, we obtained an indicator of the scientific quality of each study from a review (14) in which the authors had based their evaluation on 8 criteria (e.g., the characteristics of the study population, validity of the exposure measurement and outcome ascertainment, and comprehensiveness of adjustments for confounding factors). A higher score indicated higher quality (range, 0–12) (14). Number of Citations, Effect Size, Journal Impact Factor, and Scientific Quality for Cohort Studies on Job Strain and Coronary Heart Disease Published From 1989 to 2004 Abbreviation: CI, confidence interval. a Citations as of January 25, 2014. b Web of Science journal impact factor for 2013. c Quality score (range, 0–12) was obtained from a previous review (14). The quality score for Kuper et al. (12) is missing because it was not included in that review. Number of Citations, Effect Size, Journal Impact Factor, and Scientific Quality for Cohort Studies on Job Strain and Coronary Heart Disease Published From 1989 to 2004 Abbreviation: CI, confidence interval. a Citations as of January 25, 2014. b Web of Science journal impact factor for 2013. c Quality score (range, 0–12) was obtained from a previous review (14). The quality score for Kuper et al. (12) is missing because it was not included in that review. We computed the associations of effect size and journal impact factor with the number of citations using general linear models (procedure glm in Stata, version 11.2; StataCorp LP, College Station, Texas). Both analyses were adjusted for citation-time (the time period during which the study could have been cited). The total number of citations for the 7 studies was 1,382 in Scopus and 1,305 in Web-of-Science. In the most-cited paper, job strain was reported to double the disease risk (Table 1) (11). In contrast, the least-cited paper reported a null finding (10). There was a general trend toward higher frequencies of citations for papers that reported a higher risk ratio (7–13). After controlling for citation-time, we found that each 10% increase in reported excess risk was associated with 16.5 (95% CI: 7.9, 25.0) additional citations in Scopus (P = 0.001, adjusted R2 = 0.69) and 12.4 (95% CI: 1.1, 23.7) additional citations in Web-of-Science (P = 0.03, adjusted R2 = 0.31). Moreover, a higher journal impact factor was associated with a higher citation frequency, although this was imprecisely estimated, as evidenced by the wide confidence intervals (in Scopus, per eac

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/aje/kwu164

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 7 references

Cited by

See all 23

Metadata

Authors
Kivimaki, M., Batty, G. D., Kawachi, I., Virtanen, M., Singh-Manoux, A., & Brunner, E. J.
Venue
American Journal of Epidemiology
Published
2014-07-02
Volume
180
Issue
4
Pages
446-448
ISSN
  • 0002-9262 (ISSN-L)
  • 1476-6256
OpenAlex
W2106032541
Cited by
23
References
7
FWCI
4.16
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Don't Let the Truth Get in the Way of a Good Story: An Illustration of Citation Bias in Epidemiologic Research. https://app.cochat.ai/papers/10.1093/aje/kwu164
The paper
Kivimaki, M., Batty, G. D., Kawachi, I., Virtanen, M., Singh-Manoux, A., & Brunner, E. J. (2014). Don't Let the Truth Get in the Way of a Good Story: An Illustration of Citation Bias in Epidemiologic Research. American Journal of Epidemiology. https://doi.org/10.1093/aje/kwu164

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.