Journal prestige publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals
Abstract
CONTEXT: Citation by other authors is important in the dissemination of published science, but factors predicting it are little studied. METHODS: To identify characteristics of published research predicting citation in other journals, we searched the Science Citations Index database for a standardized 3.5 years for all citations of published articles originally submitted to a 1991 emergency medicine specialty meeting. Analysis was conducted by classification and regression trees, a nonparametric modeling technique of regression trees, to determine the impact of previously determined characteristics of the full articles on the outcome measures. We calculated the the number of times an article was cited each year and calculated the mean impact factor (citations per manuscript per year) in other citing journals. RESULTS: Of the 493 submitted manuscripts, 204 published articles met entry criteria. The mean citations per year was 2.04 (95% confidence interval, 1.6-2.4; range, 0-20.9) in 440 different journals. Nineteen articles (9.3%) were never cited. The ability to predict the citations per year was weak (pseudo R(2) = 0.14.). The strongest predictor of citations per year was the impact factor of the original publishing journal. The presence of a control group, the subjective newsworthiness score, and sample size predicted citation frequency (24.3%, 26.0%, and 26.5% as strongly, respectively). The ability to predict mean impact factor of the citing journals was even weaker (pseudo R(2) = 0.09). The impact factor of the publishing journal was the strongest predictor, followed by the newsworthiness score (89.9% as strongly) and a subjective quality score (61.5%). Positive outcome bias was not evident for either outcome measure. CONCLUSION: In this cohort of published research, commonly used measures of study methodology and design did not predict the frequency of citations or the importance of citing journals. Positive outcome bias was not evident. The impact factor of the original publishing journal was more important than any other variable, suggesting that the journal in which a study is published may be as important as traditional measures of study quality in ensuring dissemination.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The centrality of a journal to a field might mean that a weak document in a prestigious journal receives a considerable number of citations.
“The centrality of a journal to a field might mean that a weak document in a prestigious journal receives a considerable number of citations (Callaham et al., 2002).”
From Core Elements in the Process of Citing Publications: Conceptual Overview of the Literature (2018) · cites this paper as Callaham et al., 2002
Cites
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,834 · see citing works
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,028 · see citing works
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · see citing works
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · see citing works
Dickersin, K., Yuan, M.-Q., & Meinert, C. (1992) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.1992.03480030052036 · Cited by 900 · see citing works
Fortnightly Review: How can impact factors be improved?
Garfield, E. (1996) · BMJ · doi:10.1136/bmj.313.7054.411 · Cited by 491 · see citing works
Full publication of results initially presented in abstracts. A meta-analysis
Scherer, R. W. (1994) · JAMA · doi:10.1001/jama.272.2.158 · Cited by 403 · see citing works
Callaham, M. L. (1998) · JAMA · doi:10.1001/jama.280.3.254 · Cited by 304 · see citing works
Self-citationNIH clinical trials and publication bias.
Dickersin, K., & Min, Y. (1993) · PubMed · Cited by 293 · see citing works
Unpublished Research From a Medical Specialty Meeting
Weber, E. J., Callaham, M. L., Wears, R. L., Barton, C., & Young, G. P. (1998) · JAMA · doi:10.1001/jama.280.3.257 · Cited by 262 · see citing works
Self-citationBIAS AGAINST THE NULL HYPOTHESIS: THE REPRODUCTIVE HAZARDS OF COCAINE
Koren, G., Shear, H. H., Graham, K., & Einarson, T. R. (1989) · The Lancet · doi:10.1016/s0140-6736(89)92044-8 · Cited by 173 · see citing works
Fate of Cardiology Research Originally Published in Abstract Form
Goldman, L., & Loscalzo, A. (1980) · New England Journal of Medicine · doi:10.1056/nejm198007313030504 · Cited by 162 · see citing works
Measuring the Quality of Trials
Berlin, J. A., & Rennie, D. (1999) · JAMA · doi:10.1001/jama.282.11.1083 · Cited by 102 · see citing works
Random thoughts on citationology its theory and practice
Garfield, E. (1998) · Scientometrics · doi:10.1007/bf02458396 · Cited by 96 · see citing works
From Citation Indexes to Informetrics: Is the Tail Now Wagging the Dog ?
Garfield, E. (1998) · Libri · doi:10.1515/libr.1998.48.2.67 · Cited by 93 · see citing works
Garfield, E. (1982) · Medical Entomology and Zoology · Cited by 75 · see citing works
Publications Resulting from Anesthesia Meeting Abstracts
Meranze, J., Ellison, N., & Greenhow, D. E. (1982) · Anesthesia & Analgesia · doi:10.1213/00000539-198205000-00011 · Cited by 58 · see citing works
The Rise and Fall of Uncitedness
Schwartz, C. A. (1997) · College & Research Libraries · doi:10.5860/crl.58.1.19 · Cited by 57 · see citing works
Davies, H. D., Langley, J. M., & Speert, D. P. (1996) · PubMed · Cited by 24 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · see citing works
Delgado-Rodríguez, M., & Llorca, J. (2004) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2003.008466 · Cited by 1,064 · see citing works
Dissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039 · see citing works
Factors Affecting Number of Citations: A Comprehensive Review of the Literature
Tahamtan, I., Safipour Afshar, A., & Ahamdzadeh, K. (2016) · Scientometrics · doi:10.1007/s11192-016-1889-2 · Cited by 912 · see citing works
Deep impact: unintended consequences of journal rank
Brembs, B., Button, K., & Munafò, M. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00291 · Cited by 441 · see citing works
Non-publication of large randomized clinical trials: cross sectional analysis
Jones, C. W., Handler, L., Crowell, K. E., Keil, L. G., Weaver, M. A., & Platts-Mills, T. F. (2013) · BMJ · doi:10.1136/bmj.f6104 · Cited by 350 · see citing works
Testing for the Presence of Positive-Outcome Bias in Peer Review: A Randomized Controlled Trial
Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.406 · Cited by 245 · see citing works
Citation bias favoring statistically significant studies was present in medical research
Jannot, A.-S., Agoritsas, T., Gayet-Ageron, A., & Perneger, T. V. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.09.015 · Cited by 193 · see citing works
Scientific citations favor positive results: a systematic review and meta-analysis.
Duyx, B., Urlings, M. J. E., Swaen, G. M. H., Bouter, L. M., & Zeegers, M. P. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.06.002 · Cited by 197 · see citing works
Falagas, M. E., Zarkali, A., Karageorgopoulos, D. E., Bardakas, V., & Mavros, M. N. (2013) · PLoS ONE · doi:10.1371/journal.pone.0049476 · Cited by 168 · see citing works
Core Elements in the Process of Citing Publications: Conceptual Overview of the Literature
Tahamtan, I., & Bornmann, L. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.01.002 · Cited by 149 · see citing works
Concentration of the Most-Cited Papers in the Scientific Literature: Analysis of Journal Ecosystems
Ioannidis, J. P. A. (2006) · PLoS ONE · doi:10.1371/journal.pone.0000005 · Cited by 122 · see citing works
Antoniou, G. A., Antoniou, S. A., Georgakarakos, E. I., Sfyroeras, G. S., & Georgiadis, G. S. (2015) · Annals of Vascular Surgery · doi:10.1016/j.avsg.2014.09.017 · Cited by 77 · see citing works
Predicting the impact of scientific concepts using full-text features
McKeown, K., Daume, H., Chaturvedi, S., Paparrizos, J., Thadani, K., Barrio, P., Biran, O., Bothe, S., Collins, M., Fleischmann, K. R., et al. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23612 · Cited by 73 · see citing works
Determinants of the citation rate of medical research publications from a developing country
Annalingam, A., Damayanthi, H., Jayawardena, R., & Ranasinghe, P. (2014) · SpringerPlus · doi:10.1186/2193-1801-3-140 · Cited by 64 · see citing works
Citation impact was highly variable for reporting guidelines of health research: a citation analysis
Caulley, L., Cheng, W., Catalá-López, F., Whelan, J., Khoury, M., Ferraro, J., Husereau, D., Altman, D. G., & Moher, D. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.07.013 · Cited by 37 · see citing works
A retrospective analysis of dissemination biases in the brief alcohol intervention literature.
Tanner-Smith, E. E., & Polanin, J. R. (2015) · Psychology of Addictive Behaviors · doi:10.1037/adb0000014 · Cited by 26 · see citing works
Methodological quality in clinical trials and bibliometric indicators: no evidence of correlations
Akcan, D., Axelsson, S., Bergh, C., Davidson, T., & Rosén, M. (2013) · Scientometrics · doi:10.1007/s11192-013-0949-0 · Cited by 11 · see citing works
EVIDENCE-BASED IMPLANT DENTISTRY: METHODOLOGICAL FOUNDATIONS, PROGRESS, AND FUTURE PERSPECTIVES.
YAN, Q., WANG, Y., WU, X., ZHAO, Y., & ZHANG, Y. (2026) · Journal of Evidence-Based Dental Practice · doi:10.1016/j.jebdp.2026.102269
- Authors
- Callaham, M.
- Affiliations
- University of California, San Francisco (US)
- Corresponding author
- Callaham, Michael
- ORCID
- Callaham, Michael
- Venue
- JAMA
- Publisher
- American Medical Association (AMA)
- Published
- 2002-06-05
- Volume
- 287
- Issue
- 21
- Pages
- 2847
- ISSN
- 0002-9955 (ISSN-L)
- 0098-7484
- OpenAlex
- W2163978340
- Cited by
- 644
- References
- 19
- Citations by year
- 2026: 14 · 2025: 20 · 2024: 26 · 2023: 29 · 2022: 27 · 2021: 47 · 2020: 50 · 2019: 30 · 2018: 33 · 2017: 32 · 2016: 25 · 2015: 47 · 2014: 33 · 2013: 39 · 2012: 20
- FWCI
- 17.54
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- JAMA (publishedVersion)
- PubMed (publishedVersion)
- DigitalGeorgetown (Georgetown University Library) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- citation counts, journal impact factor, publication bias, citation prediction, newsworthiness, study quality, sample size, control groups, emergency medicine research, research dissemination, peer-reviewed journals
- MeSH terms
- Periodicals as Topic, Bibliometrics, Publication Bias, Peer Review, Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
