The incidence and role of negative citations in science
Open accessAbstract
Significance: Providing a detailed classification of the types of citations that an article receives is important to establish the quality of a study and to characterize how current research builds upon prior work. The methodology that we propose also informs how to improve the citation process, for example by having scientists attach additional metadata to their citations. The approach is scalable to other fields and periods and can also be used to identify other types of citations (e.g., reuse of methods, materials, empirical tests of theory, and so on). Finally, our methods provide online repositories such as Google Scholar, PubMed, ISI Web of Science, and Scopus with a way to improve their search and ranking algorithms.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In a recent study, Catalini et al. (2015) analyzed citations from papers in the Journal of Immunology and identified 2.4% of the total number of citations as negative.
“In a recent study, Catalini et al. (2015) analyzed citations from papers in the Journal of Immunology and identified 2.4% of the total number of citations as negative.”
From Core Elements in the Process of Citing Publications: Conceptual Overview of the Literature (2018) · cites this paper as Catalini et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
<i>The Structure of Scientific Revolutions</i>
Kuhn, T. S., & Schlegel, R. (1963) · Physics Today · doi:10.1063/1.3050879 · Cited by 35,538 · see citing works
Endogenous Technological Change
Romer, P. M. (1990) · Journal of Political Economy · doi:10.1086/261725 · Cited by 17,711 · see citing works
Endogenous Technological Change
Romer, P. (1989) · National Bureau of Economic Research · doi:10.3386/w3210 · Cited by 13,312 · see citing works
The Logic of Scientific Discovery
Bhaskar, R. (2013) · Routledge eBooks · doi:10.4324/9780203090732-9 · Cited by 7,765 · see citing works
The Structure of Scientific Revolutions
Hall, M. B., & Kuhn, T. S. (1963) · The American Historical Review · doi:10.2307/1847040 · Cited by 7,753 · see citing works
The Matthew Effect in Science: The reward and communication systems of science are considered.
Merton, R. K. (1968) · Science · doi:10.1126/science.159.3810.56 · Cited by 6,720 · see citing works
(1968) · PubMed · Cited by 4,803 · see citing works
Causal Inference without Balance Checking: Coarsened Exact Matching
Iacus, S. M., King, G., & Porro, G. (2012) · Political Analysis · doi:10.1093/pan/mpr013 · Cited by 3,563 · see citing works
Market value and patent citationsPreprint
Hall, B. H., Jaffe, A. B., & Trajtenberg, M. (2005) · RePEc: Research Papers in Economics · Cited by 3,314 · see citing works
A Penny for Your Quotes: Patent Citations and the Value of Innovations
Trajtenberg, M. (1990) · The RAND Journal of Economics · doi:10.2307/2555502 · Cited by 3,041 · see citing works
The Logic of Scientific Discovery.
Rescher, N., Popper, K. R., & Hanson, N. R. (1960) · Philosophy and Phenomenological Research · doi:10.2307/2104331 · Cited by 2,455 · see citing works
Toward a new economics of science
Partha, D., & David, P. A. (1994) · Research Policy · doi:10.1016/0048-7333(94)01002-1 · Cited by 2,448 · see citing works
Coauthorship networks and patterns of scientific collaboration
Newman, M. E. J. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0307545100 · Cited by 1,948 · see citing works
Priorities in Scientific Discovery: A Chapter in the Sociology of Science
Merton, R. Κ. (1957) · American Sociological Review · doi:10.2307/2089193 · Cited by 1,615 · see citing works
Citation Frequency and the Value of Patented Inventions
Harhoff, D., Narin, F., Scherer, F. M., & Vopel, K. (1999) · The Review of Economics and Statistics · doi:10.1162/003465399558265 · Cited by 1,516 · see citing works
The increasing linkage between U.S. technology and public science
Narin, F., Hamilton, K., & Olivastro, D. (1997) · Research Policy · doi:10.1016/s0048-7333(97)00013-9 · Cited by 1,307 · see citing works
Collaborative Networks as Determinants of Knowledge Diffusion Patterns
Singh, J. (2005) · Management Science · doi:10.1287/mnsc.1040.0349 · Cited by 1,192 · see citing works
Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]
Cambria, E., & White, B. (2014) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2014.2307227 · Cited by 1,106 · see citing works
Network structure and the diffusion of knowledge
Cowan, R., & Jonard, N. (2003) · Journal of Economic Dynamics and Control · doi:10.1016/j.jedc.2003.04.002 · Cited by 927 · see citing works
Stephan, P. E. (2010) · Handbook of the economics of innovation · doi:10.1016/s0169-7218(10)01005-1 · Cited by 918 · see citing works
Gittelman, M., & Kogut, B. (2003) · Management Science · doi:10.1287/mnsc.49.4.366.14420 · Cited by 686 · see citing works
Small Worlds and Regional Innovation
Fleming, L., King, C., & Juda, A. I. (2007) · Organization Science · doi:10.1287/orsc.1070.0289 · Cited by 660 · see citing works
Jumping NLP Curves: A Review of Natural Language Processing Research
Cambria, E., & White, B. (2014) · Cited by 455 · see citing works
Classificatory Notes on the Production and Transmission of Technological Knowledge
Arrow, K. J. (1969) · Cited by 403 · see citing works
Restructuring Research: Communication Costs and the Democratization of University Innovation
Agrawal, A. K., & Goldfarb, A. (2008) · American Economic Review · doi:10.1257/aer.98.4.1578 · Cited by 280 · see citing works
Models of Technical Progress and the Golden Rule of Research
Phelps, E. S. (1966) · The Review of Economic Studies · doi:10.2307/2974437 · Cited by 213 · see citing works
Knowledge Networks from Patent Data
Breschi, S., & Lissoni, F. (2004) · Springer eBooks · doi:10.1007/1-4020-2755-9_29 · Cited by 197 · see citing works
The Retraction Penalty: Evidence from the Web of Science
Lu, S. F., Jin, G. Z., Uzzi, B., & Jones, B. (2013) · Scientific Reports · doi:10.1038/srep03146 · Cited by 191 · see citing works
Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine
Furman, J. L., Jensen, K., & Murray, F. (2012) · Research Policy · doi:10.1016/j.respol.2011.11.001 · Cited by 151 · see citing works
Science controversies past and present
Sherwood, S. C. (2011) · Physics Today · doi:10.1063/pt.3.1295 · Cited by 50 · see citing works
Fraud: retracted articles are still being cited
Campanario, J. M. (2000) · Nature · doi:10.1038/35042753 · Cited by 28 · see citing works
Azoulay, P., Furman, J. L., Krieger, J., & Murray, F. (2012) · National Bureau of Economic Research · doi:10.3386/w18499 · Cited by 11 · see citing works
. . . so animals could pull their weight, and more
Goe, M. R. (2000) · Nature · doi:10.1038/35042749 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Merton RK (1968) The Matthew effect in science. Science 159(3810):56-63.
Cited by
Sorted by citations
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019) · Sage Open · doi:10.1177/2158244019829575 · Cited by 1,257 · see citing works
Core Elements in the Process of Citing Publications: Conceptual Overview of the Literature
Tahamtan, I., & Bornmann, L. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.01.002 · Cited by 149 · see citing works
Suelzer, E. M., Deal, J., Hanus, K. L., Ruggeri, B., Sieracki, R., & Witkowski, E. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.15552 · Cited by 69 · see citing works
Bordignon, F. (2020) · Scientometrics · doi:10.1007/s11192-020-03536-z · Cited by 51 · see citing works
The linguistic patterns and rhetorical structure of citation context: An approach using n-grams
Bertin, M., Atanassova, I., Sugimoto, C. R., & Lariviere, V. (2016) · Scientometrics · doi:10.1007/s11192-016-2134-8 · Cited by 50 · see citing works
Guilt by association: How scientific misconduct harms prior collaborators
Hussinger, K., & Pellens, M. (2019) · Research Policy · doi:10.1016/j.respol.2018.01.012 · Cited by 49 · see citing works
Mott, A., Fairhurst, C., & Torgerson, D. (2019) · Journal of Health Services Research & Policy · doi:10.1177/1355819618797965 · Cited by 36 · see citing works
Dal-Ré, R., & Ayuso, C. (2021) · Accountability in Research · doi:10.1080/08989621.2020.1835479 · Cited by 21 · see citing works
“Some 2% (Catalini, Lacetera, and Oettl 2015) of citations are negative, although this figure could be higher in specific disciplines [e.g., 7% in radiation oncology (Hamilton 2019) or 10% in dentistry (Rapani et al. 2020)].”
Background
- Authors
- Catalini, C., Lacetera, N., & Oettl, A.
- Affiliations
- Massachusetts Institute of Technology (US)
- University of Toronto (CA)
- Georgia Institute of Technology (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2015-10-26
- Volume
- 112
- Issue
- 45
- Pages
- 13823-13826
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4653214
- OpenAlex
- W1878144120
- Cited by
- 189
- References
- 33
- Citations by year
- 2026: 8 · 2025: 7 · 2024: 18 · 2023: 20 · 2022: 23 · 2021: 16 · 2020: 36 · 2019: 18 · 2018: 16 · 2017: 12 · 2016: 11 · 2015: 4
- FWCI
- 13.08
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, other-oa)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- citation analysis, bibliometric methodology, natural language processing, citation quality, diffusion of scientific knowledge, social distance, scholarly communication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
