CoChat
Sign Up
Cited by 1

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Citation accuracy, citation noise, and citation bias: a foundation of citation analysis

Bornmann, Lutz, Leibel, Christian·2026·Scientometrics
DOI 10.1007/s11192-026-05841-5
Open this paper in CoChat — free account

Abstract

Citation analysis is widely used in research evaluation to assess the impact of scientific papers. These analyses rest on the assumption that citation decisions by authors are accurate, representing the flow of knowledge from cited to citing papers. However, in practice, researchers often cite for reasons that are not related to the fact that there has been (intellectual) input from previous papers. Citations made for rhetorical reasons or without reading the cited work compromise the value of citations as instrument for research evaluation. Past research on threats to the accuracy of citations has mainly focused on citation bias as the primary concern. In this paper, we argue that citation noise—the undesirable variance in citation decisions—represents an equally critical but underexplored challenge in citation analysis. We define and differentiate two types of citation noise: citation level noise and citation pattern noise. Each type of noise is described in terms of how it arises and the specific ways it can undermine the validity of citation-based research assessments. By conceptually differing citation noise from citation accuracy and citation bias, we propose a framework for the foundation of citation analysis. We discuss strategies and interventions to minimize citation noise, aiming to improve the reliability and validity of citation analysis in research evaluation. We recommend that the current professional reform movement in research evaluation such as the Coalition for Advancing Research Assessment (CoARA) pick up these strategies and interventions as an additional building block for careful, responsible use of bibliometric indicators in research evaluation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11192-026-05841-5

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 70

  1. Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: An overview of basic concepts and theories. Sage Open, 9(1). https://doi.org/10.1177/2158244019829575 · DOI · Paper page
  2. Algaba, A., Mazijn, C., Holst, V., Tori, F., Wenmackers, S., & Ginis, V. (2024). Large language models reflect human citation patterns with a heightened citation bias. Retrieved May 31, 2024, from https://arxiv.org/abs/2405.15739 · arXiv
  3. Aman, V., & Gläser, J. (2025). Investigating knowledge flows in scientific communities: The potential of bibliometric methods. Minerva, 63(1), 155–182. https://doi.org/10.1007/s11024-024-09542-2 · DOI
  4. American Psychological Association. (2020). Publication manual of the American Psychological Association (7. ed.). American Psychological Association (APA).
  5. Barnes, S. B., & Dolby, R. G. (1970). The scientific ethos: A deviant viewpoint. European Journal of Sociology/Archives Européennes De Sociologie, 11(1), 3–25. · DOI
  6. Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., Shiffrin, R. M., Gershman, S. J., Popov, V., Bender, E. M., Marelli, M., Botvinick, M. M., Akata, Z., & Schulz, E. (2025). How should the advancement of large language models affect the practice of science? Proceedings of the National Academy of Sciences of the United States of America, 122(5), Article e2401227121. https://doi.org/10.1073/pnas.2401227121 · DOI · Paper page
  7. Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80. https://doi.org/10.1108/00220410810844150 · DOI · Paper page
  8. Bornmann, L., & Marewski, J. N. (2019). Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation. Scientometrics, 120(2), 419–459. https://doi.org/10.1007/s11192-019-03018-x · DOI
  9. Bruton, S. V., Macchione, A. L., Brown, M., & Hosseini, M. (2025). Citation ethics: An exploratory survey of norms and behaviors. Journal of Academic Ethics, 23, 329–346. https://doi.org/10.1007/s10805-024-09539-2 · DOI
  10. Cawkell, A. E. (1969). Citation noise. Aslib Proceedings, 21(11), 467.
  11. Chen, B., Murray, D., Liu, Y., & Barabási, A.-L. (2024). The origin, consequence, and visibility of criticism in science. Retrieved December 19, 2024, from https://arxiv.org/abs/2412.02809 · arXiv
  12. Donner, P., Stahlschmidt, S., Haunschild, R., & Bornmann, L. (2025). Does citation context information enhance the validity of citation analysis for measuring research quality? An empirical comparison of peer assessments and enriched citations. Quantitative Science Studies, 6, 967–987. https://doi.org/10.1162/qss.a.15 · DOI
  13. Dougherty, M. R., Illingworth, D. A., & Nguyen, R. (2024). A memory-theoretic account of citation propagation. Royal Society Open Science, 11(5), Article 231521. https://doi.org/10.1098/rsos.231521 · DOI
  14. Enkhbayar, A. (2025). Citational matters: Towards a performative theory of citations. Art and Technology, Simon Fraser University.
  15. Fisher, R. A. (1925). Statistical methods for research workers. Oliver & Boyd.
  16. Galton, F. (1907). Vox populi. Nature, 75, 450–451. https://doi.org/10.1038/075450a0 · DOI · Paper page
  17. Gavras, H. (2002). Inappropriate attribution: The “lazy author syndrome.” American Journal of Hypertension, 15(9), 831. https://doi.org/10.1016/s0895-7061(02)02989-8 · DOI · Paper page
  18. Gigerenzer, G., & Brighton, H. (2009). Homo heuristicus: Why biased minds make better inferences. Topics in Cognitive Science, 1(1), 107–143. https://doi.org/10.1111/j.1756-8765.2008.01006.x · DOI · Paper page
  19. Gilbert, G. N. (1977). Referencing as persuasion. Social Studies of Science, 7(1), 113–122. · DOI · Paper page
  20. Hammarfelt, B., Rushforth, A., & de Rijcke, S. (2020). Temporality in academic evaluation: ‘Trajectoral thinking’ in the assessment of biomedical researchers. Valuation Studies, 7(1), 33-63. https://doi.org/10.3384/VS.2001-5992.2020.7.1.33 · DOI
  21. Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a · DOI · Paper page
  22. Hulek, K., & Teschke, O. (2023). How do mathematicians publish? Some trends. European Mathematical Society Magazine, 129, 36–41. https://doi.org/10.4171/MAG/160 · DOI
  23. Jaffe, A. B., Trajtenberg, M., & Fogarty, M. S. (2000). Knowledge spillovers and patent citations: Evidence from a survey of inventors. American Economic Review, 90(2), 215–218. https://doi.org/10.1257/aer.90.2.215 · DOI
  24. Jannot, A. S., Agoritsas, T., Gayet-Ageron, A., & Perneger, T. V. (2013). Citation bias favoring statistically significant studies was present in medical research. Journal of Clinical Epidemiology, 66(3), 296–301. https://doi.org/10.1016/j.jclinepi.2012.09.015 · DOI · Paper page
  25. Jergas, H., & Baethge, C. (2015). Quotation accuracy in medical journal articles: A systematic review and meta-analysis. PeerJ, 3, Article e1364. https://doi.org/10.7717/peerj.1364 · DOI · Paper page
  26. Kahneman, D., & Frueh, S. (2022). Try to design an approach to making a judgment; don’t just go into it trusting your intuition. Issues in Science and Technology, 38(3), 23–26.
  27. Kahneman, D., Sibony, O., & Sunstein, C. R. (2021). Noise: A flaw in human judgment. Little, Brown.
  28. Khan, S., Banu, S. A., Pawde, A., Kumar, R., Akash, S., Dhama, K., & Amarpal, A. (2023). ChatGPT and artificial hallucinations in stem cell research: Assessing the accuracy of generated references – a preliminary study. Annals of Medicine and Surgery, 85, 5275–5278. https://doi.org/10.1097/MS9.0000000000001228 · DOI · Paper page
  29. Kim, L., Adolph, C., West, J. D., & Stovel, K. (2020). The influence of changing marginals on measures of inequality in scholarly citations: Evidence of bias and a resampling correction. Sociological Science, 7, 314–341. https://doi.org/10.15195/v7.a13 · DOI
  30. Klitzing, N., Hoekstra, R., & Strijbos, J.-W. (2019). Literature practices: Processes leading up to a citation. Journal of Documentation, 75(1), 62–77. https://doi.org/10.1108/JD-03-2018-0047 · DOI
  31. Koffi, M. (2025). Innovative ideas and gender (in)equality. American Economic Review, 115(7), 2207–2236. https://doi.org/10.1257/aer.20211811 · DOI
  32. Kousha, K., & Thelwall, M. (2024). Factors associating with or predicting more cited or higher quality journal articles: An Annual Review of Information Science and Technology (ARIST) paper. Journal of the Association for Information Science and Technology, 75(3), 215–244. https://doi.org/10.1002/asi.24810 · DOI
  33. Lawson, K. M., Murphy, B. A., Azpeitia, J., Lombard, E. J., & Pope, T. J. (2025). Citing decisions in psychology: A roadblock to cumulative and inclusive science. Advances in Methods and Practices in Psychological Science, 8(3), 1–22. https://doi.org/10.1177/25152459251351287 · DOI
  34. Masic, I. (2013). The importance of proper citation of references in biomedical articles. Acta Informatica Medica, 21(3), 148–155. https://doi.org/10.5455/aim.2013.21.148-155 · DOI
  35. Meho, L., & Yang, K. (2007). Fusion approach to citation-based quality assessment. In D. Torres‑Salinas & H. F. Moed (Eds.), Proceedings of ISSI 2007: 11th international conference of the International Society for Scientometrics and Informetrics (pp. 568–581). Centre for Scientific Information and Documentation (CINDOC), CSIC, Madrid, and ECOOM, KU Leuven.
  36. Merton, R. K. (1965). On the shoulders of giants: The post-italianate edition. University of Chicago Press.
  37. Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. University of Chicago Press.
  38. Milojević, S. (2025). Science of science. Scientometrics, 130(6), 3195–3211. https://doi.org/10.1007/s11192-025-05322-1 · DOI
  39. Mogull, S. A. (2017). Accuracy of cited “facts” in medical research articles: A review of study methodology and recalculation of quotation error rate. PLoS ONE, 12(9), Article e0184727. https://doi.org/10.1371/journal.pone.0184727 · DOI · Paper page
  40. Mohammadi, E., Thelwall, M., & Kousha, K. (2016). Can Mendeley bookmarks reflect readership? A survey of user motivations. Journal of the Association for Information Science and Technology, 67(5), 1198–1209. https://doi.org/10.1002/asi.23477 · DOI
  41. Murphy, E. J. (2011). Citations: The rules they didn’t teach you. Lipids, 46(4), 307–309. https://doi.org/10.1007/s11745-011-3543-3 · DOI
  42. Mutz, R., Wolbring, T., & Daniel, H.-D. (2017). The effect of the “very important paper” (VIP) designation in Angewandte Chemie International Edition on citation impact: A propensity score matching analysis. Journal of the Association for Information Science and Technology, 68(9), 2139–2153. https://doi.org/10.1002/asi.23701 · DOI
  43. Nicolaisen, J. (2004). Social behavior and scientific practice-missing pieces of the citation puzzle. Royal School of Library and Information Science.
  44. Nielsen, M. W., & Andersen, J. P. (2021). Global citation inequality is on the rise. Proceedings of the National Academy of Sciences of the United States of America, 118(7), Article e2012208118. https://doi.org/10.1073/pnas.2012208118 · DOI
  45. Penders, B. (2018). Ten simple rules for responsible referencing. PLoS Computational Biology, 14(4), Article e1006036. https://doi.org/10.1371/journal.pcbi.1006036 · DOI
  46. Price, D. D. (1976). General theory of bibliometric and other cumulative advantage processes. Journal of the American Society for Information Science, 27(5), 292–306. https://doi.org/10.1002/asi.4630270505 · DOI
  47. Rushforth, A., & Hammarfelt, B. (2023). The rise of responsible metrics as a professional reform movement: A collective action frames account. Quantitative Science Studies, 4(4), 879–897. https://doi.org/10.1162/qss_a_00280 · DOI
  48. Schilling, M. A., & Green, E. (2011). Recombinant search and breakthrough idea generation: An analysis of high impact papers in the social sciences. Research Policy, 40(10), 1321–1331. https://doi.org/10.1016/j.respol.2011.06.009 · DOI
  49. Small, H. (2004). On the shoulders of Robert Merton: Towards a normative theory of citation. Scientometrics, 60(1), 71–79. https://doi.org/10.1023/B:SCIE.0000027310.68393.bc · DOI · Paper page
  50. Small, H. G. (1978). Cited documents as concept symbols. Social Studies of Science, 8(3), 327–340. · DOI · Paper page
  51. Smith, D. (2014). Finding the signal in the noise of patent citations: How to focus on relevance for strategic advantage. Technology Innovation Management Review, 4(9), 36–44. https://doi.org/10.22215/timreview/830 · DOI
  52. Sula, C. A., & Miller, M. (2014). Citations, contexts, and humanistic discourse: Toward automatic extraction and classification. Literary and Linguistic Computing, 29(3), 452–464. https://doi.org/10.1093/llc/fqu019 · DOI
  53. Surowiecki, J. (2004). The wisdom of crowds. Random House.
  54. Tahamtan, I., & Bornmann, L. (2018). Core elements in the process of citing publications: Conceptual overview of the literature. Journal of Informetrics, 12(1), 203–216. https://doi.org/10.1016/j.joi.2018.01.002 · DOI · Paper page
  55. Tahamtan, I., & Bornmann, L. (2019). What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018. Scientometrics, 121(3), 1635–1684. https://doi.org/10.1007/s11192-019-03243-4 · DOI · Paper page
  56. Tahamtan, I., & Bornmann, L. (2022). The social systems citation theory (SSCT): A proposal to use the social systems theory for conceptualizing publications and their citation links. Profesional De La Información, 31(4), Article e310411. https://doi.org/10.3145/epi.2022.jul.11 · DOI
  57. Tang, B. L. (2023). Some insights into the factors influencing continuous citation of retracted scientific papers. Publications, 11(4), Article 47. https://doi.org/10.3390/publications11040047 · DOI
  58. Teplitskiy, M., Duede, E., Menietti, M., & Lakhani, K. R. (2022). How status of research papers affects the way they are read and cited. Research Policy, 51(4), Article 104484. https://doi.org/10.1016/j.respol.2022.104484 · DOI
  59. Thelwall, M. (2018). Early Mendeley readers correlate with later citation counts. Scientometrics, 115(3), 1231–1240. https://doi.org/10.1007/s11192-018-2715-9 · DOI · Paper page
  60. Thelwall, M. (2024). Quantitative methods in research evaluation: Citation indicators, altmetrics, and artificial intelligence. Retrieved July 5, 2024, from https://arxiv.org/abs/2407.00135 · arXiv
  61. Thelwall, M., Kousha, K., Stuart, E., Makita, M., Abdoli, M., Wilson, P., & Levitt, J. (2023). In which fields are citations indicators of research quality? Journal of the Association for Information Science and Technology, 74(8), 941–953. https://doi.org/10.1002/asi.24767 · DOI
  62. Traag, V. A., & Waltman, L. (2022). Causal foundations of bias, disparity and fairness. Retrieved August 4, 2022, from https://arxiv.org/abs/2207.13665 · arXiv
  63. van Raan, A. F. J. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143. https://doi.org/10.1007/s11192-005-0008-6 · DOI
  64. Varga, A. (2022). The narrowing of literature use and the restricted mobility of papers in the sciences. Proceedings of the National Academy of Sciences of the United States of America, 119(17), Article e2117488119. https://doi.org/10.1073/pnas.2117488119 · DOI
  65. Wakeling, S., Paramita, M. L., & Pinfield, S. (2025). How do authors perceive the way their work is cited? Findings from a large-scale survey on quotation accuracy. Journal of the Association for Information Science and Technology, 76(10), 1396–1410. https://doi.org/10.1002/asi.70000 · DOI
  66. Wang, D., & Barabási, A.-L. (2021). The science of science. Cambridge University Press. https://doi.org/10.1017/9781108610834 · DOI
  67. Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., Anandkumar, A., Bergen, K., Gomes, C. P., Ho, S., Kohli, P., Lasenby, J., Leskovec, J., Liu, T.-Y., Manrai, A., … Zitnik, M. (2023). Scientific discovery in the age of artificial intelligence. Nature, 620(7972), 47–60. https://doi.org/10.1038/s41586-023-06221-2 · DOI · Paper page
  68. Wei, F. F., Zhang, G. J., Zhang, L., Liang, Y. K., & Wu, J. B. (2019). Decreasing the noise of scientific citations in patents to measure knowledge flow. In G. Catalano, C. Daraio, M. Gregori, H. F. Moed, & G. Ruocco (Eds.), Proceedings of the 17th international conference of the International Society for Scientometrics and Informetrics (ISSI) on Scientometrics and Informetrics (pp. 1662–1669). Edizioni Efesto.
  69. Wouters, P. (1998). The signs of science. Scientometrics, 41(1–2), 225–241. · DOI
  70. Wouters, P. (1999). Beyond the holy grail: From citation theory to indicator theories. Scientometrics, 44(3), 561–580. · DOI

Cited by

Sorted by citations

Metadata

Authors
Bornmann, L., & Leibel, C.
ORCID
Venue
Scientometrics
Publisher
Springer Science and Business Media LLC
Published
2026-10-15
ISSN
  • 0138-9130
  • 1588-2861
DOI
10.1007/s11192-026-05841-5
Cited by
0
References
1
Language
en
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review