CoChat
Sign Up
PreprintCited by 21scientometrics and bibliometrics researchAcademic integrity and plagiarismAcademic Publishing and Open Access

Meta-Research: How problematic citing practices distort science

Open access
Horbach, Serge P. J. M., Aagaard, Kaare, Schneider, Jesper W.·2021
DOI 10.31222/osf.io/aqyhg
Open this paper in CoChat — free accountOpen PDF

Abstract

Citing practices constitute a core element in scientific research and communication in which they serve several important functions. They both comprise the principal unit of science’s social reward system and they establish epistemic genealogy, showing the foundations on which claims are built. Distorted or problematic citing behaviour can hence have major harmful consequences for the well-functioning of the scientific enterprise. In this article we employ a case study approach to show how, even in the field of research integrity, or more broadly, meta-research, researchers commonly adhere to suboptimal citing practices. Our findings highlight, among other shortcomings, an apparent lack of critical engagement with the cited literature, leading to incorrect reproductions of claims and overgeneralisation of supposed research findings. We link such problematic citing practices to the state of crisis that science is currently claimed to face, discussing how these practices might be both a symptom and a cause of these crises.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.31222/osf.io/aqyhg

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 74

  1. Anderson, M. S. (2000). Normative orientations of university faculty and doctoral students. Science and Engineering Ethics, 6(4), 443-461. doi:10.1007/s11948-000-0002-6 · DOI
  2. Anderson, M. S., Ronning, E. A., De Vries, R., & Martinson, B. C. (2010). Extending the Mertonian Norms: Scientists' Subscription to Norms of Research. The Journal of Higher Education, 81(3), 366-393.
  3. Bird, A. (2018). Understanding the Replication Crisis as a Base Rate Fallacy. The British Journal for the Philosophy of Science. doi:10.1093/bjps/axy051 · DOI
  4. Bishop, D. V. M. (2019). The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture. Quarterly Journal of Experimental Psychology, 73(1), 1-19. doi:10.1177/1747021819886519 · DOI
  5. Bordignon, F. (2020). Self-correction of science: a comparative study of negative citations and post- publication peer review. Scientometrics. doi:10.1007/s11192-020-03536-z · DOI · Paper page
  6. Bouter, L. M., Tijdink, J., Axelsen, N., Martinson, B. C., & ter Riet, G. (2016). Ranking major and minor research misbehaviors: results from a survey among participants of four World Conferences on Research Integrity. Research Integrity and Peer Review, 1(1), 17. doi:10.1186/s41073- 016-0024-5 · DOI · Paper page
  7. Callaham, M., Wears, R. L., & Weber, E. (2002). Journal Prestige, Publication Bias, and Other Characteristics Associated With Citation of Published Studies in Peer-Reviewed Journals. JAMA, 287(21), 2847-2850. doi:10.1001/jama.287.21.2847 · DOI · Paper page
  8. Callaway, E. (2011). Report finds massive fraud at Dutch universities. Nature, 479(7371), 15-15. doi:10.1038/479015a · DOI
  9. Camacho-Miñano, M.-d.-M., & Núñez-Nickel, M. (2009). The multilayered nature of reference selection. Journal of the American Society for Information Science and Technology, 60(4), 754-777. doi:https://doi.org/10.1002/asi.21018 · DOI
  10. Chambers, C. (2017). The Seven Deadly Sins of Psychology. A Manifesto for Reforming the Culture of Scientific Pratice. Princeton, NJ: Princeton University Press.
  11. Chubin, D. E., & Moitra, S. D. (1975). Content Analysis of References: Adjunct or Alternative to Citation Counting? Social Studies of Science, 5(4), 423-441. doi:10.2307/284806 · DOI
  12. Coleman, L. (2021). Research in Crisis. Blueprint to Overhaul the Broken Knolwedge Factory. London & New York, NY: Routledge, Taylor & Francis Group.
  13. Cozzens, S. E. (1989). What do citations count? The rhetoric-first model. Scientometrics, 15(5-6), 437- 447.
  14. Cronin, B. (2005). A hundred million acts of whimsy? Current Science, 89(9), 1505-1509.
  15. Cronin, B. (2016). The Incessant Chattering of Texts. In (pp. 13-19). Berlin, Boston: De Gruyter Saur.
  16. de Rijcke, S., Wouters, P. F., Rushforth, A. D., Franssen, T. P., & Hammarfelt, B. (2015). Evaluation practices and effects of indicator use-a literature review. Research Evaluation, 25(2), 161- 169. doi:10.1093/reseval/rvv038 · DOI · Paper page
  17. de Solla Price , D. J. (1963). Little science, big science. In. New York, NY.: Columbia University Press.
  18. De Solla Price , D. J. (1970). Citation measures of hard science, soft science, technology, and nonscience. In C. E. Nelson & D. K. Pollock (Eds.), Communication among scientists and enginners (pp. 3-22). Lexington, MA: Health.
  19. Edwards, M. A., & Roy, S. (2016). Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition. Environmental Engineering Science. doi:10.1089/ees.2016.0223 · DOI · Paper page
  20. Fanelli, D. (2010). "Positive" Results Increase Down the Hierarchy of the Sciences. PLoS ONE, 5(3). doi:10.1371/journal.pone.0010068 · DOI · Paper page
  21. Fiedler, K., & Schwarz, N. (2016). Questionable Research Practices Revisited. Social Psychological and Personality Science, 7(1), 45-52. doi:10.1177/1948550615612150 · DOI · Paper page
  22. Fiedler, K., & Schwarz, N. (2016). Questionable Research Practices Revisited. 7(1), 45-52. doi:10.1177/1948550615612150 · DOI · Paper page
  23. Garfield, E. (1955). Citation indexes to science: a new dimension in documentation through association of ideas. Science, 122(3159), 108-111.
  24. Greenberg, S. A. (2009a). How citation distortions create unfounded authority: analysis of a citation network. BMJ, 339. doi:10.1136/bmj.b2680 · DOI · Paper page
  25. Greenberg, S. A. (2009b). How citation distortions create unfounded authority: analysis of a citation network. 339, b2680. doi:10.1136/bmj.b2680 %J BMJ · DOI · Paper page
  26. Gross, P. L., & Gross, E. M. (1927). College libraries and chemical education. Science, 66(1713), 385- 389.
  27. Hirsch, J. E. (2005). An Index to Quantify an Individual's Scientific Research Output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569-16572.
  28. Ioannidis, J. A. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA: The Journal of the American Medical Association, 294(2), 218-228. doi:10.1001/jama.294.2.218 · DOI · Paper page
  29. Ioannidis, J. P. (2018). Massive citations to misleading methods and research tools: Matthew effect, quotation error and citation copying. European journal of epidemiology, 33(11), 1021-1023.
  30. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science, 23(5), 524-532. doi:10.1177/0956797611430953 · DOI · Paper page
  31. Johnson, B., & Oppenheim, C. (2007). How socially connected are citers to those that they cite? Journal of Documentation.
  32. Kaplan, N. (1965). The Norms of Citation Behavior: Prolegomena to the Footnote. American Documentation, 16(3), 179-184.
  33. Kurtz, M. J., Eichhorn, G., Accomazzi, A., Grant, C., Demleitner, M., Murray, S. S., . . . Elwell, B. (2005). The bibliometric properties of article readership information. Journal of the American Society for Information Science and Technology, 56(2), 111-128. doi:https://doi.org/10.1002/asi.20096 · DOI
  34. Latour, B. (1987). Science in action: How to follow scientists and engineers through society. Boston, MA: Harvard University Press.
  35. Latour, B., & Woolgar, S. (1986). Laboratory life. The construction of scientific facts (2nd ed.). Princeton, NJ: Princeton University Press.
  36. Law, J. (1986). The heterogeneity of texts. In M. Callon, J. Law, & A. Rip (Eds.), Mapping the dynamics of science and technology (pp. 67-83 ). London, England: Macmillan.
  37. Leng, G., & Leng, R. I. (2020a). The Matter of Facts. Skepticism, Persuasion, and Evidence in Science. Cambrige, MA: MIT Press.
  38. Leng, G., & Leng, R. I. (2020b). The Matter of Facts: Skepticism, Persuasion, and Evidence in Science: Mit Press.
  39. Leng, R. I. (2018). A network analysis of the propagation of evidence regarding the effectiveness of fat-controlled diets in the secondary prevention of coronary heart disease (CHD): Selective citation in reviews. Plos One, 13(5), e0197716. doi:10.1371/journal.pone.0197716 · DOI
  40. Liu, Z. (1997). Citation theories in the framework of international flow of information: New evidence with translation analysis. Journal of the American Society for Information Science, 48(1), 80- 87. doi:https://doi.org/10.1002/(SICI)1097-4571(199701)48:1<80::AID-ASI9>3.0.CO;2-Y Luukkonen, T. (1997). Why has Latour's theory of citations been ignored by the bibliometric community? discussion of sociological interpretations of citation analysis. Scientometrics, 38(1), 27-37. doi:10.1007/BF02461121 · DOI
  41. MacRoberts, M. H., & MacRoberts, B. R. (1984). The negational reference: Or the art of dissembling. Social Studies of Science, 14(1), 91-94.
  42. MacRoberts, M. H., & MacRoberts, B. R. (1989a). Citation analysis and the science policy arena. Trends in Biochemical Sciences, 14(1), 8-13. doi:https://doi.org/10.1016/0968- 0004(89)90077-7 · DOI
  43. MacRoberts, M. H., & MacRoberts, B. R. (1989b). Problems of citation analysis: A critical review. Journal of the American Society for Information Science, 40(5), 342-349. doi:10.1002/(SICI)1097-4571(198909)40:5<342::AID-ASI7>3.0.CO;2-U MacRoberts, M. H., & MacRoberts, B. R. (2010). Problems of citation analysis: A study of uncited and seldom-cited influences. Journal of the American Society for Information Science and Technology, 61(1), 1-12. doi:10.1002/asi.21228 · DOI
  44. Martin, B. R. (2013). Whither research integrity? Plagiarism, self-plagiarism and coercive citation in an age of research assessment. Research Policy, 42(5), 1005-1014. doi:http://dx.doi.org/10.1016/j.respol.2013.03.011 · DOI
  45. Merton, R. (1979). Foreword by Eugene Garfield: Citation indexing -Its theory and application in science, technology, and humanities. In. Philadelphia, PA: ISI Press.
  46. Merton, R. K. (1942). Note on science and democracy. Journal of Legal and Political Sociology, 1(1 and 2), 115-126.
  47. Merton, R. K. (1968). MATTHEW EFFECT IN SCIENCE. Science, 159(3810), 56-&. doi:10.1126/science.159.3810. · DOI
  48. Merton, R. K. (1968). Social Theory and Social Structure. New Yprk, NY: Columbia University Press.
  49. Moravcsik, M. J., & Murugesan, P. (1975). Some Results on the Function and Quality of Citations. Social Studies of Science, 5(1), 86-92.
  50. Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., . . . Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. doi:10.1038/s41562-016-0021 · DOI · Paper page
  51. Myers, G. (1985). The Social Construction of Two Biologists' Proposals. Written Communication, 2(3), 219-245. doi:10.1177/0741088385002003001 · DOI
  52. Nicolaisen, J. (2007). Citation analysis. Annual Review of Information Science and Technology, 41(1), 609-641. doi:10.1002/aris.2007.1440410120 · DOI
  53. O'Boyle, E. H. J., Banks, G. C., & Gonzalez-Mulé, E. (2017). The Chrysalis Effect. Journal of Management, 43(2), 376-399. doi:doi:10.1177/0149206314527133 · DOI · Paper page
  54. Oberauer, K., & Lewandowsky, S. (2019). Addressing the theory crisis in psychology. Psychonomic bulletin review, 26(5), 1596-1618.
  55. Pettigrew, T. F. (2018). The Emergence of Contextual Social Psychology. Personality and Social Psychology Bulletin, 44(7), 963-971. doi:10.1177/0146167218756033 · DOI
  56. Popper, K. (1945). The Open Society and its enemies. Princeton, New Jersey: Princeton University Press.
  57. Saltelli, A., & Funtowicz, S. (2017). What is science's crisis really about? Futures, 91, 5-11. doi:https://doi.org/10.1016/j.futures.2017.05.010 · DOI
  58. Schneider, J. W. (2006). Concept symbols revisited: Naming clusters by parsing and filtering of noun phrases from citation contexts of concept symbols. Scientometrics, 68(3), 573-593. doi:10.1007/s11192-006-0131-z · DOI
  59. Simkin, M., & Roychowdhury, V. (2006). Do you sincerely want to be cited? Or: read before you cite. Significance, 3(4), 179-181. doi:10.1111/j.1740-9713.2006.00202.x · DOI
  60. Small, H. (2004). On the shoulders of Robert Merton: Towards a normative theory of citation. Scientometrics Scientometrics, 60(1), 71-79. doi:10.1023/b:scie.0000027310.68393.bc · DOI · Paper page
  61. Small, H. G. (1978). Cited Documents as Concept Symbols. Social Studies of Science, 8(3), 327-340. doi:10.1177/030631277800800305 · DOI
  62. Stang, A., Jonas, S., & Poole, C. J. E. j. o. e. (2018). Case study in major quotation errors: a critical commentary on the Newcastle-Ottawa scale. 33(11), 1025-1031.
  63. Steneck, N. H. (2006). Fostering integrity in research: Definitions, current knowledge, and future directions. Science and Engineering Ethics, 12(1), 53-74. doi:10.1007/pl00022268 · DOI · Paper page
  64. Stroebe, W. (2016). Are most published social psychological findings false? Journal of Experimental Social Psychology, 66, 134-144. doi:https://doi.org/10.1016/j.jesp.2015.09.017 · DOI
  65. Teplitskiy, M., Duede, E., Menietti, M., & R., L. K. (2020, 31 Aug ). Status drives how we cite: Evidence from thousands of authors. https://arxiv.org/abs/2002.10033. · arXiv
  66. Trinquart, L., Johns, D. M., & Galea, S. (2016). Why do we think we know what we know? A metaknowledge analysis of the salt controversy. International Journal of Epidemiology, 45(1), 251-260. doi:10.1093/ije/dyv184 %J International Journal of Epidemiology · DOI · Paper page
  67. van Dalen, H. P., & Henkens, K. n. (2005). Signals in science -On the importance of signaling in gaining attention in science. Scientometrics, 64(2), 209-233. doi:10.1007/s11192-005-0248-5 · DOI
  68. van der Steen, J. T., van den Bogert, C. A., van Soest-Poortvliet, M. C., Fazeli Farsani, S., Otten, R. H. J., ter Riet, G., & Bouter, L. M. (2018). Determinants of selective reporting: A taxonomy based on content analysis of a random selection of the literature. PLoS ONE, 13(2), e0188247. doi:10.1371/journal.pone.0188247 · DOI
  69. Vazire, S. (2017). Quality Uncertainty Erodes Trust in Science. Collabra: Psychology, 3(1). doi:10.1525/collabra.74 · DOI · Paper page
  70. Voos, H., & Dagaev, K. S. (1976). Are All Citations Equal? Or, Did We Op. Cit. Your Idem? Journal of Academic Librarianship, 1(6), 19-21.
  71. Wallace, M. L., Lariviere, V., & Gingras, Y. (2012). A Small World of Citations? The Influence of Collaboration Networks on Citation Practices. PLoS ONE, 7(3). doi:10.1371/journal.pone.0033339 · DOI
  72. Wicherts, J. M. (2017). The Weak Spots in Contemporary Science (and How to Fix Them). Animals (Basel), 7(12). doi:10.3390/ani7120090 · DOI
  73. Wilson, P. (2020). Academic Fraud: Solving the Crisis in Modern Academia. Exchanges: The Interdisciplinary Research Journal, 7(3), 14-44. doi:10.31273/eirj.v7i3.546 · DOI
  74. Zipf, G. K. (1949). Human behavior and the principle of least effort. Cambridge, MA: Addison-Wesley.

Cited by

Sorted by citations

Authors
Horbach, S. P. J. M., Aagaard, K., & Schneider, J. W.
Affiliations
  • Leiden University (NL)
  • Aarhus University (DK)
Corresponding author
Horbach, Serge P. J. M.
ORCID
Publisher
Center for Open Science
Published
2021-02-22
DOI
10.31222/osf.io/aqyhg
OpenAlex
W4232249697
Cited by
21
References
12
Citations by year
2026: 2 · 2025: 3 · 2024: 2 · 2023: 6 · 2022: 6 · 2021: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • RePEc: Research Papers in Economics (submittedVersion)
  • MetArXiv (OSF Preprints) (submittedVersion)
  • MetArXiv (OSF Preprints) (submittedVersion, cc-by)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
Keywords
citation practices, meta-research, research integrity, crisis of science, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review