The Mismeasure of Science: Citation Analysis

MacRoberts, Michael H.; MacRoberts, Barbara R.

doi:10.1002/asi.23970

Cited by 101scientometrics and bibliometrics researchWeb visibility and informetricsPsychology Research and Bibliometrics

The Mismeasure of Science: Citation Analysis

MacRoberts, Michael H., MacRoberts, Barbara R.·2018·Journal of the Association for Information Science and Technology

DOI 10.1002/asi.23970

Open this paper in CoChat — free account

Abstract

For several decades we, among others, have criticized the use of citations for evaluative purposes. Although these criticisms have been noted, they have been largely brushed aside or ignored, not addressed head on. This may be for a number of reasons, but we believe the main one is that these criticisms undermine the desire to have an easy “scientific”—that is, quantitative—method of evaluation. Consequently, we continue and update our criticism of the use of citations for evaluation.

Integrity signals

Registry

Crossref · 10.1002/asi.23970

Cites

85 of 85 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations themselves are not equal because citations may have very different meanings determined by many reasons for citing.
“citations themselves are not equal, primarily because citations may have very different meaning, which is determined by the plethora of reasons for citing [266-268].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [267]
Scientific impact of any evaluated entity cannot be determined solely by citations due to different citing behaviors and incompatible citation counts between disciplines.
“It is often argued that the scientific impact of any evaluated entity cannot be determined solely by citations due to the different citing behaviors and resulting incompatible citation counts between disciplines [1-3,5,260,267].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [267]

Rapid Range Shifts of Species Associated with High Levels of Climate Warming
Chen, I., Hill, J. K., Ohlemüller, R., Roy, D. B., & Thomas, C. D. (2011) · Science · doi:10.1126/science.1206432 · Cited by 5,510 · see citing works
Little Science, Big Science
Price, D. J. D. S. (1963) · Columbia University Press eBooks · doi:10.7312/pric91844 · Cited by 4,340 · see citing works
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447 · see citing works
A Significant Upward Shift in Plant Species Optimum Elevation During the 20th Century
Lenoir, J., Gégout, J., Marquet, P. A., de Ruffray, P., & Brisse, H. (2008) · Science · doi:10.1126/science.1156831 · Cited by 2,313 · see citing works
Poleward shifts in geographical ranges of butterfly species associated with regional warming
Parmesan, C., Ryrholm, N., Stefanescu, C., Hill, J. K., Thomas, C. D., Descimon, H., Huntley, B. J., Kaila, L., Kullberg, J., Tammaru, T., et al. (1999) · Nature · doi:10.1038/21181 · Cited by 2,280 · see citing works
What do citation counts measure? A review of studies on citing behavior
Bornmann, L., & Daniel, H. (2008) · Journal of Documentation · doi:10.1108/00220410810844150 · Cited by 1,607 · see citing works
Bibliometrics: Global gender disparities in science
Larivière, V., Ni, C., Gingras, Y., Cronin, B., & Sugimoto, C. R. (2013) · Nature · doi:10.1038/504211a · Cited by 1,499 · see citing works
Problems of citation analysis: A critical review
MacRoberts, M. H., & MacRoberts, B. R. (1989) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(198909)40:5 · Cited by 807 · see citing works
Self-citation
Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods
van Raan, A. F. J. (2005) · Scientometrics · doi:10.1007/s11192-005-0008-6 · Cited by 760 · see citing works
Referencing as persuasion
Nigel Gilbert, G. (1977) · Social Studies of Science · doi:10.1177/030631277700700112 · Cited by 691 · see citing works
Google Scholar as a new source for citation analysis
Harzing, A.-W. K., & van der Wal, R. (2007) · Ethics in Science and Environmental Politics · doi:10.3354/esep00076 · Cited by 677 · see citing works
An introduction to science and technology studies
(2004) · Choice Reviews Online · doi:10.5860/choice.41-3396 · Cited by 584 · see citing works
Norms and Counter-Norms in a Select Group of the Apollo Moon Scientists: A Case Study of the Ambivalence of Scientists
Mitroff, I. I. (1974) · American Sociological Review · doi:10.2307/2094423 · Cited by 520 · see citing works
Problems of citation analysis
MacRoberts, M. H., & MacRoberts, B. R. (1996) · Scientometrics · doi:10.1007/bf02129604 · Cited by 503 · see citing works
Self-citation
Explaining Creativity
Sawyer, R. K. (2006) · doi:10.1093/oso/9780195161649.001.0001 · Cited by 494 · see citing works
Coercive Citation in Academic Publishing
Wilhite, A. W., & Fong, E. A. (2012) · Science · doi:10.1126/science.1212540 · Cited by 433 · see citing works
The Philosophical Basis of Peer Review and the Suppression of Innovation
Horrobin, D. F. (1990) · JAMA · doi:10.1001/jama.1990.03440100162024 · Cited by 420 · see citing works
Psychological relevance and information science
Harter, S. P. (1992) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199210)43:9 · Cited by 415 · see citing works
Physics and Beyond: Encounters and Conversations
Lakoff, S. A., & Heisenberg, W. (1971) · International Journal Canada s Journal of Global Policy Analysis · doi:10.2307/40201087 · Cited by 404 · see citing works
The Ortega Hypothesis
Cole, J. R., & Cole, S. (1972) · Science · doi:10.1126/science.178.4059.368 · Cited by 387 · see citing works
What determines the citation frequency of ecological papers?
Leimu, R., & Koricheva, J. (2004) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2004.10.010 · Cited by 368 · see citing works
Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results
Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H. (2007) · Ethics in Science and Environmental Politics · doi:10.3354/esep00084 · Cited by 351 · see citing works
Naked Science: Anthropological Inquiry Into Boundaries, Power, and Knowledge.
Reyna, S. P., & Nader, L. (1998) · Journal of the Royal Anthropological Institute · doi:10.2307/3034459 · Cited by 333 · see citing works
The mismeasurement of science
Lawrence, P. A. (2007) · Current Biology · doi:10.1016/j.cub.2007.06.014 · Cited by 296 · see citing works
Citation Statistics
Adler, R., Ewing, J., & Taylor, P. (2009) · arXiv.org · doi:10.1214/09-sts285 · Cited by 285 · see citing works
Quantitative Measures of Communication in Science: A Critical Review
Edge, D. O. (1979) · History of Science · doi:10.1177/007327537901700202 · Cited by 283 · see citing works
The influence of international collaboration on the impact of research results
van Raan, A. F. J. (1998) · Scientometrics · doi:10.1007/bf02458380 · Cited by 263 · see citing works
What factors determine citation counts of publications in chemistry besides their quality?
Bornmann, L., Schier, H., Marx, W., & Daniel, H. (2011) · Journal of Informetrics · doi:10.1016/j.joi.2011.08.004 · Cited by 242 · see citing works
Citation rates and journal impact factors are not suitable for evaluation of research
Seglen, P. O. (1998) · Acta Orthopaedica Scandinavica · doi:10.3109/17453679809000920 · Cited by 240 · see citing works
An Introduction to the Historiography of Science
Kragh, H. (1987) · Cambridge University Press eBooks · doi:10.1017/cbo9780511622434 · Cited by 224 · see citing works
Advice to a Young Scientist
Holyoke, T. C., & Medawar, P. B. (1980) · The Antioch Review · doi:10.2307/4638313 · Cited by 218 · see citing works
The hand of science: academic writing and its rewards
(2005) · Choice Reviews Online · doi:10.5860/choice.43-0269 · Cited by 206 · see citing works
Reviewers are blinkered by bibliometrics
Stephan, P. E., Veugelers, R., & Wang, J. (2017) · Nature · doi:10.1038/544411a · Cited by 189 · see citing works
Problems of citation analysis: A study of uncited and seldom‐cited influences
MacRoberts, M. H., & MacRoberts, B. R. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21228 · Cited by 184 · see citing works
Self-citation
How Scientists Think in the Real World
Dunbar, K. N. (2000) · Journal of Applied Developmental Psychology · doi:10.1016/s0193-3973(99)00050-7 · Cited by 177 · see citing works
Data sources for performing citation analysis: an overview
Neuhaus, C., & Daniel, H. (2008) · Journal of Documentation · doi:10.1108/00220410810858010 · Cited by 159 · see citing works
The social function of science
Oppermann, R. H. (1939) · Journal of the Franklin Institute · doi:10.1016/s0016-0032(39)91441-9 · Cited by 155 · see citing works
The scientific evolution of psychology.
Kantor, J. R. (1963) · Medical Entomology and Zoology · Cited by 142 · see citing works
Quantitative Measures of Communication in Science: A Study of the Formal Level
MacRoberts, M. H., & MacRoberts, B. R. (1986) · Social Studies of Science · doi:10.1177/030631286016001008 · Cited by 142 · see citing works
Self-citation
Naked statistics: stripping the dread from the data
(2013) · Choice Reviews Online · doi:10.5860/choice.50-5648 · Cited by 123 · see citing works
The museum of errors/horrors in Scopus
Franceschini, F., Maisano, D. A., & Mastrogiacomo, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2015.11.006 · Cited by 121 · see citing works
Bibliometrics and Citation Analysis
(2012) · doi:10.5040/9798400675966.ch-011 · Cited by 111 · see citing works
Context-Enhanced Citation Sentiment Detection
Athar, A., & Teufel, S. (2012) · Cited by 104 · see citing works
Why impact factors don't work for taxonomy
Krell, F. (2002) · Nature · doi:10.1038/415957a · Cited by 99 · see citing works
SJR and SNIP: two new journal metrics in Elsevier's Scopus
Colledge, L., Anegón, F. D. M., Guerrero‐Bote, V. P., López-Illescas, C., Aisati, M. E., & Moed, H. F. (2010) · Serials The Journal for the Serials Community · doi:10.1629/23215 · Cited by 98 · see citing works
Why has Latour's theory of citations been ignored by the bibliometric community? Discussion of sociological interpretations of citation analysis
Luukkonen, T. (1997) · Scientometrics · doi:10.1007/bf02461121 · Cited by 94 · see citing works
Is the scientific paper a fraud?
MEDAWAR, P. B. (1996) · doi:10.1093/oso/9780192861931.003.0003 · Cited by 93 · see citing works
A Reverse Engineering Approach to the Suppression of Citation Biases Reveals Universal Properties of Citation Distributions
Radicchi, F., & Castellano, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033833 · Cited by 88 · see citing works
Ike Antkare one of the great stars in the scientific firmament
Antkare, I. (2010) · Cited by 83 · see citing works
Citation analysis of research performer quality
Kostoff, R. N. (2002) · Scientometrics · doi:10.1023/a:1014831920172 · Cited by 78 · see citing works
A farewell to the MNCS and like size-independent indicators
Abramo, G., & D’Angelo, C. A. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.04.006 · Cited by 75 · see citing works
The skewed few: people and papers of quality in management studies
Macdonald, S., & Kam, J. (2011) · Organization · doi:10.1177/1350508411403533 · Cited by 70 · see citing works
SCIENTIFIC FRAUD AND THE POWER STRUCTURE OF SCIENCE
Martin, B. G. (1992) · Prometheus · doi:10.1080/08109029208629515 · Cited by 66 · see citing works
Testing the Ortega hypothesis: Facts and artifacts
MacRoberts, M. H., & MacRoberts, B. R. (1987) · Scientometrics · doi:10.1007/bf02016665 · Cited by 64 · see citing works
Self-citation
Rank injustice
Lawrence, P. A. (2002) · Nature · doi:10.1038/415835a · Cited by 62 · see citing works
Citation Autobiography: An Investigation of ISI Database Coverage in Determining Author Citedness
Nisonger, T. E. (2004) · College & Research Libraries · doi:10.5860/crl.65.2.152 · Cited by 61 · see citing works
Using complex networks concepts to assess approaches for citations in scientific papers
Amancio, D. R., Nunes, M. G. V., Oliveira, O. N., & Costa, L. D. F. (2012) · Scientometrics · doi:10.1007/s11192-012-0630-z · Cited by 60 · see citing works
A farewell to the MNCS and like size-independent indicators
Abramo, G., & D’Angelo, C. A. (n.d.) · RePEc: Research Papers in Economics · Cited by 48 · see citing works
The Correspondence of Charles Darwin 1882
Darwin, C. (2023) · Cambridge University Press eBooks · doi:10.1017/9781009233606.008 · Cited by 48 · see citing works
Science as Writing
Locke, D. C. (1992) · Yale University Press eBooks · doi:10.12987/9780300157406 · Cited by 46 · see citing works
Aberration of the Citation
Moustafa, K. (2015) · Accountability in Research · doi:10.1080/08989621.2015.1127763 · Cited by 44 · see citing works
Revisiting “Is the scientific paper a fraud?”
Howitt, S. M., & Wilson, A. N. (2014) · EMBO Reports · doi:10.1002/embr.201338302 · Cited by 43 · see citing works
Are alternative metrics still alternative?
Buschman, M., & Michalek, A. (2013) · Bulletin of the American Society for Information Science and Technology · doi:10.1002/bult.2013.1720390411 · Cited by 40 · see citing works
Why I Am Not a Co-Citationist*
Edge, D. O. (1977) · Cited by 40 · see citing works
Criteria for Evaluating Indicators
Gingras, Y. (2014) · The MIT Press eBooks · doi:10.7551/mitpress/9445.003.0009 · Cited by 32 · see citing works
Citation rates in mathematics: a study of variation by subdiscipline
Smolinsky, L., & Lercher, A. (2012) · Scientometrics · doi:10.1007/s11192-012-0647-3 · Cited by 30 · see citing works
Obliteration by Incorporation
McCain, K. W. (2014) · The MIT Press eBooks · doi:10.7551/mitpress/9445.003.0011 · Cited by 28 · see citing works
Another appraisal of Ortega, the Coles, and science policy: the Ecclesiastes hypothesis
Turner, S. P., & Chubin, D. E. (1976) · Social Science Information · doi:10.1177/053901847601500407 · Cited by 27 · see citing works
An index to evaluate the quality of taxonomic publications
Valdecasas, A. G. (2011) · Zootaxa · doi:10.11646/zootaxa.2925.1.6 · Cited by 26 · see citing works
Good practices for a literature survey are not followed by authors while preparing scientific manuscriptsPreprint
Amancio, D. R., Nunes, M. G. V., Oliveira, O. N., et al. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1005.3063 · Cited by 25 · see citing works
Bibliometric Standards for Evaluating Research Institutes in the Natural Sciences
Bornmann, L., Bowman, B. F., Bauer, J., Marx, W., Schier, H., & Palzenberger, M. (2014) · The MIT Press eBooks · doi:10.7551/mitpress/9445.003.0015 · Cited by 22 · see citing works
Advantages of Surface Mining
Kuck, D. L. (1974) · Science · doi:10.1126/science.183.4120.28 · Cited by 16 · see citing works
Users beware: implications of database errors when assessing the individual research records of ecologists and conservation biologists
Calver, M., Beatty, S. J., Bryant, K. A., Dickman, C. R., Ebner, B. C., & Morgan, D. L. (2013) · Pacific Conservation Biology · doi:10.1071/pc130320 · Cited by 14 · see citing works
Advantages of Surface Mining
Kuck, D. L. (1974) · Science · doi:10.1126/science.183.4120.28-a · Cited by 14 · see citing works
The Poverty of Citation Databases: Data Mining Is Crucial for Fair Metrical Evaluation of Research Performance
Krell, F. (2009) · BioScience · doi:10.1525/bio.2009.59.1.2 · Cited by 13 · see citing works
The false ‘Ortega Hypothesis’: a literature science case study
Száva‐Kováts, E. (2004) · Journal of Information Science · doi:10.1177/0165551504047823 · Cited by 13 · see citing works
(Non-)use of Foucault’sArchaeology of KnowledgeandOrder of Thingsin LIS journal literature, 1990-2015
Dewey, S. H. (2016) · Journal of Documentation · doi:10.1108/jd-08-2015-0096 · Cited by 12 · see citing works
Citation Analysis
Goudsmit, S. A. (1974) · Science · doi:10.1126/science.183.4120.28-b · Cited by 8 · see citing works
RAM the PI-BETA, C3PO - what the H-STAR happened to my promotion application? Or: The pros and cons of bibliometric evaluations of researchers
Calver, M. (2013) · doi:10.7882/fs.2013.019 · Cited by 7 · see citing works
Please don't aim for a highly cited paper
Calver, M. (2015) · Murdoch University Research Portal · Cited by 7 · see citing works
Indirect‐collective referencing (ICR) in the elite journal literature of physics. II. A literature science study on the level of communications
Száva‐Kováts, E. (2001) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10004 · Cited by 7 · see citing works
Evaluations of individual scientists and research institutions
Hutchinson, A. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20726 · Cited by 4 · see citing works
An increasing citation black hole in ecology and evolution
Rafferty, A. R., Wong, B. B. M., & Chapple, D. G. (2014) · Ecology and Evolution · doi:10.1002/ece3.1356 · Cited by 3 · see citing works
They’re writing a Nature paper!
Calver, M. (2014) · Pacific Conservation Biology · doi:10.1071/pc140228 · Cited by 2 · see citing works
Two cheers for research
Stokes, T. (2010) · Policy Quarterly · doi:10.26686/pq.v6i1.4315 · Cited by 1 · see citing works

Flagged first, then by citations

3 of the 88 references couldn’t be matched to a known paper.

Sorted by citations

Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696 · see citing works
Mapping graduate employability and career development in higher education research: a citation network analysis
Healy, M. G., Hammer, S. J., & McIlveen, P. (2020) · Studies in Higher Education · doi:10.1080/03075079.2020.1804851 · Cited by 172 · see citing works
Deep learning for precision agriculture: A bibliometric analysis
Coulibaly, S., Kamsu-Foguem, B., Kamissoko, D., & Traoré, D. (2022) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2022.200102 · Cited by 152 · see citing works
Social Media Metrics for New Research Evaluation
Wouters, P. F., Zahedi, Z., & Costas, R. (2019) · Springer handbooks · doi:10.1007/978-3-030-02511-3_26 · Cited by 90 · see citing works
Revisiting the Scientometric Conceptualization of Impact and Its Measurement
Abramo, G. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.05.001 · Cited by 80 · see citing works
On the relationships between bibliographic characteristics of scientific documents and citation and Mendeley readership counts: A large-scale analysis of Web of Science publications
Zahedi, Z., & Haustein, S. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2017.12.005 · Cited by 64 · see citing works
Identification of important citations by exploiting research articles’ metadata and cue-terms from content
Qayyum, F., & Afzal, M. (2018) · Scientometrics · doi:10.1007/s11192-018-2961-x · Cited by 77 · see citing works
Evaluating research quality with Large Language Models: An analysis of ChatGPT’s effectiveness with different settings and inputs
Thelwall, M. A. (2024) · Journal of Data and Information Science · doi:10.2478/jdis-2025-0011 · Cited by 54 · see citing works
Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation
Bornmann, L., & Marewski, J. N. (2019) · Scientometrics · doi:10.1007/s11192-019-03018-x · Cited by 53 · see citing works
A global exploratory comparison of country self-citations 1996-2019
Baccini, A., & Petrovich, E. (2023) · PLoS ONE · doi:10.1371/journal.pone.0294669 · Cited by 42 · see citing works
Bibliometric analysis of publications from post-Soviet countries in psychological journals in 1992–2017
Lovakov, A. V., & Рафиковна, А. Е. (2019) · Scientometrics · doi:10.1007/s11192-019-03087-y · Cited by 35 · see citing works
25 Years of Molecular Biology Databases: A Study of Proliferation, Impact, and Maintenance
Imker, H. J. (2018) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2018.00018 · Cited by 32 · see citing works
Science Mapping and Science Maps
Petrovich, E. (2021) · KNOWLEDGE ORGANIZATION · doi:10.5771/0943-7444-7-8-535 · Cited by 31 · see citing works
Gender disparities in awards to neuroscience researchers.
Melnikoff, D. E., & Valian, V. V. (2019) · Archives of Scientific Psychology · doi:10.1037/arc0000069 · Cited by 25 · see citing works
The myth of ‘older LGBT+’ people: Research shortcomings and policy/practice implications for health/care provision
Westwood, S. (2020) · Journal of Aging Studies · doi:10.1016/j.jaging.2020.100880 · Cited by 25 · see citing works
Conjoint analysis of researchers' hidden preferences for bibliometrics, altmetrics, and usage metrics
Lemke, S., Mazarakis, A., & Peters, I. (2021) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24445 · Cited by 25 · see citing works
Has Covid-19 Gone Viral? An Overview of Research by Subject Area
Ruiz–Real, J. L., Soriano, B. J. N., & Uribe-Toril, J. (2020) · Health Education & Behavior · doi:10.1177/1090198120958368 · Cited by 23 · see citing works
A temporally dynamic examination of research method usage in the Chinese library and information science community
Lou, W., Su, Z., He, J., & Li, K. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102686 · Cited by 22 · see citing works
Measures of Impact for Journals, Articles, and Authors
Suelzer, E. M., & Jackson, J. L. (2022) · Journal of General Internal Medicine · doi:10.1007/s11606-022-07475-8 · Cited by 21 · see citing works
Increasing the equitability of data citation in paleontology: capacity building for the big data future
Smith, J. A., Raja, N. B., Clements, T., Dimitrijević, D., Dowding, E. M., Dunne, E. M., Gee, B. M., Godoy, P. L., Lombardi, E. M., Mulvey, L. P. A., et al. (2023) · Paleobiology · doi:10.1017/pab.2023.33 · Cited by 20 · see citing works
The inconsistency of h-index: A mathematical analysis
Brito, R., & Navarro, A. R. (2020) · Journal of Informetrics · doi:10.1016/j.joi.2020.101106 · Cited by 20 · see citing works
Meta-Research: How problematic citing practices distort science
Horbach, S. P. J. M., Aagaard, K., & Schneider, J. W. (2021) · doi:10.31222/osf.io/aqyhg · Cited by 21 · see citing works
The publish or perish, publish and perish, publish then perish, and now retract and perish cultures in academia
da Silva, J. A. T., & Nazarovets, S. (2025) · Naunyn-Schmiedeberg s Archives of Pharmacology · doi:10.1007/s00210-025-04651-5 · Cited by 19 · see citing works
Evolución y alcance multidisciplinar de tres técnicas de análisis bibliométrico
Jorge, R. A., Vega-Almeida, R. L., & Carrillo-Calvet, H. (2020) · Palabra clave · doi:10.24215/18539912e102 · Cited by 19 · see citing works
Is biomedical research self-correcting? Modelling insights on the persistence of spurious science
Grimes, D. R. (2024) · Royal Society Open Science · doi:10.1098/rsos.231056 · Cited by 18 · see citing works
Which are the influential publications in the Web of Science subject categories over a long period of time? CRExplorer software used for big-data analyses in bibliometrics
Thor, A., Bornmann, L., Haunschild, R., & Loet Leydesdorff, A. (2020) · Journal of Information Science · doi:10.1177/0165551520913817 · Cited by 17 · see citing works
Evaluating the relationship between the academic and social impact of open access books based on citation behaviors and social media attention
Wei, M., & چاکلی, ع. ن. (2020) · Scientometrics · doi:10.1007/s11192-020-03678-0 · Cited by 17 · see citing works
Judicious Use of Bibliometrics to Supplement Peer Evaluations of Research in Kinesiology
Knudson, D. V. (2019) · Kinesiology Review · doi:10.1123/kr.2017-0046 · Cited by 17 · see citing works
Probability and expected frequency of breakthroughs: basis and use of a robust method of research assessment
Rodríguez‐Navarro, A., & Brito, R. (2019) · Scientometrics · doi:10.1007/s11192-019-03022-1 · Cited by 16 · see citing works
Citation bias in measuring knowledge flow: Evidence from the web of science at the discipline level
Lyu, H., Bu, Y., Zhao, Z., Zhang, J., & Li, J. (2022) · Journal of Informetrics · doi:10.1016/j.joi.2022.101338 · Cited by 15 · see citing works
Women in limnology: From a historical perspective to a present‐day evaluation
Catalán, N., Antón-Pardo, M., Freixa, A., Rodríguez‐Lozano, P., Bartrons, M., Bernal, S., Genua‐Olmedo, A., Mendoza‐Lera, C., Onandía, G., Benito, X., et al. (2022) · Wiley Interdisciplinary Reviews Water · doi:10.1002/wat2.1616 · Cited by 15 · see citing works
Uluslararası Dergilerde Yayımlanan Türkiye Adresli Makalelerin Atıf Etkisini Artıran Faktörler
Tonta, Y., & Akbulut, M. (2021) · Türk Kütüphaneciliği · doi:10.24146/tk.933159 · Cited by 15 · see citing works
What Kinesiology Research is Most Visible to the Academic World?
Knudson, D. V. (2022) · Quest · doi:10.1080/00336297.2022.2092880 · Cited by 15 · see citing works
Celebrating women conducting research in freshwater ecology … and how the citation game is damaging them
Downes, B. J., & Lancaster, J. (2019) · Marine and Freshwater Research · doi:10.1071/mf18436 · Cited by 13 · see citing works
Construction and validation of a short multidisciplinary research performance questionnaire (SMRPQ)
Daumiller, M. H., Siegel, S. T., & Dresel, M. (2019) · Research Evaluation · doi:10.1093/reseval/rvz009 · Cited by 13 · see citing works
The Sociology of Science and the Garfield Effect: Happy Accidents, Unanticipated Developments and Unexploited Potentials
Zuckerman, H. (2018) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2018.00020 · Cited by 12 · see citing works
In Which Fields Do ChatGPT Scores Align More Closely with Research Quality than Do Citation Rates?
Thelwall, M. A. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2026-0058 · Cited by 11 · see citing works
Towards resilience in the Anthropocene: transforming conservation biology through Indigenous perspectives
Price, M. R., Winter, K. B., & Jackson, A. (2021) · Pacific Conservation Biology · doi:10.1071/pcv27n4_fo · Cited by 11 · see citing works
Internacionalização de artigos científicos: estudo dos autores de uma universidade brasileira
Rodrigues, R. S., Abadal, E., Neubert, P. D. S., & Navas-Fernández, M. (2021) · Em Questão · doi:10.19132/1808-5245274.215-242 · Cited by 10 · see citing works
Relationship between persuasive metadiscoursal devices in research article abstracts and their attention on social media
Ngai, C. S. B., & Singh, R. G. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231305 · Cited by 10 · see citing works
Simulation of the h index use at university departments within the bibliometrics-based heuristics framework: Can the indicator be used to compare individual researchers?
Bornmann, L., Ganser, C., & Tekles, A. (2021) · Journal of Informetrics · doi:10.1016/j.joi.2021.101237 · Cited by 9 · see citing works
Citation analysis of computer systems papers
Frachtenberg, E. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1389 · Cited by 9 · see citing works
Uncited papers in the structure of scientific communication
Katchanov, Y. L., Markova, Y. V., & Anatolyevna, S. N. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2023.101391 · Cited by 8 · see citing works
Evolution of number of citations per article in Materials Science: possible causes and effect on the impact factor of journals
Ariza-Guerrero, A. M., & Blázquez, J. S. (2023) · Scientometrics · doi:10.1007/s11192-023-04863-7 · Cited by 8 · see citing works
Can ChatGPT replace citations for quality evaluation of academic articles and journals? Empirical evidence from library and information science
Thelwall, M. A. (2025) · Journal of Documentation · doi:10.1108/jd-03-2025-0075 · Cited by 7 · see citing works
New academic journals: an international overview of indexing and access models
Rodrigues, R. S., Urbano, C., Neubert, P. D. S., Rodríguez-Gairín, J. M., & Somoza‐Fernández, M. (2024) · Information Research · doi:10.47989/ir284466 · Cited by 7 · see citing works
Are There Meaningful Prestige Metrics of Kinesiology-Related Journals?
Knudson, D. V. (2024) · Measurement in Physical Education and Exercise Science · doi:10.1080/1091367x.2024.2341849 · Cited by 7 · see citing works
Bibliometrics-based heuristics: What is their definition and how can they be studied? - Research note
Bornmann, L., & Hug, S. E. (2020) · El Profesional de la Informacion · doi:10.3145/epi.2020.jul.20 · Cited by 7 · see citing works
Citations versus expert opinions: citation analysis of featured reviews of the American Mathematical Society
Smolinsky, L., Sage, D. S., Lercher, A., & Cao, A. (2021) · Scientometrics · doi:10.1007/s11192-021-03894-2 · Cited by 7 · see citing works
Mapping social innovation impact evaluation: a comprehensive literature review and prospects for future research
Nazari, M., Nezami, I. M., & Asgary, A. (2024) · Quality & Quantity · doi:10.1007/s11135-024-01868-z · Cited by 6 · see citing works
Three decades of research on innovation and inequality: Causal scenarios, explanatory factors and suggestions
Fragkandreas, T. (2022) · Prometheus · doi:10.13169/prometheus.38.2.0147 · Cited by 6 · see citing works
Identifying research quality in the social sciences
Ochsner, M. (2022) · Edward Elgar Publishing eBooks · doi:10.4337/9781800372559.00010 · Cited by 6 · see citing works
Implicit and explicit research quality score probabilities from ChatGPT
Thelwall, M., & Yang, Y. (2025) · Quantitative Science Studies · doi:10.1162/qss.a.393 · Cited by 5 · see citing works
Large language models and responsible research evaluation: an extension of the Leiden Manifesto
Thelwall, M. A. (2026) · Scientometrics · doi:10.1007/s11192-026-05552-x · Cited by 5 · see citing works
Do extraordinary science and technology scientists balance their publishing and patenting activities?
Chang, Y., Chen, D., & Huang, M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0259453 · Cited by 5 · see citing works
How academic librarians involve and contribute in research activities of universities? A systematic demonstration in practice through comparative studies of research productivities and research impacts
Shao, Z., Li, Y., Ke, W., Guo, Y., Feng, F., Hui, F., Niu, Y., & Zheng, Y. (2018) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2018.09.015 · Cited by 5 · see citing works
Evidence for Citation Networks in Studies of Free-Roaming Cats: A Case Study Using Literature on Trap–Neuter–Return (TNR)
Calver, M., & Fleming, P. A. (2020) · Animals · doi:10.3390/ani10060993 · Cited by 5 · see citing works
Assessing and predicting the quality of research master’s theses: an application of scientometrics
Xie, Z. Y., Li, Y., & Li, Z. (2020) · Scientometrics · doi:10.1007/s11192-020-03489-3 · Cited by 5 · see citing works
The Scientific Impact Derived From the Disciplinary Profiles
Allïk, J., Lauk, K., & Realo, A. (2020) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2020.569268 · Cited by 4 · see citing works
Research performance of Khalifa University of Science and Technology, Abu Dhabi
Mukundan, R., & Narayanan, N. (2019) · Performance Measurement and Metrics · doi:10.1108/pmm-06-2019-0022 · Cited by 4 · see citing works
Ontological and Epistemological Challenges in Contemporary Multidisciplinary Research: Towards an AI-Enhanced Framework for Academic Knowledge Production and EvaluationPreprint
Montgomery, R. M. (2025) · Preprints.org · doi:10.20944/preprints202506.1917.v1 · Cited by 4 · see citing works
An Imperfect List of Eminent Sport and Exercise Psychology Researchers
Smolinski, A., McIntyre, O., Martin, D., & Martin, J. J. (2024) · Kinesiology Review · doi:10.1123/kr.2024-0024 · Cited by 3 · see citing works
Scopus Citation Metrics for Top and Bottom Quintile Kinesiology-Related Journals
Knudson, D. V. (2024) · International Journal of Kinesiology in Higher Education · doi:10.1080/24711616.2024.2416184 · Cited by 3 · see citing works
Análisis bibliométrico comparativo de la actividad científica del CSIC y cuatro homólogos europeos: CNRS, HG, MPG Y CNR (2006-2015)Análisis bibliométrico comparativo de la actividad científica del CSIC y cuatro homólogos europeos: CNRS, HG, MPG Y CNR (200
Carrillo, J. M., & Montesi, M. (2018) · Revista General de Información y Documentación · doi:10.5209/rgid.60807 · Cited by 3 · see citing works
MBDBMetrics: an online metrics tool to measure the impact of biological data resources
Insana, G., Ignatchenko, A., Martin, M. J., Bateman, A. G., Bateman, A. G., Martin, M. J., Orchard, S. E., Magrane, M., Ahmad, S., Bowler-Barnett, E., et al. (2023) · Bioinformatics Advances · doi:10.1093/bioadv/vbad180 · Cited by 2 · see citing works
Features of the formation of collections and subject areas of scientific journals in the scientometric databases Scopus and Web of Science
Федорук, О. (2025) · Society Document Communication · doi:10.69587/sdc/1.2025.55 · Cited by 2 · see citing works
Les paradigmes de la revue Criminologie : auteurs, revues et disciplines qui ont marqué son histoire
Leclerc, C., Ream, F., Décary-Hêtu, D., Macaluso, B., & Larivière, V. (2018) · Criminologie · doi:10.7202/1045309ar · Cited by 2 · see citing works
Science Evaluation: Peer Review, Bibliometrics, and Research Impact Assessment
Redman, B. K. (2023) · Springer eBooks · doi:10.1007/978-3-031-27111-3_8 · Cited by 2 · see citing works
Who cites the contributions by information science?
Järvelin, K., & Vakkari, P. (2024) · Journal of Documentation · doi:10.1108/jd-01-2024-0019 · Cited by 2 · see citing works
Informed peer review for publication assessments: Are improved impact measures worth the hassle?
Abramo, G., D’Angelo, C. A., & Felici, G. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00051 · Cited by 2 · see citing works
Bibliometric Analysis of Human Resource Management Research in the Field of Health Services during the COVID-19 Pandemic: 2019-2023 Study
Jati, H. (2023) · INTERNATIONAL JOURNAL OF ECONOMICS AND MANAGEMENT REVIEW · doi:10.58765/ijemr.v1i3.121 · Cited by 2 · see citing works
Citation Concept Analysis (CCA) of concepts introduced by Pierre Bourdieu: Measuring their impact across fields and periods
Bornmann, L., Crothers, C. H. G., & Haunschild, R. (2025) · Journal of Data and Information Science · doi:10.2478/jdis-2025-0009 · Cited by 2 · see citing works
The moderating role of knowledge disruption in knowledge flow across cognitive distances
Lee, I., & Cho, K. (2025) · Asian Journal of Technology Innovation · doi:10.1080/19761597.2025.2518526 · Cited by 2 · see citing works
Still no enthusiasm for anonymous or pseudonymous publication
Calver, M. (2021) · Pacific Conservation Biology · doi:10.1071/pcv27n3_ed · Cited by 2 · see citing works
Simplifying the management of scientific reference and citation with a minimalist DOI-BibTex-LaTeX approach
Li, W. (2018) · Clinical Microbiology and Infectious Diseases · doi:10.15761/cmid.1000139 · Cited by 1 · see citing works
Exploiting Structural Similarities to Classify Citations
Ahmed, M., & Afzal, M. T. (2020) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2020.012619 · Cited by 1 · see citing works
Metrics as Proxies: A Theoretical and Empirical Study of Reasonings Based on Bibliometric Indicators
Remisiewicz, Ł. (2025) · Higher Education Quarterly · doi:10.1111/hequ.70095 · Cited by 1 · see citing works
Properties of Kinesiology-Related Journals According to Cabells Journalytics Not Apparent in Other Journal Metrics
Knudson, D. V. (2025) · Quest · doi:10.1080/00336297.2025.2603938 · Cited by 1 · see citing works
Unveiling Organizational Career Patterns: A Bibliometric Investigation
Yılmaz, S. E. (2026) · Human Resource Development Quarterly · doi:10.1002/hrdq.70010 · Cited by 1 · see citing works
Usage and impact of global biodata resources
Cummins, C., Imker, H. J., Cook, C. E., & Cochrane, G. (2026) · Bioinformatics Advances · doi:10.1093/bioadv/vbag219 · Cited by 1 · see citing works
DİJİTAL DÖNÜŞÜM IŞIĞINDA İÇ DENETİM ÜZERİNE BİBLİYOMETRİK ANALİZ YAKLAŞIMI
Erişen, O. (2024) · Denetişim · doi:10.58348/denetisim.1530516 · Cited by 1 · see citing works
Systematic Analysis of the Factors Contributing to the Variation and Change of the Microbiome
Aiello, K. D. (2018) · Cited by 1 · see citing works
Productividad científica profesional mediante indicadores de reconocimiento o visibilidad y de pertenencia repulsiva
Centro de Investigaciones Avanzadas y Formación Superior en Educación, S. Y. M. A. “. P., Pérez, G. A., Guillen, E. A., Universidad Nacional “San Luis Gonzaga”, F. D. M. H. I., Pari-Olarte, J. B., Universidad Nacional “San Luis Gonzaga”, F. D. F. Y. B. I., García, C. G. S., Universidad Nacional “San Luis Gonzaga”, F. D. O. I., Chacaltana-Ramos, L. J., Universidad Nacional “San Luis Gonzaga”, F. D. F. Y. B. I., et al. (2019) · Campus · doi:10.24265/campus.2019.v24n28.05 · Cited by 1 · see citing works
16 Messung von Forschungsleistungen? Was gemessen wird und was gemessen werden will
Ochsner, M. (2020) · De Gruyter eBooks · doi:10.1515/9783110689884-017 · Cited by 1 · see citing works
Biomedical articles share annotations with their citation neighbors
Rodriguez‐Esteban, R. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04044-4 · Cited by 1 · see citing works
Preventing information distortion in the conservation literature
Calver, M. (2019) · Pacific Conservation Biology · doi:10.1071/pcv25n3_ed
Pacific Conservation Biology and Clarivate’s Web of Science
Calver, M. (2020) · Pacific Conservation Biology · doi:10.1071/pcv26n3_ed
Bibliographic link prediction using contrast resampling technique
Краснов, Ф., Smaznevich, I. S., & Baskakova, E. N. (2021) · Computer Research and Modeling · doi:10.20537/2076-7633-2021-13-6-1317-1336
Michael H. MacRoberts (28 September 1941–18 December 2021)
Reid, C. S. (2022) · Journal of the Botanical Research Institute of Texas · doi:10.17348/jbrit.v16.i2.1271
The popularisation of self‐care: Tracing the dissemination of Orem's Self‐Care Deficiency Nursing Theory into the scientific disciplines
Yayla, K., Yilmaz, M. C., & Sarı, H. Y. (2023) · Health Information & Libraries Journal · doi:10.1111/hir.12476
Theoretical Frameworks
Petrovich, E. (2024) · Quantitative methods in the humanities and social sciences · doi:10.1007/978-3-031-53200-9_2
The Geo‐Socio‐Personal Model: Mapping Complex Information Ecosystems for Research‐related Information Behaviors
Ward, W., & Given, L. M. (2024) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1037
Citation Analysis of Journal of Breast Cancer: Targets and Therapy
Ravikumar, C. V., Raghunandana, M., Chandralekha, S. C., & Indrakumara (2025) · Pearl A Journal of Library and Information Science · doi:10.5958/0975-6922.2025.00002.4
The Impact of Metrics on Wellbeing, Meaning and Purpose in the Managerial UniversityManagerial University
Luca, E., Boell, S. K., Narayan, B., & Hovorka, D. S. (2025) · Scandinavian Journal of Information Systems · doi:10.17705/3sjis/037.08
Wissenschaftliche Bewertung: Peer-Review, Bibliometrie und Forschungsimpact-Bewertung
Redman, B. K. (2025) · Springer eBooks · doi:10.1007/978-3-031-92469-9_8
Bibliometrics of Measurement in Physical Education and Exercise Science Over the Last Twenty-Five Years
Knudson, D. V. (2026) · Measurement in Physical Education and Exercise Science · doi:10.1080/1091367x.2026.2621420
Zoological light under a bushel: using the case of natural history museum researchers to show how conventional bibliometric assessments of researcher productivity overlook important zoological research
Calver, M., & Hutchings, P. A. (2026) · Australian Zoologist · doi:10.1071/az26009
Mapping The Islam Nusantara Discourse: Bibliometric analysis of conceptual structure and key debates
Ashofi, N. A., & Anwar, R. K. (2026) · el Harakah Jurnal Budaya Islam · doi:10.18860/eh.v28i1.38094
Will Generative Artificial Intelligence-Generated Knowledge Contribute to the Junkification of Science, and Does it have Intrinsic Commercial Value?
da Silva, J. A. T. (2026) · مجلة التمكين الاجتماعي · doi:10.34118/sej.v8i4.4906

Authors: MacRoberts, M. H., & MacRoberts, B. R.
Affiliations: Louisiana State University in Shreveport (US)
Corresponding author: MacRoberts, Michael H.
Venue: Journal of the Association for Information Science and Technology
Publisher: Wiley
Published: 2017-11-17
Volume: 69
Issue: 3
Pages: 474-482
ISSN: 2330-1635 (ISSN-L)
2330-1643
DOI: 10.1002/asi.23970
OpenAlex: W2769543753
Semantic Scholar: cbf1fdb99648156e7bb31c3d04d51151f214108d
Cited by: 101
References: 88
Citations by year: 2026: 5 · 2025: 12 · 2024: 11 · 2023: 9 · 2022: 8 · 2021: 15 · 2020: 17 · 2019: 11 · 2018: 13
FWCI: 10.05
Type: journal article
Language: en
Open access: closed
Licence: onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations: Journal of the Association for Information Science and Technology (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords: citation analysis, research evaluation, bibliometric analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

The Mismeasure of Science: Citation Analysis

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks