Multidimensional Assessment of Scholarly Research Impact
Abstract
This article introduces the Multidimensional Research Assessment Matrix of scientific output. Its base notion holds that the choice of metrics to be applied in a research assessment process depends on the unit of assessment, the research dimension to be assessed, and the purposes and policy context of the assessment. An indicator may by highly useful within one assessment process, but less so in another. For instance, publication counts are useful tools to help discriminate between those staff members who are research active, and those who are not, but are of little value if active scientists are to be compared with one another according to their research performance. This paper gives a systematic account of the potential usefulness and limitations of a set of 10 important metrics, including altmetrics, applied at the level of individual articles, individual researchers, research groups, and institutions. It presents a typology of research impact dimensions and indicates which metrics are the most appropriate to measure each dimension. It introduces the concept of a “meta‐analysis” of the units under assessment in which metrics are not used as tools to evaluate individual units, but to reach policy inferences regarding the objectives and general setup of an assessment process.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Only a minority of stakeholders employing bibliometric indicators are adequately familiar with their meaning and appropriate use.
“Yet, only a minority of stakeholders employing bibliometric indicators are adequately familiar with their meaning and appropriate use [2,4,5,7].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [5]
Citations may be appropriate for assessing scientific impact of research, but they do not show impact outside the scientific community.
“Citations may be appropriate in assessing the scientific impact of the research, but they do not show the impact of the research outside the scientific community [1,3,5,6,260,264,265].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [5]
Scientific impact of any evaluated entity cannot be determined solely by citations due to different citing behaviors and incompatible citation counts between disciplines.
“It is often argued that the scientific impact of any evaluated entity cannot be determined solely by citations due to the different citing behaviors and resulting incompatible citation counts between disciplines [1-3,5,260,267].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [5]
Publication data and citation metrics continue to play the most important role in research assessments worldwide, so indicator-based research assessments remain at the center of global debate about their validity and influence.
“because publication data and citation metrics continue to play the most important role in research assessments worldwide, indicator-based research assessments also remain at the center of the global debate, not only about their validity, but also about their influence on research activities, scientific output, and its quality [1,5-7,44,260,265,289].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [5]
WoS and Scopus data are questionable for Social Sciences & Humanities because these disciplines are clearly underrepresented in the DBs.
“The same can be said about the use of WoS and Scopus data in Social Sciences & Humanities, since these disciplines are clearly underrepresented in the DBs [5,44,47,88,91,163,263].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [5]
Cites
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669 · see citing works
Handbook of Quantitative Science and Technology Research
Moed, H. F., Glänzel, W., & Schmoch, U. (2005) · Springer eBooks · doi:10.1007/1-4020-2755-9 · Cited by 1,385 · see citing works
Self-citationThe economic benefits of publicly funded basic research: a critical review
Salter, A. J., & Martin, B. R. (2001) · Research Policy · doi:10.1016/s0048-7333(00)00091-3 · Cited by 1,361 · see citing works
The Academic Ranking of World Universities
Liu, N. C., & Cheng, Y. (2005) · Higher Education in Europe · doi:10.1080/03797720500260116 · Cited by 1,078 · see citing works
Performance-based university research funding systems
Hicks, D. M. (2011) · Research Policy · doi:10.1016/j.respol.2011.09.007 · Cited by 1,025 · see citing works
van Raan, A. F. J. (2005) · Scientometrics · doi:10.1007/s11192-005-0008-6 · Cited by 760 · see citing works
Vollmar, H. C., Mayer, H., Ostermann, T., Butzlaff, M. E., Sandars, J. E., Wilm, S., & Rieger, M. A. (2010) · Implementation Science · doi:10.1186/1748-5908-5-1 · Cited by 693 · see citing works
Cultural sensitivity in substance use prevention
Resnicow, K., Soler, R. E., Braithwaite, R. L., Ahluwalia, J. S., & Butler, J. (2000) · Journal of Community Psychology · doi:10.1002/(sici)1520-6629(200005)28:3 · Cited by 637 · see citing works
Measuring Contextual Citation Impact of Scientific Journals
Moed, H. F. (2010) · Journal of Informetrics · doi:10.1016/j.joi.2010.01.002 · Cited by 628 · see citing works
Self-citationChance and Consensus in Peer Review
Cole, S., Cole, J. R., & Simon, G. A. (1981) · Science · doi:10.1126/science.7302566 · Cited by 623 · see citing works
Rankings and the Reshaping of Higher Education
Hazelkorn, E. (2011) · Palgrave Macmillan UK eBooks · doi:10.1057/9780230306394 · Cited by 566 · see citing works
Recent Developments in the Econometrics of Program Evaluation
Imbens, G. W., & Wooldridge, J. M. (2008) · National Bureau of Economic Research · doi:10.3386/w14251 · Cited by 557 · see citing works
A conceptual framework for transferring research to practice
Simpson, D. D. (2002) · Journal of Substance Abuse Treatment · doi:10.1016/s0740-5472(02)00231-3 · Cited by 538 · see citing works
Martin, B. R., & Irvine, J. M. (1983) · Research Policy · doi:10.1016/0048-7333(83)90005-7 · Cited by 529 · see citing works
Scientific and technical human capital: an alternative model for research evaluation
Bozeman, B. L., Dietz, J. S., & Gaughan, M. (2001) · International Journal of Technology Management · doi:10.1504/ijtm.2001.002988 · Cited by 429 · see citing works
"Science Citation Index"—A New Dimension in Indexing
Garfield, E. (1964) · Science · doi:10.1126/science.144.3619.649 · Cited by 427 · see citing works
Martínez‐Monés, A., Dimitriadis, Y. A., Avi, B. R., Gómez, E., & de la Fuente, P. (2003) · Computers & Education · doi:10.1016/j.compedu.2003.06.001 · Cited by 420 · see citing works
van Raan, A. F. J. (1996) · Scientometrics · doi:10.1007/bf02129602 · Cited by 390 · see citing works
Alberts, B. (2013) · Science · doi:10.1126/science.1240319 · Cited by 389 · see citing works
Mediating instruments and making markets: Capital budgeting, science and the economy
Miller, P., & O'Leary, T. (2007) · Accounting Organizations and Society · doi:10.1016/j.aos.2007.02.003 · Cited by 384 · see citing works
Narin, F. (1994) · Scientometrics · doi:10.1007/bf02017219 · Cited by 359 · see citing works
The use of multiple indicators in the assessment of basic research
Martin, B. R. (1996) · Scientometrics · doi:10.1007/bf02129599 · Cited by 356 · see citing works
Moving People and Knowledge: Scientific Mobility in the European Union1
Ackers, L. (2005) · International Migration · doi:10.1111/j.1468-2435.2005.00343.x · Cited by 341 · see citing works
Patent citation analysis.A closer look at the basic input data from patent search reports
Michel, J., & Bettels, B. (2001) · Scientometrics · doi:10.1023/a:1010577030871 · Cited by 336 · see citing works
Recent developments in the econometrics of program evaluation
Wooldridge, J. M., & Imbens, G. W. (2008) · doi:10.1920/wp.cem.2008.2408 · Cited by 323 · see citing works
van Raan, A. F. J. (2004) · Springer eBooks · doi:10.1007/1-4020-2755-9_2 · Cited by 279 · see citing works
Citation analysis in research evaluation
Taris, T. W. (2006) · Gedrag & Organisatie · doi:10.5117/2006.019.002.007 · Cited by 273 · see citing works
Abbott, A., Cyranoski, D., Jones, N., Maher, B., Schiermeier, Q., & Noorden, R. V. (2010) · Nature · doi:10.1038/465860a · Cited by 261 · see citing works
Altmetric: enriching scholarly content with article‐level discussion and metrics
Adie, E. A., & William, R. O. E. (2012) · Learned Publishing · doi:10.1087/20130103 · Cited by 224 · see citing works
Should you believe in the Shanghai ranking?: An MCDM view
Billaut, J.-C., Bouyssou, D., & Vincke, P. (2010) · Scientometrics · doi:10.1007/s11192-009-0115-x · Cited by 219 · see citing works
Fetterman, D. M. (1994) · Evaluation Practice · doi:10.1016/0886-1633(94)90055-8 · Cited by 205 · see citing works
University Rankings in Critical Perspective
Pusser, B., & Marginson, S. (2013) · The Journal of Higher Education · doi:10.1353/jhe.2013.0022 · Cited by 196 · see citing works
Articles with short titles describing the results are cited more often
Paiva, C. E., Lima, J. P. D. S. N., & Paiva, B. S. R. (2012) · Clinics · doi:10.6061/clinics/2012(05)17 · Cited by 196 · see citing works
Simons, K. (2008) · Science · doi:10.1126/science.1165316 · Cited by 178 · see citing works
Altmetrics: Rethinking the Way We Measure
Galligan, F., & Dyas-Correia, S. (2013) · Serials Review · doi:10.1016/j.serrev.2013.01.003 · Cited by 155 · see citing works
Altmetrics: Rethinking the Way We Measure
Galligan, F., & Dyas-Correia, S. (2013) · Serials Review · doi:10.1080/00987913.2013.10765486 · Cited by 155 · see citing works
Citation analysis and the complex problem of intellectual influence
Zuckerman, H. (1987) · Scientometrics · doi:10.1007/bf02016675 · Cited by 149 · see citing works
Fetterman, D. M., & Wandersman, A. (2007) · American Journal of Evaluation · doi:10.1177/1098214007301350 · Cited by 147 · see citing works
Kurtz, M. J., & Bollen, J. (2010) · Annual Review of Information Science and Technology · doi:10.1002/aris.2010.1440440108 · Cited by 146 · see citing works
Motivations for citation: A comparison of self citation and citation to others
Bonzi, S., & Snyder, H. W. (1991) · Scientometrics · doi:10.1007/bf02017571 · Cited by 145 · see citing works
Bornmann, L., & Marx, W. (2013) · Scientometrics · doi:10.1007/s11192-013-1161-y · Cited by 143 · see citing works
Achievement index climbs the ranks
Ball, P. (2007) · Nature · doi:10.1038/448737a · Cited by 138 · see citing works
Zellner, C. (2003) · Research Policy · doi:10.1016/s0048-7333(03)00080-5 · Cited by 122 · see citing works
Sugimoto, C. R., Larivière, V., Ni, C., & Cronin, B. (2013) · Journal of Informetrics · doi:10.1016/j.joi.2013.08.007 · Cited by 120 · see citing works
Towards Research Performance in the Humanities.
Moed, H. F., Luwel, M., & Nederhof, A. J. (2002) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 108 · see citing works
Self-citationThe SJR indicator: A new indicator of journals' scientific prestige
Guerrero‐Bote, V. P., Pereira, B. G., & Anegón, F. D. M. (2009) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 105 · see citing works
Presidential Address: Empowerment Evaluation.
Fetterman, D. M. (1994) · Evaluation Practice · Cited by 102 · see citing works
Usage impact factor: The effects of sample characteristics on usage‐based impact metrics
Bollen, J., & Herbert Van De Sompel, A. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20746 · Cited by 99 · see citing works
Comparison of citation and usage indicators: the case of oncology journals
Schlögl, C., & Gorraiz, J. I. (2010) · Scientometrics · doi:10.1007/s11192-010-0172-1 · Cited by 97 · see citing works
Linkage between patents and papers: An interim EPO/US comparison
Narin, F., & Olivastro, D. (1998) · Scientometrics · doi:10.1007/bf02457966 · Cited by 94 · see citing works
Evolving regimes of multi-university research evaluation
Hicks, D. M. (2008) · Higher Education · doi:10.1007/s10734-008-9154-0 · Cited by 92 · see citing works
White Paper on Electronic Journal Usage Statistics
Luther, J. (2002) · The Serials Librarian · doi:10.1300/j123v41n02_10 · Cited by 89 · see citing works
Translating three states of knowledge–discovery, invention, and innovation
Lane, J. P., & Flagg, J. L. (2010) · Implementation Science · doi:10.1186/1748-5908-5-9 · Cited by 87 · see citing works
The value and use of social media as communication tool in the plant sciences
Osterrieder, A. (2013) · Plant Methods · doi:10.1186/1746-4811-9-26 · Cited by 86 · see citing works
Counting publications and citations: Is more always better?
Waltman, L., van Eck, N. J., & Wouters, P. F. (2013) · Journal of Informetrics · doi:10.1016/j.joi.2013.04.001 · Cited by 80 · see citing works
The influences of counting methods on university rankings based on paper count and citation count
Lin, C., Huang, M., & Chen, D. (2013) · Journal of Informetrics · doi:10.1016/j.joi.2013.03.007 · Cited by 76 · see citing works
Is the impact of journal impact factors decreasing?
Reedijk, J., & Moed, H. F. (2008) · Journal of Documentation · doi:10.1108/00220410810858001 · Cited by 71 · see citing works
Self-citationBornmann, L., Mutz, R., & Daniel, H. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22857 · Cited by 64 · see citing works
Institutional Repositories: Exploration of Costs and Value
Burns, C. S., Lana, A. L., & Budd, J. M. (2013) · D-Lib Magazine · doi:10.1045/january2013-burns · Cited by 57 · see citing works
Performance-based Funding for Public Research in Tertiary Education Institutions
OECD (2010) · OECD eBooks · doi:10.1787/9789264094611-en · Cited by 55 · see citing works
Clinical and Translational Scientist Career Success: Metrics for Evaluation
Lee, L., Pusek, S. N., McCormack, W. T., Helitzer, D., Martina, C. A., Dozier, A. M., Ahluwalia, J. S., Schwartz, L., McManus, L. M., Reynolds, B. D., et al. (2012) · Clinical and Translational Science · doi:10.1111/j.1752-8062.2012.00422.x · Cited by 52 · see citing works
Which factors explain the Web impact of scientists' personal homepages?
Barjak, F., Li, X., & Thelwall, M. A. (2006) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20476 · Cited by 52 · see citing works
Nederhof, A. J., Luwel, M., & Moed, H. F. (2001) · Scientometrics · doi:10.1023/a:1010533232688 · Cited by 43 · see citing works
Self-citationEponymy and Obliteration by Incorporation: The case of the “Nash Equilibrium”
McCain, K. W. (2011) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21536 · Cited by 41 · see citing works
O’Connell, C. (2012) · Higher Education · doi:10.1007/s10734-012-9572-x · Cited by 41 · see citing works
Consuming Article-Level Metrics: Observations and Lessons
Chamberlain, S. (2013) · Information Standards Quarterly · doi:10.3789/isqv25no2.2013.02 · Cited by 40 · see citing works
Tracing scientist’s research trends realtimely
Wang, X., Wang, Z., & Xu, S. (2012) · Scientometrics · doi:10.1007/s11192-012-0884-5 · Cited by 37 · see citing works
Wandersman, A., & Snell‐Johns, J. (2005) · American Journal of Evaluation · doi:10.1177/1098214005278774 · Cited by 34 · see citing works
Katz, J. S., & Hicks, D. M. (1997) · Scientometrics · doi:10.1007/bf02461128 · Cited by 34 · see citing works
Taylor, M. (2013) · Information Standards Quarterly · doi:10.3789/isqv25no2.2013.05 · Cited by 33 · see citing works
Assessing program needs and planning change
Rowan‐Szal, G. A., Greener, J. M., Joe, G. W., & Simpson, D. D. (2007) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2006.12.028 · Cited by 27 · see citing works
Vale, R. D. (2012) · Molecular Biology of the Cell · doi:10.1091/mbc.e12-06-0490 · Cited by 27 · see citing works
Huang, M., Chen, S., Lin, C.-Y., & Chen, D. (2013) · Scientometrics · doi:10.1007/s11192-013-1054-0 · Cited by 21 · see citing works
Calero‐Medina, C., López-Illescas, C., Visser, M. S., & Moed, H. F. (2008) · Research Evaluation · doi:10.3152/095820208x280907 · Cited by 20 · see citing works
Self-citationEditorial: Assessing Academic Researchers
Zare, R. N. (2012) · Angewandte Chemie International Edition · doi:10.1002/anie.201201011 · Cited by 17 · see citing works
Björk, B., & Paetau, P. (2012) · Bulletin of the American Society for Information Science and Technology · doi:10.1002/bult.2012.1720380512 · Cited by 17 · see citing works
Measuring the international usage of US institutional repositories
Fralinger, L., & Bull, J. (2013) · OCLC Systems & Services · doi:10.1108/oclc-10-2012-0039 · Cited by 14 · see citing works
Nanotechnology Public Funding and Impact Analysis: A Tale of Two Decades (1991-2010)
Chen, H., Roco, M. C., & Son, J. (2013) · IEEE Nanotechnology Magazine · doi:10.1109/mnano.2012.2237312 · Cited by 13 · see citing works
The use of big datasets in bibliometric research
Dr, M., & Henk, F. (2012) · Cited by 10 · see citing works
Wilson, V. (2013) · Evidence Based Library and Information Practice · doi:10.18438/b8q31p · Cited by 6 · see citing works
Flagged first, then by citations
2 of the 82 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696 · see citing works
Application of bibliometrics in medicine: a historical bibliometrics analysis.
Kokol, P., Vošner, H. B., & Završnik, J. (2020) · Health Information and Libraries Journal · doi:10.1111/hir.12295 · Cited by 481 · see citing works
Guidelines for Theory Selection: The IMPACT Framework
Hollebeek, L. D., Kumar, V., Srivastava, R. K., Lim, W. M., & Urbonavičius, S. (2025) · Psychology and Marketing · doi:10.1002/mar.70009 · Cited by 142 · see citing works
Bornmann, L., Haunschild, R., & Adams, J. R. (2019) · Journal of Informetrics · doi:10.1016/j.joi.2019.01.008 · Cited by 109 · see citing works
Researcher and Author Impact Metrics: Variety, Value, and Context
Gasparyan, A. Y., Yessirkepov, M., Duisenova, A., Trukhachev, V. I., Kostyukova, E. I., & Kitas, G. D. (2018) · Journal of Korean Medical Science · doi:10.3346/jkms.2018.33.e139 · Cited by 97 · see citing works
Using altmetrics for contextualised mapping of societal impact: From hits to networks
Robinson‐García, N., van Leeuwen, T. N., & Ràfols, I. (2018) · Science and Public Policy · doi:10.1093/scipol/scy024 · Cited by 89 · see citing works
Haunschild, R., & Bornmann, L. (2017) · Scientometrics · doi:10.1007/s11192-016-2237-2 · Cited by 77 · see citing works
Altmetrics and societal impact measurements: Match or mismatch? A literature review
Tahamtan, I., & Bornmann, L. (2020) · El Profesional de la Informacion · doi:10.3145/epi.2020.ene.02 · Cited by 75 · see citing works
Haunschild, R., Loet Leydesdorff, A., Bornmann, L., Hellsten, I., & Marx, W. (2019) · Journal of Informetrics · doi:10.1016/j.joi.2019.03.008 · Cited by 68 · see citing works
Jappe, A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231735 · Cited by 64 · see citing works
Openness and Impact of Leading Scientific Countries
Wagner, C. S., Whetsell, T., Baas, J., & Jonkers, K. (2018) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2018.00010 · Cited by 64 · see citing works
Normalisation of citation impact in economics
Bornmann, L., & Wohlrabe, K. (2019) · Scientometrics · doi:10.1007/s11192-019-03140-w · Cited by 62 · see citing works
Research impact assessment in agriculture—A review of approaches and impact areas
Weißhuhn, P., Helming, K., & Ferretti, J. (2017) · Research Evaluation · doi:10.1093/reseval/rvx034 · Cited by 61 · see citing works
Envisioning the scientific paper of the future
Sopinka, N., Coristine, L. E., DeRosa, M. C., Rochman, C. M., Owens, B. L., & Cooke, S. J. (2020) · FACETS · doi:10.1139/facets-2019-0012 · Cited by 54 · see citing works
Jappe, A., Pithan, D. M., & Heinze, T. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199031 · Cited by 56 · see citing works
Data integration for research and innovation policy: an Ontology-Based Data Management approach
Daraio, C. C., Maurizio Lenzerini, B., Leporelli, C., Moed, H. F., Naggar, P., Bonaccorsi, A. J., & Bartolucci, A. (2015) · Scientometrics · doi:10.1007/s11192-015-1814-0 · Cited by 46 · see citing works
Self-citationWeb Indicators for Research Evaluation: A Practical Guide
Thelwall, M. A. (2016) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00733ed1v01y201609icr052 · Cited by 45 · see citing works
Zhang, X., Wang, X., Zhao, H., de Pablos, P. O., Sun, Y., & Xiong, H. (2019) · Scientometrics · doi:10.1007/s11192-019-03088-x · Cited by 41 · see citing works
Grand challenges in data integration—state of the art and future perspectives: an introduction
Daraio, C. C., & Glänzel, W. (2016) · Scientometrics · doi:10.1007/s11192-016-1914-5 · Cited by 38 · see citing works
Scholarly reputation building in the digital age: an activity-specific approach. Review article
Herman, E., & Nicholas, D. (2019) · El Profesional de la Informacion · doi:10.3145/epi.2019.ene.02 · Cited by 39 · see citing works
Bornmann, L., & Haunschild, R. (2016) · RePEc: Research Papers in Economics · Cited by 38 · see citing works
Braithwaite, J., Herkes, J., Churruca, K., Long, J. C., Pomare, C., Boyling, C., Bierbaum, M., Clay‐Williams, R., Rapport, F., Shih, P., et al. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-025320 · Cited by 38 · see citing works
Bornmann, L., & Haunschild, R. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.07.002 · Cited by 43 · see citing works
Artificial intelligence in higher education. A protocol paper for a systematic literature review
Ismail, F. M. M., Tan, E., Rudolph, J., Crawford, J., & Tan, S. (2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.2.34 · Cited by 39 · see citing works
Más-Bleda, A., & Thelwall, M. A. (2016) · Scientometrics · doi:10.1007/s11192-016-2118-8 · Cited by 36 · see citing works
Making visible the invisible through the analysis of acknowledgements in the humanities
Díaz-Faes, A. A., & Bordons, M. (2017) · Aslib Journal of Information Management · doi:10.1108/ajim-01-2017-0008 · Cited by 36 · see citing works
Wu, D., Li, J., Lu, X., & Li, J. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.03.008 · Cited by 33 · see citing works
Forthmann, B., & Doebler, P. (2021) · Scientometrics · doi:10.1007/s11192-021-03864-8 · Cited by 30 · see citing works
Ma, T., Jiao, J., Guo, D., Lv, S.-Z., Zhang, D., & Hou, D.-C. (2024) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-024-05099-8 · Cited by 29 · see citing works
Hou, J., Yang, X., & Chen, C. (2020) · PLoS ONE · doi:10.1371/journal.pone.0234347 · Cited by 28 · see citing works
Kurtz, M. J., & Henneken, E. A. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23689 · Cited by 26 · see citing works
Li, J., Wu, D., Li, J., & Li, J. (2017) · Information Processing & Management · doi:10.1016/j.ipm.2017.05.002 · Cited by 26 · see citing works
Investigating disciplinary differences in the relationships between citations and downloads
Vaughan, L. Q., Tang, J., & Yang, R. (2017) · Scientometrics · doi:10.1007/s11192-017-2308-z · Cited by 25 · see citing works
The inverted U-shaped relationship between knowledge diversity of researchers and societal impact
Wang, G., Gan, Y., & Yang, H. (2022) · Scientific Reports · doi:10.1038/s41598-022-21821-0 · Cited by 23 · see citing works
Openness and Impact of Leading Scientific Countries
Wagner, C. S., Whetsell, T. A., Baas, J., & Jonkers, K. (2018) · Joint Research Centre (European Commission) · doi:10.3389/frma.2018.00010/full · Cited by 23 · see citing works
Bibliometric differences – a case study in bibliometric evaluation across SSH and STEM
Melchiorsen, P. M. (2018) · Journal of Documentation · doi:10.1108/jd-07-2018-0108 · Cited by 22 · see citing works
Bornmann, L. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00089 · Cited by 21 · see citing works
Econometric Approaches to the Measurement of Research Productivity
Daraio, C. C. (2019) · Springer handbooks · doi:10.1007/978-3-030-02511-3_24 · Cited by 20 · see citing works
Koya, K., & Chowdhury, G. G. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179722 · Cited by 22 · see citing works
Evaluation and comparison of influence in international Open Access journals between China and USA
Zhao, R., & Wang, X. (2019) · Scientometrics · doi:10.1007/s11192-019-03159-z · Cited by 20 · see citing works
Identifying landmark publications in the long run using field-normalized citation data
Bornmann, L., Ye, A. Y., & Ye, F. Y. (2018) · Journal of Documentation · doi:10.1108/jd-07-2017-0108 · Cited by 18 · see citing works
Societal impact of research: a text mining study of impact types
Zheng, H., Pee, L. G., & Zhang, D. (2021) · Scientometrics · doi:10.1007/s11192-021-04096-6 · Cited by 19 · see citing works
Yang, S., Xing, X., Qi, F., & Grácio, M. C. C. (2021) · Scientometrics · doi:10.1007/s11192-020-03808-8 · Cited by 18 · see citing works
Gorraiz, J. I., Wieland, M., & Gumpenberger, C. (2016) · El Profesional de la Informacion · doi:10.3145/epi.2016.nov.07 · Cited by 18 · see citing works
HADDAD, E. A., Mena‐Chalco, J. P., & Sidone, O. J. G. (2017) · Estudos Econômicos (São Paulo) · doi:10.1590/0101-416147414ejo · Cited by 18 · see citing works
Altmetrics as Traces of the Computerization of the Research Process
Moed, H. F. (2016) · De Gruyter eBooks · doi:10.1515/9783110308464-021 · Cited by 16 · see citing works
Self-citationVȋiu, G.-A. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.02.002 · Cited by 14 · see citing works
An integrated indicator for evaluating scientific papers: considering academic impact and novelty
Yan, Z., & Fan, K. (2024) · Scientometrics · doi:10.1007/s11192-024-05150-9 · Cited by 15 · see citing works
Assessing the health research’s social impact: a systematic review
Pedrini, M., Langella, V., Battaglia, M. A., & Zaratin, P. (2017) · Scientometrics · doi:10.1007/s11192-017-2585-6 · Cited by 14 · see citing works
Difference in the impact of open-access papers published by China and the USA
Yang, S., Xing, X., & Wolfram, D. (2018) · Scientometrics · doi:10.1007/s11192-018-2697-7 · Cited by 15 · see citing works
Innovation and International High Skilled Migration
Mihi‐Ramírez, A., Rodríguez, Y., & García, E. C. (2016) · Engineering Economics · doi:10.5755/j01.ee.27.4.14396 · Cited by 15 · see citing works
How Efficiently Do Elite US Universities Produce Highly Cited Papers?
Wohlrabe, K., Anegón, F. D. M., & Bornmann, L. (2019) · Publications · doi:10.3390/publications7010004 · Cited by 15 · see citing works
Robinson‐García, N., & Ràfols, I. (2020) · Springer eBooks · doi:10.1007/978-3-030-47665-6_10 · Cited by 15 · see citing works
Efficiency of research performance and the glass researcher
Bornmann, L., & Haunschild, R. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2015.11.009 · Cited by 13 · see citing works
Performance of citations and altmetrics in the social sciences and humanities
Yang, S., & Zheng, M. (2019) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.69 · Cited by 13 · see citing works
An Initiative to Track Sentiments in Altmetrics
Halevi, G., & Schimming, L. (2018) · Journal of Altmetrics · doi:10.29024/joa.1 · Cited by 12 · see citing works
Self-citationHow to measure interdisciplinary research? A systemic design for the model of measurement
Cantone, G. G. (2024) · Scientometrics · doi:10.1007/s11192-024-05085-1 · Cited by 15 · see citing works
Bozeman, B. L., Lindsay, S., Nelson, J. P., & Bretschneider, S. I. (2023) · Public Management Review · doi:10.1080/14719037.2023.2252819 · Cited by 12 · see citing works
Research on the impact evaluation of academic journals based on altmetrics and citation indicators
Zhao, R., Wang, X., Liu, Z., Qi, Y., Zhang, Z., & Chang, R. (2019) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.27 · Cited by 12 · see citing works
Hanna, C. R., Boyd, K., & Jones, R. J. (2021) · Health Research Policy and Systems · doi:10.1186/s12961-020-00658-x · Cited by 13 · see citing works
Reasons and consequences of changes in Russian research assessment policies
Kosyakov, D. V., & Гуськов, А. Е. (2022) · Scientometrics · doi:10.1007/s11192-022-04469-5 · Cited by 11 · see citing works
Research on impact evaluation of academic journals from multidimensional perspective
Zhao, R., & Wang, X. (2019) · Library Hi Tech · doi:10.1108/lht-03-2019-0067 · Cited by 11 · see citing works
De Profundis: A Decade of Bibliometric Services Under Scrutiny
Gorraiz, J. I., Wieland, M., Ulrych, U., & Gumpenberger, C. (2020) · Springer eBooks · doi:10.1007/978-3-030-47665-6_11 · Cited by 10 · see citing works
When the Data Don’t Mean What They Say: Japan’s Comparative Underperformance in Citation Impact
Pendlebury, D. (2020) · Springer eBooks · doi:10.1007/978-3-030-47665-6_5 · Cited by 10 · see citing works
Arboledas-Lérida, L. (2021) · Social Epistemology · doi:10.1080/02691728.2021.1919237 · Cited by 10 · see citing works
Altmetrics in the evaluation of scholarly impact: a systematic and critical literature review
González, P., Fors, M. M., & Torres, A. (2025) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2025.1693304 · Cited by 9 · see citing works
The significant yet short‐term influence of research covidization on journal citation metrics
Zheng, X. M., & Ni, C. (2024) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24939 · Cited by 9 · see citing works
Sob a lente da Ciência Aberta: Olhares de Portugal, Espanha e Brasil
Borges, M. M., & Casado, E. S. (2021) · Imprensa da Universidade de Coimbra eBooks · doi:10.14195/978-989-26-2022-0 · Cited by 9 · see citing works
Martel, M. J. M., Mihi‐Ramírez, A., & Arteaga-Ortíz, J. (2017) · Engineering Economics · doi:10.5755/j01.ee.28.2.17518 · Cited by 9 · see citing works
Bibliometrics in computer science: An institution ranking
Zurita, G., Merigó, J. M., Lobos-Ossandón, V., & Mulet-Forteza, C. (2020) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-179636 · Cited by 9 · see citing works
Using Almetrics for Contextualised Mapping of Societal Impact: From Hits to NetworksPreprint
Robinson‐García, N., van Leeuwen, T. N., & Ràfols, I. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2932944 · Cited by 9 · see citing works
Bornmann, L., Gralka, S., Anegón, F. D. M., & Wohlrabe, K. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2023.101400 · Cited by 8 · see citing works
Use and Impacts of Campbell Systematic Reviews on Policy, Practice, and Research
Maynard, B. R., & Dell, N. A. (2017) · Research on Social Work Practice · doi:10.1177/1049731517722637 · Cited by 8 · see citing works
Measuring university size: A comparison of academic personnel versus scientific talent pool data
Lepori, B., Bornmann, L., & Anegón, F. D. M. (2023) · Quantitative Science Studies · doi:10.1162/qss_a_00273 · Cited by 9 · see citing works
Lazarus, S. I. (2024) · Life Writing · doi:10.1080/14484528.2024.2430666 · Cited by 7 · see citing works
Bornmann, L. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23546 · Cited by 7 · see citing works
Ismail, M. F. B. M., Crawford, J., Tan, S., Rudolph, J., Tan, E., Seah, P., Tang, F. X., Ng, F., Kaldenbach, L. V., Naidu, A. A., et al. (2024) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2024.7.1.35 · Cited by 7 · see citing works
Sapientia: The ontology of multi-dimensional research assessment
Daraio, C. C., Maurizio Lenzerini, B., Leporelli, C., Moed, H. F., Naggar, P., Bonaccorsi, A. J., & Bartolucci, A. (2015) · IRIS Research product catalog (Sapienza University of Rome) · Cited by 7 · see citing works
Self-citationToward New Indicators of a Journal’s Manuscript Peer Review Process
Moed, H. F. (2016) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2016.00005 · Cited by 7 · see citing works
Self-citationAbramo, G., & D’Angelo, C. A. (2020) · Science and Public Policy · doi:10.1093/scipol/scaa048 · Cited by 7 · see citing works
Opium in science and society: NumbersPreprint
Marewski, J. N., & Bornmann, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.11210 · Cited by 7 · see citing works
A framework of frameworks: theoretical, conceptual, contextual, methodological and impact frameworks
Lim, W. M. (2026) · Journal of Trade Science · doi:10.1108/jts-08-2025-0042 · Cited by 6 · see citing works
Zeng, Z., Jia, C., Zhang, W., Zhuang, X., & Li, X. (2024) · Journal of Information Science · doi:10.1177/01655515241238401 · Cited by 6 · see citing works
Vȋiu, G.-A. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.04.003 · Cited by 6 · see citing works
Ranking bibliométrico internacional Scimago: una realidad para las universidades peruanas
Morales, M. K., & Morales, S. K. (2022) · Revista Venezolana de Gerencia · doi:10.52080/rvgluz.27.7.28 · Cited by 8 · see citing works
Bornmann, L., Mutz, R., Haunschild, R., Anegón, F. D. M., Clemente, M. D. A. M., & Stefaner, M. (2021) · Scientometrics · doi:10.1007/s11192-021-04141-4 · Cited by 6 · see citing works
Bornmann, L., Gralka, S., Anegón, F. D. M., & Wohlrabe, K. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3555584 · Cited by 6 · see citing works
Opportunism vs. Excellence in Academia: Quality Accreditation of Collegiate Business Schools
Faria, J. R., & Mixon, F. G. (2022) · American Business Review · doi:10.37625/abr.25.1.4-24 · Cited by 5 · see citing works
Opium in science and society: numbers and other quantifications
Bornmann, L., & Marewski, J. N. (2024) · Scientometrics · doi:10.1007/s11192-024-05104-1 · Cited by 12 · see citing works
Navarro, I. L., Fernández, A. I. M., & Rey-Rocha, J. (2017) · Teknokultura Revista de Cultura Digital y Movimientos Sociales · doi:10.5209/tekn.54142 · Cited by 12 · see citing works
Measuring Impact of Academic Research in Computer and Information Science on Society
Koya, K., & Chowdhury, G. G. (2020) · Asia Pacific Information Technology Conference (APIT) · doi:10.1145/3379310.3379312 · Cited by 5 · see citing works
Ukwoma, S. C., & Ngulube, P. (2021) · American Journal of Distance Education · doi:10.1080/08923647.2021.1928432 · Cited by 5 · see citing works
Altmetrics as an Answer to the Need for Democratization of Research and Its Evaluation
Daraio, C. C. (2021) · Journal of Altmetrics · doi:10.29024/joa.43 · Cited by 5 · see citing works
Tsai, K. M., Perez‐Brena, N. J., Perez, V. M., & Camacho‐Thompson, D. E. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-818872-9.00094-7 · Cited by 6 · see citing works
International university rankings: composite indicators and methodological approaches
Latifis, K., Sidiropoulos, A., & Evangelidis, G. (2026) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2026.1828850 · Cited by 4 · see citing works
Cóser, I., Brandão, M. A. F., Raposo, N. R. B., & Gonçalves, E. (2018) · Cadernos de Saúde Pública · doi:10.1590/0102-311x00097517 · Cited by 4 · see citing works
Robayo-Castro, B. H., Rico, J. L., Hurtado‐Parrado, C., & Ortega, L. A. (2017) · Universitas Psychologica · doi:10.11144/javeriana.upsy15-5.icpa · Cited by 4 · see citing works
Altmetria jako przedmiot zainteresowania bibliologii i informatologii
Kowalska, M. (2017) · Przegląd Biblioteczny · doi:10.36702/pb.457 · Cited by 4 · see citing works
Investigating Assessment Standards in the Netherlands, Italy, and the United Kingdom
Petersohn, S., Biesenbender, S., & Thiedig, C. (2020) · Advances in standardization research · doi:10.4018/978-1-7998-2181-6.ch003 · Cited by 4 · see citing works
Daraio, C. C., Leo, S. D., & Loet Leydesdorff, A. (2022) · Scientometrics · doi:10.1007/s11192-022-04551-y · Cited by 4 · see citing works
- Authors
- Moed, H. F., & Halevi, G.
- Affiliations
- Informetric Research Group Elsevier Radarweg 29 1043 NX Amsterdam The Netherlands
- RELX Group (United States) (US)
- Venue
- Journal of the Association for Information Science and Technology
- Publisher
- Wiley
- Published
- 2015-07-07
- Volume
- 66
- Issue
- 10
- Pages
- 1988-2002
- ISSN
- 2330-1635 (ISSN-L)
- 2330-1643
- OpenAlex
- W1696956904
- Cited by
- 168
- References
- 82
- Citations by year
- 2026: 14 · 2025: 9 · 2024: 16 · 2023: 13 · 2022: 11 · 2021: 16 · 2020: 20 · 2019: 17 · 2018: 15 · 2017: 18 · 2016: 15 · 2015: 4
- FWCI
- 15.79
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the Association for Information Science and Technology (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- research evaluation, research impact, research metrics, altmetrics, publication counts, research performance, science policy, research output
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
