Assessing scientists for hiring, promotion, and tenure
Open accessAbstract
Assessment of researchers is necessary for decisions of hiring, promotion, and tenure. A burgeoning number of scientific leaders believe the current system of faculty incentives and rewards is misaligned with the needs of society and disconnected from the evidence about the causes of the reproducibility crisis and suboptimal quality of the scientific publication record. To address this issue, particularly for the clinical and life sciences, we convened a 22-member expert panel workshop in Washington, DC, in January 2017. Twenty-two academic leaders, funders, and scientists participated in the meeting. As background for the meeting, we completed a selective literature review of 22 key documents critiquing the current incentive system. From each document, we extracted how the authors perceived the problems of assessing science and scientists, the unintended consequences of maintaining the status quo for assessing scientists, and details of their proposed solutions. The resulting table was used as a seed for participant discussion. This resulted in six principles for assessing scientists and associated research and policy implications. We hope the content of this paper will serve as a basis for establishing best practices and redesigning the current approaches to assessing scientists by the many players involved in that process.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Publication data and citation metrics continue to play the most important role in research assessments worldwide, so indicator-based research assessments remain at the center of global debate about their validity and influence.
“because publication data and citation metrics continue to play the most important role in research assessments worldwide, indicator-based research assessments also remain at the center of the global debate, not only about their validity, but also about their influence on research activities, scientific output, and its quality [1,5-7,44,260,265,289].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [289]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scholarship Reconsidered: Priorities of the Professoriate
Poston, L. S., & Boyer, E. L. (1992) · Academe · doi:10.2307/40250362 · Cited by 6,881 · see citing works
Audit and feedback: effects on professional practice and healthcare outcomes
Ivers, N., Jamtvedt, G., Flottorp, S., Young, J. M., Odgaard-Jensen, J., French, S. D., O'Brien, M. A., Johansen, M., Grimshaw, J., & Oxman, A. D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000259.pub3 · Cited by 5,485 · cited 1× · see citing works
“These data can inform the development of new systems [74].”
BackgroundRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · see citing works
Bibliometrics: The Leiden Manifesto for Research Metrics
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015) · Nature · doi:10.1038/520429a · Cited by 2,828 · see citing works
The History and Meaning of the Journal Impact Factor
Garfield, E. (2006) · JAMA · doi:10.1001/jama.295.1.90 · Cited by 2,770 · see citing works
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting
Hemming, K., Haines, T. P., Chilton, P. J., Girling, A. J., & Lilford, R. J. (2015) · BMJ · doi:10.1136/bmj.h391 · Cited by 1,410 · see citing works
Reducing waste from incomplete or unusable reports of biomedical research
Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., Michie, S., Moher, D., & Wager, E. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62228-x · Cited by 1,319 · see citing works
Self-citationNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · see citing works
The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles
Piwowar, H., Priem, J., Larivière, V., Alperin, J. P., Matthias, L., Norlander, B., Farley, A., West, J., & Haustein, S. (2018) · PeerJ · doi:10.7717/peerj.4375 · Cited by 1,223 · cited 1× · see citing works
“Junior faculty may put a high value on open access publications [69].”
BackgroundKontopantelis, E., Doran, T., Springate, D. A., Buchan, I., & Reeves, D. (2015) · BMJ · doi:10.1136/bmj.h2750 · Cited by 1,001 · see citing works
Comment: Snowball versus Respondent-Driven Sampling
Heckathorn, D. D. (2011) · Sociological Methodology · doi:10.1111/j.1467-9531.2011.01244.x · Cited by 934 · see citing works
Increasing value and reducing waste: addressing inaccessible research
Chan, A.-W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., Krumholz, H. M., Ghersi, D., & van der Worp, H. B. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62296-5 · Cited by 887 · see citing works
Sharing Detailed Research Data Is Associated with Increased Citation Rate
Piwowar, H. A., Day, R. S., & Fridsma, D. B. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000308 · Cited by 881 · see citing works
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760 · see citing works
Self-citationWilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S. A., Jones, R. D., Kain, R. J. P., Kerridge, S., Thelwall, M. A., et al. (2015) · Digital Education Resource Archive (University College London) · doi:10.13140/rg.2.1.4929.1363 · Cited by 738 · see citing works
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., et al. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002456 · Cited by 700 · see citing works
Edwards, M. A., & Roy, S. (2017) · Environmental Engineering Science · doi:10.1089/ees.2016.0223 · Cited by 688 · see citing works
Hutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002541 · Cited by 500 · see citing works
Deep impact: unintended consequences of journal rank
Brembs, B., Button, K., & Munafò, M. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00291 · Cited by 441 · see citing works
Harley, D., Acord, S. K., Earl‐Novell, S., Lawrence, S., & King, C. J. (2010) · RePEc: Research Papers in Economics · Cited by 213 · see citing works
Multiple Citation Indicators and Their Composite across Scientific Disciplines
Ioannidis, J. P. A., Klavans, R., & Boyack, K. W. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002501 · Cited by 157 · see citing works
Self-citationOvseiko, P. V., Chapple, A., Edmunds, L., et al. (2017) · Health Research Policy and Systems · doi:10.1186/s12961-017-0177-9 · Cited by 150 · see citing works
Benedictus, R., Miedema, F. F., & Ferguson, M. W. J. (2016) · Nature · doi:10.1038/538453a · Cited by 149 · see citing works
Self-citationSelf-correction in science at work
Alberts, B., Cicerone, R. J., Fienberg, S. E., Kamb, A., Mcnutt, M. K., Nerem, R. M., Schekman, R. W., Shiffrin, R. M., Stodden, V., Suresh, S., et al. (2015) · Science · doi:10.1126/science.aab3847 · Cited by 143 · see citing works
Watch out for cheats in citation game
Biagioli, M. (2016) · Nature · doi:10.1038/535201a · Cited by 140 · see citing works
Citation Metrics: A Primer on How (Not) to Normalize
Ioannidis, J. P. A., Boyack, K. W., & Wouters, P. F. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002542 · Cited by 94 · see citing works
Self-citationAssessing Value in Biomedical Research
Ioannidis, J. P. A., & Khoury, M. J. (2014) · JAMA · doi:10.1001/jama.2014.6932 · Cited by 91 · see citing works
Self-citationTijdink, J. K., Schipper, K., Bouter, L. M., Pont, P. M., de Jonge, J., & Smulders, Y. M. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-008681 · Cited by 91 · see citing works
Authors' opinions on publication in relation to annual performance assessment
Walker, R. L., Sykes, L., Hemmelgarn, B. R., & Quan, H. (2010) · BMC Medical Education · doi:10.1186/1472-6920-10-21 · Cited by 86 · see citing works
Acknowledging and Overcoming Nonreproducibility in Basic and Preclinical Research
Ioannidis, J. P. A. (2017) · JAMA · doi:10.1001/jama.2017.0549 · Cited by 78 · see citing works
Self-citationBeyond Academia – Interrogating Research Impact in the Research Excellence Framework
Terämä, E., Smallman, M., Lock, S. J., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0168533 · Cited by 77 · see citing works
How should medical science change?
Kleinert, S., & Horton, R. C. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62678-1 · Cited by 75 · see citing works
No publication without confirmation
Mogil, J. S., & Macleod, M. R. (2017) · Nature · doi:10.1038/542409a · Cited by 73 · see citing works
Lapinski, S., Piwowar, H., & Priem, J. R. (2013) · College & Research Libraries News · doi:10.5860/crln.74.6.8960 · Cited by 67 · see citing works
Imagining the “open” university: Sharing scholarship to improve research and education
McKiernan, E. C. (2017) · PLoS Biology · doi:10.1371/journal.pbio.1002614 · Cited by 61 · see citing works
Powell-Smith, A., & Goldacre, B. M. (2016) · F1000Research · doi:10.12688/f1000research.10010.1 · Cited by 59 · see citing works
Evaluating Academic Scientists Collaborating in Team-Based Research
Mazumdar, M., Messinger, S., Finkelstein, D. M., et al. (2015) · Academic Medicine · doi:10.1097/acm.0000000000000759 · Cited by 57 · see citing works
Recognition and reward in the academy
Hammarfelt, B. (2017) · Aslib Journal of Information Management · doi:10.1108/ajim-01-2017-0006 · Cited by 57 · see citing works
The State of OA: A large-scale analysis of the prevalence and impact of Open Access articlesPreprint
Piwowar, H., Priem, J. R., Larivière, V., Alperín, J. P., Matthias, L., Norlander, B., Farley, A., West, J. D., & Haustein, S. (2017) · doi:10.7287/peerj.preprints.3119v1 · Cited by 51 · see citing works
Time to remodel the journal impact factor
(2016) · Nature · doi:10.1038/535466a · Cited by 50 · see citing works
A critical evaluation of the algorithm behind the Relative Citation Ratio (RCR)
Janssens, A. C. J. W., Goodman, M., Powell, K. R., & Gwinn, M. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002536 · Cited by 42 · see citing works
Schekman, R. W., & Patterson, M. N. (2013) · eLife · doi:10.7554/elife.00855 · Cited by 39 · see citing works
Faculty promotion must assess reproducibility
Flier, J. S. (2017) · Nature · doi:10.1038/549133a · Cited by 35 · see citing works
Stürmer, S., Oeberst, A., Trötschel, R., & Decker, O. (2017) · Social Psychology · doi:10.1027/1864-9335/a000324 · Cited by 32 · see citing works
Incentivizing Data Sharing and Collaboration in Medical Research—The S-Index
Olfson, M., Wall, M. M., & Blanco, C. (2016) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.2610 · Cited by 29 · see citing works
The missing metric: quantifying contributions of reviewers
Cantor, M., & Gero, S. (2015) · Royal Society Open Science · doi:10.1098/rsos.140540 · Cited by 28 · see citing works
Academic criteria for appointment, promotion and rewards in medical research: where's the evidence?
Moher, D., Goodman, S. N., & Ioannidis, J. P. A. (2016) · European Journal of Clinical Investigation · doi:10.1111/eci.12612 · Cited by 26 · see citing works
Self-citationOpen Science at an institutional level: an interview with Guy Rouleau
Rouleau, G. A. (2017) · Genome biology · doi:10.1186/s13059-017-1152-z · Cited by 18 · see citing works
Editorial: Assessing Academic Researchers
Zare, R. N. (2012) · Angewandte Chemie International Edition · doi:10.1002/anie.201201011 · Cited by 17 · see citing works
Pasterkamp, G., Hoefer, I. E., & Prakken, B. J. (2016) · Nature Biotechnology · doi:10.1038/nbt.3691 · Cited by 16 · see citing works
Rewarding open access scholarship in promotion and tenure: Driving institutional change
Odell, J. D., Coates, H., & Palmer, K. L. (2016) · College & Research Libraries News · doi:10.5860/crln.77.7.9518 · Cited by 15 · see citing works
Money back guarantees for non-reproducible results?
Topol, E. J. (2016) · BMJ · doi:10.1136/bmj.i2770 · Cited by 14 · see citing works
Defending Biomedical Science in an Era of Threatened Funding
Ioannidis, J. P. A. (2017) · JAMA · doi:10.1001/jama.2017.5811 · Cited by 9 · see citing works
Self-citationFew UK universities have adopted rules against impact-factor abuse
Gaind, N. (2018) · Nature · doi:10.1038/d41586-018-01874-w · Cited by 7 · see citing works
Coens, C., Bogaerts, J., & Collette, L. (2017) · F1000Research · doi:10.12688/f1000research.10503.1 · Cited by 4 · see citing works
Gadd, E. A. (2015) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [2]Quan W, Chen B, Shu F. Publish Or Impoverish: An Investigation Of The Monetary Reward System Of Science In China (1999–2016).[Internet]. Available from: https://arxiv.org/ftp/arxiv/papers/1707/1707.01162.pdf. Last accessed: 22Feb2018.
- [34]Larivière V, Kiermer V, MacCallum CJ, McNutt M, Patterson M, Pulverer B, et al. A simple proposal for the publication of journal citation distributions. bioRxiv. Available from: https://www.biorxiv.org/content/biorxiv/early/2016/09/11/062109.full.pdf. Last accessed: 22Feb2018.
- [45]Public Library of Science. PLOS and DORA. [Internet]. Available at: https://www.plos.org/dora. Last Accessed: 15-Feb-2018.
- [66]Terämä E, Smallman M, Lock SJ, Johnson C, Austwick MZ. Beyond Academia–Interrogating Research Impact in the Research Excellence Framework. PLoS ONE 2016; 11(12): e0168533 doi: 10.1371/journal.pone.0168533 27997599 PMC5173344 · DOI
- [71]Assessing current practices in the review, promotion and tenure process. [Internet] Available from: https://publishing.sfu.ca/7297-review-promotion-tenure-project/. Last Accessed: 22Feb2018.
Cited by
Sorted by citations
Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696 · see citing works
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919 · see citing works
Open science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608 · see citing works
A standardized citation metrics author database annotated for scientific field
Ioannidis, J. P. A., Baas, J., Klavans, R., & Boyack, K. W. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000384 · Cited by 356 · see citing works
Self-citationRice, D. B., Raffoul, H., Ioannidis, J. P. A., & Moher, D. (2020) · BMJ · doi:10.1136/bmj.m2081 · Cited by 241 · see citing works
Self-citationHardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187 · see citing works
Self-citationDanchev, V., Min, Y., Borghi, J., Baiocchi, M., & Ioannidis, J. P. A. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.33972 · Cited by 94 · see citing works
Self-citationWho games metrics and rankings? Institutional niches and journal impact factor inflation
Siler, K., & Larivière, V. (2022) · Research Policy · doi:10.1016/j.respol.2022.104608 · Cited by 70 · see citing works
Glonti, K., Boutron, I., Moher, D., & Hren, D. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-033421 · Cited by 69 · see citing works
Self-citationThe Pricing of Open Access Journals: Diverse Niches and Sources of Value in Academic Publishing
Siler, K., & Frenken, K. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00016 · Cited by 64 · see citing works
Siebert, M., Gaba, J. F., Caquelin, L., Gouraud, H., Dupuy, A., Moher, D., & Naudet, F. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-038887 · Cited by 45 · see citing works
Self-citationIncrease value and reduce waste in research on psychological therapies
Cristea, I. A., & Naudet, F. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103479 · Cited by 22 · see citing works
Self-citationTen simple rules for building and maintaining a responsible data science workflow
Stoudt, S., Jernite, Y., Marshall, B., Marwick, B., Sharan, M., Whitaker, K., & Danchev, V. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1012232 · Cited by 11 · see citing works
- Authors
- Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P. A., & Goodman, S. N.
- Affiliations
- Ottawa Hospital Research Institute (CA)
- Inserm (FR)
- Babeș-Bolyai University (RO)
- Utrecht University (NL)
- Stanford University (US)
- Corresponding author
- Moher, David
- ORCID
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-03-29
- Volume
- 16
- Issue
- 3
- Pages
- e2004089
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC5892914
- OpenAlex
- W2794944797
- Cited by
- 462
- References
- 58
- Citations by year
- 2026: 40 · 2025: 36 · 2024: 77 · 2023: 51 · 2022: 55 · 2021: 81 · 2020: 52 · 2019: 45 · 2018: 24 · 2015: 1
- FWCI
- 38.37
- Type
- journal article
- Language
- en
- Update notices
- new_version (2018-04-10)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Padua Research Archive (University of Padova) (submittedVersion, cc-by)
- Funders
- Wellcome Trust
- Imperial College London
- European Commission
- Universitair Medisch Centrum Utrecht
- Nature
- National Institutes of Health
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Decision Sciences › Information Systems and Management › Research, Science, and Academia
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Keywords
- research evaluation, academic hiring, academic promotion, tenure, replication crisis, publication quality, faculty incentives, science policy, evaluation principles
- MeSH terms
- Employee Incentive Plans, Faculty, Humans, Research, Reproducibility of Results, Laboratory Personnel
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
