The effect of rater accountability on performance ratings: a meta-analytic review
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Through iterative prompt engineering, the authors mitigated several known limitations of LLM-based evaluators, including a predisposition toward overly supportive or affirming judgments, overreliance on linguistic polish as a proxy for empathic or clinical competence, and the absence of external evaluative pressure that typically constrains human raters.
“Through iterative prompt engineering, we mitigated several known limitations of LLM-based evaluators, including (1) a predisposition toward overly supportive or affirming judgments [51], (2) overreliance on linguistic polish as a proxy for empathic or clinical competence [52], and (3) the absence of external evaluative pressure that typically constrains human raters [50].”
From Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study (2026) · cites this paper as [50]
The shift from disagreement in overall scoring for the clinical research team to consistently higher ratings by both psychiatrists for residents may reflect a favorable bias toward psychiatry residents, potentially influenced by expectations of trainee competence.
“This shift, from disagreement in overall scoring for the clinical research team to consistently higher ratings by both psychiatrists for residents, may reflect a favorable bias toward psychiatry residents, potentially influenced by expectations of trainee competence [50].”
From Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study (2026) · cites this paper as [50]
Cites
The need to belong: Desire for interpersonal attachments as a fundamental human motivation.
Baumeister, R. F., & Leary, M. R. (1995) · Psychological Bulletin · doi:10.1037//0033-2909.117.3.497 · Cited by 23,245 · see citing works
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771 · see citing works
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542 · see citing works
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · see citing works
Accounting for the effects of accountability.
Lerner, J. S., & Tetlock, P. E. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.2.255 · Cited by 2,383 · see citing works
Shafir, E. (1993) · Cognition · doi:10.1016/0010-0277(93)90034-s · Cited by 1,453 · see citing works
How Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294 · see citing works
Understanding performance appraisal: social, organizational, and goal-based perspectives
(1995) · Choice Reviews Online · doi:10.5860/choice.32-5748 · Cited by 1,088 · see citing works
Accountability and complexity of thought.
Tetlock, P. E. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.45.1.74 · Cited by 995 · see citing works
The Impact of Accountability on Judgment and Choice: Toward A Social Contingency Model
Tetlock, P. E. (1992) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60287-7 · Cited by 926 · see citing works
Wayne, S. J., & Ferris, G. R. (1990) · Journal of Applied Psychology · doi:10.1037/0021-9010.75.5.487 · Cited by 860 · see citing works
Lee, B.-M., Kacew, S., & Kim, H. S. (2015) · Productivity Press eBooks · doi:10.1201/b18778-13 · Cited by 805 · see citing works
Comparative analysis of the reliability of job performance ratings.
Viswesvaran, C., Öneş, D. S., & Schmidt, F. L. (1996) · Journal of Applied Psychology · doi:10.1037/0021-9010.81.5.557 · Cited by 774 · see citing works
The Social Context of Performance Appraisal: A Review and Framework for the Future
Levy, P. E., & Williams, J. (2004) · Journal of Management · doi:10.1016/j.jm.2004.06.005 · Cited by 770 · see citing works
Performance appraisal and management: The developing research agenda
Fletcher, C. (2001) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317901167488 · Cited by 626 · see citing works
Tetlock, P. E., Skitka, L. J., & Boettger, R. (1989) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.57.4.632 · Cited by 610 · see citing works
Tetlock, P. E., Skitka, L. J., & Boettger, R. D. (1989) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.57.4.632 · Cited by 541 · see citing works
Accountability and the Perseverance of First Impressions
Tetlock, P. E. (1983) · Social Psychology Quarterly · doi:10.2307/3033716 · Cited by 513 · see citing works
Bretz, R. D., Milkovich, G. T., & Read, W. (1992) · Journal of Management · doi:10.1177/014920639201800206 · Cited by 505 · see citing works
Multiple uses of performance appraisal: Prevalence and correlates.
Cleveland, J. N., Murphy, K. R., & Williams, R. E. (1989) · Journal of Applied Psychology · doi:10.1037/0021-9010.74.1.130 · Cited by 423 · see citing works
Cawley, B. D., Keeping, L. M., & Levy, P. E. (1998) · Journal of Applied Psychology · doi:10.1037/0021-9010.83.4.615 · Cited by 391 · see citing works
Performance appraisal reactions: Measurement, modeling, and method bias.
Keeping, L. M., & Levy, P. E. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.5.708 · Cited by 353 · see citing works
Performance Appraisal: Alternative Perspectives
Dobbins, G., & Cardy, R. L. (1993) · Medical Entomology and Zoology · Cited by 311 · see citing works
Self–other agreement in job performance ratings: A meta-analytic test of a process model.
Heidemeier, H., & Moser, K. (2009) · Journal of Applied Psychology · doi:10.1037/0021-9010.94.2.353 · Cited by 296 · see citing works
Methods of meta-analysis: Correcting error and bias in research findings
Pigott, T. (2006) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2006.06.002 · Cited by 291 · see citing works
WHERE ALL THE CHILDREN ARE ABOVE AVERAGE: THE PERFORMANCE APPRAISAL PURPOSE EFFECT
Jawahar, I. M. “., & Williams, C. R. (1997) · Personnel Psychology · doi:10.1111/j.1744-6570.1997.tb01487.x · Cited by 253 · see citing works
Performance appraisal : state of the art in practice
Smither, J. W. (1998) · Jossey-Bass Publishers eBooks · Cited by 252 · see citing works
Explaining the Weak Relationship Between Job Performance and Ratings of Job Performance
Murphy, K. R. (2008) · Industrial and Organizational Psychology · doi:10.1111/j.1754-9434.2008.00030.x · Cited by 195 · see citing works
Larson, J. R. (1989) · Academy of Management Review · doi:10.5465/amr.1989.4279075 · Cited by 194 · see citing works
Ferris, G. R., Munyon, T., Basik, K. J., & Buckley, M. (2008) · Human Resource Management Review · doi:10.1016/j.hrmr.2008.07.006 · Cited by 189 · see citing works
THE EFFECTS OF FEEDBACK ACCOUNTABILITY ON UPWARD APPRAISAL RATINGS
Antonioni, D. (1994) · Personnel Psychology · doi:10.1111/j.1744-6570.1994.tb01728.x · Cited by 181 · see citing works
Effects of rater accountability on the accuracy and the favorability of performance ratings.
Mero, N. P., & Motowidlo, S. J. (1995) · Journal of Applied Psychology · doi:10.1037/0021-9010.80.4.517 · Cited by 180 · see citing works
Rater motivation in the performance appraisal context: A theoretical framework
Harris, M. M. (1994) · Journal of Management · doi:10.1016/0149-2063(94)90028-0 · Cited by 174 · see citing works
Sackett, P. R., Harris, M. M., & Orr, J. M. (1986) · Journal of Applied Psychology · doi:10.1037/0021-9010.71.2.302 · Cited by 155 · see citing works
Mero, N. P., Guidice, R. M., & Brownlee, A. L. (2007) · Journal of Management · doi:10.1177/0149206306297633 · Cited by 141 · see citing works
Viswesvaran, C., Schmidt, F. L., & Öneş, D. S. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.2.345 · Cited by 126 · see citing works
A social relationship conceptualization of trust and accountability in organizations
Ammeter, A. P., Douglas, C., Ferris, G. R., & Goka, H. (2004) · Human Resource Management Review · doi:10.1016/j.hrmr.2004.02.003 · Cited by 121 · see citing works
Spence, J. R., & Keeping, L. M. (2010) · Human Resource Management Review · doi:10.1016/j.hrmr.2010.09.013 · Cited by 103 · see citing works
EFFECT OF INSTRUCTIONAL CONDITIONS IN PRODUCING LENIENCY ON TWO TYPES OF RATING SCALES1
Sharon, A. T., & Bartlett, C. J. (1969) · Personnel Psychology · doi:10.1111/j.1744-6570.1969.tb00330.x · Cited by 93 · see citing works
Managing five paradoxes of 360-degree feedback
Ghorpade, J. V. (2000) · Academy of Management Perspectives · doi:10.5465/ame.2000.2909846 · Cited by 88 · see citing works
Comparison of online and traditional performance appraisal systems
Payne, S. C., Horner, M. T., Boswell, W. R., Schroeder, A. N., & Kelleen, S.-C. (2009) · Journal of Managerial Psychology · doi:10.1108/02683940910974116 · Cited by 86 · see citing works
Bernardin, H. J., Thomason, S. J., Buckley, M. R., & Kane, J. S. (2015) · Human Resource Management · doi:10.1002/hrm.21678 · Cited by 84 · see citing works
Contextual and Rater Factors Affecting Rating Behavior
Tziner, A. E., Murphy, K. R., & Cleveland, J. N. (2004) · Group & Organization Management · doi:10.1177/1059601104267920 · Cited by 83 · see citing works
Rater Personality, Rating Format, and Social Context: Implications for Performance Appraisal Ratings
Yun, G. J., Donahue, L. M., Dudley, N. M., & McFarland, L. A. (2005) · International Journal of Selection and Assessment · doi:10.1111/j.0965-075x.2005.00304.x · Cited by 64 · see citing works
Creating quality-driven performance appraisal systems
Ghorpade, J. V., & Chen, M. M. (1995) · Academy of Management Perspectives · doi:10.5465/ame.1995.9503133485 · Cited by 63 · see citing works
An Investigation of Factors Influencing Accountability and Performance Ratings
Roch, S. G., & McNall, L. A. (2007) · The Journal of Psychology · doi:10.3200/jrlp.141.5.499-524 · Cited by 52 · see citing works
Police officer performance appraisal systems
Coutts, L. M., & Schneider, F. W. (2004) · Policing An International Journal · doi:10.1108/13639510410519921 · Cited by 51 · see citing works
Smither, J. W. (2012) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199928309.013.0010 · Cited by 41 · see citing works
Does rater personality matter? A meta‐analysis of rater Big Five–performance rating relationships
Harari, M. B., Rudolph, C. W., & Laginess, A. J. (2014) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12086 · Cited by 38 · see citing works
Self-citationWood, R. E., & Marshall, V. (2008) · Human Resource Management Journal · doi:10.1111/j.1748-8583.2008.00067.x · Cited by 37 · see citing works
An information-processing approach to leniency in performance judgments.
Hauenstein, N. M. A. (1992) · Journal of Applied Psychology · doi:10.1037//0021-9010.77.4.485 · Cited by 34 · see citing works
Effects of anonymity and retaliatory potential on student evaluations of faculty performance
Stone, E. F., Spool, M. D., & Rabinowitz, S. (1977) · Research in Higher Education · doi:10.1007/bf00992176 · Cited by 28 · see citing works
Peer Assessment, Individual Performance, and Contribution to Group Processes
Bamberger, P. A., Erev, I., Kimmel, M., & Oref-Chen, T. (2005) · Group & Organization Management · doi:10.1177/1059601104267619 · Cited by 26 · see citing works
Discussion and Consensus in Rater Groups: Implications for Behavioral and Rating Accuracy
Roch, S. G. (2006) · Human Performance · doi:10.1207/s15327043hup1902_1 · Cited by 22 · see citing works
An investigation of motivational factors influencing performance ratings
Roch, S. G. (2005) · Journal of Managerial Psychology · doi:10.1108/02683940510631453 · Cited by 19 · see citing works
An information-processing approach to leniency in performance judgments.
Hauenstein, N. M. A. (1992) · Journal of Applied Psychology · doi:10.1037/0021-9010.77.4.485 · Cited by 19 · see citing works
Roch, S. G. (2006) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2006.08.003 · Cited by 16 · see citing works
Effect of Identifiability, Rating Audience, and Conscientiousness on Rating Level
Roch, S. G., Ayman, R., Newhouse, N. K., & Harris, M. (2005) · International Journal of Selection and Assessment · doi:10.1111/j.0965-075x.2005.00299.x · Cited by 15 · see citing works
A Field Study of Rating Conditions and Leniency in Performance Appraisal
Waldman, D. A., & Thornton, G. C. (1988) · Psychological Reports · doi:10.2466/pr0.1988.63.3.835 · Cited by 15 · see citing works
Shore, T. H., & Tashchian, A. (2007) · Psychological Reports · doi:10.2466/pr0.100.4.1091-1100 · Cited by 9 · see citing works
Effects of accountability procedures upon the utility of peer ratings of present performance
Warmke, D. L. (1979) · OhioLink ETD Center (Ohio Library and Information Network) · Cited by 5 · see citing works
Smith, B. D. (2012) · TigerPrints (Clemson University) · Cited by 4 · see citing works
Motivated strategies in the performance appraisal process: Effects of rater accountability
Swerdlin, M. R. (1992) · Rice Research Repository (Rice University) · Cited by 2 · see citing works
Promotability Ratings: An Underused Promotion Method for Public Safety Organizations
Cederblom, D. (1991) · Public Personnel Management · doi:10.1177/009102609102000103 · Cited by 2 · see citing works
Beaulieu, R. P. (2005) · Psychological Reports · doi:10.2466/pr0.97.2.389-399 · Cited by 2 · see citing works
Flagged first, then by citations
2 of the 69 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Managing Employee Performance and Reward
Shields, J. L., Brown, M., Kaine, S., Dolle-Samuel, C., North-Samardzic, A., McLean, P., Johns, R., O’Leary, P., Plimmer, G., & Robinson, J. (2007) · Cambridge University Press eBooks · doi:10.1017/9781009520355 · Cited by 141 · see citing works
Aleksovska, M., Schillemans, T., & Grimmelikhuijsen, S. (2019) · Journal of Behavioral Public Administration · doi:10.30636/jbpa.22.66 · Cited by 92 · see citing works
The Advantages and Limitations of Using Meta-analysis in Human Resource Management Research
Stone, D. L., & Rosopa, P. J. (2016) · Human Resource Management Review · doi:10.1016/j.hrmr.2016.09.001 · Cited by 80 · see citing works
Overman, S., Schillemans, T., & Grimmelikhuijsen, S. (2020) · Public Management Review · doi:10.1080/14719037.2020.1751254 · Cited by 72 · see citing works
Understanding Felt Accountability
Schillemans, T., Overman, S., Fawcett, P., Flinders, M. V., Fredriksson, M., Lægreid, P., Maggetti, M., Papadopoulos, Y., Rübecksen, K., Rykkja, L. H., et al. (2020) · Governance · doi:10.1111/gove.12547 · Cited by 67 · see citing works
Employee performance management: charting the field from 1998 to 2018
Maley, J. F., Dabić, M., & Moeller, M. (2020) · International Journal of Manpower · doi:10.1108/ijm-10-2019-0483 · Cited by 42 · see citing works
Speer, A. B. (2020) · Organizational Research Methods · doi:10.1177/1094428120930815 · Cited by 36 · see citing works
Ramdani, Z., Marliani, R., & Rahman, A. A. (2019) · Humanities & Social Sciences Reviews · doi:10.18510/hssr.2019.7545 · Cited by 30 · see citing works
The Performance of Performance Appraisal Systems: A Theoretical Framework for Public Organizations
Barbieri, M., Micacchi, L., Vidè, F., & Valotti, G. (2021) · Review of Public Personnel Administration · doi:10.1177/0734371x211043560 · Cited by 30 · see citing works
Happily unaccountable? Perceptions of accountability by public managers
Maggetti, M., & Papadopoulos, Y. (2022) · Public Policy and Administration · doi:10.1177/09520767221074487 · Cited by 20 · see citing works
Evaluating the quality of medical content on YouTube using large language models
Khalil, M., Mohamed, F., & Shoufan, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-94208-6 · Cited by 22 · see citing works
Creation and Validation of the Performance Appraisal Motivation Scale (PAMS)
Speer, A. B., Tenbrink, A. P., & Schwendeman, M. G. (2020) · Human Performance · doi:10.1080/08959285.2020.1776713 · Cited by 15 · see citing works
Let’s Talk it Out: The Effects of Calibration Meetings on Performance Ratings
Speer, A. B., Tenbrink, A. P., & Schwendeman, M. G. (2019) · Human Performance · doi:10.1080/08959285.2019.1609477 · Cited by 13 · see citing works
Sources of accountability inside the boardroom
Goodman, J., Pearson, H., & Mthombeni, M. (2021) · European Business Review · doi:10.1108/ebr-05-2020-0119 · Cited by 10 · see citing works
Kaur, S., & Kaur, G. (2021) · South African Journal of Business Management · doi:10.4102/sajbm.v52i1.2070 · Cited by 8 · see citing works
Tenbrink, A. P., & Speer, A. B. (2022) · Human Performance · doi:10.1080/08959285.2021.2023876 · Cited by 7 · see citing works
Schillemans, T. (2022) · Public Performance & Management Review · doi:10.1080/15309576.2022.2040034 · Cited by 6 · see citing works
Employer Responsiveness to Online Reviews: A Signal of Caring About Employees
Höllig, C. E., Ademi, E., & Tumasjan, A. (2025) · Human Resource Management · doi:10.1002/hrm.22316 · Cited by 5 · see citing works
Managing employee performance and reward systems, practices and prospects
Shields, J. L., Rooney, J., Brown, M., & Kaine, S. (2020) · Cited by 4 · see citing works
Means, ends and meaning in accountability for strategic education governance
Fahey, G., & Köster, F. (2019) · OECD education working papers · doi:10.1787/1d516b5c-en · Cited by 7 · see citing works
The study of the effect of performance accountability on performance management
Sun, Y. (2024) · Highlights in Business, Economics and Management · doi:10.54097/pvfcd854 · Cited by 4 · see citing works
Accountability and referent Selection: Evidence from Chinese mutual funds
Yang, Z., & Zhi, Y. (2025) · Journal of Business Research · doi:10.1016/j.jbusres.2025.115368 · Cited by 2 · see citing works
The root cause of perceived appraisal politics using evidence from the South Korean public sector
Yoo, J.-W., Roh, S., Haddoud, M. Y., & Tripathi, S. (2025) · The International Journal of Human Resource Management · doi:10.1080/09585192.2025.2579560 · Cited by 2 · see citing works
Implementation of Performance Appraisal System in Public Sector: New Line Managers’ Perspective
Astuti, I. F., & Rachmawati, R. (2023) · Advances in economics, business and management research/Advances in Economics, Business and Management Research · doi:10.2991/978-94-6463-234-7_147 · Cited by 2 · see citing works
Isolating the effect of rater experience as a time‐variant predictor of performance ratings
Borba, D., & Spence, J. R. (2024) · Human Resource Management Journal · doi:10.1111/1748-8583.12563 · Cited by 2 · see citing works
Wu, P., Jin, Y., Cheng, L., & Wang, Y. (2021) · E3S Web of Conferences · doi:10.1051/e3sconf/202123301166 · Cited by 2 · see citing works
Lu, Y., Miazek, K., & Bocian, K. (2025) · Personality and Social Psychology Bulletin · doi:10.1177/01461672251364308 · Cited by 2 · see citing works
Sockbeson, C. S. (2018) · doi:10.1093/obo/9780199828340-0213 · Cited by 2 · see citing works
Rueda, A., Al-Shamali, H. F., Cote, Z., Roy, N., Janssen-Aguilar, R., Joseph, J., Teferra, B. G., Kamaleddin, M. A., Burback, L., Winkler, O., et al. (2026) · JMIR Medical Education · doi:10.2196/88580 · Cited by 1 · see citing works
Obliterating rating distortions: role of rater disposition and rating context
Ikramullah, M., Khalil, A., Iqbal, M., & Hassan, F. S. U. (2022) · Management Decision · doi:10.1108/md-08-2021-1060
Liu, J. (2022) · Mobile Information Systems · doi:10.1155/2022/8890859 · Cited by 1 · see citing works
Yazid, H., Wiyantoro, L. S., & Ismawati, I. (2023) · JRAK · doi:10.23969/jrak.v15i2.6243 · Cited by 1 · see citing works
(2018) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-78743-785-220181022 · Cited by 1 · see citing works
Kaur, S., & Kaur, G. (2021) · South African Journal of Business Management · doi:10.4102/sajbm.v52i2.2070
Aleksovska, M. (2021) · doi:10.33540/741
Wibbeke, L.-M., & Lachmann, M. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4307659
Reciprocity in Peer Assessments
Li, L., Louis, P., Maniadis, Z., & Xefteris, D. (2026) · AEA Randomized Controlled Trials · doi:10.1257/rct.18376
The effects of process transparency and prior evaluation results on subjective evaluation decisionsPreprint
Wibbeke, L.-M., & Lachmann, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5158528
Overview and Evaluation of Supervisory Performance Ratings
Roch, S. G., & Woehr, D. J. (2025) · doi:10.1093/9780197809013.003.0024
- Authors
- Harari, M. B., & Rudolph, C. W.
- Affiliations
- Florida Atlantic University (US)
- Saint Louis University (US)
- Corresponding author
- Harari, Michael B.
- Venue
- Human Resource Management Review
- Publisher
- Elsevier BV
- Published
- 2017-03
- Volume
- 27
- Issue
- 1
- Pages
- 121-133
- ISSN
- 1053-4822 (ISSN-L)
- OpenAlex
- W2520439239
- Cited by
- 38
- References
- 69
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 2 · 2023: 2 · 2022: 6 · 2021: 6 · 2020: 6 · 2019: 4 · 2018: 2 · 2016: 1
- FWCI
- 4.54
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Human Resource Management Review (publishedVersion)
- Topics
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › Job Satisfaction and Organizational Behavior
- Social Sciences › Psychology › Applied Psychology › Human Resource Development and Performance Evaluation
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › Employee Performance and Motivation
- Keywords
- performance ratings, performance evaluation (human resources), rater behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
