Rating the ratings: Assessing the psychometric quality of rating data.

Saal, Frank E., Downey, Ronald G., Lahey, Mary A.·1980·Psychological Bulletin

Cited by (567)

  1. Externalizing behavior problems and academic underachievement in childhood and adolescence: Causal relationships and underlying mechanisms.
    Stephen P. Hinshaw · 1992 · Psychological Bulletin
  2. On the distinction between attentional deficits/hyperactivity and conduct problems/aggression in child psychopathology.
    Hinshaw, Stephen P. · 1987 · Psychological Bulletin
  3. Community readiness: Research to practice
    Edwards, Ruth W., Jumper-Thurman, Pamela, Plested, Barbara A., Oetting, Eugene R., Swanson, Louis · 2000 · Journal of Community Psychology
  4. The criterion problem: 1917-1992.
    Austin, James T., Villanova, Peter · 1992 · Journal of Applied Psychology
  5. PROJECT A VALIDITY RESULTS: THE RELATIONSHIP BETWEEN PREDICTOR AND CRITERION DOMAINS
    MCHENRY, JEFFREY J., HOUGH, LEAETTA M., TOQUAM, JODY L., HANSON, MARY ANN, ASHWORTH, STEVEN · 1990 · Personnel Psychology
  6. Detecting and measuring rater effects using many-facet Rasch measurement: Part II.
    Carol M. Myford, Edward W. Wolfe · 2003 · PubMed
  7. Journal of Management Development
    2006 · Journal of Management Development
  8. A role set analysis of managerial reputation
    Tsui, Anne S. · 1984 · Organizational Behavior and Human Performance
  9. Rater bias in psychological research: When is it a problem and what can we do about it?
    Hoyt, William T. · 2000 · Psychological Methods
  10. People nominated as wise: A comparative study of wisdom-related knowledge.
    Baltes, Paul B., Staudinger, Ursula M., Maercker, Andreas, Smith, Jacqui · 1995 · Psychology and Aging
  11. The faultline activation process and the effects of activated faultlines on coalition formation, conflict, and group outcomes
    Jehn, Karen A., Bezrukova, Katerina · 2010 · Organizational Behavior and Human Decision Processes
  12. The Determinants and Performance Effects of Managers’ Performance Evaluation Biases
    Bol, Jasmijn C. · 2010 · SSRN Electronic Journal
  13. Nature and consequences of halo error: A critical analysis.
    Murphy, Kevin R., Jako, Robert A., Anhalt, Rebecca L. · 1993 · Journal of Applied Psychology
  14. Comparison of text preprocessing methods
    Chai, Christine P. · 2023 · Natural Language Engineering
  15. A Meta-Analysis of the Distinction between Reactive and Proactive Aggression in Children and Adolescents
    Polman, Hanneke, Orobio de Castro, Bram, Koops, Willem, van Boxtel, Herman W., Merk, Welmoet W. · 2007 · Journal of Abnormal Child Psychology
  16. Towards a Theory of Country Image Effect on Product Evaluation
    Israel D. Nebenzahl, Eugene D. Jaffe, Schlomo I. Lampert · 1997 · Management International Review
  17. Examining Rater Errors in the Assessment of Written Composition With a Many‐Faceted Rasch Model
    Engelhard, George · 1994 · Journal of Educational Measurement
  18. THE BEST AND THE REST: REVISITING THE NORM OF NORMALITY OF INDIVIDUAL PERFORMANCE
    O’BOYLE JR., ERNEST, AGUINIS, HERMAN · 2012 · Personnel Psychology
  19. Halo and performance appraisal research: A critical examination.
    Balzer, William K., Sulsky, Lorne M. · 1992 · Journal of Applied Psychology
  20. Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns.
    Sulsky, Lorne M., Balzer, William K. · 1988 · Journal of Applied Psychology
  21. A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research
    Aryadoust, Vahid, Ng, Li Ying, Sayama, Hiroki · 2021 · Language Testing
  22. Wisdom-related knowledge in a life review task: Age differences and the role of professional specialization.
    Ursula M. Staudinger, Jacqui Smith, Paul B. Baltes · 1992 · Psychology and Aging
  23. Mood, categorization breadth, and performance appraisal: The effects of order of information acquisition and affective state on halo, accuracy, information retrieval, and evaluations
    Sinclair, Robert C · 1988 · Organizational Behavior and Human Decision Processes
  24. RESEARCH NOTES. INTERPERSONAL AFFECT AND RATING ERRORS.
    Tsui, A. S., Barry, B. · 1986 · Academy of Management Journal
  25. Sentiment analysis of political communication: combining a dictionary approach with crowdcoding
    Haselmayer, Martin, Jenny, Marcelo · 2017 · Quality & Quantity
  26. Magnitude and moderators of bias in observer ratings: A meta-analysis.
    Hoyt, William T., Kerns, Michael-David · 1999 · Psychological Methods
  27. Rater training revisited: An updated meta‐analytic review of frame‐of‐reference training
    Roch, Sylvia G., Woehr, David J., Mishra, Vipanchi, Kieszczynska, Urszula · 2012 · Journal of Occupational and Organizational Psychology
  28. Measuring User Engagement
    Lalmas, Mounia, O’Brien, Heather, Yom-Tov, Elad · 2015 · Synthesis Lectures on Information Concepts, Retrieval, and Services
  29. Reexamining the Relationship Between Academic Achievement and Social Behavior
    Algozzine, Bob, Wang, Chuang, Violette, Amy S. · 2011 · Journal of Positive Behavior Interventions
  30. Independent dimensions of hyperactivity and aggression: A validation with playroom observation data.
    Milich, Richard, Loney, Jan, Landau, Steven · 1982 · Journal of Abnormal Psychology
  31. Behavioral and Cognitive Subtypes of ADHD
    AUGUST, GERALD J., GARFINKEL, BARRY D. · 1989 · Journal of the American Academy of Child & Adolescent Psychiatry
  32. Rater errors and rating accuracy.
    Kevin R. Murphy, William K. Balzer · 1989 · Journal of Applied Psychology
  33. Accuracy of performance ratings as affected by rater training and perceived purpose of rating.
    McIntyre, Robert M., Smith, David E., Hassett, Catherine E. · 1984 · Journal of Applied Psychology
  34. A comparison of objective classroom measures and teacher ratings of attention deficit disorder
    Atkins, Marc S., Pelham, William E., Licht, Mark H. · 1985 · Journal of Abnormal Child Psychology
  35. The language of motivational interviewing and feedback: Counselor language, client language, and client drinking outcomes.
    Amanda M. Vader, Scott T. Walters, Gangamma Chenenda Prabhu, Jon M. Houck, Craig A. Field · 2010 · Psychology of Addictive Behaviors
  36. A comparison of rater training programs: Error training and accuracy training.
    Elaine D. Pulakos · 1984 · Journal of Applied Psychology
  37. Raters who pursue different goals give different ratings.
    Kevin R. Murphy, Jeanette N. Cleveland, Amie L. Skattebo, Ted B. Kinney · 2004 · Journal of Applied Psychology
  38. RATER SOURCE EFFECTS ARE ALIVE AND WELL AFTER ALL
    Brian James Hoffman, Charles E. Lance, Bethany H. Bynum, William A. Gentry · 2010 · Personnel Psychology
  39. Changes in parenting practices and adolescent drug abuse during multidimensional family therapy.
    Susan E. Schmidt, Howard Arthur Liddle, Gayle A. Dakof · 1996 · Journal of Family Psychology
  40. Peer evaluation in self-managing work groups.
    Richard Saavedra, Seog Kyeun Kwun · 1993 · Journal of Applied Psychology
  41. MULTIPLE ASSESSMENT OF MANAGERIAL EFFECTIVENESS: INTERRATER AGREEMENT AND CONSENSUS IN EFFECTIVENESS MODELS
    Anne S. Tsui, Patricia J. Ohlott · 1988 · Personnel Psychology
  42. The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty.
    Chockalingam Viswesvaran, Frank L. Schmidt, Deniz S. Öneş · 2002 · Journal of Applied Psychology
  43. Automated essay scoring: Psychometric guidelines and practices
    Ramineni, Chaitanya, Williamson, David M. · 2013 · Assessing Writing
  44. Modulation of ongoing cognitive processes by emotionally intense words
    Luis Carretié, José Antonio Hinojosa, Jacobo Albert, Sara López‐Martín, Belén S. De La Gándara, José Manuel Igoa, María Sotillo · 2007 · Psychophysiology
  45. What predicts wisdom-related performance? A first look at personality, intelligence, and facilitative experiential contexts
    Ursula M. Staudinger, Anna G. Maciel, Jacqui Smith, Paul B. Baltes · 1998 · European Journal of Personality
  46. Cost of quality usage and its relationship to quality system maturity
    Victor E. Sower, Ross Quarles, Eric Broussard · 2007 · International Journal of Quality & Reliability Management
  47. Personnel Selection and Classification
    Mary L. Tenopyr, P D Oeltjen · 1982 · Annual Review of Psychology
  48. Exploring the role of first impressions in rater-based assessments
    Timothy J. Wood · 2013 · Advances in Health Sciences Education
  49. Rating leniency and halo in multisource feedback ratings: Testing cultural assumptions of power distance and individualism-collectivism.
    Kok‐Yee Ng, Christine Siew Kuan Koh, Soon Ang, Jeffrey C. Kennedy, Kim-Yin Chan · 2011 · Journal of Applied Psychology
  50. Rater Effects on Essay Scoring: A Multilevel Analysis of Severity Drift, Central Tendency, and Rater Experience
    George Leckie, Jo‐Anne Baird · 2011 · Journal of Educational Measurement
  51. STABILITY OF RATER LENIENCY: THREE STUDIES.
    Joanne Kane, H. John Bernardin, Peter D. Villanova, Joseph Peyrefitte · 1995 · Academy of Management Journal
  52. Data Mesh: A Systematic Gray Literature Review
    Abel Goedegebuure, Indika Kumara, Stefan W. Driessen, Willem‐Jan van den Heuvel, Geert Monsieur, Damian A. Tamburri, Dario Di Nucci · 2024 · ACM Computing Surveys
  53. EmoMadrid: An emotional pictures database for affect research
    Luis Carretié, Manuel Tapia, Sara López‐Martín, Jacobo Albert · 2019 · Motivation and Emotion
  54. Research methods for subgrouping low back pain
    Peter M. Kent, Jennifer Lyn Keating, Charlotte Leboeuf‐Yde · 2010 · BMC Medical Research Methodology
  55. Evidence of Halo Effects in Student Evaluations of Communication Instruction
    Thomas Hugh Feeley · 2002 · Communication Education
  56. The Stability of Rater Severity in Large‐Scale Assessment Programs
    Peter J. Congdon, Joy MeQueen · 2000 · Journal of Educational Measurement
  57. Trends in Classroom Observation Scores
    Casabianca, Jodi M., Lockwood, J. R., McCaffrey, Daniel F. · 2015 · Educational and Psychological Measurement
  58. Convergent and Discriminant Validity of Assessment Center Dimensions: A Conceptual and Empirical Reexamination of the Assessment Center Construct-Related Validity Paradox
    Winfred Arthur, David J. Woehr, Robyn Maldegen · 2000 · Journal of Management
  59. The relationship between rater affect and three sources of 360-degree feedback ratings
    David Antonioni, Heejoon Park · 2001 · Journal of Management
  60. Accuracy in behavioral measurement: An alternative definition based on raters' cognitive schema and signal detection theory.
    Robert G. Lord · 1985 · Journal of Applied Psychology
  61. Comment on Halo Effects in Rating and Evaluation Research
    Thomas Hugh Feeley · 2002 · Human Communication Research
  62. Development and Validation of an Instrument to Measure Maturity of IT Business Strategic Alignment Mechanisms
    Deb Sledgianowski, Jerry N. Luftman, Richard R. Reilly · 2006 · Information Resources Management Journal
  63. Understanding and Using Informants’ Reporting Discrepancies of Youth Victimization: A Conceptual Model and Recommendations for Research
    Kimberly L. Goodman, Andres De Los Reyes, Catherine P. Bradshaw · 2010 · Clinical Child and Family Psychology Review
  64. COMMUNITY READINESS AND HEALTH SERVICES
    EUGENE R. OETTING, Pamela Jumper‐Thurman, Barbara Plested, R. W. Edwards · 2001 · Substance Use & Misuse
  65. Halo in customer satisfaction measures: The role of purpose of rating, number of attributes and customer involvement
    Jochen Wirtz · 2003 · International Journal of Service Industry Management
  66. Rater source factors represent important subcomponents of the criterion construct space, not rater bias
    Charles E. Lance, Brian James Hoffman, William A. Gentry, Lisa E. Baranik · 2008 · Human Resource Management Review
  67. REHAB: A New Assessment Instrument for Chronic Psychiatric Patients
    Roger Denio Baker, John Hall · 1988 · Schizophrenia Bulletin
  68. Implications of Three Causal Models for the Measurement of Halo Error
    Sebastiano A. Fisicaro, Charles E. Lance · 1990 · Applied Psychological Measurement
  69. Psychometric Properties of Teacher-Judgment Measures of Pupil Aptitudes, Classroom Behaviors, and Achievement Levels
    Robert D. Hoge · 1983 · The Journal of Special Education
  70. Rater Errors in a Clinical Skills Assessment of Medical Students
    Cherdsak Iramaneerat, Rachel Yudkowsky · 2007 · Evaluation & the Health Professions
  71. A systematic review of methods for evaluating rating quality in language assessment
    Stefanie A. Wind, Meghan E. Peterson · 2017 · Language Testing
  72. A test of the context dependency of three causal models of halo rater error.
    Charles E. Lance, Julie A. LaPointe, Amy M. Stewart · 1994 · Journal of Applied Psychology
  73. Identifying Rater Effects Using Latent Trait Models
    Edward W. Wolfe · 2004
  74. Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability.
    Putka, Dan J., Le, Huy, McCloy, Rodney A., Diaz, Tirso · 2008 · Journal of Applied Psychology
  75. The Halo Effect in Store Image Measurement
    Bob T. W. Wu, Susan M. Petroshius · 1987 · Journal of the Academy of Marketing Science
  76. Halo Effects in Trait Assessment Depend on Information Valence: Why Being Honest Makes You Industrious, but Lying Does Not Make You Lazy
    Michael Gräf, Christian Unkelbach · 2016 · Personality and Social Psychology Bulletin
  77. Climate for creative productivity as a predictor of research usefulness and organizational effectiveness in an r&d organization
    Lawrence Alan Witt, Mark N. Beorkrem · 1989 · Creativity Research Journal
  78. A CLOSER LOOK AT HALO ERROR IN PERFORMANCE RATINGS.
    R. Jacobs, Steve W. J. Kozlowski · 1985 · Academy of Management Journal
  79. Shifting Standards and the Inference of Incompetence: Effects of Formal and Informal Evaluation Tools
    Monica Biernat, Kathleen Fuegen, Diane Kobrynowicz · 2010 · Personality and Social Psychology Bulletin
  80. The Relationship of Performance Appraisal System Characteristics to Verdicts in Selected Employment Discrimination Cases
    Hubert S. Feild, William H. Holley · 1982 · Academy of Management Journal
  81. The Role of Facial Attractiveness and Facial Masculinity/Femininity in Sex Classification of Faces
    Rebecca A. Hoss, Jennifer L. Ramsey, Angela M Griffin, Judith H. Langlois · 2005 · Perception
  82. SELF‐APPRAISAL BASED UPON SUPERVISORY FEEDBACK
    Robert P. Steel, Nestor K. Ovalle · 1984 · Personnel Psychology
  83. Rater effects: Ego engagement in rater decision-making
    Cynthia S. Wiseman · 2012 · Assessing Writing
  84. Work and Play: An Experiment in Enterprise Gamification
    Laurentiu Catalin Stanculescu, Alessandro Bozzon, Robert-Jan Sips, Geert‐Jan Houben · 2016 · ACM Conference on Computer Supported Cooperative Work (CSCW)
  85. Towards a science of user engagement.
    Simon Attfield, Gabriella Kazai, Mounia Lalmas, Benjamin Piwowarski · 2011
  86. Readiness for Drug Use Prevention in Rural Minority Communities
    Barbara Plested, Debra M. Smitham, Pamela Jumper‐Thurman, EUGENE R. OETTING, Ruth W. Edwards · 1999 · Substance Use & Misuse
  87. Facilitating Peer Evaluation in Team Contexts: The Impact of Frame-of-Reference Rater Training
    Andrew C. Loignon, David J. Woehr, Jane Shumski Thomas, Misty L. Loughry, Matthew Ohland, Daniel Michael Ferguson · 2016 · Academy of Management Learning and Education
  88. Multimethod assessment of covert antisocial behavior in children: Laboratory observations, adult ratings, and child self-report.
    Stephen P. Hinshaw, Cassandra Simmel, Tracy L. Heller · 1995 · Psychological Assessment
  89. Supervisor assessment of clinical and professional competence of medical trainees: a reliability study using workplace data and a focused analytical literature review
    Darryl Andrew McGill, Cees P. M. van der Vleuten, Mike J Clarke · 2011 · Advances in Health Sciences Education
  90. Pseudotaxonicity in MAMBAC and MAXCOV analyses of rating-scale data: Turning continua into classes by manipulating observer's expectations.
    Theodore P. Beauchaine, Everett Waters · 2003 · Psychological Methods
  91. Difficulties in the statistical control of halo.
    Kevin R. Murphy · 1982 · Journal of Applied Psychology
  92. Does double‐blind peer review reduce bias? Evidence from a top computer science conference
    Mengyi Sun, Jainabou Barry Danfa, Misha Teplitskiy · 2021 · Journal of the Association for Information Science and Technology
  93. A reexamination of the relation between halo error and accuracy.
    Sebastiano A. Fisicaro · 1988 · Journal of Applied Psychology
  94. Distinguishing anxiety and depression: Reexamination of the reconstructed Hamilton scales.
    Karla Moras, Peter A. Di Nardo, David Harrison Barlow · 1992 · Psychological Assessment
  95. An Evaluative Review of Social Validation Research Involving Persons with Handicaps
    Keith Storey, Robert H. Horner · 1991 · The Journal of Special Education
  96. The Role of Calibration Committees in Subjective Performance Evaluation Systems
    B. William Demeré, Karen L. Sedatole, Alexander Woods · 2018 · Management Science
  97. Norms for Parental Ratings on Conners' Abbreviated Parent-Teacher Questionnaire: Implications for the Design of Behavioral Rating Inventories and Analyses of Data Derived from Them
    Katherine S. Rowe, Ken Rowe · 1997 · Research on Child and Adolescent Psychopathology
  98. Can AI Provide Useful Holistic Essay Scoring?
    Tate, Tamara P., Steiss, Jacob, Bailey, Drew, Graham, Steve, Moon, Youngsun, Ritchie, Daniel, Tseng, Waverly, Warschauer, Mark · 2024 · Computers and Education: Artificial Intelligence
  99. Applying the social relations model to self and peer evaluations
    Gary J. Greguras, Chet Robie, Marise Ph. Born · 2001 · Journal of Management Development
  100. Influence of halo error on appraisal effectiveness: A conceptual and empirical reconsideration.
    Brian E. Becker, Robert L. Cardy · 1986 · Journal of Applied Psychology

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.