Rating the ratings: Assessing the psychometric quality of rating data.

Saal, Frank E., Downey, Ronald G., Lahey, Mary A.·1980·Psychological Bulletin

Cited by (567)

Items 301–400 of 567

  1. Evaluating Rater Effects of Large Language Models in Automated Essay Scoring: GPT, Claude, Gemini, and DeepSeek
    Hong Jiao, Dan Song, Won‐Chan Lee · 2026 · Educational Measurement Issues and Practice
  2. Can two psychotherapy process measures be dependably rated simultaneously? A generalizability study.
    Pål Ulvenes, Lene Berggraf, Asle Hoffart, Raymon A. Levy, J. Stuart Ablon, Leigh McCullough, Bruce E. Wampold · 2012 · Journal of Counseling Psychology
  3. Reflection of GDPR by the Czech Population
    Radek Tahal, Tomáš Formánek · 2020 · Management & Marketing
  4. Voice stress analysis: Use of telephone recordings
    Robert F. Wain, Ronald G. Downey · 1987 · Journal of Business and Psychology
  5. Sequential Effects in Essay Ratings: Evidence of Assimilation Effects Using Cross-Classified Models
    Haiyan Zhao, Björn Andersson, Boliang Guo, Tao Xin · 2017 · Frontiers in Psychology
  6. Rating teacher performance to determine career ladder advancement: An analysis of bias and reliability
    David W. Peterson · 1988 · Iowa State University Digital Repository (Iowa State University)
  7. When raters generalize: Examining sources of halo effects with mixture Rasch facets models
    Kuan‐Yu Jin, Thomas Eckes · 2025 · Behavior Research Methods
  8. Enhancing Cultural Heritage Archive Analysis via Automated Entity Extraction and Graph-Based Representation Learning
    A. Alper. Ozdemir, Berke Odaci, Lorans Tanatar Baruh, Onur Varol, Selim Saffet Balcisoy · 2025 · Journal on Computing and Cultural Heritage
  9. Peer effects in subjective performance evaluation
    Gavin Cassar, Taeho Ko · 2023 · Contemporary Accounting Research
  10. Exploring rater judging behaviour using the many-facet rasch model
    Noor Lide Abu Kassim, Noor Lide · 2007 · Universiti Utara Malaysia Institutional Repository (Universiti Utara Malaysia)
  11. The Economic Role of Rating Behavior in Third-Party Application Market
    Hao Lin, Xiaofei Li, Yong Tan, Lei Xu · 2011 · Journal of the Association for Information Systems
  12. Rational Dictators in the Dictator Game Are Seen as Cold and Agentic but Not Intelligent
    Janna Katrin Ruessmann, Christian Unkelbach · 2021 · Personality and Social Psychology Bulletin
  13. Automated Essay Scoring
    Beata Beigman Klebanov, Nitin Madnani · 2021 · Synthesis lectures on human language technologies
  14. Towards an Operational Definition of Clinical Competency in Pharmacy
    L. Douglas Ried, Charles Allen Douglas · 2015 · American Journal of Pharmaceutical Education
  15. Investigating the Judges Performance in a National Competition of Sport Dance
    Laura Anderlucci, Alessandro Lubisco, Stefania Mignani · 2020 · Social Indicators Research
  16. Estimating Interrater Reliability of Examiner Scoring for a State Quality Award
    Garry D. Coleman, Eileen M. Van Aken, Shen Jianming · 2002 · Quality Management Journal
  17. Nurse appraisal systems: Characteristics and effectiveness
    Robert L. Cardy, Cathy Korodi · 1991 · Social Science & Medicine
  18. The Utility of Utility Analysis
    Robert J. Vance, Adrienne J. Colella · 1990 · Human Performance
  19. The Development and Validation of an Informal Rating Matrix to Measure Social Skills in Early Adolescents
    Alan Bain · 1991 · Educational Psychology
  20. Evaluating the Validity of Educational Rating Data
    Michael R. Harwell · 1999 · Educational and Psychological Measurement
  21. Good News: Work Sample Administrators' Global Performance Judgments are (About) as Valid as We've Suspected
    Charles E. Lance, C. Douglas Johnson, Shane S. Douthitt, Winston Bennett, Donald L. Harville · 2000 · Human Performance
  22. Using Automated Scoring to Monitor Reader Performance and Detect Reader Drift in Essay Scoring
    Susan M. Lottridge, E. Matthew Schulz, Howard C. Mitzel · 2015 · Routledge eBooks
  23. Short‐listing of applicants from UCCA forms: the structure of pre‐selection judgements
    I. C. McManus, Sally Maitlis, Peter Richards · 1989 · Medical Education
  24. The latent distribution of a rating observed
    Jeffrey C. Bodington · 2022 · Journal of Wine Research
  25. Combining Dual Scaling with Semi-Structured Interviews to Interpret Rating Differences
    Ruth A. Childs, Anita Ram, Yunmei Xu · 2009 · Practical assessment, research & evaluation
  26. A framework to capture the dependency between prerequisite and advanced courses in higher education
    Raghda Hriez, Ghazi Al‐Naymat · 2021 · Journal of Computing in Higher Education
  27. Performance Appraisal and Development
    Jeffrey R. Spence, Patricia L. Baratta · 2014 · Wiley eBooks
  28. What Guides Subjective Performance Evaluation: Incentive Provision or Norm Enforcement?
    Robert M. Gillenkirch, Heike Kreienbaum · 2015 · Review of Managerial Science
  29. Stochastic error and biases remain in blind wine ratings
    Jeffrey C. Bodington · 2022 · Journal of Wine Economics
  30. Gleichgestelltenbeurteilung: Eine Erweiterung traditioneller Personalbeurteilungsansätze in Unternehmen
    Torsten J. Gerpott · 1992 · Gabler Verlag eBooks
  31. The effects of rater training, job analysis format and congruence of training on job evaluation ratings
    Douglas F. Cellar, John R. Curtis, Kim Kohlepp, Patricia Poczapski, Sameena Mohiuddin · 1989 · Journal of Business and Psychology
  32. An individual differences approach to the halo-accuracy paradox
    Chris J. Jackson · 1996 · Personality and Individual Differences
  33. Rater Fatigue in Job Analysis Surveys
    Edmond W. Israelski, Jill S. Lenoble · 1982 · Proceedings of the Human Factors and Ergonomics Society Annual Meeting
  34. Systematic Rater Training Model: An Aid to Counselors in Collecting Observational Data
    F. Robert Wilson · 1982 · Measurement and Evaluation in Guidance
  35. INTERNAL HOMOGENEITY, DESCRIPTIVENESS, AND HALO: RESURRECTING SOME ANSWERS AND QUESTIONS ABOUT THE STRUCTURE OF JOB PERFORMANCE RATING CATEGORIES
    WILLIAM H. COOPER · 1983 · Personnel Psychology
  36. Color harmony : experimental and computational modeling
    Christel Chamaret · 2016 · theses.fr (ABES)
  37. Scoring Stability in a Large‐Scale Assessment Program: A Longitudinal Analysis of Leniency/Severity Effects
    Corey Palermo, Michael B. Bunch, Kirk Ridge · 2019 · Journal of Educational Measurement
  38. Development of an Evaluation Capacity-Building Program for Nurse Teachers in Students’ Practicum
    Netrachanee Kamonratananun, Siridej Sujiva, Kamonwan Tangdhanakanond · 2016 · Procedia - Social and Behavioral Sciences
  39. Modeling random and non-random decision uncertainty in ratings data: a fuzzy beta model
    Antonio Calcagnì, Luigi Lombardi · 2021 · AStA Advances in Statistical Analysis
  40. Relating Cultural Distance to Self-Other Agreement of Leader–Observer Dyads: The Role of Hierarchical Position
    Tim Vriend, Caroline Rook, Harry Garretsen, Janka Ireen Stoker, Manfred Kets De Vries · 2021 · Frontiers in Psychology
  41. Reassigning sources of misophonic trigger sounds to change their unpleasantness: Testing alternative mechanisms with a new set of movies, paintings, and words
    Laurie M. Heller, Urszula Oszczapinska, Jessica M. Smith, Megan M. Julien · 2025 · PLoS ONE
  42. CKH: Causal Knowledge Hierarchy for Estimating Structural Causal Models from Data and Priors
    Riddhiman Adib, Md Mobasshir Arshed Naved, Chih-Hao Fang, Md Osman Gani, Ananth Y. Grama, Paul M. Griffin, Uzma Hasan, Sheikh Iqbal Ahamed, Mohammad Adibuzzaman · 2025 · IEEE Annual International Computer Software and Applications Conference (COMPSAC)
  43. VALIDATING THE GRE ANALYTICAL ABILITY MEASURE AGAINST FACULTY RATINGS OF ANALYTICAL REASONING SKILLS
    Mary K. Enright, Donald E. Powers · 1990 · ETS Research Report Series
  44. Direct Behavior Rating Instrumentation: Evaluating the Impact of Scale Formats
    Faith G. Miller, T. Chris Riley‐Tillman, Sandra M. Chafouleas, Alyssa A. Schardt · 2016 · Assessment for Effective Intervention
  45. Harmanlanmış Öğrenme Ortamlarında Bilgisayar Cebiri Sistemlerinin Akademik Başarıya Etkisi
    Kemal ŞİMŞEK, Jale İpek · 2019 · Necatibey Eğitim Fakültesi Elektronik Fen ve Matematik Eğitimi Dergisi
  46. Does Social Content Influence the Subjective Evaluation of Affective Pictures?
    Vladimir Kosonogov, José M. Martínez‐Selva, Ginesa Torrente, Eduvigis Carrillo, Juan P. Sánchez‐Navarro · 2020 · The Spanish Journal of Psychology
  47. Developing Reliable Lab Rubrics Using Only Two Columns
    Joshua Enszer · 2020 · ASEE Annual Conference & Exposition
  48. A quantitative analysis of accuracy, reliability and bias in judgements of functional analyses
    Allie E. Rader, Michael E. Young, Justin B. Leaf · 2021 · Journal of the Experimental Analysis of Behavior
  49. Flexible Endoscopic Evaluation of Swallowing: A Proposed Checklist
    Ryan J. Burdick, Justine Dallal-York, Yael Shapira‐Galitz · 2024 · American Journal of Speech-Language Pathology
  50. Employee Selection and Performance Evaluation Biases and Organizational Efficiency: A Mathematical Modeling Attempt
    Arie Reichel, Abraham Mehrez · 1994 · Journal of Management Inquiry
  51. Category‐Based and Feature‐Based Cognitive Processes: The Role of Unfavorable Information1
    Carol T. Kulik, Sue Campbell Clark · 1994 · Journal of Applied Social Psychology
  52. THE EMPLOYEES’ PERCEPTION OF THE PERFORMANCE APPRAISAL SYSTEM OF THE LOCAL GOVERNMENT OF ILOILO CITY, PHILIPPINES
    Phillip Bernard Capadosa · 2013 · Institutional Repositories DataBase (IRDB)
  53. A mixed methods investigation of leadership and performance in practice-based research networks
    Brandon James Patterson · 2013
  54. The dynamics of place ratings: platforms, effects, and their discontents
    Andrea Ballatore · 2025 · GeoJournal
  55. Measuring the Accuracy of True Score Predictions for AI Scoring Evaluation
    Daniel F. McCaffrey, Jodi M. Casabianca, Matthew S. Johnson · 2025 · Journal of Educational Measurement
  56. How Subjective Performance Evaluations of Management Accountants Can be Biased by the News that They Report
    Sascha Matanovic, Maximilian Schmidt, Arnt Wöhrmann · 2022 · Behavioral Research in Accounting
  57. Effects of raters’ professional backgrounds on assessing interpreting quality: An exploratory mixed-methods investigation into rater behavior
    Yang Li, Xini Liao, Jia Jia · 2025 · System
  58. Three Research Essays on the Effects of Charity Website Design on Online Donations
    Dong-Heon Kwak · 2014 · UWM Digital Commons (University of Wisconsin–Milwaukee)
  59. Does the Time Between Scoring Sessions Impact Scoring Accuracy? An Evaluation of Constructed‐Response Essay Responses on theGRE® General Test
    Bridgid Finn, Cathy L. W. Wendler, Kathryn L. Ricker‐Pedley, Burcu Arslan · 2018 · ETS Research Report Series
  60. A Latent Class Signal Detection Model for Rater Scoring with Ordered Perceptual Distributions
    Lawrence T. DeCarlo, Xiaoliang Zhou · 2020 · Journal of Educational Measurement
  61. Can local staff reliably assess their own programs? A confirmatory test-retest study of Lot Quality Assurance Sampling data collectors in Uganda
    Colin Beckworth, Robert Anguyo, Francis Kyakulaga, Stephen K. Lwanga, Joseph James Valadez · 2016 · BMC Health Services Research
  62. Profiling children's emotion regulation behaviours
    Angela Callear, Shane T. Harvey, DAVID L. BIMLER, Nicholas Catto · 2018 · British Journal of Developmental Psychology
  63. Visualizations in High School Chemistry Textbooks Used in Turkey
    Sevil Akaygün · 2018 · ACS symposium series
  64. Classification Consistency and Accuracy With Atypical Score Distributions
    Stella Yun Kim, Won‐Chan Lee · 2019 · Journal of Educational Measurement
  65. Relationship Between Student-Perceived Reading Strategy Instruction and Students’ Reading Comprehension: A Serial Multiple Mediation Model
    Quan Qian, Guomin Zheng, Xianhan Huang, Xian Liao · 2023 · Reading & Writing Quarterly
  66. Raters and Examinations
    Steven Barrett · 2005 · Springer eBooks
  67. The Relevance of Leadership Variables for Performance Appraisal
    Amira Galin · 1989 · Personnel Review
  68. MEASURING CHARACTER STRENGTHS: A METHODOLOGICAL STUDY OF MILITARY CADETS DURING A FIELD EXERCISE
    Ole Christian Boe, Bengt Andre Heiskel, Øyvind Moholdt Grande, Fredrik Nilsen, Henning Bang · 2016 · ICERI proceedings
  69. Journalistic Learning: Rethinking and Redefining Language Arts Curricula
    Ed Madison · 2012 · Scholars' Bank (University of Oregon)
  70. Detecting Rater Centrality Effect Using Simulation Methods and Rasch Measurement Analysis
    Xiaohui Yue · 2011 · VTechWorks (Virginia Tech)
  71. Operationalisierung
    Nicola M. Döring · 2023 · Springer eBooks
  72. On the applicability of time series anomaly detection methods to real-world traffic volume data
    Iman Taheri Sarteshnizi, Majid Sarvi, Saeed Asadi Bagloee, Neema Nassir · 2026 · Transportation Research Part C Emerging Technologies
  73. Towards an operational definition of pharmacy clinical competency
    Charles Allen Douglas · 2011 · PhDT
  74. Objectivity of International Aerobics Referee decision
    Hyungdon Kim, 우동일, EungJoon Kim · 2011 · The Korean Journal of Measurement and Evaluation in Physical Education and Sports Science
  75. The Science and Best Practices of Behavioral Safety: The Source for Reducing Injuries on the Front Line
    Timothy D. Ludwig, Matthew M. Laske · 2023 · Routledge eBooks
  76. The Second Database of Emotional Videos from Ottawa (DEVO-2): Over 1300 brief video clips rated on valence, arousal, impact, and familiarity
    Patrick S. R. Davidson, Alex W. Castro, Steven Carton, Vanessa Conceicao Cunha, Charles A. Collin · 2025 · Behavior Research Methods
  77. Geographically Weighted Cronbach's Alpha (GWalpha): An Exploratory Local Measure of Reliability for Scale Construction
    Sui Zhang, Ziqi Li · 2025 · Geographical Analysis
  78. Standard-Setting bei C-Tests: Bestimmung von Kompetenzniveaus mit der Prototypgruppenmethode
    Thomas Eckes · 2010 · Diagnostica
  79. A Pilot Test of the CM-GAF among Offenders with Mental Disorders
    Deborah Shelton, Sara T. Wakai · 2014 · Issues in Mental Health Nursing
  80. HR Practices in Selected RMG Factories in Dhaka
    Silvia Akter, Raju Mohammad Kamrul Alam · 2016 · Journal of Business
  81. Raters Knowledge of Students Proficiency Levels as a Source of Measurement Error in Oral Assessments
    Fatma TANRIVERDİ-KÖKSAL · 2017 · Hacettepe University Journal of Education
  82. RecSys'17 Joint Workshop on Interfaces and Human Decision Making for Recommender Systems
    Peter L. Brusilovsky, Marco de Gemmis, Alexander Felfernig, Pasquale Lops, John O’Donovan, Nava Tintarev, Martijn C. Willemsen · 2017 · ACM Conference on Recommender Systems (RecSys)
  83. Exploring difficult-to-score essays with a hyperbolic cosine accuracy model and Coh-Metrix indices
    Jue Wang, George Engelhard, Trenton Combs · 2021 · The Journal of Experimental Education
  84. Exploring Rater Accuracy Using Unfolding Models Combined with Topic Models: Incorporating Supervised Latent Dirichlet Allocation
    Jordan M. Wheeler, George Engelhard, Jue Wang · 2022 · Measurement Interdisciplinary Research and Perspectives
  85. CONSIDERATIONS FOR DEVELOPING MEASURES OF SPEAKING AND LISTENING
    Donald E. Powers · 1984 · ETS Research Report Series
  86. Rater Severity in Large-Scale Assessment: Is It Invariant?.
    Joy McQueen, P J Congdon · 1997
  87. THE EFFECT OF POSITIVE CORE SELF AND EXTERNAL EVALUATIONS ON PERFORMANCE APPRAISALS
    Lale Güven · 2007
  88. How smart does your profile image look? Estimating intelligence from social network profile images
    Xingjie Wei, David J. Stillwell · 2016 · White Rose Research Online (University of Leeds, The University of Sheffield, University of York)
  89. Format and scale effects on leniency error and halo effect in 360° feedback
    MyoungSo Kim, 이광진 · 2008 · Korean Journal of Industrial and Organizational Psychology
  90. Metrics for estimating validity, reliability and bias in peer assessment
    Rafael Molina-Carmona, Rosana Satorre Cuerda, Patricia Compañ, Faraón Llorens Largo · 2018 · Repositorio Institucional de la Universidad de Alicante (Universidad de Alicante)
  91. Analyzing the Judges’ rating in Modern Dance Competitors of Dong-A Concours through Many-Faceted Rasch Model
    최윤선 · 2011 · The Korean Journal of Measurement and Evaluation in Physical Education and Sports Science
  92. An Empirical Study for the Statistical Adjustment of Rater Bias
    Mustafa İlhan · 2019 · International Journal of Assessment Tools in Education
  93. Evaluating Differential Rater Functioning in Performance Ratings: Using a Goal-Based Approach
    Kevin B. Tamanini · 2008 · OhioLink ETD Center (Ohio Library and Information Network)
  94. A teacher-judgment measure of social competence: Validity data.
    Nishat A. Khan, Robert D. Hoge · 1983 · Journal of Consulting and Clinical Psychology
  95. Reliability and Accuracy of Surgical Resident Peer Ratings
    Larry Lutsky, Donald A. Risucci, Anthony J. Tortólani · 1993 · Evaluation Review
  96. An Empirical Analysis of Rating Effectiveness for a State Quality Award
    Ronald Theodore Sienknecht · 1999 · VTechWorks (Virginia Tech)
  97. Evaluating the Evaluation of Child Care Accreditation
    Sally Murray · 1996 · Australasian Journal of Early Childhood
  98. Psychometric Properties of the Indonesian Version of Religious Orientation Scale in Muslim Adolescents Sample
    Muthia Rahmah, Dewi Maulina, Rena Latifa, Indy Hurun Ein · 2025 · Tazkiya Journal of Psychology
  99. Efectos del corrector en las evaluaciones educativas de alto impacto./ Rater effects in high-impact educational assessments.
    Pamela Woitschach, Carlota Díaz-Pérez, Daniel Fernández Argüelles, Jaime Fernández-Castañón, Alba Fernández-Castillo, Lara Fernández Rodríguez, María Cristina González Canal, Iris López Marqués, David Martín Espinosa, Rubén Navarro Cabrero et al. · 2018 · R E M A Revista electrónica de metodología aplicada
  100. Analysis of Multiple Error Source in results by Judges of Artitstic Gymnastics Women’s Uneven Bars Events
    조은형, 한연숙, Eom Han Joo · 2017 · The Korean Journal of Measurement and Evaluation in Physical Education and Sports Science

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.