Rating the ratings: Assessing the psychometric quality of rating data.

Saal, Frank E., Downey, Ronald G., Lahey, Mary A.·1980·Psychological Bulletin

Cited by (567)

Items 101–200 of 567

  1. Comparing the Effectiveness of Self‐Paced and Collaborative Frame‐of‐Reference Training on Rater Accuracy in a Large‐Scale Writing Assessment
    Kevin R. Raczynski, Allan S. Cohen, George Engelhard, Zhenqiu Laura Lu · 2015 · Journal of Educational Measurement
  2. Measurement and assessment issues in performance appraisal.
    Theresa J. B. Kline, Lorne M. Sulsky · 2009 · Canadian Psychology/Psychologie canadienne
  3. THE ROLE OF THE TRADITIONAL RESEARCH DESIGN IN UNDERESTIMATING THE VALIDITY OF THE EMPLOYMENT INTERVIEW
    George F. Dreher, Ronald A. Ash, PRISCILLA HANCOCK · 1988 · Personnel Psychology
  4. Comparing Consensus- and Aggregation-Based Methods of Measuring Team-Level Variables: The Role of Relationship Conflict and Conflict Management Processes
    Narda R. Quigley, Amanuel G. Tekleab, Paul E. Tesluk · 2007 · Organizational Research Methods
  5. Dependability of Measurement in Counseling Psychology: An Introduction to Generalizability Theory
    William T. Hoyt, Janet Nieuwsma Melby · 1999 · The Counseling Psychologist
  6. Detecting differential rater functioning over time (DRIFT) using a Rasch multi-faceted rating scale model.
    Edward W. Wolfe, Brad Moulder, Carol M. Myford · 2001 · PubMed
  7. How invariant and accurate are domain ratings in writing assessment?
    Stefanie A. Wind, George Engelhard · 2013 · Assessing Writing
  8. How Work Orientation Impacts Objective Career Outcomes via Managerial (Mis)perceptions
    Yuna Cho, Winnie Yun Jiang · 2021 · Academy of Management Journal
  9. A Direct Study of Halo Effect
    Sheldon J. Lachman, Alan R. Bass · 1985 · The Journal of Psychology
  10. Individualized outcome measures of internal change: Methodological considerations.
    Larry E. Beutler, David Hamblin · 1986 · Journal of Consulting and Clinical Psychology
  11. ORGANIZATION AND RATER DIFFERENCES IN PERFORMANCE APPRAISALS
    Raymond F. Zammuto, Manuel London, Kendrith M. Rowland · 1982 · Personnel Psychology
  12. METHODOLOGICAL ARTICLE: A Brief Taxometrics Primer
    Theodore P. Beauchaine · 2007 · Journal of Clinical Child & Adolescent Psychology
  13. Automated Analysis of Optic Nerve Images for Detection and Staging of Papilledema
    Sebastian Echegaray, Gilberto Zamora, Honggang Yu, Wenbin Luo, Peter Solíz, Randy H. Kardon · 2011 · Investigative Ophthalmology & Visual Science
  14. Examining the Impacts of Rater Effects in Performance Assessments
    Stefanie A. Wind · 2018 · Applied Psychological Measurement
  15. Trifactor Models for Multiple-Ratings Data
    Shin, Hyo Jeong, Rabe-Hesketh, Sophia, Wilson, Mark · 2019 · Multivariate Behavioral Research
  16. Real‐Time Feedback on Rater Drift in Constructed‐Response Items: An Example From the Golden State Examination
    Hoskens, Machteld, Wilson, Mark · 2001 · Journal of Educational Measurement
  17. Teacher candidate performance assessments: Local scoring and implications for teacher preparation program improvement
    Kevin C. Bastian, Gary T. Henry, Yi Pan, Diana B. Lys · 2016 · Teaching and Teacher Education
  18. Corrections for Criterion Reliability in Validity Generalization: A False Prophet in a Land of Suspended Judgment
    James M. LeBreton, Kelly T. Scherer, Lawrence R. James · 2014 · Industrial and Organizational Psychology
  19. Student evaluation of teaching and matters of reliability
    Dennis E. Clayson · 2017 · Assessment & Evaluation in Higher Education
  20. Rater fairness in music performance assessment: evaluating model-data fit and differential rater functioning
    Wesolowski, Brian C., Wind, Stefanie A., Engelhard, George · 2015 · Musicae Scientiae
  21. The Impact of Performance Measure Discriminability on Ratee Incentives
    Tae Sik Ahn, Iny Hwang, Myung‐In Kim · 2010 · The Accounting Review
  22. Satisfaction with a performance appraisal system and appraisal discussion
    Michael K. Mount · 1984 · Journal of Organizational Behavior
  23. The Student–Teacher Relationship Scale revisited: Testing factorial structure, measurement invariance and validity criteria in German-speaking samples
    Anne Milatz, Michael Glüer, Elena Harwardt-Heinecke, Gregor Kappler, Lieselotte Ahnert · 2014 · Early Childhood Research Quarterly
  24. Many-Faceted Rasch Modeling Expert Judgment in Test Development
    Weimo Zhu, Catherine D. Ennis, Ang Chen · 1998 · Measurement in Physical Education and Exercise Science
  25. Behaviorally anchored rating scales: An application for evaluating teaching practice
    Michelle P. Martín‐Raugh, Richard J. Tannenbaum, Cynthia M. Tocci, Clyde M. Reese · 2016 · Teaching and Teacher Education
  26. Team Dating Leads to Better Online Ad Hoc Collaborations
    Ioanna Lykourentzou, Robert E. Kraut, Steven P. Dow · 2017 · ACM Conference on Computer Supported Cooperative Work (CSCW)
  27. Robust Neural Automated Essay Scoring Using Item Response Theory
    Masaki Uto, Masashi Okano · 2020 · Lecture notes in computer science
  28. Interrater reliability and agreement of performance ratings: A methodological comparison
    John W. Fleenor, Julie Fleenor, William F. Grossnickle · 1996 · Journal of Business and Psychology
  29. As if sand were stone. New concepts and metrics to probe the ground on which to build trustable AI
    Federico Cabitza, Andrea Campagner, Luca Maria Sconfienza · 2020 · BMC Medical Informatics and Decision Making
  30. Cognitive categorization processes in performance evaluation: Confirmatory tests of two models
    Charles E. Lance, David J. Woehr, Sebastiano A. Fisicaro · 1991 · Journal of Organizational Behavior
  31. Immediate effectiveness of brief psychological training for health professionals dealing with terminally ill cancer patients: A controlled study
    Darius Razavi, Nicole Delvaux, Christine Farvacques, Edmond Robaye · 1988 · Social Science & Medicine
  32. Inter-rater reliability in clinical assessments: do examiner pairings influence candidate ratings?
    Aileen Faherty, Tim Counihan, Thomas Kropmans, Yvonne Finn · 2020 · BMC Medical Education
  33. Social Context Disambiguates the Interpretation of Laughter
    William J. Curran, Gary John McKeown, Magdalena Rychlowska, Elisabeth André, Johannes Wagner, Florian Lingenfelser · 2018 · Frontiers in Psychology
  34. Empirical comparison of item response theory models with rater's parameters
    Masaki Uto, Maomi Ueno · 2018 · Heliyon
  35. Learning Automated Essay Scoring Models Using Item-Response-Theory-Based Scores to Decrease Effects of Rater Biases
    Masaki Uto, Masashi Okano · 2021 · IEEE Transactions on Learning Technologies
  36. Scoring best-worst data in unbalanced many-item designs, with applications to crowdsourcing semantic judgments
    Geoff Hollis · 2017 · Behavior Research Methods
  37. The impact of training on rater variability
    Steven Barrett · 2001
  38. Creativity and dementia: Does artistic activity affect well-being beyond the art class?
    Susan M. Gross, Deana Danilova, Michael A. Vandehey, George M. Diekhoff · 2013 · Dementia
  39. Rating ability in performance judgments: The joint influence of implicit theories and intelligence
    Neil M. A. Hauenstein, Ralph A. Alexander · 1991 · Organizational Behavior and Human Decision Processes
  40. Examining the relationship between cognitive inflexibility and internalizing and externalizing symptoms in autistic children and adolescents: A systematic review and meta‐analysis
    Jiedi Lei, Tony Charman, Eleanor Leigh, Ailsa Russell, Zameer Mohamed, Matthew J. Hollocks · 2022 · Autism Research
  41. Purposes of Performance Appraisal System: A Perceptual Study of Civil Servants in District Dera Ismail Khan Pakistan
    Malik Ikramullah, Bahadar Shah, Shadiullah Khan, Faqir Sajjad Ul Hassan, Tariq Zaman · 2012 · International Journal of Business and Management
  42. A generalized many-facet Rasch model and its Bayesian estimation using Hamiltonian Monte Carlo
    Uto, Masaki, Ueno, Maomi · 2020 · Behaviormetrika
  43. The Influence of Rater Effects in Training Sets on the Psychometric Quality of Automated Scoring for Writing Assessments
    Stefanie A. Wind, Edward W. Wolfe, George Engelhard, Peter W. Foltz, Mark Rosenstein · 2017 · International Journal of Testing
  44. Focusing the international personnel performance appraisal process
    Michael G. Harvey · 1997 · Human Resource Development Quarterly
  45. Integration of Prediction Scores From Various Automated Essay Scoring Models Using Item Response Theory
    Masaki Uto, Itsuki Aomi, Emiko Tsutsumi, Maomi Ueno · 2023 · IEEE Transactions on Learning Technologies
  46. Effectiveness of performance feedback from behaviorally anchored rating scales.
    Peter W. Hom, Angelo S. DeNisi, Angelo J. Kinicki, Brendan D. Bannister · 1982 · Journal of Applied Psychology
  47. Faculty Evaluation by Residents in an Emergency Medicine Program: A New Evaluation Instrument
    Ivan P. Steiner, Jeffrey Michael Franc, Karen D. Kelly, Brian Hunter Rowe · 2000 · Academic Emergency Medicine
  48. Automated Short-Answer Grading Using Deep Neural Networks and Item Response Theory
    Masaki Uto, Yuto Uchida · 2020 · Lecture notes in computer science
  49. Cognitive shifting and externalising problem behaviour in intellectual disability and autism spectrum disorder
    Eelke Visser, H. J. C. Berger, H. M. J. van Schrojenstein Lantman‐de Valk, J. B. Prins, Jan‐Pieter Teunisse · 2015 · Journal of Intellectual Disability Research
  50. EI competencies as a related but different characteristic than intelligence
    Richard Eleftherios Boyatzis, Joan Manuel Batista‐Foguet, Xavier Fernández‐i‐Marín, Margarida Truninger · 2015 · Frontiers in Psychology
  51. Examining Rater Effects of the TGMD-2 on Children With Intellectual Disability
    Youngdeok Kim, Ilhyeok Park, Minsoo Kang · 2012 · Adapted Physical Activity Quarterly
  52. Reliability and Accuracy of Resident Evaluations of Surgical Faculty
    Donald A. Risucci, Larry Lutsky, Robert J. Rosati, Anthony J. Tortólani · 1992 · Evaluation & the Health Professions
  53. A LONGITUDINAL EXAMINATION OF RATER AND RATEE EFFECTS IN PERFORMANCE RATINGS
    Robert J. Vance, Peter S. Winne, E. SCOTT WRIGHT · 1983 · Personnel Psychology
  54. Emotion work as a source of employee well- and ill-being: the moderating role of service interaction type
    Marcel Kern, Kai Trumpold, Dieter Zapf · 2021 · European Journal of Work and Organizational Psychology
  55. A Method for Analyzing Sparse Data Matrices in the Generalizability Theory Framework
    Christopher W. T. Chiu, Edward W. Wolfe · 2002 · Applied Psychological Measurement
  56. A comparative review of family rating scales: Guidelines for clinicians and researchers.
    Cindy I. Carlson, Harold D. Grotevant · 1987 · Journal of Family Psychology
  57. Brief psychological training for health care professionals dealing with cancer patients: A one-year assessment
    Darius Razavi, Nicole Delvaux, Christine Farvacques, Edmond Robaye · 1991 · General Hospital Psychiatry
  58. The Nosology of Attention-deficit Hyperactivity Disorder
    Gerald J. August, Barry D. Garfinkel · 1993 · Journal of the American Academy of Child & Adolescent Psychiatry
  59. Inflated Reputations: Uncertainty, Leniency, and Moral Wiggle Room in Trader Feedback Systems
    Gary Eugene Bolton, David J. Kusterer, Johannes Mans · 2019 · Management Science
  60. Student Engagement and Its Predictors: An Exploratory Study in an Indian Business School
    Baldev R. Sharma, Pradip K. Bhaumik · 2013 · Global Business Review
  61. Assessing Self-Appraisal of Job Performance as an Evaluation Device: Are the Poor Results a Function of Method or Methodology?
    Mark John Somers, Dee Birnbaum · 1991 · Human Relations
  62. THE BEST AND THE REST: REVISITING THE NORM OF NORMALITY OF INDIVIDUAL PERFORMANCE
    Ernest H. O’Boyle, Herman Aguinis · 2011 · Academy of Management Proceedings
  63. EFFECTS OF ASSESSORSHIP ON MANAGERs' PROFICIENCY IN ACQUIRING, EVALUATING, AND COMMUNICATING INFORMATION ABOUT PEOPLE
    Robert V. Lorenzo · 1984 · Personnel Psychology
  64. Teaching Design Freedom: Understanding the Effects of Variations in Design for Additive Manufacturing Education on Students’ Creativity
    Rohan Prabhu, Scarlett R. Miller, Timothy W. Simpson, Nicholas Alexander Meisel · 2020 · Journal of Mechanical Design
  65. Database of Emotional Videos from Ottawa (DEVO)
    Kylee T. Ack Baraly, Lydia Muyingo, Christine Beaudoin, Sanaz Karami, Melina Langevin, Patrick S. R. Davidson · 2020 · Collabra Psychology
  66. Analyzing rater severity in a freshman composition course using many facet Rasch measurement
    Inan Deniz Erguvan, Beyza Aksu Dünya · 2020 · Language Testing in Asia
  67. Examiner error in curriculum-based measurement of oral reading
    Kelli D. Cummings, Gina Biancarosa, Andrew Schaper, Deborah K. Reed · 2014 · Journal of School Psychology
  68. Evaluating Teacher Effectiveness Using Classroom Observations: A Rasch Analysis of the Rater Effects of Principals
    Eli A. Jones, Christi Crosby Bergin · 2019 · Educational Assessment
  69. Statistical analysis of ordinal categorical status after therapies.
    Robert J. Grissom · 1994 · Journal of Consulting and Clinical Psychology
  70. Kohut's psychology of the self: Theory and measures of counseling outcome.
    Michael John Patton, Gary E. Connor, Kathleen J. Scott · 1982 · Journal of Counseling Psychology
  71. An examination of the presence, magnitude and impact of halo on consumer satisfaction measures
    Jochen Wirtz · 2000 · Journal of Retailing and Consumer Services
  72. Evidence for the Effectiveness of an Alternative Multisource Performance Rating Methodology
    Brian James Hoffman, Charles Allen Gorman, Carrie A. Blair, John Patrick Meriac, Benjamin L. Overstreet, E. Kate Atchley · 2012 · Personnel Psychology
  73. Prediction of global ratings of fluency and delivery in narrative discourse by linguistic and phonetic measures- oral performances of students aged 11-12 years
    Amos J. S. van Gelderen · 1994 · Language Testing
  74. Performance Appraisers as Test Developers
    Cristina G. Banks, Loriann Roberson · 1985 · Academy of Management Review
  75. Intersection of the Web-Based Vaping Narrative With COVID-19: Topic Modeling Study
    Kamila Janmohamed, Abdul‐Nasah Soale, Laura Forastiere, Weiming Tang, Yongjie Sha, Jakob Demant, Edoardo M. Airoldi, Navin Kumar · 2020 · Journal of Medical Internet Research
  76. A Trustworthy Human–Machine framework for collective decision making in Food–Energy–Water management: The role of trust sensitivity
    Süleyman Uslu, Davinder Kaur, Samuel J. Rivera, Arjan Durresi A, Meghna Babbar‐Sebens, Jenna H. Tilt · 2020 · Knowledge-Based Systems
  77. Improving Rater Calibration in Aviation: A Case Study
    Robert W. Holt, Jeffrey T. Hansberger, Deborah A. Boehm‐Davis · 2002 · International Journal of Aviation Psychology
  78. The Kestenberg Movement Profile: Performance of Novice Raters
    Sabine C. Koch, Robyn Flaum Cruz, Sherry W. Goodill · 2001 · American Journal of Dance Therapy
  79. How Network Embeddedness Affects Real-Time Performance Feedback: An Empirical Investigation
    Mariia Petryk, Michael Rivera, Siddharth Bhattacharya, Liangfei Qiu, Subodha Kumar · 2022 · Information Systems Research
  80. Taxometric Methods: Enhancing Early Detection and Prevention of Psychopathology by Identifying Latent Vulnerability Traits
    Theodore P. Beauchaine, Penny Marsh · 2015 · Wiley eBooks
  81. Interobserver relational agreement.
    Wm Wren Stine · 1989 · Psychological Bulletin
  82. Psychometric Characteristics of Behavior Rating Scales: Definitions, Problems, and Solutions
    Michael J. Wilson, Lyndal M. Bullock · 1989 · Behavioral Disorders
  83. Cross-sectional and longitudinal observations of activity level and sustained attention in a normative sample
    Richard S. Milich · 1984 · Research on Child and Adolescent Psychopathology
  84. Unravelling Vietnamese Students' Critical Thinking and Its Relationship with Argumentative Writing
    Tuyen Son Nguyen, Huan Buu Nguyen · 2020 · Universal Journal of Educational Research
  85. A model of volitional rating behavior
    Jeffrey S. Kane · 1994 · Human Resource Management Review
  86. Outlier Aversion in Subjective Evaluation: Evidence From World Figure Skating Championships
    Jungmin Lee · 2007 · Journal of Sports Economics
  87. How Smart Does Your Profile Image Look? Estimating Intelligence from Social Network Profile Images
    Xingjie Wei, David J. Stillwell · 2017 · ACM International Conference on Web Search and Data Mining (WSDM)
  88. The Critical Role of Anchor Paper Selection in Writing Assessment
    Sharon E. Osborn Popp, Joseph M. Ryan, Marilyn S. Thompson · 2009 · Applied Measurement in Education
  89. Development and validation of the Bush-Francis Catatonia Rating Scale – Brazilian version
    Ana Letícia Santos Nunes, Alberto Filgueiras, Rodrigo Nicolato, Jussara Mendonça Alvarenga, Luciana Angélica Silva Silveira, Rafael de Assis da Silva, Elie Cheniaux · 2017 · Arquivos de Neuro-Psiquiatria
  90. But Will It Build? Assessing Student Engineering Designers’ Use of Design for Additive Manufacturing Considerations in Design Outcomes
    Rohan Prabhu, Scarlett R. Miller, Timothy W. Simpson, Nicholas Alexander Meisel · 2020 · Journal of Mechanical Design
  91. Beurteilereffekte bei der Messung von Unterrichtsqualität: Das Multifacetten-Rasch-Modell und die Generalisierbarkeitstheorie als Methoden der Qualitätssicherung in der externen Evaluation von Schulen
    Marcus Pietsch, Simone Tosana · 2008 · Zeitschrift für Erziehungswissenschaft
  92. Resident evaluation of surgical faculty
    Anthony J. Tortólani, Donald A. Risucci, Robert J. Rosati · 1991 · Journal of Surgical Research
  93. Evaluation of students' clinical competence.
    Evelyn R. Dienst, Patricia M. Armstrong · 1988 · Professional Psychology Research and Practice
  94. Effects of rating criteria order on the halo effect in L2 writing assessment: a many-facet Rasch measurement analysis
    Hyun-Woo Kim · 2020 · Language Testing in Asia
  95. Don't Just Judge the Spelling! The Influence of Spelling on Assessing Second-Language Student Essays
    Thorben Jansen, Cristina Vögelin, Nils Machts, Stefan Daniel Keller, Jens Møller · 2021 · Frontline Learning Research
  96. Need-solution pair recognition by household sector individuals: Evidence, and a cognitive mechanism explanation
    Ruth Stock‐Homburg, Shannon L. M. Heald, Christian Holthaus, Nils Lennart Gillert, Eric von Hippel · 2020 · Research Policy
  97. Dynamic Fuzzy Rating Tracker (DYFRAT): a novel methodology for modeling real-time dynamic cognitive processes in rating scales
    Antonio Calcagnì, Luigi Lombardi · 2014 · Applied Soft Computing
  98. Comments on the Measurement of Halo
    Sebastiano A. Fisicaro, Robert J. Vance · 1994 · Educational and Psychological Measurement
  99. Hawks and doves: The influence of nurse assessor stringency and leniency on pass grades in clinical skills assessments
    Miranda Daly, Yenna Salamonson, Paul J. Glew, Bronwyn Everett · 2016 · Collegian Journal of the Royal College of Nursing Australia
  100. On the design and validation of a rubric for the evaluation of performance in a musical contest
    Marcos Álvarez-Díaz, Luis Magín Muñiz-Bascón, Antonio Soria-Alemany, Alberto Veintimilla-Bonet, Rubén Fernández-Alonso · 2020 · International Journal of Music Education

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.