Initial investigation into computer scoring of candidate essays for personnel selection.
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
[Correction Notice: An Erratum for this article was reported in Vol 101(7) of Journal of Applied Psychology (see record 2016-32115-001). In the article the affiliations for Emily D. Campion and Matthew H. Reider were originally incorrect. All versions of this article have been corrected.] Emerging advancements including the exponentially growing availability of computer-collected data and increasingly sophisticated statistical software have led to a "Big Data Movement" wherein organizations have begun attempting to use large-scale data analysis to improve their effectiveness. Yet, little is known regarding how organizations can leverage these advancements to develop more effective personnel selection procedures, especially when the data are unstructured (text-based). Drawing on literature on natural language processing, we critically examine the possibility of leveraging advances in text mining and predictive modeling computer software programs as a surrogate for human raters in a selection context. We explain how to "train" a computer program to emulate a human rater when scoring accomplishment records. We then examine the reliability of the computer's scores, provide preliminary evidence of their construct validity, demonstrate that this practice does not produce scores that disadvantage minority groups, illustrate the positive financial impact of adopting this practice in an organization (N ∼ 46,000 candidates), and discuss implementation issues. Finally, we discuss the potential implications of using computer scoring to address the adverse impact-validity dilemma. We suggest that it may provide a cost-effective means of using predictors that have comparable validity but have previously been too expensive for large-scale screening. (PsycINFO Database Record
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Landauer, T. K., & Dumais, S. T. (1997) · Psychological Review · doi:10.1037/0033-295x.104.2.211 · Cited by 6,282
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608 · cited 1×
“The ICC(1) values reflect the reliabilities of single raters, and ICC(3) values reflect the reliabilities of the means of the three raters (based on intraclass correlations, LeBreton & Senter, 2008).”
BackgroundAutomated essay scoring: Psychometric guidelines and practices
Ramineni, C., & Williamson, D. M. (2013) · Assessing Writing · doi:10.1016/j.asw.2012.10.004 · Cited by 127 · cited 2×
“AES programs can score writing skill at a level of reliability comparable to that of a human grader, but they are often limited in their ability to evaluate content and other higher order skills (Attali et al., 2013; Deane, 2013; Ramineni & Williamson, 2013; Shermis et al., 2010).”
Background
Cited by
Schwarzmüller, T., Brosi, P., Duman, D., & Welpe, I. M. (2018) · management revu · doi:10.5771/0935-9915-2018-2-114 · Cited by 409
Text Preprocessing for Text Mining in Organizational Research: Review and Recommendations
Hickman, L., Thapa, S., Tay, L., Cao, M., & Srinivasan, P. (2022) · Organizational Research Methods · doi:10.1177/1094428120971683 · Cited by 362
Highly automated job interviews: Acceptance under the influence of stakes
Langer, M., König, C. J., & Papathanasiou, M. (2019) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12246 · Cited by 203
“…job interview training might be more relevant for entry‐level jobs (cf. Langer et al., 2016) and because highly automated solutions might be more useful to screen a large applicant pool (cf. Campion et al., 2016), which is more likely for entry‐level positions compared to jobs in higher management.”
Sajjadiani, S., Sojourner, A. J., Kammeyer-Mueller, J. D., & Mykerezi, E. (2019) · Journal of Applied Psychology · doi:10.1037/apl0000405 · Cited by 173
Lacroux, A., & Martin-Lacroux, C. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.895997 · Cited by 67
“As mentioned previously, this scarcity is of particular concern for personnel selection, in which the market for predictive hiring tools is growing fast but academic research remains at an early stage (Campion et al., 2016; Langer et al., 2019b, 2021a; Tambe et al., 2019; Oberst et al., 2020).”
BackgroundCheck the box! How to deal with automation bias in AI-based personnel selection
Kupfer, C., Prassl, R., Fleiß, J., Malin, C., Thalmann, S., & Kubicek, B. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1118723 · Cited by 59
Metadata
- Authors
- Campion, M. C., Campion, M. A., Campion, E. D., & Reider, M. H.
- Venue
- Journal of Applied Psychology
- Published
- 2016-07
- Volume
- 101
- Issue
- 7
- Pages
- 958-975
- ISSN
- 0021-9010 (ISSN-L)
- 1939-1854
- OpenAlex
- W2336430626
- Cited by
- 130
- References
- 3
- FWCI
- 13.48
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.