Automated video interview personality assessments: Reliability, validity, and generalizability investigations.
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Organizations are increasingly adopting automated video interviews (AVIs) to screen job applicants despite a paucity of research on their reliability, validity, and generalizability. In this study, we address this gap by developing AVIs that use verbal, paraverbal, and nonverbal behaviors extracted from video interviews to assess Big Five personality traits. We developed and validated machine learning models within (using nested cross-validation) and across three separate samples of mock video interviews (total N = 1,073). Also, we examined their test–retest reliability in a fourth sample (N = 99). In general, we found that the AVI personality assessments exhibited stronger evidence of validity when they were trained on interviewer-reports rather than self-reports. When cross-validated in the other samples, AVI personality assessments trained on interviewer-reports had mixed evidence of reliability, exhibited consistent convergent and discriminant relations, used predictors that appear to be conceptually relevant to the focal traits, and predicted academic outcomes. On the other hand, there was little evidence of reliability or validity for the AVIs trained on self-reports. We discuss the implications for future work on AVIs and personality theory, and provide practical recommendations for the vendors marketing such approaches and organizations considering adopting them.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
Mischel, W., & Shoda, Y. (1995) · Psychological Review · doi:10.1037/0033-295x.102.2.246 · Cited by 3,889
A meta-analysis of the five-factor model of personality and academic performance.
Poropat, A. E. (2009) · Psychological Bulletin · doi:10.1037/a0014996 · Cited by 2,729
A personality trait-based interactionist model of job performance
Tett, R. P., & Burnett, D. D. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.3.500 · Cited by 2,241
Openface 2.0: facial behavior analysis toolkit
Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00019 · Cited by 1,861
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818
Cited by
Lacroux, A., & Martin-Lacroux, C. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.895997 · Cited by 67
“Resume screening is a time-consuming task for which a wide range of solutions exists, from automatic extraction and applications’ ranking to advice on choosing the most suitable candidates, and ADSSs are already a viable option for decision support (Hickman et al., 2021).”
BackgroundHow asynchronous video interviews are used in practice: A study of an Australian‐based AVI vendor
Dunlop, P. D., Holtrop, D., & Wee, S. (2022) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12372 · Cited by 46
Breil, S. M., Lievens, F., Forthmann, B., & Back, M. D. (2023) · Personnel Psychology · doi:10.1111/peps.12507 · Cited by 22
From Tasks to Traits: Can LLMs Infer Personality From Brief Assessment Center Behavior?
Härtel, T. M. (2026) · International Journal of Selection and Assessment · doi:10.1111/ijsa.70074 · Cited by 1
Sharma, G., & Molamohammadi, M. (2026) · arXiv · doi:10.48550/arxiv.2604.22679
Metadata
- Authors
- Hickman, L., Bosch, N., Ng, V., Saef, R., Tay, L., & Woo, S. E.
- Venue
- Journal of Applied Psychology
- Published
- 2022-08
- Volume
- 107
- Issue
- 8
- Pages
- 1323-1351
- ISSN
- 0021-9010 (ISSN-L)
- 1939-1854
- OpenAlex
- W3161452233
- Cited by
- 179
- References
- 25
- FWCI
- 16.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.