Automated video interview personality assessments: Reliability, validity, and generalizability investigations.
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Organizations are increasingly adopting automated video interviews (AVIs) to screen job applicants despite a paucity of research on their reliability, validity, and generalizability. In this study, we address this gap by developing AVIs that use verbal, paraverbal, and nonverbal behaviors extracted from video interviews to assess Big Five personality traits. We developed and validated machine learning models within (using nested cross-validation) and across three separate samples of mock video interviews (total N = 1,073). Also, we examined their test–retest reliability in a fourth sample (N = 99). In general, we found that the AVI personality assessments exhibited stronger evidence of validity when they were trained on interviewer-reports rather than self-reports. When cross-validated in the other samples, AVI personality assessments trained on interviewer-reports had mixed evidence of reliability, exhibited consistent convergent and discriminant relations, used predictors that appear to be conceptually relevant to the focal traits, and predicted academic outcomes. On the other hand, there was little evidence of reliability or validity for the AVIs trained on self-reports. We discuss the implications for future work on AVIs and personality theory, and provide practical recommendations for the vendors marketing such approaches and organizations considering adopting them.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
Mischel, W., & Shoda, Y. (1995) · Psychological Review · doi:10.1037/0033-295x.102.2.246 · Cited by 3,889
A meta-analysis of the five-factor model of personality and academic performance.
Poropat, A. E. (2009) · Psychological Bulletin · doi:10.1037/a0014996 · Cited by 2,729
A personality trait-based interactionist model of job performance
Tett, R. P., & Burnett, D. D. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.3.500 · Cited by 2,241
Openface 2.0: facial behavior analysis toolkit
Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00019 · Cited by 1,861
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818
Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019) · Psychological Science in the Public Interest · doi:10.1177/1529100619832930 · Cited by 1,748
On the accuracy of personality judgment: A realistic approach
Funder, D. C. (1995) · Psychological Review · doi:10.1037/0033-295x.102.4.652 · Cited by 1,342
Amazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
Kuncel, N. R., Credé, M., & Thomas, L. L. (2005) · Review of Educational Research · doi:10.3102/00346543075001063 · Cited by 1,137
Personality predictors of academic outcomes: Big five correlates of GPA and SAT scores.
Noftle, E. E., & Robins, R. W. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.93.1.116 · Cited by 1,068
Connelly, B. S., & Ones, D. S. (2010) · Psychological Bulletin · doi:10.1037/a0021212 · Cited by 1,059
Meaning and Values in Test Validation: The Science and Ethics of Assessment
Messick, S. (1989) · Educational Researcher · doi:10.3102/0013189x018002005 · Cited by 948
Automatic personality assessment through social media language.
Park, G., Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Kosinski, M., Stillwell, D. J., Ungar, L. H., & Seligman, M. E. P. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000020 · Cited by 802
Correlational effect size benchmarks
Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015) · Journal of Applied Psychology · doi:10.1037/a0038047 · Cited by 768
Judge, T. A., & Zapata, C. P. (2015) · Academy of Management Journal · doi:10.5465/amj.2010.0837 · Cited by 738
RECONSIDERING THE USE OF PERSONALITY TESTS IN PERSONNEL SELECTION CONTEXTS
MORGESON, F. P., CAMPION, M. A., DIPBOYE, R. L., HOLLENBECK, J. R., MURPHY, K., & SCHMITT, N. (2007) · Personnel Psychology · doi:10.1111/j.1744-6570.2007.00089.x · Cited by 638
Measuring Bias in Contextualized Word Representations
Kurita, K., Vyas, N., Pareek, A., Black, A. W., & Tsvetkov, Y. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3823 · Cited by 348
Highly automated job interviews: Acceptance under the influence of stakes
Langer, M., König, C. J., & Papathanasiou, M. (2019) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12246 · Cited by 203
Perceiver effects as projective tests: What your perceptions of others say about you.
Wood, D., Harms, P., & Vazire, S. (2010) · Journal of Personality and Social Psychology · doi:10.1037/a0019390 · Cited by 184
Madera, J. M., & Hebl, M. R. (2012) · Journal of Applied Psychology · doi:10.1037/a0025799 · Cited by 134
Hi YouTube!: personality impressions and verbal content in social video
Biel, J.-I., Tsiminaki, V., Dines, J., & Gatica-Perez, D. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2522877 · Cited by 66
Mitigating Bias in Algorithmic Employment Screening: Evaluating Claims and Practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3408010 · Cited by 51
Cited by
Lacroux, A., & Martin-Lacroux, C. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.895997 · Cited by 67
“Resume screening is a time-consuming task for which a wide range of solutions exists, from automatic extraction and applications’ ranking to advice on choosing the most suitable candidates, and ADSSs are already a viable option for decision support (Hickman et al., 2021).”
BackgroundHow asynchronous video interviews are used in practice: A study of an Australian‐based AVI vendor
Dunlop, P. D., Holtrop, D., & Wee, S. (2022) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12372 · Cited by 46
Breil, S. M., Lievens, F., Forthmann, B., & Back, M. D. (2023) · Personnel Psychology · doi:10.1111/peps.12507 · Cited by 22
From Tasks to Traits: Can LLMs Infer Personality From Brief Assessment Center Behavior?
Härtel, T. M. (2026) · International Journal of Selection and Assessment · doi:10.1111/ijsa.70074 · Cited by 1
Sharma, G., & Molamohammadi, M. (2026) · arXiv · doi:10.48550/arxiv.2604.22679
Metadata
- Authors
- Hickman, L., Bosch, N., Ng, V., Saef, R., Tay, L., & Woo, S. E.
- Venue
- Journal of Applied Psychology
- Published
- 2022-08
- Volume
- 107
- Issue
- 8
- Pages
- 1323-1351
- ISSN
- 0021-9010 (ISSN-L)
- 1939-1854
- OpenAlex
- W3161452233
- Cited by
- 179
- References
- 27
- FWCI
- 16.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.