The Performance of Large Language Models on Quantitative and Verbal Ability Tests: Initial Evidence and Implications for Unproctored High-stakes Testing

Abstract

Unproctored assessments are widely used in pre-employment assessment. However, the recent emergence of widely accessible large language models (LLMs) poses challenges for unproctored personnel assessments, given that applicants may use them to artificially inflate their scores beyond their true abilities. This may be particularly concerning in cognitive ability testing, which is widely used and is thought to be less fakeable by humans than personality tests. Thus, this study compares the performance of LLMs on two common types of cognitive tests: quantitative ability and verbal ability. The tests investigated are used in real-world, high-stakes selection. We also examine the performance of the LLMs across different test formats (i.e., open-ended vs. multiple choice). Further, we contrast the performance of two LLMs (GPT-3.5 and GPT-4) across multiple prompt approaches and ‘temperature’ settings (i.e., a parameter that determines the amount of randomness in the model’s output). We found that the LLMs performed extremely poorly on the quantitative ability test and well on the verbal ability test, even when accounting for the test format. GPT-4 outperformed GPT-3.5 across both types of tests. Notably, although prompt approaches and temperature settings did affect LLM test performance, those effects were mostly minor relative to differences across tests and language models. We provide recommendations for securing pre-employment testing against LLM influences. Additionally, we call for rigorous research investigating the prevalence of LLM usage in pre-employment testing as well as on how LLM usage influences selection test validity.