Using Natural Language Processing to Predict Item Response Times and Improve Test Construction
Open accessAbstract
In this article, it is shown how item text can be represented by (a) 113 features quantifying the text's linguistic characteristics, (b) 16 measures of the extent to which an information‐retrieval‐based automatic question‐answering system finds an item challenging, and (c) through dense word representations (word embeddings). Using a random forests algorithm, these data then are used to train a prediction model for item response times and predicted response times then are used to assemble test forms. Using empirical data from the United States Medical Licensing Examination, we show that timing demands are more consistent across these specially assembled forms than across forms comprising randomly‐selected items. Because an exam's timing conditions affect examinee performance, this result has implications for exam fairness whenever examinees are compared with each other or against a common standard.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Quinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · see citing works
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · see citing works
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Flesch, R. (1948) · Journal of Applied Psychology · doi:10.1037/h0057532 · Cited by 5,391 · see citing works
The MRC Psycholinguistic Database
Coltheart, M. (1981) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640748108400805 · Cited by 2,484 · see citing works
SMOG Grading - A New Readability Formula.
McLaughlin, G. H. (1969) · The Journal of Reading · Cited by 2,067 · see citing works
The Technique of Clear Writing.
Gunning, R. C. (1968) · CERN Document Server (European Organization for Nuclear Research) · Cited by 1,675 · see citing works
Coh-Metrix: Analysis of text on cohesion and language
Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195564 · Cited by 1,556 · see citing works
Introduction to Information Retrieval
Larson, R. R. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21234 · Cited by 845 · see citing works
Pedersen, T., Patwardhan, S., & Michelizzi, J. (2004) · doi:10.3115/1614025.1614037 · Cited by 829 · see citing works
Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests
Wise, S. L., & Kong, X. J. (2005) · Applied Measurement in Education · doi:10.1207/s15324818ame1802_2 · Cited by 617 · see citing works
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · see citing works
Linear Models for Optimal Test Design
van der Linden, W. J. (2005) · Springer eBooks · doi:10.1007/0-387-29054-0 · Cited by 295 · see citing works
Benjamin, R. G. (2011) · Educational Psychology Review · doi:10.1007/s10648-011-9181-8 · Cited by 281 · see citing works
An Application of Item Response Time: The Effort‐Moderated IRT Model
Wise, S. L., & DeMars, C. E. (2006) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2006.00002.x · Cited by 260 · see citing works
Rapid‐Guessing Behavior: Its Identification, Interpretation, and Implications
Wise, S. L. (2017) · Educational Measurement Issues and Practice · doi:10.1111/emip.12165 · Cited by 226 · see citing works
Sanderson, M. (2010) · Natural Language Engineering · doi:10.1017/s1351324909005129 · Cited by 210 · see citing works
100 Million Words of English:The British National Corpus (BNC)
Leech, G. (1992) · Seoul National University Open Repository (Seoul National University) · Cited by 198 · see citing works
The UMLS Metathesaurus: representing different views of biomedical concepts.
Schuyler, P., Hole, W. T., Tuttle, M. S., et al. (1993) · PubMed · Cited by 173 · see citing works
Timed Testing: An Approach Using Item Response Theory
Thissen, D. (1983) · Elsevier eBooks · doi:10.1016/b978-0-12-742780-5.50019-6 · Cited by 152 · see citing works
van der Linden, W. J., Scrams, D. J., & Schnipke, D. L. (1999) · Applied Psychological Measurement · doi:10.1177/01466219922031329 · Cited by 123 · see citing works
A Survey of Text Question Answering Techniques
Gupta, P., & Gupta, V. (2012) · International Journal of Computer Applications · doi:10.5120/8406-2030 · Cited by 115 · see citing works
Fundamentals of Artificial Neural Networks
Waszczyszyn, Z. (1999) · Courses and lectures · doi:10.1007/978-3-7091-2484-0_1 · Cited by 114 · see citing works
Using Response Times to Detect Aberrant Responses in Computerized Adaptive Testing
van der Linden, W. J., & van Krimpen-Stoop, E. M. L. A. (2003) · Psychometrika · doi:10.1007/bf02294800 · Cited by 106 · see citing works
How readers construct situation models for stories. The role of suntactic cues and causal inferences
Kintsch, W. (1995) · Typological studies in language · doi:10.1075/tsl.31.06kin · Cited by 98 · see citing works
Development and Calibration of an Item Response Model That Incorporates Response Time
Wang, T., & Hanson, B. A. (2005) · Applied Psychological Measurement · doi:10.1177/0146621605275984 · Cited by 98 · see citing works
Testing for Aberrant Behavior in Response Time Modeling
Marianti, S., Fox, J., Avetisyan, M., et al. (2014) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998614559412 · Cited by 90 · see citing works
Utilizing Response Time Distributions for Item Selection in CAT
Fan, Z., Wang, C., Chang, H., et al. (2012) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998611422912 · Cited by 72 · see citing works
Automated Test Assembly Using lp_Solve Version 5.5 in R
Diao, Q., & van der Linden, W. J. (2011) · Applied Psychological Measurement · doi:10.1177/0146621610392211 · Cited by 54 · see citing works
Clercq, O. D., & Hoste, V. (2016) · Computational Linguistics · doi:10.1162/coli_a_00255 · Cited by 45 · see citing works
A Comparison of Item‐Selection Methods for Adaptive Tests with Content Constraints
van der Linden, W. J. (2005) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2005.00015.x · Cited by 43 · see citing works
Use of Data Mining Methods to Detect Test Fraud
Man, K., Harring, J. R., & Sinharay, S. (2019) · Journal of Educational Measurement · doi:10.1111/jedm.12208 · Cited by 42 · see citing works
Predictive Control of Speededness in Adaptive Testing
van der Linden, W. J. (2008) · Applied Psychological Measurement · doi:10.1177/0146621607314042 · Cited by 34 · see citing works
van der Linden, W. J. (2011) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2010.00130.x · Cited by 33 · see citing works
An NCME Instructional Module on Data Mining Methods for Classification and Regression
Sinharay, S. (2016) · Educational Measurement Issues and Practice · doi:10.1111/emip.12115 · Cited by 30 · see citing works
Swanson, D. B., Case, S. M., RIPKEY, D. R., et al. (2001) · Academic Medicine · doi:10.1097/00001888-200110001-00038 · Cited by 26 · see citing works
Self-citationAutomatic Question Answering for Medical MCQs: Can It Go Further than Information Retrieval?
Ha, L. A., & Yaneva, V. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_049 · Cited by 19 · see citing works
Self-citationEstimating Testing Time: The Effects of Item Characteristics on Response Latency
Halkitis, P. N. (1996) · American Educational Research Association Annual Meeting · Cited by 16 · see citing works
TIME ALLOCATION AND PATTERNS OF ITEM RESPONSE
Miller, M. D. (1986) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1986.tb00240.x · Cited by 16 · see citing works
Response Latency: An Investigation into Determinants of Item-Level Timing.
Parshall, C. G. (1994) · Cited by 10 · see citing works
Harik, P., Clauser, B. E., Grabovsky, I., et al. (2018) · Journal of Educational Measurement · doi:10.1111/jedm.12177 · Cited by 10 · see citing works
Self-citationZenisky, A. L., & Baldwin, P. (2006) · Cited by 10 · see citing works
Self-citationThe Use of Item Response Time Measurements in the Construction of Educational Achievement Tests
Ebel, R. L. (1953) · Educational and Psychological Measurement · doi:10.1177/001316445301300303 · Cited by 8 · see citing works
Flagged first, then by citations
3 of the 50 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Fitzgerald C. Fremer J. &Maynes D.(2005 October).Test security issues and best practices in detecting test fraud. Paper presented at the Innovations in Testing Research Conference October 7 2005 Victoria B.C.
- Loeve M., 1977, Graduate texts in mathematics: Vol. 45. Probability theory
- Tatsuoka K. K., 1979, Proceedings of the 1979 Computerized Adaptive Testing Conference, 236
- 1944 Bureau of Publications Teachers Co Oxford England E. L. Thorndike I. Lorge The teacher's word book of 30 000 words
Cited by
Sorted by citations
Gin, B. C., O’Sullivan, P., Hauer, K. E., Abdulnour, R.-E. E., Mackenzie, M. E., ten Cate, O., & Boscardin, C. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005930 · Cited by 42 · see citing works
Complexity and Difficulty of Items in Learning Systems
Pelánek, R., Effenberger, T., & Čechák, J. (2021) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00252-4 · Cited by 30 · see citing works
“The typical application of complexity and difficulty measures in the case of MCQs is the prediction of difficulty based on complexity measures, e.g., in the context of medical questions (Yaneva et al., 2019; Baldwin et al., 2020) or English vocabulary learning (Susanti et al., 2016).”
BackgroundMethod“Predicting the difficulty and response time of multiple choice questions using transfer learning,”
Xue, K., Yaneva, V., Runyon, C., & Baldwin, P. (2020) · Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2020.bea-1.20 · Cited by 27 · see citing works
“A more recent study by Baldwin et al. (2020) modeled the response time of medical MCQs using a broad range of linguistic features and embeddings (similar to Yaneva et al. (2019)) and showed that the predicted response times can be used to improve fairness by reducing the time intensity variance of…”
BackgroundSelf-citationUsing Attributes of Survey Items to Predict Response Times May Benefit Survey Research
Schneider, S., Jin, H., Orriens, B., Junghaenel, D. U., Kapteyn, A., Meijer, E., & Stone, A. A. (2022) · Field Methods · doi:10.1177/1525822x221100904 · Cited by 13 · see citing works
“We hypothesize that the attributes of questions, based on their stems and response options, can robustly predict item response times, which could be applied to new sets of items (Baldwin et al. 2021).”
Predicting Item Survival for Multiple Choice Questions in a High-Stakes Medical Exam.
Yaneva, V., Ha, L. A., Baldwin, P., & Mee, J. (2020) · Language Resources and Evaluation · Cited by 13 · see citing works
Self-citationCohen, M. L., Boulton, A. J., Lanzi, A. M., Sutherland, E., & Pompon, R. H. (2021) · Quality of Life Research · doi:10.1007/s11136-021-02778-5 · Cited by 9 · see citing works
Historical Perspectives on Score Comparability Issues Raised by Innovations in Testing
Baldwin, P., & Clauser, B. E. (2022) · Journal of Educational Measurement · doi:10.1111/jedm.12318 · Cited by 9 · see citing works
Self-citationItem Difficulty Modeling Using Fine-tuned Small and Large Language Models
Li, M., Jiao, H., Zhou, T., Zhang, N., Peters, S., & Lissitz, R. W. (2025) · Educational and Psychological Measurement · doi:10.1177/00131644251344973 · Cited by 7 · see citing works
“These explorations thoroughly covered the most widely accepted methods off the shelf, which can serve as detailed guidance for future endeavors to other datasets.”
Predicting Item Response Theory Parameters Using Question Statements Texts
Marinho, W., Clua, E., Martí, L., & Marinho, K. (2023) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3576050.3576139 · Cited by 5 · see citing works
Extracting Linguistic Signal From Item Text and Its Application to Modeling Item Characteristics
Yaneva, V., Baldwin, P., Ha, L. A., et al. (2023) · Routledge eBooks · doi:10.4324/9781003278658-14 · Cited by 4 · see citing works
Self-citationYaneva, V., Jurich, D. P., Ha, L. A., et al. (2021) · Wolverhampton Intellectual Repository and E-Theses (University of Wolverhampton) · Cited by 2 · see citing works
Self-citationApplication of Network Analysis to Description and Prediction of Assessment Outcomes
Thompson, J. J. (2022) · Measurement Interdisciplinary Research and Perspectives · doi:10.1080/15366367.2021.1971024 · Cited by 2 · see citing works
“The response time (rt) information has proven useful for test design (Baldwin et al., 2021; Choe et al., 2018; Van der Linden & Xiong, 2013), motivation evaluation (Goldhammer, 2015; Lafit et al., 2019; Wise & Kuhfeld, 2021), enhancement of accuracy measurement (Bolsinova & Molenaar, 2018; Van der…”
MethodPredicting Response Latencies on Test Questions Based on Features of the Questions
Denner, M., Xu, X., Ober, T. M., Pei, B., & Cheng, Y. (2024) · Springer eBooks · doi:10.1007/978-981-99-9379-6_7 · Cited by 1 · see citing works
Content Modeling in Smart Learning Environments: A systematic literature review
Jiménez, A., Muñoz‐Merino, P. J., Ortíz-Rojas, M., Muñoz-Organero, M., & Kloos, C. D. (2024) · JUCS - Journal of Universal Computer Science · doi:10.3897/jucs.106023 · Cited by 1 · see citing works
Toutkoushian, E., Sun, H., Keegan, M. T., et al. (2024) · Measurement Interdisciplinary Research and Perspectives · doi:10.1080/15366367.2023.2273581
“Recent studies have examined multiple characteristics of medical licensure and certification items (e.g., Baldwin et al., 2021; Brucia, 2020; Ha et al., 2019; Yaneva et al., 2020) for the purposes of improving item quality and implementing AIG processes.”
Hurtz, G. M., & Mucino, R. (2024) · Journal of Educational Measurement · doi:10.1111/jedm.12395
Models of Internet of Things Detection System for Translation Based on Artificial Intelligence
He, J. (2024) · International Conference on Software, Systems and Information Technology (SSITCON) · doi:10.1109/ssitcon62437.2024.10796564
“In the process of English translation IoT detection, text information is analyzed to identify the features of the text, obtain feature information, and process the text [11].”
Botelho, A. F., Huang, Z., Martin, N. S., & Shin, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29755-6_21
Xiong, J., Tang, C., Liu, Q., & Li, F. (2026) · Journal of Intelligence · doi:10.3390/jintelligence14090216
“Baldwin et al. (2021) is the closest prior work in this respect, since it carries natural language processing predictions forward into test construction, though it predicts response times rather than item parameters and does not propagate predictive uncertainty into the assembly objective.”
- Authors
- Baldwin, P., Yaneva, V., Mee, J., Clauser, B. E., & Ha, L. A.
- Affiliations
- National Board of Medical Examiners (US)
- University of Wolverhampton (GB)
- Venue
- Journal of Educational Measurement
- Publisher
- Wiley
- Published
- 2020-02-24
- Volume
- 58
- Issue
- 1
- Pages
- 4-30
- ISSN
- 0022-0655 (ISSN-L)
- 1745-3984
- OpenAlex
- W3009010129
- Semantic Scholar
- fc677244d2a81602d033256e8cbd98d2c971f7c5
- Cited by
- 20
- References
- 50
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 6 · 2023: 2 · 2022: 3 · 2021: 3 · 2020: 3
- FWCI
- 1.75
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of Educational Measurement (publishedVersion)
- Wolverhampton Intellectual Repository and E-Theses (University of Wolverhampton) (submittedVersion, cc-by-nc-nd)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- item response time, test development, linguistic features, information retrieval, automatic question answering, word embeddings, random forest, USMLE, exam fairness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
