“Predicting the difficulty and response time of multiple choice questions using transfer learning,”
Open accessAbstract
This paper reports on whether transfer learning can improve the prediction of the difficulty and response time parameters for ≈ 18,000 multiple-choice questions from a high-stakes medical exam.The type of the signal that best predicts difficulty and response time is also explored, both in terms of representation abstraction and item component used as input (e.g., whole item, answer options only, etc.).The results indicate that, for our sample, transfer learning can improve the prediction of item difficulty when response time is used as an auxiliary task but not the other way around.In addition, difficulty was best predicted using signal from the item stem (the description of the clinical case), while all parts of the item were important for predicting the response time.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
With automatic testing systems and large quantities of historical test logs, deep learning has been increasingly used for feature extraction and/or difficulty estimation in QDP.
“With automatic testing systems and ready availability of large quantities of historical test logs, deep learning has been increasingly used for feature extraction (word embeddings, question representations, etc.) and/or difficulty estimation (Fang et al., 2019; Lin et al., 2019; Xue et al., 2020).”
From Proactive and reactive engagement of artificial intelligence methods for education: a review (2023) · cites this paper as Xue et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition
Graves, A., Fernández, S., & Schmidhuber, J. (2005) · Lecture Notes in Computer Science · doi:10.1007/11550907_126 · Cited by 925 · see citing works
One billion word benchmark for measuring progress in statistical language modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-564 · Cited by 627 · see citing works
Question Difficulty Prediction for READING Problems in Standard Tests
Huang, Z., Liu, Q., Chen, E., Zhao, H., Gao, M., Wei, S., Su, Y., & Hu, G. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10740 · Cited by 118 · see citing works
Predicting the Difficulty of Multiple Choice Questions in a High-stakes Medical Exam
Ha, L. A., Yaneva, V., Baldwin, P., et al. (2019) · doi:10.18653/v1/w19-4402 · Cited by 48 · see citing works
Self-citationA similarity-based theory of controlling MCQ difficulty
Alsubait, T., Parsia, B., & Sattler, U. (2013) · International Conference on E-Learning and E-Technologies in Education · doi:10.1109/icelete.2013.6644389 · Cited by 37 · see citing works
Benedetto, L., Cappelli, A., Turrin, R., et al. (2020) · International Conference on Learning Analytics & Knowledge (LAK) · doi:10.1145/3375462.3375517 · Cited by 35 · see citing works
Swanson, D. B., Case, S. M., RIPKEY, D. R., et al. (2001) · Academic Medicine · doi:10.1097/00001888-200110001-00038 · Cited by 26 · see citing works
Ha, L. A., & Yaneva, V. (2018) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w18-0548 · Cited by 25 · see citing works
Self-citationTextual complexity as a predictor of difficulty of listening items in language proficiency tests.
Loukina, A., Yoon, S., Sakano, J., Wei, Y., & Sheehan, K. H. (2016) · International Conference on Computational Linguistics · Cited by 24 · see citing works
Using Natural Language Processing to Predict Item Response Times and Improve Test Construction
Baldwin, P., Yaneva, V., Mee, J., Clauser, B. E., & Ha, L. A. (2021) · Journal of Educational Measurement · doi:10.1111/jedm.12264 · Cited by 20 · cited 1× · see citing works
“A more recent study by Baldwin et al. (2020) modeled the response time of medical MCQs using a broad range of linguistic features and embeddings (similar to Yaneva et al. (2019)) and showed that the predicted response times can be used to improve fairness by reducing the time intensity variance of…”
BackgroundSelf-citationEstimating Testing Time: The Effects of Item Characteristics on Response Latency
Halkitis, P. N. (1996) · American Educational Research Association Annual Meeting · Cited by 16 · see citing works
Candidate evaluation strategies for improved difficulty prediction of language tests
Beinborn, L., Zesch, T., & Gurevych, I. (2015) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.3115/v1/w15-0601 · Cited by 16 · see citing works
Predicting Item Survival for Multiple Choice Questions in a High-Stakes Medical Exam.
Yaneva, V., Ha, L. A., Baldwin, P., & Mee, J. (2020) · Language Resources and Evaluation · Cited by 13 · see citing works
Self-citationResponse Latency: An Investigation into Determinants of Item-Level Timing.
Parshall, C. G. (1994) · Cited by 10 · see citing works
R2DE: a NLP approach to estimating IRT parameters of newly generated questionsPreprint
Benedetto, L., Cappelli, A., Turrin, R., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.07569 · Cited by 10 · see citing works
An Experimental Evaluation of Automatically Generated Multiple Choice Questions from Ontologies
Kurdi, G. R., Parsia, B., & Sattler, U. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-54627-8_3 · Cited by 9 · see citing works
Smith, R. W. (2000) · Lincoln (University of Nebraska) · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proc. of NAACL.
- Victoria Yaneva, Peter Baldwin, Janet Mee, et al. 2019. Predicting the difficulty of multiple choice questions in a high-stakes medical exam. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 11-20.
Cited by
Sorted by citations
Proactive and reactive engagement of artificial intelligence methods for education: a review
Mallik, S., & Gangopadhyay, A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1151391 · Cited by 139 · see citing works
- Authors
- Xue, K., Yaneva, V., Runyon, C., & Baldwin, P.
- Affiliations
- University of Georgia (US)
- National Board of Medical Examiners (US)
- Venue
- Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications
- Published
- 2020
- Pages
- 193-197
- OpenAlex
- W3037535674
- Cited by
- 27
- References
- 18
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 4 · 2023: 7 · 2022: 3 · 2021: 8 · 2020: 1
- FWCI
- 5.51
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on Innovative Use of NLP for Building Educational Applications (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- transfer learning, item difficulty prediction, response time prediction, multiple-choice questions, medical licensing examinations, answer options, high-stakes testing, item response theory
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
