CoChat
Sign Up
PreprintCited by 16Psychometric Methodologies and TestingIntelligent Tutoring Systems and Adaptive LearningTopic Modeling

Position: AI Evaluation Should Learn from How We Test Humans

Open access
Zhuang, Yan, Liu, Qi, Pardos, Zachary A., Kyllonen, Patrick C., Zu, Jiyun, Huang, Zhenya, Wang, Shijin, Chen, Enhong·2023·arXiv
DOI 10.48550/arxiv.2306.10512arXiv 2306.10512
Open this paper in CoChat — free accountOpen PDF

Abstract

As AI systems continue to evolve, their rigorous evaluation becomes crucial for their development and deployment. Researchers have constructed various large-scale benchmarks to determine their capabilities, typically against a gold-standard test set and report metrics averaged across all items. However, this static evaluation paradigm increasingly shows its limitations, including high evaluation costs, data contamination, and the impact of low-quality or erroneous items on evaluation reliability and efficiency. In this Position, drawing from human psychometrics, we discuss a paradigm shift from static evaluation methods to adaptive testing. This involves estimating the characteristics or value of each test item in the benchmark, and tailoring each model's evaluation instead of relying on a fixed test set. This paradigm provides robust ability estimation, uncovering the latent traits underlying a model's observed scores. This position paper analyze the current possibilities, prospects, and reasons for adopting psychometrics in AI evaluation. We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.10512

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Zhuang et al. used IRT for adaptive testing, making testing more efficient, but do not propose a performance estimator for LLMs, only rank models based on ability parameters.

    “More recently, Zhuang et al. (2023) used IRT for adaptive testing, making testing more efficient.However, the authors do not propose a performance estimator for LLMs but only rank models based on their ability parameters.”

    From tinyBenchmarks: evaluating LLMs with fewer examples (2024) · cites this paper as Zhuang et al. (2023)

    In “Comparison of Rankings with Full Dataset” (page unavailable), the paper reports that IRT-based adaptive testing achieves strong ranking consistency using about 60% of the items, supporting the efficiency and ranking portions of the claim. But “Hard Easy” (page unavailable) specifies a procedure for estimating each LLM’s ability and provides the final estimated ability as the assessment result; the work does not only rank models. The cited passages do not show a separate estimator of overall benchmark performance. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • ChatGPT surpasses highly capable college students in dynamic programming, greedy algorithm, and search, but struggles in data structure, tree, and graph theory.

    “Specifically, ChatGPT has developed superior skills in dynamic programming, greedy algorithm, and search, surpassing highly capable college students, but it struggles in data structure, tree, and graph theory.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [265]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 121

  1. Ackerman, T. A., Gierl, M. J., and Walker, C. M. Using multidimensional item response theory to evaluate educa- tional and psychological tests. Educational Measurement: Issues and Practice, 22(3):37-51, 2003.
  2. Alex, N., Lifland, E., Tunstall, L., Thakur, A., Maham, P., Riedel, C., Hine, E., Ashurst, C., Sedille, P., Carlier, A., Noetel, M., and Stuhlmüller, A. Raft: A real-world few- shot text classification benchmark. In Vanschoren, J. and Yeung, S. (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
  3. Allen-Zhu, Z. ICML 2024 Tutorial: Physics of Language Models, July 2024.
  4. Allen-Zhu, Z. and Li, Y. Physics of language models: Part 1, learning hierarchical language structures, 2024. URL https://arxiv.org/abs/2305.13673. · arXiv
  5. Arnsten, A. F. Stress signalling pathways that impair pre- frontal cortex structure and function. Nature reviews neuroscience, 10(6):410-422, 2009.
  6. Arora, D., Singh, H. G., et al. Have llms advanced enough? a challenging problem solving benchmark for large lan- guage models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
  7. Bachmann, D., van der Wal, O., Chvojka, E., Zuidema, W. H., van Maanen, L., and Schulz, K. fl-irt-ing with psychometrics to improve nlp bias measurement. Minds and Machines, 34(4):37, 2024.
  8. Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. A mul- titask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. In Pro- ceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 675-718, 2023.
  9. Bellman, R. Dynamic programming. Science, 153(3731): 34-37, 1966.
  10. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
  11. Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., and Wal- lach, H. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1004-1015, 2021.
  12. Boiko, D. A., MacKnight, R., Kline, B., and Gomes, G. Au- tonomous chemical research with large language models. Nature, 624(7992):570-578, 2023.
  13. Bridgeman, B., Payne, D., and Briel, J. Graduate admissions test has some merit. Nature, 511(7508):155-155, 2014.
  14. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
  15. Burden, J. Evaluating ai evaluation: Perils and prospects. arXiv preprint arXiv:2407.09221, 2024. · arXiv
  16. Chang, H.-H. Psychometrics behind computerized adaptive testing. Psychometrika, 80(1):1-20, 2015.
  17. Chang, H.-H. and Ying, Z. A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20(3):213-229, 1996.
  18. Chang, K., Cramer, M., Soni, S., and Bamman, D. Speak, memory: An archaeology of books known to chatgpt/gpt- 4. In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pp. 7312- 7327, 2023.
  19. Chang, W.-C. and Yang, H.-C. Applying irt to estimate learning ability and k-means clustering in web based learning. J. Softw., 4(2):167-174, 2009.
  20. Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
  21. Cheng, C., Barceló, J., Hartnett, A. S., Kubinec, R., and Messerschmidt, L. Covid-19 government response event dataset (coronanet v. 1.0). Nature Human Behaviour, 4 (7):756-768, 2020.
  22. Cheng, M., Zhang, H., Yang, J., Liu, Q., Li, L., Huang, X., Song, L., Li, Z., Huang, Z., and Chen, E. Towards personalized evaluation of large language models with an anonymous crowd-sourcing platform. In Companion Proceedings of the ACM Web Conference 2024, pp. 1035- 1038, 2024.
  23. Cheng, Y. When cognitive diagnosis meets computerized adaptive testing: Cd-cat. Psychometrika, 74(4):619-632, 2009.
  24. Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonza- lez, J. E., et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024. · arXiv
  25. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
  26. Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., et al. Using large language models in psychology. Nature Reviews Psychology, 2(11):688-701, 2023.
  27. Deng, C., Zhao, Y., Tang, X., Gerstein, M., and Cohan, A. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8698-8711, 2024.
  28. Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., and Narasimhan, K. Toxicity in chatgpt: Analyzing persona-assigned language models. In Findings of the As- sociation for Computational Linguistics: EMNLP 2023, pp. 1236-1270, 2023.
  29. DiBello, L., Roussos, L., and Stout, W. Review of cogni- tively diagnostic assessment and a summary of psycho- metric models. cr rao, & s. sinharay (eds.), handbook of statistics, vol. 26: Psychometrics (pp. 970-1030), 2007.
  30. Efron, B. and Hinkley, D. V. Assessing the accuracy of the maximum likelihood estimator: Observed versus ex- pected fisher information. Biometrika, 65(3):457-483, 1978.
  31. Embretson, S. E. and Reise, S. P. Item response theory. Psychology Press, 2013.
  32. Fang, X., Che, S., Mao, M., Zhang, H., Zhao, M., and Zhao, X. Bias of ai-generated content: an examination of news produced by large language models. Scientific Reports, 14(1):1-20, 2024.
  33. Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. In Interna- tional Conference on Machine Learning, pp. 1126-1135.
  34. PMLR, 2017.
  35. Franzoni, V. Gender differences and bias in artificial in- telligence. In Gender in AI and robotics: The gender challenges from an interdisciplinary perspective, pp. 27- 43. Springer, 2023.
  36. Freund, R. J. and Wilson, W. J. Statistical methods. Elsevier, 2003.
  37. Gao, W., Liu, Q., Huang, Z., Yin, Y., Bi, H., Wang, M.-C., Ma, J., Wang, S., and Su, Y. Rcd: Relation map driven cognitive diagnosis for intelligent education systems. In Proceedings of the 44th international ACM SIGIR con- ference on research and development in information re- trieval, pp. 501-510, 2021.
  38. Gao, W., Liu, Q., Yue, L., Yao, F., Wang, H., Gu, Y., and Zhang, Z. Collaborative cognitive diagnosis with disen- tangled representation learning for learner modeling. In The Thirty-eighth Annual Conference on Neural Informa- tion Processing Systems, 2024.
  39. Gepshtein, S., Wang, Y., He, F., Diep, D., and Albright, T. D. A perceptual scaling approach to eyewitness identifica- tion. Nature Communications, 11(1):3380, 2020.
  40. Ghosh, A. and Lan, A. Bobcat: Bilevel optimization-based computerized adaptive testing. pp. 2410-2417. Interna- tional Joint Conferences on Artificial Intelligence Organi- zation, 8 2021.
  41. Green, P. E. and Srinivasan, V. Conjoint analysis in con- sumer research: issues and outlook. Journal of consumer research, 5(2):103-123, 1978.
  42. Greenwald, A. G., McGhee, D. E., and Schwartz, J. L. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology, 74(6):1464, 1998.
  43. Guinet, G., Omidvar-Tehrani, B., Deoras, A., and Callot, L. Automated evaluation of retrieval-augmented language models with task-specific exam generation. In Forty-first International Conference on Machine Learning, 2024.
  44. He, L., Xu, Y., He, W., Lin, Y., Tian, Y., Wu, Y., Wang, W., Zhang, Z., Han, J., Tian, Y., et al. Network model with internal complexity bridges artificial intelligence and neuroscience. Nature Computational Science, pp. 1-16, 2024.
  45. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
  46. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2), 2021.
  47. Huang, Y., Huang, W., Tong, S., Huang, Z., Liu, Q., Chen, E., Ma, J., Wan, L., and Wang, S. Stan: adversarial network for cross-domain question difficulty prediction. In 2021 IEEE International Conference on Data Mining (ICDM), pp. 220-229. IEEE, 2021.
  48. Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, P. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
  49. Kejriwal, M., Santos, H., Shen, K., Mulvehill, A. M., and McGuinness, D. L. A noise audit of human-labeled bench- marks for machine commonsense reasoning. Scientific Reports, 14(1):8609, 2024.
  50. Kipnis, A., Voudouris, K., Buschoff, L. M. S., and Schulz, E. metabench-a sparse benchmark to measure gen- eral ability in large language models. arXiv preprint arXiv:2407.12844, 2024. · arXiv
  51. Kline, P. Handbook of psychological testing. Routledge, 2013.
  52. Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., and Grefenstette, E. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
  53. Kusne, A. G., Yu, H., Wu, C., Zhang, H., Hattrick-Simpers, J., DeCost, B., Sarker, S., Oses, C., Toher, C., Curtarolo, S., Davydov, A. V., Agarwal, R., Bendersky, L. A., Li, M., Mehta, A., and Takeuchi, I. On-the-fly closed-loop materials discovery via bayesian active learning. Nature Communications, 11(1):5966, Nov 2020. ISSN 2041- 1723.
  54. Lalor, J. P., Wu, H., and Yu, H. Building an evaluation scale using item response theory. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2016, pp. 648. NIH Public Access, 2016.
  55. Lalor, J. P., Wu, H., Munkhdalai, T., and Yu, H. Understand- ing deep learning performance through an examination of test set difficulty: A psychometric case study. In Proceed- ings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2018, pp. 4711. NIH Public Access, 2018.
  56. Li, C. and Flanigan, J. Task contamination: Language mod- els may not be few-shot anymore. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18471-18480, 2024.
  57. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C. A., Manning, C. D., Re, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao, H., WANG, J., Santhanam, K., Orr, L., Zheng, L., Yuksekgonul, M., Suzgun, M., Kim, N., Guha, N., Chatterji, N. S., Khattab, O., Henderson, P., Huang, Q., Chi, R. A., Xie, S. M., Santurkar, S., Ganguli, S., Hashimoto, T., Icard, T., Zhang, T., Chaudhary, V., Wang, W., Li, X., Mai, Y., Zhang, Y., and Koreeda, Y. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
  58. Liu, Q., Huang, Z., Yin, Y., Chen, E., Xiong, H., Su, Y., and Hu, G. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering, 33(1):100-115, 2019.
  59. Liu, Q., Zhuang, Y., Bi, H., Huang, Z., Huang, W., Li, J., Yu, J., Liu, Z., Hu, Z., Hong, Y., et al. Survey of computerized adaptive testing: A machine learning perspective. arXiv preprint arXiv:2404.00712, 2024. · arXiv
  60. Liu, Y., Bhandari, S., and Pardos, Z. A. Leveraging llm respondents for item evaluation: A psychometric analysis. British Journal of Educational Technology, 56(3):1028- 1052, 2025.
  61. Loo, N., Hasani, R., Lechner, M., and Rus, D. Dataset distillation with convexified implicit gradients. In Inter- national Conference on Machine Learning, pp. 22649- 22674. PMLR, 2023.
  62. Lord, F., Novick, M., and Birnbaum, A. Statistical theories of mental test scores. Addison-Wesley, 1968.
  63. Lord, F. M. Applications of Item Response Theory to Prac- tical Testing Problems. Routledge, 1980.
  64. Louviere, J. J., Flynn, T. N., and Marley, A. A. J. Best-worst scaling: Theory, methods and applications. Cambridge University Press, 2015.
  65. Magno, C. Demonstrating the difference between classi- cal test theory and item response theory using derived test data. The international Journal of Educational and Psychological assessment, 1(1):1-11, 2009.
  66. Martínez-Plumed, F., Prudêncio, R. B., Martínez-Usó, A., and Hernández-Orallo, J. Making sense of item response theory in machine learning. In ECAI 2016, pp. 1140- 1148. IOS Press, 2016.
  67. Marx, V. The big challenges of big data. Nature, 498(7453): 255-260, 2013.
  68. Masters, G. N. A rasch model for partial credit scoring. Psychometrika, 47(2):149-174, 1982.
  69. McLeod, L., Lewis, C., and Thissen, D. A bayesian method for the detection of item preknowledge in computerized adaptive testing. Applied Psychological Measurement, 27 (2):121-137, 2003.
  70. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, 2018.
  71. Mirzasoleiman, B., Bilmes, J., and Leskovec, J. Coresets for data-efficient training of machine learning models. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6950-6960. PMLR, 13-18 Jul 2020.
  72. Nay, J. J., Karamardian, D., Lawsky, S. B., Tao, W., Bhat, M., Jain, R., Lee, A. T., Choi, J. H., and Kasai, J. Large language models as tax attorneys: a case study in legal capabilities emergence. Philosophical Transactions of the Royal Society A, 382(2270):20230159, 2024.
  73. Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., and Kiela, D. Adversarial nli: A new benchmark for nat- ural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, 2020.
  74. Otani, N., Nakazawa, T., Kawahara, D., and Kurohashi, S. Irt-based aggregation model of crowdsourced pairwise comparison for evaluating machine translations. In Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 511-520, 2016.
  75. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
  76. Peña, A., Morales, A., Fierrez, J., Serna, I., Ortega-Garcia, J., Puente, I., Cordova, J., and Cordova, G. Leveraging large language models for topic classification in the do- main of public affairs. In International Conference on Document Analysis and Recognition, pp. 20-33. Springer, 2023.
  77. Piloto, L. S., Weinstein, A., Battaglia, P., and Botvinick, M. Intuitive physics learning in a deep-learning model inspired by developmental psychology. Nature human behaviour, 6(9):1257-1267, 2022.
  78. Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., and Yurochkin, M. tinybenchmarks: evaluating LLMs with fewer examples. In Forty-first International Conference on Machine Learning, 2024.
  79. Possati, L. M. Algorithmic unconscious: why psychoanaly- sis helps in understanding ai. Palgrave Communications, 6(1):1-13, 2020.
  80. Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bon- nefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. Machine behaviour. Nature, 568(7753):477-486, 2019.
  81. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. · arXiv
  82. Rajpurkar, P., Jia, R., and Liang, P. Know what you don't know: Unanswerable questions for squad. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, 2018.
  83. Ramachandran, R., Kulkarni, T., Sharma, C., Vijaykeerthy, D., and Balasubramanian, V. N. On evaluation of vision datasets and models using human competency frame- works. arXiv preprint arXiv:2409.04041, 2024. · arXiv
  84. Rittler, N. and Chaudhuri, K. A two-stage active learning algorithm for k-nearest neighbors. In International Con- ference on Machine Learning, pp. 29103-29129.
  85. PMLR, 2023.
  86. Rodriguez, P., Barrow, J., Hoyle, A. M., Lalor, J. P., Jia, R., and Boyd-Graber, J. Evaluation examples are not equally informative: How should that change nlp leader- boards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pp. 4486-4503, 2021.
  87. Ross, S. M. A first course in probability. Pearson, 2014.
  88. Samejima, F. Graded response models. In Handbook of item response theory, pp. 95-107.
  89. Sandmann, S., Riepenhausen, S., Plagwitz, L., and Vargh- ese, J. Systematic analysis of chatgpt, google search and llama 2 for clinical decision support tasks. Nature Communications, 15(1):2050, 2024.
  90. Schneider, W. J. and McGrew, K. S. The cattell-horn-carroll theory of cognitive abilities. Contemporary intellectual assessment: Theories, tests, and issues, pp. 73-163, 2018.
  91. Sedoc, J. and Ungar, L. Item response theory for efficient human evaluation of chatbots. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pp. 21-33, 2020.
  92. Shanks, D. R. The psychology of associative learning. Cam- bridge University Press, 1995.
  93. Spearman, C. " general intelligence," objectively determined and measured. The American Journal of Psychology, 15 (2):201-292, 1904.
  94. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. · arXiv
  95. Trognon, A., Cherifi, Y. I., Habibi, I., Demange, L., and Prudent, C. Using machine-learning strategies to solve psychometric problems. Scientific Reports, 12(1):18922, 2022.
  96. Valmeekam, K., Sreedharan, S., Marquez, M., Olmo, A., and Kambhampati, S. On the planning abilities of large language models (a critical investigation with a proposed benchmark). arXiv preprint arXiv:2302.06706, 2023. · arXiv
  97. van der Linden, W. J. Bayesian item selection criteria for adaptive testing. Psychometrika, 63(2):201-216, 1998.
  98. Van Essen, D. C. and Dierker, D. L. Surface-based and probabilistic atlases of primate cerebral cortex. Neuron, 56(2):209-225, 2007.
  99. Vania, C., Htut, P. M., Huang, W., Mungra, D., Yuanzhe Pang, R., Phang, J., Liu, H., Cho, K., and Bow- man, S. R. Comparing test sets with item response theory. In Annual Meeting of the Association for Computational Linguistics, 2021.
  100. Vie, J.-J., Popineau, F., Bruillard, É., and Bourda, Y. A review of recent advances in adaptive assessment. Learn- ing analytics: fundaments, applications, and trends, pp. 113-142, 2017.
  101. Von Davier, M. The dina model as a constrained general diagnostic model: Two variants of a model equivalency. British Journal of Mathematical and Statistical Psychol- ogy, 67(1):49-71, 2014.
  102. Wang, C., Dong, Q., Wang, X., and Sui, Z. Statistical dataset evaluation: A case study on named entity recognition. Natural Language Processing, pp. 1-21, 2024a. doi: 10.1017/nlp.2024. · DOI
  103. Wang, F., Liu, Q., Chen, E., Huang, Z., Yin, Y., Wang, S., and Su, Y. Neuralcd: a general framework for cognitive diagnosis. IEEE Transactions on Knowledge and Data Engineering, 35(8):8312-8327, 2022.
  104. Wang, H., Liu, X., Fan, W., Zhao, X., Kini, V., Yadav, D., Wang, F., Wen, Z., Tang, J., and Liu, H. Rethinking large language model architectures for sequential recommen- dations. arXiv preprint arXiv:2402.09543, 2024b. · arXiv
  105. Wang, T., Zhu, J.-Y., Torralba, A., and Efros, A. A. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018. · arXiv
  106. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. · arXiv
  107. William, C. B. Computer-managed instruction: State of the art. AEDS Journal, 12(3):117-137, 1979.
  108. Xia, X., Liu, J., Zhang, S., Wu, Q., Wei, H., and Liu, T. Refined coreset selection: Towards minimal coreset size under model performance constraints. In Forty-first Inter- national Conference on Machine Learning, 2024.
  109. Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., and Zhang, Y. A survey on large language model (llm) security and pri- vacy: The good, the bad, and the ugly. High-Confidence Computing, pp. 100211, 2024.
  110. Ye, Q., Fu, H., Ren, X., and Jia, R. How predictable are large language model capabilities? a case study on big- bench. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7493-7517, 2023.
  111. Yu, J., Zhuang, Y., Huang, Z., Liu, Q., Li, X., Rui, L., and Chen, E. A unified adaptive testing system enabled by hierarchical structure search. In Forty-first International Conference on Machine Learning, 2024.
  112. Yuan, L., Chen, Y., Cui, G., Gao, H., Zou, F., Cheng, X., Ji, H., Liu, Z., and Sun, M. Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and llms eval- uations. Advances in Neural Information Processing Systems, 36, 2024.
  113. Zanette, A. and Wainwright, M. Stabilizing q-learning with linear architectures for provable efficient learning. In International Conference on Machine Learning, pp. 25920-25954.
  114. PMLR, 2022.
  115. Zhang, Z., Wu, L., Liu, Q., Liu, J., Huang, Z., Yin, Y., Zhuang, Y., Gao, W., and Chen, E. Understanding and improving fairness in cognitive diagnosis. Science China Information Sciences, 67(5):152106, 2024.
  116. Zhao, D., Andrews, J., Papakyriakopoulos, O., and Xiang, A. Position: Measure dataset diversity, don't just claim it. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 60644-60673. PMLR, 21-27 Jul 2024.
  117. Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N. Z., and Xie, X. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts, 2023. URL https://arxiv.org/abs/2306.04528. · arXiv
  118. Zhuang, Y., Liu, Q., Huang, Z., Li, Z., Jin, B., Bi, H., Chen, E., and Wang, S. A robust computerized adaptive testing approach in educational question retrieval. In Proceed- ings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 416-426, 2022a.
  119. Zhuang, Y., Liu, Q., Huang, Z., Li, Z., Shen, S., and Ma, H. Fully adaptive framework: Neural computerized adaptive testing for online education. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4):4734-4742, Jun. 2022b.
  120. Zhuang, Y., Liu, Q., Ning, Y., Huang, W., Lv, R., Huang, Z., Zhao, G., Zhang, Z., Mao, Q., Wang, S., et al. Efficiently measuring the cognitive ability of llms: An adaptive test- ing perspective. arXiv preprint arXiv:2306.10512, 2023.
  121. Zhuo, T. Y., Li, Z., Huang, Y., Shiri, F., Wang, W., Haffari, G., and Li, Y.-F. On robustness of prompt-based seman- tic parsing with large pre-trained language model: An empirical study on codex. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1090-1102, 2023.

Cited by

Sorted by citations

Authors
Zhuang, Y., Liu, Q., Pardos, Z. A., Kyllonen, P. C., Zu, J., Huang, Z., Wang, S., & Chen, E.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2306.10512
arXiv
2306.10512
OpenAlex
W4381556776
Cited by
16
References
0
Citations by year
2026: 2 · 2025: 8 · 2024: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2201888)
  • National Natural Science Foundation of China (62337001)
  • National Key Research and Development Program of China (2024YFC3308200)
  • Fundamental Research Funds for the Central Universities
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
AI evaluation, adaptive testing, psychometrics, item response theory, latent traits, assessment reliability, contaminated data, evaluation costs

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review