INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models
Open accessAbstract
Instruction-tuned large language models have revolutionized natural language processing and have shown great potential in applications such as conversational agents. These models, such as GPT-4, can not only master language but also solve complex tasks in areas like mathematics, coding, medicine, and law. Despite their impressive capabilities, there is still a lack of comprehensive understanding regarding their full potential, primarily due to the black-box nature of many models and the absence of holistic evaluation studies. To address these challenges, we present INSTRUCTEVAL, a more comprehensive evaluation suite designed specifically for instruction-tuned large language models. Unlike previous works, our evaluation involves a rigorous assessment of models based on problem-solving, writing ability, and alignment to human values. We take a holistic approach to analyze various factors affecting model performance, including the pretraining foundation, instruction-tuning data, and training methods. Our findings reveal that the quality of instruction data is the most crucial factor in scaling model performance. While open-source models demonstrate impressive writing abilities, there is substantial room for improvement in problem-solving and alignment. We are encouraged by the rapid development of models by the open-source community, but we also highlight the need for rigorous evaluation to support claims made about these models. Through INSTRUCTEVAL, we aim to foster a deeper understanding of instruction-tuned models and advancements in their capabilities. INSTRUCTEVAL is publicly available at https://github.com/declare-lab/instruct-eval.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Part of this paper’s reference list could not be split into single entries.
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“While instructed LLMs are capable of performing many tasks in a zero-shot fashion, their generalization may be enhanced by providing few-shot demonstrations during inference (Brown et al., 2020).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 2× · see citing works
“GPT-NeoX (Black et al., 2022) Decoder 472B The Pile Allowed StableLM (StabilityAI, 2023) Decoder 800B StableLM Pile Allowed LLaMA (Touvron et al., 2023) Decoder 1.4T LLaMA No Pythia (Biderman et al., 2023) Decoder 472B The Pile Allowed OPT (Zhang et al., 2022) Decoder 180B The Pile Allowed UL2 (Tay…”
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1× · see citing works
“In this subsection, we detail the benchmarks used for evaluating various problem-solving aspects.”
MethodTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2× · see citing works
“While most instruction-tuned LLMs are trained with supervised fine-tuning, this may not capture the nuances of human preferences compared to reinforcement learning from human feedback (Ouyang et al., 2022).”
BackgroundLoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · cited 1× · see citing works
“Notably, we find that parameter-efficient methods such as LoRA [Hu et al., 2021] are more effective as the instructed LLM scales in parameter count.”
MethodScaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243 · see citing works
The Flan Collection: Designing Data and Methods for Effective Instruction TuningPreprint
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2301.13688 · Cited by 1,029 · cited 11× · see citing works
“Alpaca Data (Taori et al., 2023) 52K 52K General GPT-3 Flan Collection (Longpre et al., 2023) 15M 1836 General Human-Annotation Self-Instruct (Wang et al., 2023) 82K 52K General GPT-3 Natural Instructions (Mishra et al., 2022) 620K 61 General Human-Annotation Super-Natural Instructions (Mishra et…”
BackgroundInfluentialGLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,028 · cited 2× · see citing works
“…2022) Decoder 180B The Pile Allowed UL2 (Tay et al., 2023) Encoder-Decoder 1T C4 Allowed T5 (Raffel et al., 2020) Encoder-Decoder 1T C4 Allowed GLM (Du et al., 2022) Hybrid-Decoder 1T The Pile, Wudao Corpora No RWKV (Peng et al., 2023) Parallelizable RNN 472B The Pile Allowed Mosaic (MosaicML,…”
BackgroundFantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2021) · arXiv · doi:10.48550/arxiv.2104.08786 · Cited by 615 · cited 1× · see citing works
“However, this area of in-context learning (Wei et al., 2022; Wu et al., 2023; Lu et al., 2022; Liu et al., 2022) is still an emerging research area, and there are few studies that involve diverse models and tasks.”
BackgroundMultitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · see citing works
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.556 · Cited by 610 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · cited 1× · see citing works
“Complex Instructions BIG-Bench Hard (BBH) is a subset of 23 challenging tasks from the BIG-Bench benchmark (Srivastava et al., 2022), which focuses on tasks believed to be beyond the capabilities of current language models (Suzgun et al., 2022).”
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563 · see citing works
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · cited 1× · see citing works
“…Labs, 2023) 15K 12K General Human-Annotation OpenAssistant Conversations (Köpf et al., 2023) 161K 161K Dialogue Human-Annotated Anthropic HH (Bai et al., 2022) 161K 161K Safety Human-Annotated such efforts, we are deeply concerned that the rate of development of new models may outpace the…”
BackgroundGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q. G., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · doi:10.18653/v1/2022.bigscience-1.9 · Cited by 407 · see citing works
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.936 · Cited by 390 · see citing works
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · arXiv · doi:10.48550/arxiv.2305.01937 · Cited by 331 · cited 1× · see citing works
“Specifically, we provide suitable rubrics of relevance and coherence to the evaluation model, where relevance measures how well the answer engages with the given prompt and coherence covers the general text quality such as organization and logical flow (Chiang and Lee, 2023).”
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 7× · see citing works
“While previous evaluation studies have assessed models on benchmarks such as exams across diverse topics Hendrycks et al. [2021], Zhong et al.”
BackgroundMethodInfluentialGLM-130B: An Open Bilingual Pre-trained Model
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02414 · Cited by 281 · see citing works
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.244 · Cited by 269 · see citing works
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247 · see citing works
Wu, Z., Wang, Y., Ye, J., & Kong, L. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.10375 · Cited by 237 · cited 1× · see citing works
“However, this area of in-context learning (Wei et al., 2022; Wu et al., 2023; Lu et al., 2022; Liu et al., 2022) is still an emerging research area, and there are few studies that involve diverse models and tasks.”
BackgroundOpenAssistant Conversations - Democratizing Large Language Model Alignment
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07327 · Cited by 218 · cited 2× · see citing works
“…et al., 2022) 12M 62 General Human-Annotation Databricks Dolly (Databricks Labs, 2023) 15K 12K General Human-Annotation OpenAssistant Conversations (Köpf et al., 2023) 161K 161K Dialogue Human-Annotated Anthropic HH (Bai et al., 2022) 161K 161K Safety Human-Annotated such efforts, we are deeply…”
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.149 · Cited by 211 · see citing works
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211 · cited 1× · see citing works
“While previous evaluation studies have assessed models on benchmarks such as exams across diverse topics (Hendrycks et al., 2021; Zhong et al., 2023), they do not consider holistic aspects such as general writing ability and alignment with human values.”
BackgroundUL2: Unifying Language Learning ParadigmsPreprint
Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Shakeri, S., Bahri, D., Schuster, T., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05131 · Cited by 194 · cited 10× · see citing works
“[2023] OPT 1-30B OPT-IML Bench Supervised Flan-UL2 Tay et al. [2023] UL2 20B Flan-Collection Supervised Tk-Instruct Wang et al. [2022] T5 3-11B Super-Natural Instructions Supervised Flan-Alpaca Chia et al. [2023] T5 3-11B Alpaca Data Supervised Flan-T5 Chung et al. [2022] T5 3-11B Flan-Collection Supervised Vicuna Chi…”
BackgroundInfluentialPythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01373 · Cited by 160 · cited 2× · see citing works
“…472B The Pile Allowed StableLM (StabilityAI, 2023) Decoder 800B StableLM Pile Allowed LLaMA (Touvron et al., 2023) Decoder 1.4T LLaMA No Pythia (Biderman et al., 2023) Decoder 472B The Pile Allowed OPT (Zhang et al., 2022) Decoder 180B The Pile Allowed UL2 (Tay et al., 2023) Encoder-Decoder 1T…”
BackgroundWhat Makes Good In-Context Examples for GPT-$3$?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2101.06804 · Cited by 148 · see citing works
A General Language Assistant as a Laboratory for Alignment
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00861 · Cited by 146 · cited 2× · see citing works
“We integrate the Helpful, Honest, and Harmless (HHH) benchmark (Askell et al., 2021) in I NSTRUCT E VAL to assess the understanding of instructed models with respect to human values: 1.”
MethodSelf-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118 · cited 2× · see citing works
“Alpaca Data (Taori et al., 2023) 52K 52K General GPT-3 Flan Collection (Longpre et al., 2023) 15M 1836 General Human-Annotation Self-Instruct (Wang et al., 2023) 82K 52K General GPT-3 Natural Instructions (Mishra et al., 2022) 620K 61 General Human-Annotation Super-Natural Instructions (Mishra et…”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.12017 · Cited by 81 · see citing works
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67 · cited 9× · see citing works
“Model Architecture Training Tokens Data Source Commercial? GPT-NeoX Black et al. [2022] Decoder 472B The Pile Allowed StableLM StabilityAI [2023] Decoder 800B StableLM Pile Allowed LLaMA Touvron et al.”
BackgroundInfluentialChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.09261 · Cited by 44 · cited 2× · see citing works
“Complex Instructions BIG-Bench Hard (BBH) is a subset of 23 challenging tasks from the BIG-Bench benchmark (Srivastava et al., 2022), which focuses on tasks believed to be beyond the capabilities of current language models (Suzgun et al., 2022).”
BackgroundAnonymous (2018) · doi:10.5194/cp-2018-145-rc1 · Cited by 42
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
OpenAssistant Conversations - Democratizing Large Language Model Alignment
Köpf, A., Kilcher, Y., Rütte, D. V., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2064
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068 · cited 6×
“This may be explained by the multiple-choice question format which is easy to grasp and hence does not require demonstrations, while some models such as OPT-IML were optimized for zero-shot settings.”
BackgroundInfluential
Flagged first, then by citations
14 of the 54 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
“, 2023c) X (Chia et al., 2023) X (Pu and Demberg, 2023) X X (Xu et al.”
Meyer, J., Jansen, T., Schiller, R., Liebenow, L. W., Steinbach, M., Horbach, A., & Fleckenstein, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100199 · Cited by 311 · see citing works
Chan, S. T. S., Lo, N. P. K., & Wong, A. M. H. (2024) · Contemporary Educational Technology · doi:10.30935/cedtech/15607 · Cited by 56 · see citing works
Large Language Model Instruction Following: A Survey of Progresses and Challenges
Lou, R., Zhang, K., & Yin, W. (2023) · Computational Linguistics · doi:10.1162/coli_a_00523 · Cited by 74 · see citing works
Chan, S., Lo, N., & Wong, A. (2024) · Cogent Education · doi:10.1080/2331186x.2024.2440182 · Cited by 43 · see citing works
“On the other hand, LLMs typically perform better at creative tasks, which may include providing feedback (Chia et al., 2023).”
RetractedUncovering the Causes of Emotions in Software Developer Communication Using Zero-shot LLMs
Imran, M. M., Chatterjee, P., & Damevski, K. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639223 · Cited by 33 · see citing works
“GPT-4 introduced a rule-based reward model (RBRM) approach on top of RLHF. • flan-alpaca [19]: This is a variation of the Alpaca [37] fine-tuned model.”
MethodZhou, D., Wang, K., Gu, J., Peng, X., Lian, D., Zhang, Y., You, Y., & Feng, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01578 · Cited by 32 · see citing works
“To evaluate the fine-tuned LLMs, we follow the benchmark proposed in [9], which consists of MMLU [23], BBH [48], DROP [17], and HumanEval [7] datasets.”
MethodBenchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A SurveyPreprint
Li, Z., Wu, X., Du, H., Nghiem, H. T., & Shi, G. (2025) · Qeios · doi:10.32388/gxr68q · Cited by 30 · see citing works
Chan, S., Lo, N., & Wong, A. (2024) · rEFLections · doi:10.61508/refl.v31i3.277514 · Cited by 29 · see citing works
“Nevertheless, LLMs tend to excel in creative tasks, which may include generating nuanced and adaptive feedback (Chia et al., 2023).”
Revisiting Automated Topic Model Evaluation with Large Language Models
Stammbach, D., Zouhar, V., Hoyle, A., Sachan, M., & Ash, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.581 · Cited by 37 · see citing works
Can Large Language Models Understand Real-World Complex Instructions?
He, Q., Zeng, J., Huang, W., Chen, L., Xiao, J., He, Q., Zhou, X., Liang, J., & Xiao, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29777 · Cited by 24 · see citing works
Yang, X., Wu, W., Feng, S., Wang, M., Wang, D., Li, Y., Sun, Q., Zhang, Y., Fu, X., & Poria, S. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103204 · Cited by 26 · see citing works
Self-citationTang, Y., Puduppully, R., Liu, Z., & Chen, N. F. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.6 · Cited by 21 · see citing works
Systematic Assessment of Factual Knowledge in Large Language Models
Luo, L., Vu, T.-T., Phung, D., & Haf, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.885 · Cited by 21 · see citing works
Building hybrid AI models in chemical engineering: A tutorial review
Chakraborty, A., Taskiran, N. P., Kottooru, R., Mann, V., & Venkatasubramanian, V. (2025) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2025.109236 · Cited by 19 · see citing works
CRAB: Assessing the Strength of Causal Relationships Between Real-world Events
Romanou, A., Montariol, S., Paul, D., Laugier, L., Aberer, K., & Bosselut, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.940 · Cited by 22 · see citing works
Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets
Moghe, N., Fazla, A., Amrhein, C., Kocmi, T., Steedman, M. J., Birch, A., Sennrich, R., & Guillou, L. (2024) · Computational Linguistics · doi:10.1162/coli_a_00537 · Cited by 15 · see citing works
Delmas, M., Wysocka, M., & Freitas, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00520 · Cited by 10 · see citing works
Knowledge-embedded large language models for emergency triage
Shen, Q., Zhang, X., Ren, H., Guo, Q., & Zhang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113431 · Cited by 8 · see citing works
Efficient Benchmarking (of Language Models)
Perlitz, Y., Bandel, E., Gera, A., Arviv, O., Ein-Dor, L., Shnarch, E., Slonim, N., Shmueli-Scheuer, M., & Choshen, L. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.139 · Cited by 7 · see citing works
Yang, D., & Malik, M. S. A. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3090 · Cited by 6 · see citing works
CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks
Ismayilzada, M., Paul, D., Montariol, S., Geva, M., & Bosselut, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.607 · Cited by 7 · see citing works
FlanEC: Exploring Flan-T5 for Post-ASR Error Correction
Quatra, M. L., Salerno, V. M., Tsao, Y., & Siniscalchi, S. M. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832257 · Cited by 3 · see citing works
“Flan-T5 has demonstrated superior performance in handling instruction-based tasks, even when compared to decoder-only LLMs of similar parameter scale, as evidenced by its results in the InstructEval benchmark [22].”
Latency Adjustable Transformer Encoder for Language Understanding
Kachuee, S., & Sharifkhani, M. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3539727 · Cited by 3 · see citing works
Exploring Advanced Methodologies in Security Evaluation for Large Language Models
Huang, J., Zhang, J., Wang, Q., Han, W., & Zhang, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-4519-7_10 · Cited by 2 · see citing works
Giving Examples Instead of Answering Questions: Introducing Legal Concept-Example Systems
Libal, T., & Smywiński-Pohl, A. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230976 · Cited by 2 · see citing works
DINGO: Towards Diverse and Fine-Grained Instruction-Following Evaluation
Gu, Z., Sun, X., Lian, F., Kang, Z., Xu, C., & Fan, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29768 · Cited by 2 · see citing works
TabVer: Tabular Fact Verification with Natural Logic
Aly, R., & Vlachos, A. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00722 · Cited by 2 · see citing works
LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning
Shi, Y., Wu, Y., Wei, J., Ran, R., Sun, C., He, S., & Yang, Y. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2026.3655172 · Cited by 3 · see citing works
“…of LoLDU across various model architectures, scales, and task types, we conduct an extensive set of experiments on tasks including instruction following [16], [17], [23], natural language understanding (NLU) [24], image classification [25], [26], [27], [28], [29], [30], and image generation [6].”
Aiyappa, R., Senthilmani, S., An, J., Kwak, H., & Ahn, Y. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3540 · Cited by 7 · see citing works
AdapSNE: Adaptive Fireworks-Optimized and Entropy-Guided Dataset Sampling for Edge DNN Training
Zhao, B., Liu, H., Yuan, Z., Zhu, L., Yang, F., Xie, L., Xia, T., Zhao, W., & Ren, P. (2026) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2026.3667183 · Cited by 1 · see citing works
“To evaluate the performance of the fine-tuned LLMs, we followed the protocol in [45] and selected InstructEval as the evaluation dataset.”
LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-TuningPreprint
Shi, Y., Wei, J., Wu, Y., Ran, R., & Sun, C. (2024) · Qeios · doi:10.32388/in8jl4 · Cited by 1 · see citing works
Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
Rao, D., & Callison-Burch, C. (2026) · arXiv · doi:10.48550/arxiv.2606.00093
Vitale, E., Conte, L., Cioce, M., Cornacchione, P., Capuano, A., Massafra, R., Panzanaro, L., Nunzio, G. D., Steduto, M., Visintini, C., et al. (2025) · Palliative & Supportive Care · doi:10.1017/s1478951525100771
iBRAHIMOĞLU, N., Demi̇rezen, M. U., & Yıldız, F. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01490-x · Cited by 1 · see citing works
Learning, memory, and belonging: how older adults use conversational AI beyond information seeking
Gómez, J. E. A., Hernández, J. I. S., Chiappe, A., & Mella-Norambuena, J. (2026) · Cogent Education · doi:10.1080/2331186x.2026.2720995
OrthCoder: Balancing Security and Utility in Secure Code Generation via Orthogonal Adaptation
Tian, M., Yuan, Y., Zhong, Z., & He, P. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837499
Untitled work
Chan, C. K. Y., Lo, N., & Wong, G. (2024)
- Authors
- Chia, Y. K., Hong, P., Bing, L., & Poria, S.
- Affiliations
- Singapore University of Technology and Design (SG)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 35-64
- arXiv
- 2306.04757
- OpenAlex
- W7126425137
- Cited by
- 36
- References
- 54
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 14 · 2023: 9
- FWCI
- 7.96
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- instruction-tuned large language models, holistic assessment, problem-solving skills, writing skills, training methods, open-source language models, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
