Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges

Attention Is All You Need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works

MizAR 60 for Mizar 50

Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works

Language Models are Few-Shot Learners

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works

Neural Machine Translation by Jointly Learning to Align and Translate

Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works

High-Resolution Image Synthesis with Latent Diffusion Models

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · see citing works

Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics

Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works

Deep Contextualized Word Representations

Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works

Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead

Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works

LLaMA: Open and Efficient Foundation Language Models

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works

Large language models encode clinical knowledge

Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works

A review on the attention mechanism of deep learning

Niu, Z., Zhong, G., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.091 · Cited by 3,379 · see citing works

HuggingFace's Transformers: State-of-the-art Natural Language Processing

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works

A Survey on Evaluation of Large Language Models

Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.03109 · Cited by 3,159 · see citing works

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works

A Survey on Evaluation of Large Language Models

Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works

On the Opportunities and Risks of Foundation Models

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works

Llama 2: Open Foundation and Fine-Tuned Chat Models

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works

PaLM: Scaling Language Modeling with Pathways

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works

LoRA: Low-Rank Adaptation of Large Language Models

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works

ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope

Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · see citing works

Parameter-efficient transfer learning for NLP

Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · see citing works

A Survey of Large Language Models

Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works

A Survey of Large Language Models

Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090

Large Language Models are Zero-Shot Reasoners

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · see citing works

Large language models are zero-shot reasoners

Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · see citing works

QLoRA: efficient finetuning of quantized LLMs

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-0441 · Cited by 1,814 · see citing works

Scaling Laws for Neural Language Models

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 2× · see citing works

Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly

Xian, Y., Lampert, C. H., Schiele, B., & Akata, Z. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2857768 · Cited by 1,678 · see citing works

Artificial hallucinations in ChatGPT: implications in scientific writing

Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · see citing works

Sparks of Artificial General Intelligence: Early experiments with GPT-4

Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · see citing works

Domain Generalization: A Survey

Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195549 · Cited by 1,315 · see citing works

The Curious Case of Neural Text Degeneration

Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works

AllenNLP: A Deep Semantic Natural Language Processing Platform

Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M. E., Schmitz, M., & Zettlemoyer, L. (2018) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,273 · see citing works

Emergent Abilities of Large Language Models

Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · see citing works

A Primer on Neural Network Models for Natural Language Processing

Goldberg, Y. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4992 · Cited by 1,259 · see citing works

Parameter-efficient fine-tuning of large-scale pre-trained language models

Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00626-4 · Cited by 1,191 · see citing works

On Faithfulness and Factuality in Abstractive Summarization

Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · see citing works

Tree of Thoughts: Deliberate Problem Solving with Large Language Models

Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158 · see citing works

An Embarrassingly Simple Approach to Zero-Shot Learning

Romera‐Paredes, B., & Torr, P. H. S. (2017) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-319-50077-5_2 · Cited by 1,113 · see citing works

Beyond accuracy: behavioral testing of NLP models with CheckList

Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890 · see citing works

LaMDA: Language Models for Dialog Applications

Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · see citing works

ChatGPT: Jack of all trades, master of none

Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758 · see citing works

Extracting Training Data from Large Language Models

Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · see citing works

Mathematical Capabilities of ChatGPT

Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P., Chevalier, A., & Berner, J. (2023) · Neural Information Processing Systems · doi:10.52202/075280-1205 · Cited by 630 · see citing works

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works

Graph of Thoughts: Solving Elaborate Problems with Large Language Models

Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29720 · Cited by 556 · see citing works

Code Llama: Open Foundation Models for Code

Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · cited 1× · see citing works

Jailbreaking Black Box Large Language Models in Twenty Queries

Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00010 · Cited by 523 · see citing works

Jailbreaking Black Box Large Language Models in Twenty Queries

Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2023) · arXiv · doi:10.48550/arxiv.2310.08419 · Cited by 522 · see citing works

QLoRA: Efficient Finetuning of Quantized LLMs

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 507 · see citing works

How is ChatGPT's behavior changing over time?

Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507 · see citing works

How Is ChatGPT’s Behavior Changing Over Time?

Chen, L., Zaharia, M., & Zou, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.5317da47 · Cited by 507 · see citing works

Holistic Evaluation of Language Models

Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · see citing works

Orca: Progressive Learning from Complex Explanation Traces of GPT-4Preprint

Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., & Awadallah, A. (2023) · arXiv.org · doi:10.48550/arxiv.2306.02707 · Cited by 466 · see citing works

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · cited 2× · see citing works

Mathematical Capabilities of ChatGPT

Frieder, S., Pinchetti, L., Chevalier, A., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., & Berner, J. (2023) · arXiv · doi:10.48550/arxiv.2301.13867 · Cited by 385 · see citing works

A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity

Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337 · cited 1× · see citing works

Language Model Tokenizers Introduce Unfairness Between LanguagesPreprint

Bibi, A., Malfa, E. L., Petrov, A., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.15425 · Cited by 303 · see citing works

Automatic Chain of Thought Prompting in Large Language Models

Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022) · arXiv · doi:10.48550/arxiv.2210.03493 · Cited by 274 · see citing works

Economics of ChatGPT: a labor market view on the occupational impact of artificial intelligence

Zarifhonarvar, A. (2023) · Journal of Electronic Business & Digital Economics · doi:10.1108/jebde-10-2023-0021 · Cited by 273 · see citing works

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262 · cited 1× · see citing works

An Empirical Study of the Non-Determinism of ChatGPT in Code Generation

Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258 · cited 1× · see citing works

Dynabench: Rethinking Benchmarking in NLP

Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254 · see citing works

A Survey on In-context Learning

Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00234 · Cited by 248 · see citing works

ART: Automatic multi-step reasoning and tool-use for large language modelsPreprint

Paranjape, B., Lundberg, S. M., Singh, S., Hajishirzi, H., Zettlemoyer, L., & Ribeiro, M. T. (2023) · arXiv.org · doi:10.48550/arxiv.2303.09014 · Cited by 239 · see citing works

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · see citing works

RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning

Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H. W., Shu, T., Song, M., Xing, E. P., & Hu, Z. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.222 · Cited by 233 · see citing works

Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and FaithfulnessPreprint

Li, B., Fang, G., Yang, Y., Wang, Q.-S., Ye, W., Zhao, W., & Zhang, S.-K. (2023) · arXiv.org · doi:10.48550/arxiv.2304.11633 · Cited by 223 · see citing works

Importance of semantic representation: dataless classification

Chang, M., Ratinov, L., Roth, D., & Srikumar, V. (2008) · National Conference on Artificial Intelligence · Cited by 222 · see citing works

Survey of vector database management systems

Pan, J. J., Wang, J., & Li, G. (2024) · The VLDB Journal · doi:10.1007/s00778-024-00864-x · Cited by 221 · see citing works

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211 · cited 1× · see citing works

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.149 · Cited by 211 · see citing works

Emergent and Predictable Memorization in Large Language ModelsPreprint

Biderman, S., Prashanth, U. S., Sutawika, L., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2304.11158 · Cited by 205 · see citing works

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

Sinha, K., Jia, R., Hupkes, D., Pineau, J., Williams, A., & Kiela, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.230 · Cited by 204 · see citing works

Quantifying Memorization Across Neural Language Models

Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022) · arXiv · doi:10.48550/arxiv.2202.07646 · Cited by 204 · see citing works

Are Emergent Abilities of Large Language Models a Mirage?

Schaeffer, R., Miranda, B., & Koyejo, S. (2023) · arXiv · doi:10.48550/arxiv.2304.15004 · Cited by 200 · see citing works

Untitled work

(2023) · doi:10.52202/075280-2425 · Cited by 200 · see citing works

Universal and Transferable Adversarial Attacks on Aligned Language Models

Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15043 · Cited by 197 · see citing works

The Internal State of an LLM Knows When It's Lying

Azaria, A., & Mitchell, T. (2023) · arXiv · doi:10.48550/arxiv.2304.13734 · Cited by 179 · see citing works

The Internal State of an LLM Knows When It’s Lying

Azaria, A., & Mitchell, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.68 · Cited by 175 · see citing works

Delivering Trustworthy AI through Formal XAI

Marques-Silva, J., & Ignatiev, A. (2022) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21499 · Cited by 171 · see citing works

Reasoning with Language Model is Planning with World Model

Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., & Hu, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.507 · Cited by 168 · see citing works

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01373 · Cited by 160 · see citing works

Champions of Revealing - the Role of Open Source Developers in Commercial Firms

Henkel, J. (2008) · Industrial and Corporate Change · doi:10.1093/icc/dtn046 · Cited by 136 · see citing works

Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language ModelsPreprint

Ding, N., Qin, Y., Yang, G. Z., et al. (2022) · Research Square · doi:10.21203/rs.3.rs-1553541/v1 · Cited by 134 · see citing works

Wikipedia and the Politics of Openness

Tkacz, N. (2014) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226192444.001.0001 · Cited by 131 · see citing works

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130 · see citing works

On the Explainability of Natural Language Processing Deep Models

Zini, J. E., & Awad, M. (2022) · ACM Computing Surveys · doi:10.1145/3529755 · Cited by 124 · see citing works

Faith and Fate: Limits of Transformers on Compositionality

Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., et al. (2023) · arXiv · Cited by 124 · see citing works

Faith and Fate: Limits of Transformers on Compositionality

Dziri, N., Lu, X., Sclar, M., Li, X. (., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Bras, R. L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3081 · Cited by 124 · cited 1× · see citing works

A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets

Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.29 · Cited by 117 · see citing works

Whose Opinions Do Language Models Reflect?

Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2303.17548 · Cited by 114 · see citing works

Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators

Liesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3604316 · Cited by 111 · see citing works

Jailbreaking ChatGPT via Prompt Engineering: An Empirical StudyPreprint

Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., Wang, K., & Liu, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13860 · Cited by 109 · see citing works

Black-Box Tuning for Language-Model-as-a-Service

Sun, T., Shao, Y., Qian, H., Huang, X., & Qiu, X. (2022) · arXiv · doi:10.48550/arxiv.2201.03514 · Cited by 100 · see citing works

On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective

Wang, J., Hu, X., Hou, W., Chen, H., Zheng, R., Wang, Y., Yang, L., Huang, H., Ye, W., Geng, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.12095 · Cited by 86 · see citing works

Is ChatGPT a Good Sentiment Analyzer? A Preliminary StudyPreprint

Wang, Z., Xie, Q., Feng, Y., Ding, Z., Yang, Z., & Xia, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.04339 · Cited by 85 · see citing works

AgentBench: Evaluating LLMs as Agents

Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.03688 · Cited by 79 · see citing works

On the Adversarial Robustness of Multi-Modal Foundation Models

Schlarmann, C., & Hein, M. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00395 · Cited by 78 · see citing works

How Language Model Hallucinations Can Snowball

Zhang, M., Press, O., Merrill, W., Liu, A., & Smith, N. A. (2023) · arXiv · doi:10.48550/arxiv.2305.13534 · Cited by 77 · see citing works

In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT

Shen, X., Chen, Z., Backes, M., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.08979 · Cited by 76 · see citing works

Open Source and Accessibility: Advantages and Limitations

Heron, M. J., Hanson, V. L., & Ricketts, I. W. (2013) · Journal of Interaction Science · doi:10.1186/2194-0827-1-2 · Cited by 74 · see citing works

Can we trust the evaluation on ChatGPT?

Aiyappa, R., An, J., Kwak, H., & Ahn, Y.-Y. (2023) · Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) · doi:10.18653/v1/2023.trustnlp-1.5 · Cited by 72 · see citing works

How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN

McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70 · see citing works

How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN

McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70 · see citing works

Talking About Large Language ModelsPreprint

Shanahan, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.03551 · Cited by 69 · see citing works

Towards Revealing the Mystery behind Chain of Thought: a Theoretical Perspective

Feng, G., Zhang, B., Gu, Y., Ye, H., He, D., & Wang, L. (2023) · arXiv · doi:10.48550/arxiv.2305.15408 · Cited by 69 · cited 1× · see citing works

Overview of Catastrophic AI Risks

Hendrycks, D. (2024) · CRC Press eBooks · doi:10.1201/9781003530336-1 · Cited by 66 · see citing works

We’re Afraid Language Models Aren’t Modeling Ambiguity

Liu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.51 · Cited by 63 · see citing works

Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models

Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D. R., Smith, N. A., & Tsvetkov, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.614 · Cited by 63 · see citing works

When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and LimitationsPreprint

Petrov, A., Torr, P., & Bibi, A. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.19698 · Cited by 59 · see citing works

Language Model Tokenizers Introduce Unfairness Between Languages

Petrov, A., Malfa, E. L., Torr, P., & Bibi, A. (2023) · neural information processing systems · doi:10.52202/075280-1608 · Cited by 58 · see citing works

Execution-Guided Neural Program Synthesis.

Chen, X., Liu, C., & Song, D. X. (2018) · International Conference on Learning Representations · Cited by 52 · see citing works

TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

Eldan, R., & Li, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.07759 · Cited by 52 · see citing works

Petals: Collaborative Inference and Fine-tuning of Large Models

Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) · doi:10.18653/v1/2023.acl-demo.54 · Cited by 52 · see citing works

FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models

Yao, D., Zhang, J., Harris, I. G., & Carlsson, M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448041 · Cited by 51 · see citing works

The False Promise of Imitating Proprietary LLMs

Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., & Song, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15717 · Cited by 48 · cited 1× · see citing works

Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark

Peng, W., Yi, J., Wu, F., Wu, S., Zhu, B. B., Lyu, L., Jiao, B., Xu, T., Sun, G., & Xie, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.423 · Cited by 45 · see citing works

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.09261 · Cited by 44 · see citing works

Sentiment Analysis Is Not Solved! Assessing and Probing Sentiment Classification

Barnes, J., Øvrelid, L., & Velldal, E. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4802 · Cited by 42 · see citing works

Abduction-Based Explanations for Machine Learning Models

Ignatiev, A., Narodytska, N., & Marques-Silva, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011511 · Cited by 40 · see citing works

Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve

McCoy, R. T., Yao, S., Friedman, D., Hardy, M., & Griffiths, T. L. (2023) · arXiv · Cited by 37 · see citing works

Detecting Pretraining Data from Large Language Models

Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2310.16789 · Cited by 37 · see citing works

On Guaranteed Optimal Robust Explanations for NLP Models

Malfa, E. L., Michelmore, R., Zbrzezny, A. M., Paoletti, N., & Kwiatkowska, M. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/366 · Cited by 37 · see citing works

The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., & Evans, O. (2023) · arXiv · doi:10.48550/arxiv.2309.12288 · Cited by 36 · cited 1× · see citing works

Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks

Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.308 · Cited by 36 · see citing works

Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks

Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.10160 · Cited by 36 · see citing works

A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challenge

Ma, L., Zhang, R., Han, Y., Yu, S., Wang, Z., Ning, Z., Zhang, J., Xu, P., Li, P., Qiao, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.11703 · Cited by 34 · see citing works

Measuring Faithfulness in Chain-of-Thought Reasoning

Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.13702 · Cited by 33 · see citing works

Untitled work

(2023) · doi:10.52202/075280-1219 · Cited by 33 · see citing works

Aligning Generative Language Models with Human Values

Liu, R., Zhang, G., Feng, X., & Vosoughi, S. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.18 · Cited by 31 · see citing works

Inverse Scaling: When Bigger Isn't Better

McKenzie, I. R., Lyzhov, A., Pieler, M., Parrish, A., Mueller, A., Prabhu, A., McLean, E., Kirtland, A., Ross, A., Liu, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.09479 · Cited by 29 · see citing works

The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification TasksPreprint

Møller, A. G., Dalsgaard, J. A., Pera, A., & Aiello, L. M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.13861 · Cited by 28 · see citing works

Post Hoc Explanations of Language Models Can Improve Language ModelsPreprint

Krishna, S., Ma, J., Slack, D., Ghandeharioun, A., Singh, S., & Lakkaraju, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3006112/v1 · Cited by 23 · see citing works

Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations

Huang, S., Mamidanna, S., Jangam, S., Zhou, Y., & Gilpin, L. H. (2023) · arXiv · doi:10.48550/arxiv.2310.11207 · Cited by 22 · see citing works

Preventing Verbatim Memorization in Language Models Gives a False Sense of PrivacyPreprint

Ippolito, D., Tramèr, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette-Choo, C. A., & Carlini, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.17546 · Cited by 20 · see citing works

Reinforced Self-Training (ReST) for Language Modeling

Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08998 · Cited by 20 · see citing works

Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models

Zhang, S. J., Florin, S., Lee, A. N., Niknafs, E., Marginean, A., Wang, A., Tyser, K., Chin, Z., Hicke, Y., Singh, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.08997 · Cited by 19 · see citing works

Are Emergent Abilities in Large Language Models just In-Context Learning?

Lu, S., Bigoulaeva, I., Sachdeva, R., Madabushi, H. T., & Gurevych, I. (2023) · arXiv · doi:10.48550/arxiv.2309.01809 · Cited by 12 · cited 1× · see citing works

Delivering Inflated Explanations

Izza, Y., Ignatiev, A., Stuckey, P. J., & Marques-Silva, J. P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29170 · Cited by 12 · see citing works

A Measure-Theoretic Characterization of Tight Language Models

Du, L., Hennigen, L. T., Pimentel, T., Meister, C., Eisner, J. M., & Cotterell, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.543 · Cited by 10 · see citing works

Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMsPreprint

Cohn, A. G., & Hernández‐Orallo, J. (2023) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · doi:10.48550/arxiv.2304.11164 · Cited by 9 · see citing works

LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework

Zhao, M., Mi, F., Wang, Y., Li, M., Jiang, X., Liu, Q., & Schuetze, H. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.51 · Cited by 8 · see citing works

Separating form and meaning: Using self-consistency to quantify task understanding across multiple sensesPreprint

Ohmer, X., Bruni, E., & Hupkes, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.11662 · Cited by 6 · see citing works

Revealing the structure of language model capabilitiesPreprint

Burnell, R., Hao, H., Conway, A. R. A., & Orallo, J. H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.10062 · Cited by 6 · see citing works

Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelsPreprint

Yu, D., Kaur, S., Gupta, A., Brown-Cohen, J., Goyal, A., & Arora, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.17567 · Cited by 6 · see citing works

The King Is Naked: On the Notion of Robustness for Natural Language Processing

Malfa, E. L., & Kwiatkowska, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21353 · Cited by 4 · see citing works

Explanatory Paradigms in Neural NetworksPreprint

AlRegib, G., & Prabhushankar, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.11838 · Cited by 3 · see citing works

Mitigating Data Scarcity for Large Language ModelsPreprint

Van, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.01806 · Cited by 3 · see citing works

Foveate, Attribute, and Rationalize: Towards Physically Safe and Trustworthy AI

Mei, A., Levy, S., & Wang, W. Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.701 · Cited by 3 · see citing works

ArK: Augmented Reality with Knowledge Interactive Emergent AbilityPreprint

Huang, Q., Park, J. S., Gupta, A., Bennett, P. N., Gong, R., Som, S., Peng, B., Mohammed, O. K., Pal, C., Choi, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.00970 · Cited by 2 · see citing works

Invalid Logic, Equivalent Gains: The Bizarreness of Reasoning in Language Model PromptingPreprint

Schaeffer, R., Pistunova, K., Khanna, S., Consul, S., & Koyejo, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.10573 · Cited by 2 · see citing works

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805

GPT-4 Technical Report

OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100

The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool

Resnik, D. B., & Hosseini, M. (2025) · AI and Ethics · doi:10.1007/s43681-024-00493-8 · Cited by 228 · see citing works

ML-Asset Management: Curation, Discovery, and Utilization

Wang, M., Duan, M., Huang, Y., Li, C., He, B., & Wu, Y. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3750601.3750701 · Cited by 2 · see citing works

Exploring model-as-a-service for generative ai on cloud platforms

Pitkar, H., Bauskar, S. R., Parmar, D., & Saran, H. K. (2024) · Review of Computer Engineering Research · doi:10.18488/76.v11i4.4017 · Cited by 2 · see citing works

From natural language to executable filter code: LLM-Assisted behavioural customisation in trigger–Action Platforms

Cimino, G., Santis, L. D., & Deufemia, V. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2711012 · Cited by 1 · see citing works

Staying Young at the Edge: A Software Aging Perspective for Foundation Models as a Service

Picano, B., & Fantacci, R. (2026) · Computers · doi:10.3390/computers15030158

Reporting Transparency for AI in Applied Linguistics: An L2 Writing Audit and Proposed StandardPreprint

Carlo, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7079541

Prism: Proactive Workload-Aware Optimization for Hybrid-Service in LMaaS Systems

Li, Y., Qiu, C., Huang, S., Zhang, T., Zhao, Y., & Wang, X. (2026) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/2575-8411.2026.00046

PreServe: Intelligent Management for LMaaS Systems via Hierarchical Prediction

Jiang, Z., Huang, Y., Yu, G., Huang, J., Gu, J., & Lyu, M. R.-T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773173

ISERA-KGC: Enhancing Knowledge Graph Completion with Interactive Semantic Enhancement and Representation Alignment

ZhePing, Y., Zhang, T., & Zhang, G. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729167

Lessons from the Trenches on Reproducible Evaluation of Language Models

Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9 · see citing works

Holistic Evaluation of Language Models

Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255 · see citing works

Evaluation of Language Model-Based Services on the CAISE Platform: Methods, Metrics, and Applications

Majkutewicz, J., Olewniczak, S., & Gierszewski, M. (2026) · DOAJ (DOAJ: Directory of Open Access Journals) · doi:10.34808/tq2026/30.1/c

The ARRT of Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges.Preprint

Malfa, E. L., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Cohn, A. G., Shadbolt, N., & Wooldridge, M. (2023) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York)

A fine-grained evaluation framework for language models: Combining pointwise grading and pairwise comparison

Li, Y., & Sun, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104270 · Cited by 2 · see citing works