CoChat
Sign Up
PreprintCited by 196Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Challenges and Applications of Large Language Models

Open access
Kaddour, Jean, Harris, Joshua, Mozes, Maximilian, Bradley, Herbie, Raileanu, Roberta, McHardy, Robert·2023·arXiv
DOI 10.48550/arxiv.2307.10169arXiv 2307.10169
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) went from non-existent to ubiquitous in the machine learning discourse within a few years. Due to the fast pace of the field, it is difficult to identify the remaining challenges and already fruitful application areas. In this paper, we aim to establish a systematic set of open problems and application successes so that ML researchers can comprehend the field's current state more quickly and become productive.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.10169

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Large language models have shown remarkable versatility in various generative tasks such as writing, drawing, and question answering, significantly reducing the time required for many routine tasks.

    “On one hand, large language models (LLMs) have shown remarkable versatility in various generative tasks such as writing, drawing, and question answering, significantly reducing the time required for many routine tasks.”

    From LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024) · cites this paper as Kaddour et al., 2023

    Section 3.4 (Creative Work; page unavailable) describes LLM applications in writing and in guiding image-generation models, while §3.7.1 describes medical question answering. The visual examples use LLMs to guide image generation rather than draw independently. The passages do not establish that LLMs significantly reduce the time needed for many routine tasks. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 196

Research works printed but not linked: 551

  1. A. Abbas, K. Tirumala, D. Simig, S. Ganguli and A. S. Morcos. 2023. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540. · arXiv
  2. J. D. Abernethy, A. Agarwal, T. V. Marinov and M. K. War- muth. 2023. A mechanism for sample-efficient in-context learning for sparse retrieval tasks. ArXiv, abs/2305.17040. · arXiv
  3. D. Adiwardana, M.-T. Luong, D. R. So, J. Hall, N. Fiedel, R. Thoppilan, Z. Yang, A. Kulshreshtha et al. 2020. To- wards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977. · arXiv
  4. R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville and M. Bellemare. 2021. Deep Reinforcement Learning at the Edge of the Statistical Precipice. In Advances in Neural Information Processing Systems, volume 34, pages 29304-29320.
  5. M. Agrawal, S. Hegselmann, H. Lang, Y. Kim and D. Son- tag. 2022. Large language models are zero-shot clinical information extractors. arXiv preprint arXiv:2205.12689. · arXiv
  6. P. Agrawal, C. Alberti, F. Huot, J. Maynez, J. Ma, S. Ruder, K. Ganchev, D. Das et al. 2022. Qameleon: Multilingual qa with only 5 examples. arXiv preprint arXiv:2211.08264. · arXiv
  7. G. Aher, R. I. Arriaga and A. T. Kalai. 2022. Using large language models to simulate multiple humans. arXiv preprint arXiv:2208.10264. · arXiv
  8. O. Ahia, S. Kumar, H. Gonen, J. Kasai, D. R. Mortensen, N. A. Smith and Y. Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. arXiv preprint arXiv:2305.13707. · arXiv
  9. M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan et al. 2022. Do as i can, not as i say: Grounding language in robotic affor- dances. arXiv preprint arXiv:2204.01691. · arXiv
  10. J. Ainslie, T. Lei, M. de Jong, S. Ontañón, S. Brahma, Y. Zemlyanskiy, D. Uthus, M. Guo et al. 2023. Colt5: Faster long-range transformers with conditional computa- tion. arXiv preprint arXiv:2303.09752. · arXiv
  11. E. Akyürek, D. Schuurmans, J. Andreas, T. Ma and D. Zhou. 2023. What learning algorithm is in-context learn- ing? investigations with linear models. In The Eleventh International Conference on Learning Representations.
  12. L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra et al. 2023. Santa- coder: don't reach for the stars! · DOI · Paper page
  13. J. Andreas. 2022. Language models as agent models. · DOI · Paper page
  14. C. Anil, Y. Wu, A. Andreassen, A. Lewkowycz, V. Misra, V. Ramasesh, A. Slone, G. Gur-Ari et al. 2022. Explor- ing Length Generalization in Large Language Models. ArXiv:2207.04901 [cs]. · arXiv
  15. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa et al. 2023. Palm 2 techni- cal report. arXiv preprint arXiv:2305.10403. · arXiv
  16. F. Antaki, S. Touma, D. Milad, J. El-Khoury and R. Duval. 2023. Evaluating the performance of chatgpt in ophthal- mology: An analysis of its successes and shortcomings. medRxiv. · DOI · Paper page
  17. L. P. Argyle, E. C. Busby, N. Fulda, J. Gubler, C. Rytting and D. Wingate. 2022. Out of one, many: Using lan- guage models to simulate human samples. arXiv preprint arXiv:2209.06899. · arXiv
  18. V. Aribandi, Y. Tay, T. Schuster, J. Rao, H. S. Zheng, S. V. Mehta, H. Zhuang, V. Q. Tran et al. 2022. Ext5: Towards extreme multi-task scaling for transfer learning. In International Conference on Learning Representations.
  19. S. Arora, A. Narayan, M. F. Chen, L. Orr, N. Guha, K. Bhatia, I. Chami, F. Sala et al. 2022. Ask me anything: A simple strategy for prompting language models. · DOI
  20. A. Asai, T. Schick, P. Lewis, X. Chen, G. Izacard, S. Riedel, H. Hajishirzi and W.-t. Yih. 2022. Task-aware retrieval with instructions. · DOI
  21. N. Asher, S. Bhar, A. Chaturvedi, J. Hunter and S. Paul. 2023. Limits for Learning with Language Models. ArXiv:2306.12213 [cs]. · arXiv
  22. M. C. Ashton and K. Lee. 2009. The hexaco-60: A short measure of the major dimensions of personality. Journal of personality assessment, 91(4):340-345.
  23. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. · arXiv
  24. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie et al. 2022. Constitu- tional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. · arXiv
  25. P. Bajaj, C. Xiong, G. Ke, X. Liu, D. He, S. Tiwary, T.-Y. Liu, P. Bennett et al. 2022. Metro: Efficient denoising pre- training of large scale autoencoding language models with model generated signals. arXiv preprint arXiv:2204.06644. · arXiv
  26. A. Bakhtin, S. Gross, M. Ott, Y. Deng, M. Ranzato and A. Szlam. 2019. Real or Fake? Learning to Discriminate Machine from Human Generated Text. ArXiv:1906.03351 [cs, stat]. · arXiv
  27. R. Balestriero, J. Pesenti and Y. LeCun. 2021. Learning in high dimension always amounts to extrapolation. arXiv preprint arXiv:2110.09485. · arXiv
  28. J. Bandy and N. Vincent. 2021. Addressing "documenta- tion debt" in machine learning research: A retrospective datasheet for bookcorpus. · DOI · Paper page
  29. P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard, H. Lim et al. 2022. Pathways: Asynchronous distributed dataflow for ml. Proceedings of Machine Learning and Systems, 4:430-449.
  30. M. Bavarian, H. Jun, N. Tezak, J. Schulman, C. McLeavey, J. Tworek and M. Chen. 2022. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255. · arXiv
  31. E. Ben Zaken, Y. Goldberg and S. Ravfogel. 2022. Bit- Fit: Simple parameter-efficient fine-tuning for transformer- based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland. Association for Computational Linguistics. · DOI · Paper page
  32. S. Biderman, K. Bicheno and L. Gao. 2022. Datasheet for the pile. arXiv preprint arXiv:2201.07311. · arXiv
  33. S. Biderman, U. S. Prashanth, L. Sutawika, H. Schoelkopf, Q. Anthony, S. Purohit and E. Raff. 2023. Emergent and Predictable Memorization in Large Language Mod- els. ArXiv:2304.11158 [cs]. · arXiv
  34. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 2397-2430. PMLR.
  35. A. Birhane, V. U. Prabhu and E. Kahembwe. 2021. Mul- timodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963. · arXiv
  36. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy et al. 2022. Gpt-neox-20b: An open-source autoregressive language model. · DOI · Paper page
  37. A. Blair-Stanek, N. Holzenberger and B. Van Durme. 2023. Can gpt-3 perform statutory reasoning? arXiv preprint arXiv:2302.06100. · arXiv
  38. J. Bommarito, M. Bommarito, D. M. Katz and J. Katz. 2023. Gpt as knowledge worker: A zero-shot evaluation of (ai) cpa capabilities. arXiv preprint arXiv:2301.04408. · arXiv
  39. M. Bommarito II and D. M. Katz. 2022. Gpt takes the bar exam. arXiv preprint arXiv:2212.14402. · arXiv
  40. L. Bonifacio, H. Abonizio, M. Fadaee and R. Nogueira. 2022. Inpars: Unsupervised dataset generation for infor- mation retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 2387-2392, New York, NY, USA. Association for Computing Machinery. · DOI
  41. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Ruther- ford, K. Millican, G. v. d. Driessche, J.-B. Lespiau et al. 2021. Improving language models by retrieving from tril- lions of tokens. arXiv preprint arXiv:2112.04426. · arXiv
  42. A. Borji. 2023. A Categorical Archive of ChatGPT Fail- ures. ArXiv:2302.03494 [cs]. · arXiv
  43. A. Borzunov, D. Baranchuk, T. Dettmers, M. Ryabinin, Y. Belkada, A. Chumachenko, P. Samygin and C. Raffel. 2022. Petals: Collaborative inference and fine-tuning of large models. arXiv preprint arXiv:2209.01188. · arXiv
  44. L. Bos, C. Schemer, N. Corbu, M. Hameleers, I. An- dreadis, A. Schulz, D. Schmuck, C. Reinemann et al. 2020. The effects of populism as a social identity frame on persua- sion and mobilisation: Evidence from a 15-country experi- ment. European Journal of Political Research, 59(1):3-24.
  45. D. Britz, M. Y. Guan and M.-T. Luong. 2017. Efficient attention using a fixed-size memory representation. arXiv preprint arXiv:1707.00110. · arXiv
  46. A. Z. Broder, M. Charikar, A. M. Frieze and M. Mitzen- macher. 1998. Min-wise independent permutations. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pages 327-336.
  47. G. Brown, M. Bun, V. Feldman, A. Smith and K. Talwar. 2021. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd annual ACM SIGACT symposium on theory of computing, pages 123-132.
  48. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
  49. M. Brundage, S. Avin, J. Clark, H. Toner, P. Eckersley, B. Garfinkel, A. Dafoe, P. Scharre et al. 2018. The mali- cious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv:1802.07228. · arXiv
  50. A. Calderwood, N. Wardrip-Fruin and M. Mateas. 2022. Spinning coherent interactive fiction through foundation model prompts. International Conference of Computation and Creativity.
  51. N. Carlini, M. Jagielski, C. A. Choquette-Choo, D. Paleka, W. Pearce, H. Anderson, A. Terzis, K. Thomas et al. 2023. Poisoning Web-Scale Training Datasets is Practical. ArXiv:2302.10149 [cs]. · arXiv
  52. N. Carlini, C. Liu, Ú. Erlingsson, J. Kos and D. Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium, volume 267.
  53. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert- Voss, K. Lee, A. Roberts, T. Brown et al. 2020. Extracting training data from large language models. · DOI · Paper page
  54. S. Casper, J. Lin, J. Kwon, G. Culp and D. Hadfield- Menell. 2023. Explore, establish, exploit: Red team- ing language models from scratch. arXiv preprint arXiv:2306.09442. · arXiv
  55. T. Chakrabarty, V. Padmakumar and H. He. 2022. Help me write a poem: Instruction tuning as a vehicle for collab- orative poetry writing. arXiv preprint arXiv:2210.13669. · arXiv
  56. I. Chalkidis, I. Androutsopoulos and N. Aletras. 2019. Neural legal judgment prediction in english. arXiv preprint arXiv:1906.02059. · arXiv
  57. I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Ale- tras and I. Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. arXiv preprint arXiv:2010.02559. · arXiv
  58. Y. Chang, X. Wang, J. Wang, Y. Wu, K. Zhu, H. Chen, L. Yang, X. Yi et al. 2023. A Survey on Evaluation of Large Language Models. ArXiv:2307.03109 [cs]. · DOI · Paper page
  59. B. Chen, X. Cheng, L. ao Gengyang, S. Li, X. Zeng, B. Wang, G. Jing, C. Liu et al. 2023. xtrimopglm: Uni- fied 100b-scale pre-trained transformer for deciphering the language of protein. bioRxiv. · DOI
  60. C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre and J. Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. · arXiv
  61. L. Chen, M. Zaharia and J. Zou. 2023. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. ArXiv:2305.05176 [cs]. · DOI · Paper page
  62. L. Chen, M. Zaharia and J. Zou. 2023. How is ChatGPT's behavior changing over time? ArXiv:2307.09009 [cs]. · arXiv
  63. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda et al. 2021. Evaluating large language models trained on code. · DOI · Paper page
  64. M. Chen, A. Papangelis, C. Tao, S. Kim, A. Rosenbaum, Y. Liu, Z. Yu and D. Hakkani-Tur. 2023. Places: Prompting language models for social conversation synthesis. arXiv preprint arXiv:2302.03269. · arXiv
  65. X. Chen, M. Lin, N. Schärli and D. Zhou. 2023. Teach- ing large language models to self-debug. arXiv preprint arXiv:2304.05128. · arXiv
  66. D. Choe, R. Al-Rfou, M. Guo, H. Lee and N. Constant. 2019. Bridging the Gap for Tokenizer-Free Language Mod- els. ArXiv:1908.10322 [cs]. · arXiv
  67. J. H. Choi, K. E. Hickman, A. Monahan and D. Schwarcz. 2023. Chatgpt goes to law school. Available at SSRN.
  68. K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis et al. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794. · arXiv
  69. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. · arXiv
  70. P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg and D. Amodei. 2017. Deep reinforcement learning from human preferences. · DOI · Paper page
  71. K. Christianson, A. Hollingworth, J. F. Halliwell and F. Ferreira. 2001. Thematic roles assigned along the garden path linger. Cognitive psychology, 42(4):368-407.
  72. H. W. Chung, X. Garcia, A. Roberts, Y. Tay, O. Firat, S. Narang and N. Constant. 2023. Unimax: Fairer and more effective language sampling for large-scale multilin- gual pretraining. In The Eleventh International Conference on Learning Representations.
  73. H. W. Chung, D. Garrette, K. C. Tan and J. Riesa. 2020. Improving multilingual models with language-clustered vo- cabularies. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 4536-4546, Online. Association for Computational Linguistics. · DOI
  74. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang et al. 2022. Scaling instruction- finetuned language models. · DOI · Paper page
  75. J. H. Clark, D. Garrette, I. Turc and J. Wieting. 2022. Ca- nine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73-91. · DOI
  76. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek et al. 2021. Training verifiers to solve math word problems. · DOI · Paper page
  77. D. Cohen, M. Ryu, Y. Chow, O. Keller, I. Greenberg, A. Hassidim, M. Fink, Y. Matias et al. 2022. Dynamic plan- ning in open-ended dialogue using reinforcement learning. arXiv preprint arXiv:2208.02294. · arXiv
  78. R. Cohen, M. Hamri, M. Geva and A. Globerson. 2023. LM vs LM: Detecting Factual Errors via Cross Examina- tion. ArXiv:2305.13281 [cs]. · arXiv
  79. A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimer- sheim and A. Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. arXiv preprint arXiv:2304.14997. · arXiv
  80. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott et al. 2020. Unsu- pervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics. · DOI · Paper page
  81. A. B. Cyphert. 2021. A human being wrote this law review article: Gpt-3 and the practice of law. UC Davis L. Rev., 55:401.
  82. D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang and F. Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493-8502, Dublin, Ireland. Association for Com- putational Linguistics. · DOI · Paper page
  83. D. Dai, Y. Sun, L. Dong, Y. Hao, Z. Sui and F. Wei. 2022. Why can gpt learn in-context? language models se- cretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559. · arXiv
  84. Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le and R. Salakhutdinov. 2019. Transformer-XL: Attentive lan- guage models beyond a fixed-length context. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
  85. H. Dalla-Torre, L. Gonzalez, J. Mendoza Revilla, N. Lopez Carranza, A. Henryk Grywaczewski, F. Oteri, C. Dallago, E. Trop et al. 2023. The nucleotide trans- former: Building and evaluating robust foundation models for human genomics. bioRxiv, pages 2023-01.
  86. T. Dao, D. Y. Fu, S. Ermon, A. Rudra and C. Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. arXiv preprint arXiv:2205.14135. · arXiv
  87. T. Dao, D. Y. Fu, K. K. Saab, A. W. Thomas, A. Rudra and C. Ré. 2023. Hungry Hungry Hippos: Towards Language Modeling with State Space Models. ArXiv:2212.14052 [cs]. · arXiv
  88. J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet et al. 2022. Robust deep learning–based protein sequence de- sign using proteinmpnn. Science, 378(6615):49-56. · DOI · Paper page
  89. N. De Cao, W. Aziz and I. Titov. 2021. Editing fac- tual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6491-6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
  90. M. Dehghani, A. Arnab, L. Beyer, A. Vaswani and Y. Tay. 2022. The Efficiency Misnomer. ArXiv:2110.12894 [cs, stat]. · arXiv
  91. M. Dehghani, Y. Tay, A. A. Gritsenko, Z. Zhao, N. Houlsby, F. Diaz, D. Metzler and O. Vinyals. 2021. The benchmark lottery. arXiv preprint arXiv:2107.07002. · arXiv
  92. L. Del Corro, A. Del Giorno, S. Agarwal, B. Yu, A. Awadallah and S. Mukherjee. 2023. SkipDecode: Au- toregressive Skip Decoding with Batching and Caching for Efficient LLM Inference. ArXiv:2307.02628 [cs]. · DOI
  93. A. Deshpande, V. Murahari, T. Rajpurohit, A. Kalyan and K. Narasimhan. 2023. Toxicity in chatgpt: Analyz- ing persona-assigned language models. arXiv preprint arXiv:2304.05335. · arXiv
  94. T. Dettmers, M. Lewis, Y. Belkada and L. Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transform- ers at scale. · DOI · Paper page
  95. T. Dettmers, A. Pagnoni, A. Holtzman and L. Zettle- moyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. ArXiv:2305.14314 [cs]. · arXiv
  96. T. Dettmers, R. Svirschevski, V. Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler et al. 2023. Spqr: A sparse-quantized represen- tation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078. · arXiv
  97. J. Devlin, M.-W. Chang, K. Lee and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171- 4186, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI · Paper page
  98. S. Diao, X. Li, Y. Lin, Z. Huang and T. Zhang. 2022. Black-box prompt learning for pre-trained language mod- els. arXiv preprint arXiv:2201.08531. · arXiv
  99. B. Ding, C. Qin, L. Liu, L. Bing, S. Joty and B. Li. 2022. Is gpt-3 a good data annotator? arXiv preprint arXiv:2212.10450. · arXiv
  100. J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Il- harco, D. Groeneveld, M. Mitchell and M. Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758. · arXiv
  101. R. Dominguez-Olmedo, M. Hardt and C. Mendler- Dünner. 2023. Questioning the survey responses of large language models. arXiv preprint arXiv:2306.07951. · arXiv
  102. Q. Dong, D. Dai, Y. Song, J. Xu, Z. Sui and L. Li. 2022. Calibrating factual knowledge in pretrained language mod- els. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5937-5947, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
  103. D. R. Dowty, R. Wall and S. Peters. 2012. Introduction to Montague semantics, volume 11. Springer Science & Business Media.
  104. D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378. · arXiv
  105. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In Interna- tional Conference on Machine Learning, pages 5547-5569. PMLR.
  106. Y. Du, S. Li, A. Torralba, J. B. Tenenbaum and I. Mordatch. 2023. Improving Factuality and Reason- ing in Language Models through Multiagent Debate. ArXiv:2305.14325 [cs]. · arXiv
  107. Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang and J. Tang. 2022. GLM: General language model pretrain- ing with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 320-335, Dublin, Ireland. Association for Computational Linguistics. · DOI
  108. A. Dunn, J. Dagdelen, N. Walker, S. Lee, A. S. Rosen, G. Ceder, K. Persson and A. Jain. 2022. Struc- tured information extraction from complex scientific text with fine-tuned large language models. arXiv preprint arXiv:2212.05238. · arXiv
  109. D. Duong and B. D. Solomon. 2023. Analysis of large- language model versus human performance for genetics questions. European Journal of Human Genetics, pages 1-3.
  110. N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y. Lin, P. West, C. Bhagavatula et al. 2023. Faith and Fate: Limits of Transformers on Compositionality. ArXiv:2305.18654 [cs]. · arXiv
  111. N. Dziri, A. Madotto, O. Zaiane and A. J. Bose. 2021. Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding. ArXiv:2104.08455 [cs]. · arXiv
  112. E.-M. El-Mhamdi, S. Farhadkhani, R. Guerraoui, N. Gupta, L.-N. Hoang, R. Pinot, S. Rouault and J. Stephan. 2023. On the Impossible Safety of Large AI Models. ArXiv:2209.15259 [cs]. · arXiv
  113. A. Elnaggar, M. Heinzinger, C. Dallago, G. Rihawi, Y. Wang, L. Jones, T. Gibbs, T. Feher et al. 2020. Prottrans: towards cracking the language of life's code through self- supervised deep learning and high performance computing. arXiv preprint arXiv:2007.06225. · arXiv
  114. F. Faal, K. Schmitt and J. Y. Yu. 2023. Reward model- ing for mitigating toxicity in transformer-based language models. Applied Intelligence, 53(7):8421-8435.
  115. A. Fan, C. Gardent, C. Braud and A. Bordes. 2021. Aug- menting transformers with KNN-based composite memory for dialog. Transactions of the Association for Computa- tional Linguistics, 9:82-99. · DOI
  116. A. Fan, E. Grave and A. Joulin. 2020. Reducing trans- former depth on demand with structured dropout. In Inter- national Conference on Learning Representations.
  117. M. Fathi, J. Pilault, P.-L. Bacon, C. Pal, O. Fi- rat and R. Goroshin. 2023. Block-State Transformer. ArXiv:2306.09539 [cs]. · arXiv
  118. W. Fedus, B. Zoph and N. Shazeer. 2021. Switch trans- formers: Scaling to trillion parameter models with simple and efficient sparsity. · DOI · Paper page
  119. V. Feldman. 2020. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954-959.
  120. S. Feng, C. Y. Park, Y. Liu and Y. Tsvetkov. 2023. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models. ArXiv:2305.08283 [cs]. · arXiv
  121. W. Feng, W. Zhu, T.-j. Fu, V. Jampani, A. Akula, X. He, S. Basu, X. E. Wang et al. 2023. LayoutGPT: Compo- sitional Visual Planning and Generation with Large Lan- guage Models. ArXiv:2305.15393 [cs]. · DOI · Paper page
  122. E. Ferrara. 2023. Should chatgpt be biased? challenges and risks of bias in large language models. arXiv preprint arXiv:2304.03738. · arXiv
  123. A. Ficek, F. Liu and N. Collier. 2022. How to tackle an emerging topic? combining strong and weak labels for covid news NER. In Proceedings of the 2nd Confer- ence of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 488-496, Online only. Association for Computational Linguistics.
  124. E. Frantar and D. Alistarh. 2023. Massive language mod- els can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774. · arXiv
  125. E. Frantar, S. Ashkboos, T. Hoefler and D. Alis- tarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. · arXiv
  126. D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih et al. 2022. Incoder: A genera- tive model for code infilling and synthesis. · DOI · Paper page
  127. A. Frömmgen and L. Kharatyan. 2023. Resolv- ing code review comments with ml. Available from: https://ai.googleblog.com/2023/05/ resolving-code-review-comments-with-ml. html. Accessed: 26/06/2023.
  128. J. Fu, S.-K. Ng, Z. Jiang and P. Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. · arXiv
  129. T. Fujii, K. Shibata, A. Yamaguchi, T. Morishita and Y. Sogawa. 2023. How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in japanese. arXiv preprint arXiv:2306.09572. · arXiv
  130. I. Gabriel. 2020. Artificial intelligence, values, and align- ment. Minds and machines, 30(3):411-437.
  131. S. Gadgil, A. R. Tadipatri, A. Agrawal, A. Narayanan and N. Goyal. 2022. Towards automating formalisation of theorem statements using large language models. 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Workshop on MATH-AI.
  132. T. Gale, D. Narayanan, C. Young and M. Zaharia. 2022. Megablocks: Efficient sparse training with mixture-of- experts. arXiv preprint arXiv:2211.15841. · arXiv
  133. T. Gale, M. Zaharia, C. Young and E. Elsen. 2020. Sparse GPU Kernels for Deep Learning. ArXiv:2006.10901 [cs, stat]. · arXiv
  134. K. Gandhi, J.-P. Fränken, T. Gerstenbrg and N. D. Goodman. 2023. Understanding social reasoning in lan- guage models with language models. arXiv preprint arXiv:2306.15448. · arXiv
  135. D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858. · arXiv
  136. J. Gao, X. Ding, B. Qin and T. Liu. 2023. Is chatgpt a good causal reasoner? a comprehensive evaluation. arXiv preprint arXiv:2305.07375. · arXiv
  137. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. · arXiv
  138. S. Gehman, S. Gururangan, M. Sap, Y. Choi and N. A. Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462. · arXiv
  139. S. Gehrmann, H. Strobelt and A. M. Rush. 2019. GLTR: Statistical Detection and Visualization of Generated Text. ArXiv:1906.04043 [cs]. · arXiv
  140. R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge and F. A. Wichmann. 2020. Short- cut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673.
  141. A. Glaese, N. McAleese, M. Trębacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger et al. 2022. Improving alignment of dialogue agents via targeted human judgements. · DOI · Paper page
  142. D. Goldberg. 1991. What every computer scientist should know about floating-point arithmetic. ACM Com- puting Surveys, 23(1):5-48. · DOI
  143. A. N. Gomez, O. Key, K. Perlin, S. Gou, N. Frosst, J. Dean and Y. Gal. 2022. Interlocking backpropagation: Improving depthwise model-parallelism. The Journal of Machine Learning Research, 23(1):7714-7741.
  144. L. Gong, D. He, Z. Li, T. Qin, L. Wang and T. Liu. 2019. Efficient training of BERT by progressively stacking. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2337-2346. PMLR.
  145. Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, N. Duan and W. Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738. · arXiv
  146. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz and M. Fritz. 2023. More than you've asked for: A comprehensive analysis of novel prompt injection threats to application-integrated large language models. arXiv preprint arXiv:2302.12173. · arXiv
  147. L. D. Griffin, B. Kleinberg, M. Mozes, K. T. Mai, M. Vau, M. Caldwell and A. Marvor-Parker. 2023. Susceptibil- ity to influence of large language models. arXiv preprint arXiv:2303.06074. · arXiv
  148. Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao et al. 2021. Domain-specific language model pretraining for biomedical natural language pro- cessing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1-23.
  149. Y. Gu, X. Han, Z. Liu and M. Huang. 2022. PPT: Pre-trained prompt tuning for few-shot learning. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8410-8423, Dublin, Ireland. Association for Com- putational Linguistics. · DOI
  150. A. Gudibande, E. Wallace, C. Snell, X. Geng, H. Liu, P. Abbeel, S. Levine and D. Song. 2023. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15717. · arXiv
  151. J. P. Guilford. 1967. Creativity: Yesterday, today and tomorrow. The Journal of Creative Behavior, 1(1):3-14.
  152. M. Guo, J. Ainslie, D. Uthus, S. Ontanon, J. Ni, Y.-H. Sung and Y. Yang. 2022. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Associ- ation for Computational Linguistics: NAACL 2022, pages 724-736, Seattle, United States. Association for Computa- tional Linguistics. · DOI · Paper page
  153. A. Gupta. 2023. Probing Quantifier Comprehension in Large Language Models. ArXiv:2306.07384 [cs]. · arXiv
  154. K. Guu, K. Lee, Z. Tung, P. Pasupat and M. Chang. 2020. Retrieval augmented language model pre-training. In International Conference on Machine Learning, pages 3929-3938. PMLR.
  155. M. Hahn and N. Goyal. 2023. A theory of emergent in-context learning as implicit structure induction. ArXiv, abs/2303.07971. · arXiv
  156. S. Hamilton. 2023. Blind judgement: Agent-based supreme court modelling with gpt. arXiv preprint arXiv:2301.05327. · arXiv
  157. C. Han, Z. Wang, H. Zhao and H. Ji. 2023. In-context learning of large language models explained as kernel re- gression. ArXiv, abs/2305.12766. · arXiv
  158. T. Hartvigsen, S. Sankaranarayanan, H. Palangi, Y. Kim and M. Ghassemi. 2022. Aging with grace: Lifelong model editing with discrete key-value adaptors. arXiv preprint arXiv:2211.11031. · arXiv
  159. A. Haviv, O. Ram, O. Press, P. Izsak and O. Levy. 2022. Transformer language models without positional encodings still learn positional information. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 1382-1390, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  160. J. Hazell. 2023. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972. · arXiv
  161. E. L. Henderson, S. J. Westwood and D. J. Simons. 2022. A reproducible systematic map of research on the illusory truth effect. Psychonomic Bulletin & Review, pages 1-24.
  162. P. Henderson, M. S. Krass, L. Zheng, N. Guha, C. D. Manning, D. Jurafsky and D. E. Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256GB open-source legal dataset. In Thirty-sixth Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track.
  163. D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song and J. Steinhardt. 2020. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275. · arXiv
  164. D. Hendrycks, N. Carlini, J. Schulman and J. Steinhardt. 2021. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916. · arXiv
  165. D. Hendrycks and M. Mazeika. 2022. X-risk analysis for ai research. arXiv preprint arXiv:2206.05862. · arXiv
  166. D. Hernandez, T. Brown, T. Conerly, N. DasSarma, D. Drain, S. El-Showk, N. Elhage, Z. Hatfield-Dodds et al. 2022. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10487. · arXiv
  167. J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali et al. 2017. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409. · arXiv
  168. P. Hingston and M. Preuss. 2011. Red teaming with coevolution. In 2011 IEEE Congress of Evolutionary Com- putation (CEC), pages 1155-1163. IEEE.
  169. J. Hoelscher-Obermaier, J. Persson, E. Kran, I. Kon- stas and F. Barez. 2023. Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark. ArXiv:2305.17553 [cs]. · DOI
  170. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de las Casas, L. A. Hendricks et al. 2022. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems.
  171. A. Holtzman, J. Buys, L. Du, M. Forbes and Y. Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
  172. N. Holzenberger, A. Blair-Stanek and B. Van Durme. 2020. A dataset for statutory reasoning in tax law entailment and question answering. arXiv preprint arXiv:2005.05257. · arXiv
  173. O. Honovich, T. Scialom, O. Levy and T. Schick. 2022. Unnatural instructions: Tuning language models with (al- most) no human labor. arXiv preprint arXiv:2212.09689. · arXiv
  174. S. Hooker. 2021. The hardware lottery. Communications of the ACM, 64(12):58-65.
  175. J. J. Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? arXiv preprint arXiv:2301.07543. · arXiv
  176. M. Horton, S. Mehta, A. Farhadi and M. Rastegari. 2023. Bytes Are All You Need: Transformers Operating Directly On File Bytes. ArXiv:2306.00238 [cs]. · arXiv
  177. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan and S. Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR.
  178. D. Houser and K. McCabe. 2014. Experimental eco- nomics and experimental game theory. In Neuroeconomics, pages 19-34. Elsevier.
  179. J. Howard and S. Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computa- tional Linguistics. · DOI · Paper page
  180. Z. Hu, Y. Lan, L. Wang, W. Xu, E.-P. Lim, R. K.-W. Lee, L. Bing and S. Poria. 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933. · arXiv
  181. W. Hua, Z. Dai, H. Liu and Q. Le. 2022. Transformer Quality in Linear Time. In Proceedings of the 39th Interna- tional Conference on Machine Learning, pages 9099-9117. PMLR. ISSN: 2640-3498.
  182. C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, I. Simon, C. Hawthorne, N. Shazeer, A. M. Dai, M. D. Hoffman et al. 2019. Music transformer. In International Conference on Learning Representations.
  183. J. Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu and J. Han. 2022. Large language models can self-improve. · DOI · Paper page
  184. J. Huang and K. C.-C. Chang. 2023. Towards Reasoning in Large Language Models: A Survey. ArXiv:2212.10403 [cs]. · arXiv
  185. W. Huang, P. Abbeel, D. Pathak and I. Mordatch. 2022. Language models as zero-shot planners: Extracting action- able knowledge for embodied agents. In International Con- ference on Machine Learning, pages 9118-9147. PMLR.
  186. W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson et al. 2022. Inner monologue: Em- bodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608. · arXiv
  187. Y. Huang, Y. Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam et al. 2018. Gpipe: Efficient training of giant neural networks using pipeline parallelism. · DOI · Paper page
  188. Z. Huang, Y. Shen, X. Zhang, J. Zhou, W. Rong and Z. Xiong. 2023. Transformer-patcher: One mistake worth one neuron. In The Eleventh International Conference on Learning Representations.
  189. I. Hubara, B. Chmiel, M. Island, R. Banner, J. Naor and D. Soudry. 2021. Accelerated sparse neural training: A provable and efficient method to find n:m transposable masks. In Advances in Neural Information Processing Sys- tems, volume 34, pages 21099-21111.
  190. C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas et al. 2022. Tutel: Adaptive mixture-of- experts at scale. arXiv preprint arXiv:2206.03382. · arXiv
  191. J. P. A. Ioannidis. 2005. Why Most Published Research Findings Are False. PLoS Medicine, 2(8):e124. · DOI · Paper page
  192. D. Ippolito, A. Yuan, A. Coenen and S. Burnam. 2022. Creative writing with an ai-powered writing assistant: Perspectives from professional writers. arXiv preprint arXiv:2211.05030. · arXiv
  193. G. Irving, P. Christiano and D. Amodei. 2018. Ai safety via debate. arXiv preprint arXiv:1805.00899. · arXiv
  194. K. Y. Iu and V. M.-Y. Wong. 2023. Chatgpt by openai: The end of litigation lawyers? Available at SSRN.
  195. S. Iyer, X. V. Lin, R. Pasunuru, T. Mihaylov, D. Simig, P. Yu, K. Shuster, T. Wang et al. 2022. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. · DOI · Paper page
  196. G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin et al. 2022. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299. · arXiv
  197. A. Jacovi, A. Caciularu, O. Goldman and Y. Goldberg. 2023. Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evalua- tion Benchmarks. ArXiv:2305.10160 [cs]. · arXiv
  198. N. Jain, K. Saifullah, Y. Wen, J. Kirchenbauer, M. Shu, A. Saha, M. Goldblum, J. Geiping et al. 2023. Bring your own data! self-supervised evaluation for large language models. arXiv preprint arXiv:23062.13651.
  199. J. Jang, S. Kim, S. Ye, D. Kim, L. Logeswaran, M. Lee, K. Lee and M. Seo. 2023. Exploring the Benefits of Training Expert Language Models over Instruction Tuning. ArXiv:2302.03202 [cs]. · arXiv
  200. J. R. Jeliazkov, D. del Alamo and J. D. Karpiak. 2023. Esmfold hallucinates native-like protein sequences. bioRxiv. · DOI

200 of 551 shown

Authors
Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., & McHardy, R.
Affiliations
  • UK Health Security Agency (GB)
  • University of Cambridge (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2307.10169
arXiv
2307.10169
OpenAlex
W4384920109
Cited by
196
References
0
Citations by year
2026: 13 · 2025: 59 · 2024: 102 · 2023: 22
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, open problems, natural language processing, machine learning challenges

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review