CoChat
Sign Up
PreprintCited by 160Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Open access
Biderman, Stella, Schoelkopf, Hailey, Anthony, Quentin, Bradley, Herbie, O'Brien, Kyle, Hallahan, Eric, Khan, Mohammad Aflah, Purohit, Shivanshu +5 more·2023·arXiv
DOI 10.48550/arxiv.2304.01373arXiv 2304.01373
Open this paper in CoChat — free accountOpen PDF

Abstract

How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce \textit{Pythia}, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend \textit{Pythia} to facilitate research in many areas, and we present several case studies including novel results in memorization, term frequency effects on few-shot performance, and reducing gender bias. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics. Trained models, analysis code, training code, and training data can be found at \url{https://github.com/EleutherAI/pythia}.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.01373

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs are typically trained on sequences containing no more than 8K tokens.

    “LLMs are typically trained on sequences containing no more than 8K tokens (Touvron et al., 2023; Penedo et al., 2023; Biderman et al., 2023), and thus cannot well handle contexts exceeding 8K tokens.”

    From ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024) · cites this paper as Biderman et al., 2023

    Section 2.4 (Training; page not specified) says all Pythia models were trained with sequences of 2,048 tokens, supporting an example below 8K. It does not establish that LLMs generally are typically trained on sequences of no more than 8K tokens. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Prior work has found large effects of input probability on LLM performance on arithmetic and factual recall.

    “Additional situations where input probability has a large effect can be found in prior work studying LLM performance on arithmetic (33) and factual recall (34, 35).”

    From Embers of autoregression show how large language models are shaped by the problem they are trained to solve (2024) · cites this paper as [35]

  • Dolly is Pythia 12B fine-tuned on DataBricks Dolly, human-written data created by Databricks.

    “Dolly 9 is Pythia 12B (Biderman et al., 2023) fine-tuned on DataBricks Dolly, human-written data created by Databricks.”

    From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Biderman et al., 2023

  • Oasst-pythia is Pythia 12B fine-tuned on human-written data collected through Open Assistant.

    “Oasst-pythia 11 is Pythia 12B fine-tined on humanwritten data collected through Open Assistant.”

    From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Biderman et al., 2023

    The Task tables list a 12B Pythia model, but they do not identify Oasst-pythia. Searches for Oasst-pythia, Open Assistant, and fine-tuning found no passage in our copy establishing the claimed model lineage or the origin of its training data. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 160

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 81

  1. Ahdritz, G., Bouatta, N., Kadyan, S., Xia, Q., Gerecke, W., O'Donnell, T. J., Berenberg, D., Fisk, I., Zanichelli, N., Zhang, B., et al. Openfold: Retraining alphafold2 yields new insights into its learning mechanisms and capacity for generalization. bioRxiv, 2022.
  2. Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., and Steinhardt, J. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023. · arXiv
  3. Biderman, S., Bicheno, K., and Gao, L. Datasheet for the Pile. Computing Research Repository, 2022. doi: 10.48550/arXiv.2201.07311. URL https://arxiv. org/abs/2201.07311v1. Version 1. Biderman, S., Prashanth, U. S., Sutawika, L., Purohit, S., Schoelkopf, H., Anthony, Q., and Raff, E. Emergent and predictable memorization in large language models. Preprint under review, 2023. · DOI · Paper page
  4. Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., and Bao, M. The values encoded in machine learning research. Computing Research Repository, 2021. doi: 10.48550/arXiv.2106.15590. URL https://arxiv. org/abs/2106.15590v2. Version · DOI · Paper page
  5. Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5-Workshop on Challenges & Perspectives in Creating Large Language Models, pp. 95-136, 2022.
  6. Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., and Wallach, H. Stereotyping Norwegian salmon: An in- ventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Process- ing (Volume 1: Long Papers), pp. 1004-1015, Online, August 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/2021.acl-long.81. URL https: //aclanthology.org/2021.acl-long. · DOI · Paper page
  7. Bordia, S. and Bowman, S. Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 7-15, 2019.
  8. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877- 1901, 2020.
  9. Brunet, M.-E., Alkalay-Houlihan, C., Anderson, A., and Zemel, R. Understanding the origins of bias in word embeddings. In International conference on machine learning, pp. 803-811. PMLR, 2019.
  10. Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Se- curity Symposium (USENIX Security 19), pp. 267-284, 2019.
  11. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021.
  12. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neu- ral language models. arXiv preprint arXiv:2202.07646, 2022. · arXiv
  13. Caswell, I., Breiner, T., van Esch, D., and Bapna, A. Lan- guage id in the wild: Unexpected challenges on the path to a thousand-language web text corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6588-6608, 2020.
  14. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. Computing Research Repository, 2021. doi: 10.48550/arXiv.2107.03374. URL https: //arxiv.org/abs/2107.03374v2. Version 2. · DOI · Paper page
  15. Choenni, R., Shutova, E., and van Rooij, R. Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1477-1491, 2021.
  16. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. PaLM: Scaling language model- ing with Pathways. Computing Research Repository, 2022. doi: 10.48550/arXiv.2204.0231. URL https: //arxiv.org/abs/2204.02311v5. Version 5. · DOI
  17. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, Guillaume andGuzmán, F., Grave, E., Ott, Myle andZettlemoyer, L., and Stoyanov, V. Unsuper- vised cross-lingual representation learning at scale. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. R. (eds.), Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pp. 8440-8451, Online, 07 2020. Association for Computational Linguis- tics. doi: 10.18653/v1/2020.acl-main.747. URL https: //aclanthology.org/2020.acl-main.747. · DOI · Paper page
  18. Crowson, K., Biderman, S., Kornis, D., Stander, D., Halla- han, E., Castricato, L., and Raff, E. VQGAN-CLIP: Open domain image generation and editing with nat- ural language guidance. Computing Research Repos- itory, 2022. doi: 10.48550/arXiv.2204.08583. URL https://arxiv.org/abs/2204.08583v2. Ver- sion 2. · DOI
  19. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. arXiv preprint arXiv:2205.14135, 2022. · arXiv
  20. Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., and Gardner, M. Docu- menting large webtext corpora: A case study on the colos- sal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, 2021.
  21. D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Ali- panahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 2020.
  22. Elazar, Y., Kassner, N., Ravfogel, S., Feder, A., Ravichan- der, A., Mosbach, M., Belinkov, Y., Schütze, H., and Goldberg, Y. Measuring causal effects of data statistics on language model'sfactual'predictions. arXiv preprint arXiv:2207.14251, 2022. · arXiv
  23. Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-t., Zettlemoyer, L., and Lewis, M. InCoder: A generative model for code infilling and synthesis. Computing Research Repository, 2022. doi: 10.48550/arXiv.2204.05999. URL https://arxiv. org/abs/2204.05999v2. Version · DOI · Paper page
  24. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The Pile: An 800GB dataset of diverse text for language modeling. Computing Research Repository, 2020. doi: 10.48550/arXiv.2101.00027. URL https://arxiv.org/abs/2101.00027v1. Ver- sion 1. · DOI · Paper page
  25. Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., and Cherry, C. Scaling laws for neural machine translation. Computing Research Repository, 2021. doi: 10.48550/arXiv.2109.07740. URL https://arxiv.org/abs/2109.07740v1. Ver- sion 1. · DOI
  26. Ginart, A., van der Maaten, L., Zou, J., and Guo, C. Sub- mix: Practical private prediction for large-scale language models. arXiv preprint arXiv:2201.00971, 2022. · arXiv
  27. Gira, M., Zhang, R., and Lee, K. Debiasing pre-trained language models via efficient fine-tuning. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, pp. 59-69, 2022.
  28. Goyal, P., Dollár, P., Girshick, R. B., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch SGD: training ima- genet in 1 hour. CoRR, abs/1706.02677, 2017. URL http://arxiv.org/abs/1706.02677. · arXiv
  29. Hall, M., van der Maaten, L., Gustafson, L., and Adcock, A. A systematic study of bias amplification. arXiv preprint arXiv:2201.11706, 2022. · arXiv
  30. Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. Scaling laws for autoregres- sive generative modeling. Computing Research Repos- itory, 2020. doi: 10.48550/arXiv.2010.14701. URL https://arxiv.org/abs/2010.14701v2. Ver- sion 2. · DOI · Paper page
  31. Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer. Computing Research Repos- itory, 2021. doi: 10.48550/arXiv.2102.01293. URL https://arxiv.org/abs/2102.01293v1. Ver- sion 1. · DOI · Paper page
  32. Hernandez, D., Brown, T., Conerly, T., DasSarma, N., Drain, D., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Henighan, T., Hume, T., Johnston, S., Mann, B., Olah, C., Olsson, C., Amodei, D., Joseph, N., Kaplan, J., and McCandlish, S. Scaling laws and interpretability of learn- ing from repeated data. Computing Research Reposi- tory, 05 2022. doi: 10.48550/arXiv.2205.10487. URL https://arxiv.org/abs/2205.10487v1. Ver- sion 1. · DOI · Paper page
  33. Hirota, Y., Nakashima, Y., and Garcia, N. Quantifying soci- etal bias amplification in image captioning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13450-13459, 2022.
  34. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. · arXiv
  35. Hu, H., Salcic, Z., Sun, L., Dobbie, G., Yu, P. S., and Zhang, X. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54(11s):1-37, 2022.
  36. Huang, J., Shao, H., and Chang, K. C.-C. Are large pre- trained language models leaking your personal informa- tion? arXiv preprint arXiv:2205.12628, 2022. · arXiv
  37. Ippolito, D., Tramèr, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette-Choo, C. A., and Carlini, N. Prevent- ing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546, 2022. · arXiv
  38. Jagielski, M., Thakkar, O., Tramer, F., Ippolito, D., Lee, K., Carlini, N., Wallace, E., Song, S., Thakurta, A., Papernot, N., et al. Measuring forgetting of memorized training examples. arXiv preprint arXiv:2207.00099, 2022. · arXiv
  39. Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. Data governance in the age of large-scale data-driven language technology. In 2022 ACM Conference on Fairness, Accountability, and Trans- parency, pp. 2206-2222, 2022.
  40. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Trivi- aqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
  41. Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589, 2021.
  42. Kandpal, N., Deng, H., Roberts, A., Wallace, E., and Raffel, C. Large language models struggle to learn long-tail knowledge. arXiv preprint arXiv:2211.08411, 2022. · arXiv
  43. Kirtane, N., Manushree, V., and Kane, A. Efficient gender debiasing of pre-trained indic language models. arXiv preprint arXiv:2209.03661, 2022. · arXiv
  44. Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., Setyawan, M., Sarin, S., Samb, S., Sagot, B., Rivera, C., Rios, A., Papadimitriou, I., Osei, S., Suárez, P. O., Orife, I., Ogueji, K., Rubungo, A. N., Nguyen, T. Q., Müller, M., Müller, A., Muhammad, S. H., Muhammad, N., Mnyakeni, A., Mirzakhalov, J., Matan- gira, T., Leong, C., Lawson, N., Kudugunta, S., Jernite, Y., Jenny, M., Firat, O., Dossou, B. F. P., Dlamini, S., de Silva, N., C ¸abuk Ballı, S., Biderman, S., Battisti, A., Baruwa, A., Bapna, A., Baljekar, P., Azime, I. A., Awokoya, A., Ataman, D., Ahia, O., Ahia, O., Agrawal, S., and Adeyemi, M. Quality at a glance: An audit of web-crawled multilingual datasets. Computing Research Repository, 2021. doi: 10.48550/arXiv.2103.12028. URL https://arxiv.org/abs/2103.12028v3. Ver- sion 4. Laurenc ¸on, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Werra, L. V., Mou, C., Ponferrada, E. G., Nguyen, H., Frohberg, J., Šaško, M., Lhoest, Q., McMillan-Major, A., Dupont, G., Biderman, S., Rogers, A., allal, L. B., Toni, F. D., Pistilli, G., Nguyen, O., Nikpoor, S., Masoud, M., Colombo, P., de la Rosa, J., Villegas, P., Thrush, T., Longpre, S., Nagel, S., Weber, L., Muñoz, M. R., Zhu, J., Strien, D. V., Alyafeai, Z., Almubarak, K., Chien, V. M., Gonzalez-Dios, I., Soroa, A., Lo, K., Dey, M., Suarez, P. O., Gokaslan, A., Bose, S., Adelani, D. I., Phan, L., Tran, H., Yu, I., Pai, S., Chim, J., Lepercq, V., Ilic, S., Mitchell, M., Luccioni, S., and Jernite, Y. The bigscience ROOTS corpus: A 1.6TB composite multilingual dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https: //openreview.net/forum?id=UoEw6KigkUn. · DOI · Paper page
  45. Le Scao, T., Wang, T., Hesslow, D., Saulnier, L., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Phang, J., Press, O., et al. What language model to train if you have one million GPU hours? In Proceedings of BigScience Episode #5-Workshop on Challenges & Perspectives in Creating Large Language Models, 2022.
  46. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating train- ing data makes language models better. In Annual Meet- ing of the Association for Computational Linguistics, 2021.
  47. Levy, S., Lazar, K., and Stanovsky, G. Collecting a large- scale gender bias dataset for coreference resolution and machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2470- 2480, 2021.
  48. Mallen, A., Asai, A., Zhong, V., Das, R., Hajishirzi, H., and Khashabi, D. When not to trust language models: Investi- gating effectiveness and limitations of parametric and non- parametric memories. arXiv preprint arXiv:2212.10511, 2022. · arXiv
  49. McCandlish, S., Kaplan, J., Amodei, D., and Team, O. D. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018. · arXiv
  50. McGrath, T., Kapishnikov, A., Tomašev, N., Pearce, A., Hassabis, D., Kim, B., Paquet, U., and Kramnik, V. Ac- quisition of chess knowledge in AlphaZero. Computing Research Repository, 2021. doi: 10.48550/arXiv.2111. 09259. URL https://arxiv.org/abs/2111. 09259v3. Version 3. McMillan-Major, A., Alyafeai, Z., Biderman, S., Chen, K., De Toni, F., Dupont, G., Elsahar, H., Emezue, C., Aji, A. F., Ilić, S., et al. Documenting geographically and con- textually diverse data sources: The BigScience catalogue of language data and resources. Computing Research Repository, 2022. doi: 10.48550/arXiv.2201.10066. URL https://arxiv.org/abs/2201.10066v1. Ver- sion 1. · DOI
  51. Mikami, H., Fukumizu, K., Murai, S., Suzuki, S., Kikuchi, Y., Suzuki, T., Maeda, S.-i., and Hayashi, K. A scaling law for synthetic-to-real transfer: How much is your pre-training effective? Computing Research Repository, 2021. doi: 10.48550/arXiv.2108.11018. URL https: //arxiv.org/abs/2108.11018v3. Version 3. Nangia, N., Vania, C., Bhalerao, R., and Bowman, S. R. CrowS-pairs: A challenge dataset for measuring so- cial biases in masked language models. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1953- 1967, Online, November 2020. Association for Computa- tional Linguistics. doi: 10.18653/v1/2020.emnlp-main. 154. URL https://aclanthology.org/2020. emnlp-main.154. · DOI
  52. Névéol, A., Dupont, Y., Bezanc ¸on, J., and Fort, K. French CrowS-pairs: Extending a challenge dataset for measur- ing social bias in masked language models to a language other than English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8521-8531, Dublin, Ire- land, May 2022. Association for Computational Linguis- tics. doi: 10.18653/v1/2022.acl-long.583. URL https: //aclanthology.org/2022.acl-long.583. · DOI
  53. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. · arXiv
  54. Pfeiffer, J., Goyal, N., Lin, X. V., Li, X., Cross, J., Riedel, S., and Artetxe, M. Lifting the curse of multilinguality by pre-training modular transformers. Computing Re- search Repository, 05 2022. doi: 10.48550/arXiv.2205. 06266. URL https://arxiv.org/abs/2205. 06266v1. Version 1. · DOI
  55. Phang, J., Bradley, H., Gao, L., Castricato, L., and Bider- man, S. Eleutherai: Going beyond" open science" to" science in the open". arXiv preprint arXiv:2210.06413, 2022. · arXiv
  56. Pu, J., Yang, Y., Li, R., Elibol, O., and Droppo, J. Scaling ef- fect of self-supervised speech models. Proc. Interspeech 2021, pp. 1084-1088, 2021.
  57. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (140):1-67, 01 2020. ISSN 1532-4435. URL http: //jmlr.org/papers/v21/20-074.html.
  58. Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. ZeRO: Memory optimizations toward training trillion parameter models. In Proceedings of the Interna- tional Conference for High Performance Computing, Net- working, Storage and Analysis, SC '20. IEEE Press, 2020. ISBN 9781728199986. doi: 10.5555/3433701. 3433727. URL https://dl.acm.org/doi/10. 5555/3433701.3433727. · DOI
  59. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022. · arXiv
  60. Razeghi, Y., Logan IV, R. L., Gardner, M., and Singh, S. Impact of pretraining term frequencies on few-shot rea- soning. arXiv preprint arXiv:2202.07206, 2022. · arXiv
  61. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
  62. Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H., Manica, M., Shen, S., Yong, Z. X., Pandey, H., Bawden, R., Wang, T., Neeraj, T., Rozen, J., Sharma, A., Santilli, A., Févry, T., Fries, J. A., Teehan, R., Biderman, S., Gao, L., Bers, T., Wolf, T., and Rush, A. M. Multitask prompted training en- ables zero-shot task generalization. Computing Research Repository, 2021. doi: 10.48550/arXiv.2110.08207. URL https://arxiv.org/abs/2110.08207v3. Ver- sion 3. · DOI · Paper page
  63. Sellam, T., Yadlowsky, S., Wei, J., Saphra, N., D'Amour, A., Linzen, T., Bastings, J., Turc, I., Eisenstein, J., Das, D., et al. The multiberts: Bert reproductions for robustness analysis. arXiv preprint arXiv:2106.16163, 2021. · arXiv
  64. Sharma, U. and Kaplan, J. A neural scaling law from the dimension of the data manifold. Computing Research Repository, 2020. doi: 10.48550/arXiv.2004.10802. URL https://arxiv.org/abs/2004.10802v1. Ver- sion 1. · DOI
  65. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM: Training multi- billion parameter language models using model paral- lelism. Computing Research Repository, 2019. doi: 10.48550/arXiv.1909.08053. URL https://arxiv. org/abs/1909.08053v4. Version · DOI · Paper page
  66. Silva, A., Chopra, R., and Gombolay, M. Cross-loss influ- ence functions to explain deep network representations. In International Conference on Artificial Intelligence and Statistics, pp. 1-17. PMLR, 2022.
  67. Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhan- dari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., Zhang, E., Child, R., Aminabadi, R. Y., Bernauer, J., Song, X., Shoeybi, M., He, Y., Hous- ton, M., Tiwary, S., and Catanzaro, B. Using deepspeed and megatron to train megatron-turing nlg 530b, a large- scale generative language model. Computing Research Repository, 2022. doi: 10.48550/arXiv.2201.11990. URL https://arxiv.org/abs/2201.11990v3. Ver- sion 3. · DOI · Paper page
  68. Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. Ro- Former: Enhanced transformer with rotary position em- bedding. Computing Research Repository, 2021. doi: 10.48550/arXiv.2104.09864. URL https://arxiv. org/abs/2104.09864v4. Version · DOI · Paper page
  69. Suárez, P. J. O., Sagot, B., and Romary, L. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Chal- lenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache, 2019.
  70. Tang, J. WuDao: Pretrain the world. Keynote adress at the European Conference on Machine Learning and Princi- ples and Practice of Knowledge Discovery in Databases, 2021.
  71. Tirumala, K. N. B., Markosyan, A. H., Zettlemoyer, L., and Aghajanyan, A. Memorization without overfitting: An- alyzing the training dynamics of large language models. ArXiv, abs/2205.10770, 2022. · arXiv
  72. Van der Wal, O., Bachmann, D., Leidinger, A., van Maanen, L., Zuidema, W., and Schulz, K. Undesirable biases in nlp: Averting a crisis of measurement. arXiv preprint arXiv:2211.13709, 2022a. · arXiv
  73. Van der Wal, O., Jumelet, J., Schulz, K., and Zuidema, W. The birth of bias: A case study on the evolution of gender bias in an english language model. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp. 75-75, 2022b.
  74. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. · arXiv
  75. Xia, M., Artetxe, M., Zhou, C., Lin, X. V., Pasunuru, R., Chen, D., Zettlemoyer, L., and Stoyanov, V. Training trajectories of language models across scales, 2022. URL https://arxiv.org/abs/2212.09803. · arXiv
  76. Xu, F. F., Alon, U., Neubig, G., and Hellendoorn, V. J. A systematic evaluation of large language models of code. Computing Research Repository, 2022. doi: 10.48550/arXiv.2202.13169. URL https://arxiv. org/abs/2202.13169v3. Version · DOI
  77. Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022. · arXiv
  78. Zhang, G., Li, L., Nado, Z., Martens, J., Sachdeva, S., Dahl, G., Shallue, C., and Grosse, R. B. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. Advances in neural information processing systems, 32, 2019.
  79. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mi- haylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettle- moyer, L. OPT: Open pre-trained transformer language models. Computing Research Repository, 2022. doi: 10.48550/arXiv.2205.01068. URL https://arxiv. org/abs/2205.01068v4. Version · DOI · Paper page
  80. Zhao, J., Wang, T., Yatskar, M., Ordonez, V., and Chang, K.-W. Men also like shopping: Reducing gender bias am- plification using corpus-level constraints. arXiv preprint arXiv:1707.09457, 2017. · arXiv
  81. Zhao, J., Wang, T., Yatskar, M., Ordonez, V., and Chang, K.-W. Gender bias in coreference resolution: Evalua- tion and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 15-20, 2018.

Authors
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., & van der Wal, O.
Affiliations
  • Booz Allen Hamilton (United States) (US)
  • Yale University (US)
  • University of Cambridge (GB)
  • Indraprastha Institute of Information Technology Delhi (IN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2304.01373
arXiv
2304.01373
OpenAlex
W4362655426
Cited by
160
References
0
Citations by year
2026: 5 · 2025: 41 · 2024: 47 · 2023: 67
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
  • UK Research and Innovation (EP/S022961/1)
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (406.DI.19.059)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
Pythia, large language models, emergence timing, training dynamics, model scaling, memorization, training checkpoints, training data reconstruction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review