CoChat
Sign Up
PreprintCited by 605Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

Galactica: A Large Language Model for Science

Open access
Taylor, Ross, Kardas, Marcin, Cucurull, Guillem, Scialom, Thomas, Hartshorn, Anthony, Saravia, Elvis, Poulton, Andrew, Kerkez, Viktor +1 more·2022·arXiv
DOI 10.48550/arxiv.2211.09085arXiv 2211.09085
Open this paper in CoChat — free accountOpen PDF

Abstract

Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2211.09085

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Galactica models were trained for 4.25 epochs and exhibit continually decreasing validation loss and improving downstream performance throughout training.

    “An exception is the recent Galactica models (Taylor et al., 2022) that were trained for 4.25 epochs and exhibit continually decreasing validation loss and improving downstream performance throughout training.”

    From Scaling Data-Constrained Language Models (2023) · cites this paper as Taylor et al., 2022

    Section 5.1 (p. 11) reports approximately 4.25 epochs and says validation and downstream performance generally improved with repeated training; it also reports periods of flat or rising validation loss and overfitting in the 120B model at the start of the fifth epoch. The same section (p. 11) says repeated tokens improved downstream performance, but the claim that validation loss decreased continually throughout training overstates the findings. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • The creators of Galactica trained a 120 billion parameter model on 450 billion tokens, a significant overallocation to parameters even in Chinchilla terms.

    “The creators of Galactica decided to train a 120 billion parameter model on 450 billion tokens, a significant overallocation to parameters even in Chinchilla terms (black efficient frontier).”

    From Scaling Data-Constrained Language Models (2023) · cites this paper as Taylor et al., 2022

    Table 5 (4.2 Models, p. 10) lists a 120-billion-parameter Galactica model, and §5.1 (p. 11) says the models were trained for 450 billion tokens. Section 5.5 (p. 19) discusses Chinchilla and the authors’ emphasis on high-quality tokens, but does not characterize this parameter-to-token allocation as significant or establish its position relative to a Chinchilla-efficient frontier. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The experiments of Galactica do not compare this setup to an alternative non-data-constrained model trained for one epoch on unique data.

    “However, the experiments of Galactica do not compare this setup to an alternative non-data-constrained model trained for one epoch on unique data.”

    From Scaling Data-Constrained Language Models (2023) · cites this paper as Taylor et al., 2022

  • The Galactica models are pre-trained on a predominantly scientific data set, which includes code data among other data sources.

    “The Galactica models are pre-trained on a predominantly scientific data set, which includes code data among other data sources.”

    From Scaling Data-Constrained Language Models (2023) · cites this paper as Taylor et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. [1]1. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 'Attention Is All You Need.' ArXiv abs/1706.03762 (2017). · arXiv
  2. [2]2. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 'BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.' ArXiv abs/1810.04805 (2019). · arXiv

Cited by

Sorted by citations

Authors
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R.
ORCID
Stojnic, Robert
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2211.09085
arXiv
2211.09085
OpenAlex
W4320005767
Cited by
605
References
1
Citations by year
2026: 121 · 2025: 169 · 2024: 180 · 2023: 131 · 2022: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, scientific knowledge representation, scientific publications, mathematical reasoning, MATH benchmark, MMLU, PubMedQA, MedMCQA, scientific knowledge organization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review