CoChat
Sign Up
PreprintCited by 412Topic ModelingDomain Adaptation and Few-Shot LearningNatural Language Processing Techniques

Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping

Open access
Dodge, Jesse, Ilharco, Gabriel, Schwartz, Roy, Farhadi, Ali, Hajishirzi, Hannaneh, Smith, Noah·2020·arXiv
DOI 10.48550/arxiv.2002.06305arXiv 2002.06305
Open this paper in CoChat — free accountOpen PDF

Abstract

Fine-tuning pretrained contextual word embedding models to supervised downstream tasks has become commonplace in natural language processing. This process, however, is often brittle: even with the same hyperparameter values, distinct random seeds can lead to substantially different results. To better understand this phenomenon, we experiment with four datasets from the GLUE benchmark, fine-tuning BERT hundreds of times on each while varying only the random seeds. We find substantial performance increases compared to previously reported results, and we quantify how the performance of the best-found model varies as a function of the number of fine-tuning trials. Further, we examine two factors influenced by the choice of random seed: weight initialization and training data order. We find that both contribute comparably to the variance of out-of-sample performance, and that some weight initializations perform well across all tasks explored. On small datasets, we observe that many fine-tuning trials diverge part of the way through training, and we offer best practices for practitioners to stop training less promising runs early. We publicly release all of our experimental data, including training and validation scores for 2,100 trials, to encourage further analysis of training dynamics during fine-tuning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2002.06305

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Promptless fine-tuning may overfit or not learn stably on small datasets.

    “This is a simple, powerful, and widely used method, but it may overfit or not learn stably on small datasets [21].”

    From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [21]

    The Figure 4 caption (Section 5, p. 6) says training is unstable on smaller datasets, and Section 5 (p. 6) reports that some fine-tuning runs fail to converge. These passages support the instability part of the claim, but our copy does not establish that the models overfit small datasets. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Multiple training runs under the same configuration can lead to different performance as a result of different initialization or data order.

    “Previous work has shown multiple training runs under the same configuration can lead to different performance as a result of a different initialization or data order [14, 13].”

    From Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (2025) · cites this paper as [14]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., and Szpektor, I. The second pascal recognising textual entailment challenge. In Proc. of the II PASCAL challenge , 2006.
  2. Bergstra, J., Bardenet, R., Bengio, Y., and Kegl, B. Algorithms for hyper-parameter optimization. In Proc. of NeurIPS , 2011.
  3. Dagan, I., Glickman, O., and Magnini, B. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop , 2005.
  4. Dodge, J., Gururangan, S., Card, D., Schwartz, R., and Smith, N. A. Show your work: Improved reporting of experimental results. In Proc. of EMNLP , 2019.
  5. Dolan, B. and Brockett, C. Automatically constructing a corpus of sentential paraphrases. In Proc. of IWP , 2005.
  6. Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. The third pascal recognizing textual entailment challenge. In Proc. of the ACL-PASCAL workshop on textual entailment and paraphrasing , 2007.
  7. Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. of AISTATS , 2010.
  8. Jamieson, K. and Talwalkar, A. Non-stochastic best arm identification and hyperparameter optimization. In Proc. of AISTATS , 2016.
  9. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. arXiv:1909.11942 , 2019. · arXiv
  10. Lewis, D. D. and Gale, W. A. A sequential algorithm for training text classifiers. In Proc. of SIGIR , 1994.
  11. Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research , 2018.
  12. Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pretraining task-agnostic visiolinguistic representations for visionand-language tasks. In Proc. of NeurIPS , 2019.
  13. Melis, G., Dyer, C., and Blunsom, P. On the state of the art of evaluation in neural language models. In Proc. of ICLR , 2018.
  14. Phang, J., F´ evry, T., and Bowman, S. R. Sentence encoders on stilts: Supplementary training on intermediate labeleddata tasks. arXiv:1811.01088 , 2018. · arXiv
  15. Raffe, C., Shazeer, N., Roberts, A., Lee, K. L., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910.10683 , 2019. · arXiv
  16. Saxe, A. M., McClelland, J. L., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Proc. of ICLR , 2014.
  17. Schwartz, D., Toneva, M., and Wehbe, L. Inducing brainrelevant bias in natural language processing models. In Proc. of NeurIPS , 2019.
  18. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP , 2013.
  19. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proc. of the EMNLP Workshop BlackboxNLP , 2018.
  20. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Superglue: A stickier benchmark for general-purpose language understanding systems. In Proc. of NeuRIPS , 2019.
  21. Warstadt, A., Singh, A., and Bowman, S. R. Neural network acceptability judgments. TACL , 7:625-641, 2019.
  22. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. Huggingface's transformers: State-of-theart natural language processing. ArXiv , abs/1910.03771, 2019. · arXiv
  23. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. In Proc. of NeuRIPS , 2019.
  24. Zhang, C., Kjellstrm, H., and Mandt, S. Determinantal point processes for mini-batch diversification. In Proc. of UAI , 2017.
  25. Zhu, C., Cheng, Y., Gan, Z., Sun, S., Goldstein, T., and Liu, J. Freelb: Enhanced adversarial training for language understanding. arXiv:1909.11764 , 2019. · arXiv

Cited by

Sorted by citations

Authors
Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., & Smith, N.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2002.06305
arXiv
2002.06305
OpenAlex
W3005700362
Cited by
412
References
4
Citations by year
2026: 4 · 2025: 29 · 2024: 56 · 2023: 74 · 2022: 83 · 2021: 100 · 2020: 65 · 2019: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
fine-tuning, pre-trained language models, BERT, random seed sensitivity, weight initialization, early stopping, GLUE benchmark, training dynamics, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review