CoChat
Sign Up
PreprintCited by 90Topic ModelingNatural Language Processing TechniquesSpeech Recognition and Synthesis

Dynamic Evaluation of Neural Sequence Models

Open access
Krause, Ben, Kahembwe, Emmanuel, Murray, Iain, Renals, Steve·2017·arXiv
DOI 10.48550/arxiv.1709.07432arXiv 1709.07432
Open this paper in CoChat — free accountOpen PDF

Abstract

We present methodology for using dynamic evaluation to improve neural sequence models. Models are adapted to recent history via a gradient descent based mechanism, causing them to assign higher probabilities to re-occurring sequential patterns. Dynamic evaluation outperforms existing adaptation approaches in our comparisons. Dynamic evaluation improves the state-of-the-art word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1 and 44.3 respectively, and the state-of-the-art character-level cross-entropies on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char respectively.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1709.07432

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • At training time, prior TTT/dynamic evaluation optimized the model for its loss out of the box, not for its loss after TTT.

    “This form of Test-Time Training (TTT), similar to an old idea known as dynamic evaluation [72, 60], still has a missing piece: At training time, we were optimizing the model for its loss out of the box, not for its loss after TTT.”

    From End-to-End Test-Time Training for Long Context (2025) · cites this paper as [60]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Neural Machine Translation in Linear Time

    Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. V. D., Graves, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.10099 · Cited by 414 · cited 1× · see citing works

    “…LSTM (Wu et al., 2016) 17M 1.44 HyperLSTM (Ha et al., 2017) 27M 1.34 Hierarchical multiscale LSTM (Chung et al., 2017) 1.32 Bytenet decoder (Kalchbrenner et al., 2016) 1.31 LSTM + BB tuning (Melis et al., 2017) 46M 1.30 Recurrent highway networks (Zilly et al., 2017) 46M 1.27 Fast-slow…”
    Influential

Flagged first, then by citations

Research works printed but not linked: 34

  1. Z. Aharoni, G. Rattner, and H. Permuter. Gradual learning of deep recurrent neural networks. arXiv preprint arXiv:1708.08863 , 2017. · arXiv
  2. J. R. Bellegarda. Statistical language model adaptation: review and perspectives. Speech Communication , 42(1): 93-108, 2004.
  3. J. Chung, S. Ahn, and Y. Bengio. Hierarchical multiscale recurrent neural networks. ICLR , 2017.
  4. T. Cooijmans, N. Ballas, C. Laurent, and A. Courville. Recurrent batch normalization. ICLR , 2017.
  5. M. Fortunato, C. Blundell, and O. Vinyals. Bayesian recurrent neural networks. arXiv preprint arXiv:1704.02798 , 2017. · arXiv
  6. Y. Gal and Z. Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems , pp. 1019-1027, 2016.
  7. E. Grave, A. Joulin, and N. Usunier. Improving neural language models with a continuous cache. ICLR , 2017.
  8. A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 , 2013. · arXiv
  9. D. Ha, A. Dai, and Q. Lee. Hypernetworks. ICLR , 2017.
  10. H. Inan, K. Khosravi, and R. Socher. Tying word vectors and word classifiers: A loss framework for language modeling. ICLR , 2017.
  11. F. Jelinek, B. Merialdo, S. Roukos, and M. Strauss. A dynamic language model for speech recognition. In HLT , volume 91, pp. 293-295, 1991.
  12. Y. Kim, Y. Jernite, D. Sontag, and A. M. Rush. Character-aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence , 2016.
  13. D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
  14. P. Koehn. Europarl: A parallel corpus for statistical machine translation. In MT Summit , volume 5, pp. 79-86, 2005.
  15. B. Krause, L. Lu, I. Murray, and S. Renals. Multiplicative LSTM for sequence modelling. arXiv preprint arXiv:1609.07959 , 2016. · arXiv
  16. B. Krause, I. Murray, S. Renals, and L. Lu. Multiplicative LSTM for sequence modelling. ICLR Workshop track , 2017. URL https://openreview.net/forum?id=SJCS5rXFl .
  17. R. Kuhn. Speech recognition and the frequency of recently used words: A modified Markov model for natural language. In Proceedings of the 12th conference on Computational linguistics-Volume 1 , pp. 348-350. Association for Computational Linguistics, 1988.
  18. M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics , 19(2):313-330, 1993.
  19. G. Melis, C. Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589 , 2017. · arXiv
  20. S. Merity, N. S. Keskar, and R. Socher. Regularizing and optimizing LSTM language models. arXiv preprint arXiv:1708.02182 , 2017a. · arXiv
  21. S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. ICLR , 2017b.
  22. T. Mikolov, M. Karafiát, L. Burget, J. Cernock` y, and S. Khudanpur. Recurrent neural network based language model. In Interspeech , volume 2, pp. 3, 2010.
  23. T. Mikolov, I. Sutskever, A. Deoras, H. Le, S. Kombrink, and J. Cernocky. Subword language modeling with neural networks. preprint (http://www. fit. vutbr. cz/imikolov/rnnlm/char. pdf) , 2012.
  24. A. Mujika, F. Meier, and A. Steger. Fast-slow recurrent neural networks. arXiv preprint arXiv:1705.08639 , 2017. · arXiv
  25. B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization , 30(4):838-855, 1992.
  26. O. Press and L. Wolf. Using the output embedding to improve language models. EACL 2017 , pp. 157, 2017.
  27. T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems , pp. 901-909, 2016.
  28. T. Tieleman and G. E. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning , 4(2), 2012.
  29. L. Wan, M. Zeiler, S. Zhang, Yann L. Cun, and R. Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th international conference on machine learning (ICML-13) , pp. 1058-1066, 2013.
  30. P. J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE , 78: 1550-1560, 1990.
  31. Y. Wu, S. Zhang, Y. Zhang, Y. Bengio, and R. Salakhutdinov. On multiplicative integration with recurrent neural networks. In NIPS , 2016.
  32. W. Zaremba, I. Sutskever, and O. Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 , 2014. · arXiv
  33. J. G. Zilly, R. K. Srivastava, J. Koutník, and J. Schmidhuber. Recurrent highway networks. ICLR , 2017.
  34. B. Zoph and Quoc V Le. Neural architecture search with reinforcement learning. ICLR , 2017.

Cited by

Sorted by citations

Authors
Krause, B., Kahembwe, E., Murray, I., & Renals, S.
Affiliations
University of Edinburgh (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Pages
2766-2775
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1709.07432
arXiv
1709.07432
OpenAlex
W2963938518
Cited by
90
References
1
Citations by year
2024: 1 · 2023: 4 · 2022: 10 · 2021: 19 · 2020: 16 · 2019: 18 · 2018: 18 · 2017: 3 · 2016: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Funders
Horizon 2020 Framework Programme
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
dynamic assessment, gradient-based adaptation, language model evaluation, Penn Treebank, WikiText-2

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review