CoChat
Sign Up
PreprintCited by 2,894GNSS positioning and interferenceStatistical and numerical algorithmsNeural Networks and Applications

Generating Sequences With Recurrent Neural Networks

Open access
Graves, Alex·2013·arXiv
DOI 10.48550/arxiv.1308.0850arXiv 1308.0850
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1308.0850

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RMSProp with momentum generates its parameter updates using a momentum on the rescaled gradient, whereas Adam updates are directly estimated using a running average of first and second moment of the gradient.

    “RMSProp with momentum generates its parameter updates using a momentum on the rescaled gradient, whereas Adam updates are directly estimated using a running average of first and second moment of the gradient.”

    From Adam: A Method for Stochastic Optimization (2014) · cites this paper as Graves, 2013

    Section 4.2 (p. 23) describes RMSProp as dividing gradients by a running average of their recent magnitude. Our copy does not state that momentum is applied to those rescaled gradients; the momentum mentioned in Section 3.1 (p. 7) concerns a separate stochastic-gradient-descent experiment. It also does not discuss Adam or compare the two optimizers. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 23

  1. [1][1] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks , 5(2):157-166, March 1994. -
  2. [3][3] C. Bishop. Neural Networks for Pattern Recognition . Oxford University Press, Inc., 1995. -
  3. [4][4] N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent. Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. In Proceedings of the Twenty-nine International Conference on Machine Learning (ICML'12) , 2012. -
  4. [5][5] J. G. Cleary, Ian, and I. H. Witten. Data compression using adaptive coding and partial string matching. IEEE Transactions on Communications , 32:396-402, 1984. -
  5. [7][7] F. Gers, N. Schraudolph, and J. Schmidhuber. Learning precise timing with LSTM recurrent networks. Journal of Machine Learning Research , 3:115-143, 2002. -
  6. [8][8] A. Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems , volume 24, pages 2348-2356. 2011. -
  7. [9][9] A. Graves. Sequence transduction with recurrent neural networks. In ICML Representation Learning Worksop , 2012. -
  8. [10][10] A. Graves, A. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In Proc. ICASSP , 2013. -
  9. [12][12] A. Graves and J. Schmidhuber. Offline handwriting recognition with multidimensional recurrent neural networks. In Advances in Neural Information Processing Systems , volume 21, 2008. -
  10. [13][13] P. D. Gr¨ unwald. The Minimum Description Length Principle (Adaptive Computation and Machine Learning) . The MIT Press, 2007. -
  11. [18][18] K.-C. Jim, C. Giles, and B. Horne. An analysis of noise in recurrent neural networks: convergence and generalization. Neural Networks, IEEE Transactions on , 7(6):1424 -1438, 1996. -
  12. [20][20] B. Knoll and N. de Freitas. A machine learning perspective on predictive coding with paq. CoRR , abs/1108.3298, 2011. - · arXiv
  13. [21][21] M. Liwicki and H. Bunke. IAM-OnDB - an on-line English sentence database acquired from handwritten text on a whiteboard. In Proc. 8th Int. Conf. on Document Analysis and Recognition , volume 2, pages 956961, 2005. -
  14. [22][22] M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. COMPUTATIONAL LINGUISTICS , 19(2):313-330, 1993. -
  15. [23][23] T. Mikolov. Statistical Language Models based on Neural Networks . PhD thesis, Brno University of Technology, 2012. -
  16. [25][25] A. Mnih and G. Hinton. A Scalable Hierarchical Distributed Language Model. In Advances in Neural Information Processing Systems , volume 21, 2008. -
  17. [26][26] A. Mnih and Y. W. Teh. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of the 29th International Conference on Machine Learning , pages 1751-1758, 2012. -
  18. [27][27] T. N. Sainath, A. Mohamed, B. Kingsbury, and B. Ramabhadran. Lowrank matrix factorization for deep neural network training with highdimensional output targets. In Proc. ICASSP , 2013. -
  19. [28][28] M. Schuster. Better generative models for sequential data problems: Bidirectional recurrent mixture density networks. pages 589-595. The MIT Press, 1999. -
  20. [29][29] I. Sutskever, G. E. Hinton, and G. W. Taylor. The recurrent temporal restricted boltzmann machine. pages 1601-1608, 2008. -
  21. [30][30] I. Sutskever, J. Martens, and G. Hinton. Generating text with recurrent neural networks. In ICML , 2011. -
  22. [31][31] G. W. Taylor and G. E. Hinton. Factored conditional restricted boltzmann machines for modeling motion style. In Proc. 26th Annual International Conference on Machine Learning , pages 1025-1032, 2009. -
  23. [33][33] R. Williams and D. Zipser. Gradient-based learning algorithms for recurrent networks and their computational complexity. In Back-propagation: Theory, Architectures and Applications , pages 433-486. 1995.

Cited by

Sorted by citations

Authors
Graves, A.
Affiliations
University of Toronto (CA)
Corresponding author
Graves, Alex
Venue
arXiv
Publisher
arXiv
Published
2013
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1308.0850
arXiv
1308.0850
OpenAlex
W1810943226
Cited by
2,894
References
2
Citations by year
2026: 14 · 2025: 64 · 2024: 120 · 2023: 194 · 2022: 204 · 2021: 428 · 2020: 408 · 2019: 447 · 2018: 354 · 2017: 299 · 2016: 213 · 2015: 109 · 2014: 34 · 2013: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Aerospace Engineering › GNSS positioning and interference
  • Physical Sciences › Mathematics › Applied Mathematics › Statistical and numerical algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
GNSS time series, long short-term memory networks, recurrent neural networks, time series denoising, coordinate prediction, position time series

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review