CoChat
Sign Up
PreprintCited by 7

The DOI matches its DataCite record; 13 of the 13 references it cites were found in a registry or an index.

Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks

Reimers, Nils, Gurevych, Iryna·2017·arXiv
DOI 10.48550/arxiv.1707.06799arXiv 1707.06799
Open this paper in CoChat — free account

Abstract

Selecting optimal parameters for a neural network architecture can often make the difference between mediocre and state-of-the-art performance. However, little is published which parameters and design choices should be evaluated or selected making the correct hyperparameter optimization often a "black art that requires expert experiences" (Snoek et al., 2012). In this paper, we evaluate the importance of different network design choices and hyperparameters for five common linguistic sequence tagging tasks (POS, Chunking, NER, Entity Recognition, and Event Detection). We evaluated over 50.000 different setups and found, that some parameters, like the pre-trained word embeddings or the last layer of the network, have a large impact on the performance, while other parameters, for example the number of LSTM layers or the number of recurrent units, are of minor importance. We give a recommendation on a configuration that performs well among different tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1707.06799

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 13 references

Research works printed but not linked: 22

  1. Y. Bengio, P. Simard, and P. Frasconi. 1994. Learning Long-term Dependencies with Gradient Descent is Difficult. Trans. Neur. Netw., 5(2):157-166, March.
  2. James Bergstra and Yoshua Bengio. 2012. Random Search for Hyper-parameter Optimization. Journal of Ma- chine Learning Research, 13(1):281-305, February.
  3. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for Hyper-Parameter Opti- mization. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Infor- mation Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain., pages 2546-2554.
  4. James Bergstra, Daniel Yamins, and David D. Cox. 2013. Making a Science of Model Search: Hyperparame- ter Optimization in Hundreds of Dimensions for Vision Architectures. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pages 115-123.
  5. Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1019-1027.
  6. Robert Gramacy, Matt Taddy, and Stefan Wild. 2013. Variable selection and sensitivity analysis using dynamic trees, with an application to computer code performance tuning. The Annals of Applied Statistics, 7(1):51-80.
  7. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735- 1780, November.
  8. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. CoRR, abs/1508.01991. · arXiv
  9. Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. 2014. An Efficient Approach for Assessing Hyperparam- eter Importance. In Proceedings of the 31st International Conference on International Conference on Machine Learning -Volume 32, ICML'14, pages I-754-I-762. JMLR.org.
  10. Frank Hutter, Jörg Lücke, and Lars Schmidt-Thieme. 2015. Beyond Manual Tuning of Hyperparameters. KI - Künstliche Intelligenz, 29(4):329-337.
  11. Alexandros Komninos and Suresh Manandhar. 2016. Dependency based embeddings for sentence classification tasks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1490-1500, San Diego, California, June. Association for Computational Linguistics.
  12. Omer Levy and Yoav Goldberg. 2014. Dependency-Based Word Embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers, pages 302-308.
  13. Qi Li, Heng Ji, and Liang Huang. 2013. Joint Event Extraction via Structured Prediction with Global Features. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 73-82, Sofia, Bulgaria, August. Association for Computational Linguistics.
  14. Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF. CoRR, abs/1603.01354. · arXiv
  15. Héctor Martínez Alonso and Barbara Plank. 2017. When is multitask learning effective? semantic sequence pre- diction under varying data conditions. In In EACL 2017 (long paper). Association for Computational Linguistics (ACL).
  16. Tomáš Mikolov. 2012. Statistical language models based on neural networks. Ph.D. thesis, Brno University of Technology.
  17. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the Difficulty of Training Recurrent Neural Networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning -Volume 28, ICML'13, pages III-1310-III-1318. JMLR.org.
  18. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word repre- sentation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
  19. M. Schuster and K.K. Paliwal. 1997. Bidirectional Recurrent Neural Networks. Trans. Sig. Proc., 45(11):2673- 2681, November.
  20. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res., 15(1):1929-1958, January.
  21. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich Part-of-speech Tagging with a Cyclic Dependency Network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology -Volume 1, NAACL 2003, pages 173-180, Stroudsburg, PA, USA. Association for Computational Linguistics.
  22. Naushad UzZaman, Hector Llorens, James F. Allen, Leon Derczynski, Marc Verhagen, and James Pustejovsky. 2012. TempEval-3: Evaluating Events, Time Expressions, and Temporal Relations. CoRR, abs/1206.5333. · arXiv

Cited by

Sorted by citations

See all 7

Metadata

Authors
Reimers, N., & Gurevych, I.
Venue
arXiv
Publisher
arXiv
Published
2017
DOI
10.48550/arxiv.1707.06799
arXiv
1707.06799
References
13
Type
Preprint
Licence
creativecommons.org/licenses/by-sa/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review