Automatic Speech Recognition With Very Large Conversational Finnish and Estonian Vocabularies

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors are aware of one earlier work where colloquial Finnish has been translated to standard Finnish using both rule-based normalization and statistical machine translation.

“We are aware of one earlier work where colloquial Finnish has been translated to standard Finnish using both rule-based normalization and statistical machine translation [22].”

• cites [22] [22] I. Listenmaa and F. M. Tyers, "Automatic conversion of colloquial Finnish to standard Finnish," in Proc. Nordic Conference of Computational Linguistics (NODALIDA), B. Megyesi, Ed. Linköping University Electronic Press / ACL, May 2015, pp. 219–223.Supported

Probably not a substantial claim: may not pass S3 (load-bearing).

Morfessor has been successfully applied in speech recognition of many agglutinative languages.

“which has been successfully applied in speech recognition of many agglutinative languages [4], [25].”

• cites [4] [4] M. Kurimo, A. Puurula, E. Arisoy, V. Siivola, T. Hirsimäki, J. Pylkkönen, T. Alumäe, and M. Saraclar, "Unlimited vocabulary speech recognition for agglutinative languages," in Proc. HLT-NAACL. Stroudsburg, PA, USA: Association for Computational Linguistics, Jun. 2006, pp. 487–494. sourcePartly supported

  The Introduction identifies Morfessor as the algorithm used, and the Discussion reports successful speech-recognition results for all three agglutinative languages tested. The paper supports the application to Finnish, Estonian and Turkish, but not the broader claim of application to “many” languages. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [25] [25] M. Creutz, T. Hirsimäki, M. Kurimo, A. Puurula, J. Pylkkönen, V. Siivola, M. Varjokallio, E. Arisoy, M. Saraçlar, and A. Stolcke, "Morph-based speech recognition and modeling of out-of-vocabulary words across languages," ACM Transactions on Speech and Language Processing (TSLP), vol. 5, no. 1, pp. 3:1–3:29, Dec. 2007. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Distributed word representations can be created quickly using shallow networks, such as the Continuous Bag-of-Words (CBOW) model.

“Distributed word representations can be created quickly using shallow networks, such as the Continuous Bag-of-Words (CBOW) model [19].”

• cites [19] [19] T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," in Proc. International Conference on Learning Representations (ICLR) Workshops, 2013. [Online]. Available: https://arxiv.org/abs/1301.3781 sourceSupported

May combine two claims.

The best previously published results are 48.4% WER in the Finnish task and 52.7% WER in the Estonian task.

“The best previously published results are 48.4 % WER in the Finnish task [11] and 52.7 % WER in the Estonian task [49].”

• cites [11] [11] S. Enarvi and M. Kurimo, "TheanoLM – an extensible toolkit for neural network language modeling," in Proc. INTERSPEECH, Sep. 2016, pp. 3052–3056. sourceSupported
• cites [49] [49] M. Kurimo, S. Enarvi, O. Tilk, M. Varjokallio, A. Mansikkaniemi, and T. Alumäe, "Modeling under-resourced languages for speech recognition," Language Resources and Evaluation (LRE), Feb. 2016. [Online]. Available: https://doi.org/10.1007/s10579-016-9336-9 sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Both data sets contain around two million unique word forms.

“Both data sets contain around two million unique word forms.”

May combine two claims.

While the authors did not have the possibility to systematically explore different network architectures, one LSTM layer followed by a highway network seemed to perform well.

“While we did not have the possibility to systematically explore different network architectures, one LSTM layer followed by a highway network seemed to perform well.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Even though the authors did not perform the experiments in such a way that they could get a fair comparison of the training times, they noticed that their exchange implementation needed less time to converge than what the Brown clustering needed to finish.

“Even though we did not perform the experiments in such a way that we could get a fair comparison of the training times, we noticed that our exchange implementation needed less time to converge than what the Brown clustering needed to finish.”

May combine two claims.

Using time delay neural network (TDNN) acoustic models, the authors achieved new state of the art in Finnish and Estonian conversational speech recognition, 27.1 % word error rate in the Finnish task and 21.9 % in the Estonian task.

“Using time delay neural network (TDNN) acoustic models, we were able to achieve new state of the art in Finnish and Estonian conversational speech recognition, 27.1 % word error rate in the Finnish task and 21.9 % in the Estonian task.”

• Also stated in: § I. INTRODUCTION · boosted

  “In this paper, we show how subword models can be used in the FST-based Kaldi speech recognition toolkit and obtain the best results to date by rescoring subword lattices using subword NNLMs, 27.1 % WER for spontaneous Finnish conversations, and 21.9 % WER for spontaneous Estonian conversations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Adding a normal 500-unit layer with hyperbolic tangent activation before the output layer for 500k-word models reduced memory consumption and speeded up training.

“With 500k words we added a normal 500-unit layer with hyperbolic tangent activation before the output layer, which reduced memory consumption and speeded up training.”

May combine two claims.

With the smaller shortlist, NCE was better on Finnish and hierarchical softmax was better on Estonian.

“With the smaller shortlist NCE was better on Finnish and hierarchical softmax was better on Estonian.”

Probably not a substantial claim: may not pass S3 (load-bearing).

First-pass decoding was very fast with a real-time factor less than 0.5.

“First-pass decoding was very fast with real-time factor less than 0.5.”

May combine two claims.

BlackOut training was slightly faster than NCE but less stable, and training could not be finished without numerical errors.

“BlackOut training was slightly faster than NCE, but even less stable, and we were unable to finish the training without numerical errors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Compiling the FST-based decoding graph from the language model, pronunciation dictionary, context dependency information, and HMM structure consumed around 60 GB of memory.

“compiling the FST-based decoding graph from the language model, pronunciation dictionary, context dependency information, and HMM structure did consume around 60 GB of memory.”

May combine two claims.

Class-based models were the fastest to converge, taking 5 to 8 days on Finnish data and 4 to 6 days on Estonian data.

“The class-based models were clearly the fastest to converge, 5 to 8 days on Finnish data and 4 to 6 days on Estonian data.”

May combine two claims.

Sampling web data improves perplexity but not word error rate in NNLM training.

“Sampling seems to improve perplexity, but not word error rate.”

May combine two claims.

For the Finnish evaluation set, the out-of-vocabulary word rate is 6.67% with a 100,000-word vocabulary, 3.36% with a 500,000-word vocabulary, and 2.31% with the full 2.4M-word vocabulary.

“100,000 6.67 3.89”

• Also stated in: § V. EXPERIMENTS · plain

  “500,000 3.36 1.59”

• Also stated in: § V. EXPERIMENTS · plain

  “2.4M (Fin) / 1.8M (Est) 2.31 1.01”

May combine two claims.

For the Estonian evaluation set, the out-of-vocabulary word rate is 3.89% with a 100,000-word vocabulary, 1.59% with a 500,000-word vocabulary, and 1.01% with the full 1.8M-word vocabulary.

“100,000 6.67 3.89”

• Also stated in: § V. EXPERIMENTS · plain

  “500,000 3.36 1.59”

• Also stated in: § V. EXPERIMENTS · plain

  “2.4M (Fin) / 1.8M (Est) 2.31 1.01”

May combine two claims.

In the Finnish task, the smallest 42.5k-subword vocabulary worked well and was small enough to use normal softmax without classes.

“In the Finnish task, the smallest 42.5k-subword vocabulary worked well, which is small enough to use normal softmax without classes.”

May combine two claims.

Training even 40,000 classes was possible in 15 days, but the results did not improve.

“Training even 40,000 classes was possible in 15 days, but the results did not improve, so they are not reported.”

May combine two claims.

Small β values flatten the distribution too much and the optimal model is not reached.

“Small β values flatten the distribution too much and the optimal model is not reached.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Using a larger beam would have improved the results, but the gain would have been small compared to the increase in decoding time.

“Using a larger beam would have improved the results, but the gain would have been small compared to the increase in decoding time.”

Probably not a substantial claim: may not pass S3 (load-bearing).

CBOW is the fastest algorithm among the compared clustering methods.

“CBOW is clearly the fastest algorithm, which is probably why it has gained some popularity.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Finnish experiments, the transcribed data received a weight slightly less than 0.5 in the mixture.

“In the Finnish experiments this gave the transcribed data a weight slightly less than 0.5.”

May combine two claims.

The NCE implementation required more GPU memory than hierarchical softmax and could not be run with the larger shortlist.

“Our NCE implementation required more GPU memory than hierarchical softmax and we were unable to run it with the larger shortlist.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Optimization of the interpolation parameters was quite difficult with the development data, so equal weight was given to both models.

“We noticed that optimization of the interpolation parameters was quite difficult with our development data, so we gave equal weight to both models.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The difference between training Morfessor on a combined word list and other options such as token-based training was negligible.

“we trained Morfessor on a word list combined from all training corpora; the difference to other options such as token-based training was negligible.”

The quote only partly states this claim.

The chosen pruning parameters n=22 and c=62 were large enough that their effect on the results was negligible.

“We performed a few tests with different pruning parameters and chose large enough n and c so that their effect in the results was negligible.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Differences between different initializations of the exchange algorithm seemed mostly insignificant.

“Mostly the differences between the different initializations of the exchange algorithm seemed insignificant.”

May combine two claims.

In the Estonian experiments, the weights of the spontaneous conversations and the web data were typically around 0.4, while the broadcasts were given a weight less than 0.2.

“In the Estonian experiments the weights of the spontaneous conversations and the web data were typically around 0.4, while the broadcasts were given a weight less than 0.2.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Increasing the number of threads increases memory consumption.

“Increasing the number of threads increases memory consumption.”

May combine two claims.

In the Estonian task, the best result was 21.9% WER, a 16.1% relative improvement from the 26.1% baseline WER.

“The best result in the Estonian task, 21.9 %, is a 16.1 % relative improvement from our 26.1 % baseline WER.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The corpus weighting methods used in NNLM training showed potential for improvement.

“The corpus weighting methods that we used in NNLM training showed potential for improvement, but more thorough research should be done on how to select optimal weights.”

May combine two claims.

In the Finnish task, the best result was 27.1% WER, a 14.5% relative improvement from the 31.7% WER given by the baseline 4-gram model.

“In the Finnish task the best result was 27.1 %, which is a 14.5 % relative improvement from the 31.7 % WER given by our baseline 4-gram model.”

May combine two claims.

These are the best results achieved in these tasks and better than the authors' previously best results by a large margin.

“These are the best results achieved in these tasks, and better than our previously best results by a large margin.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A language model can be incorporated to disambiguate between the alternative candidates when normalizing text.

“A language model can be incorporated to disambiguate between the alternative candidates when normalizing text.”

Probably not a substantial claim: may not pass S4 (based).

The pronunciation variation in colloquial Finnish is also written down, making it very difficult to reliably estimate the probability of the rare words in new contexts.

“The pronunciation variation in colloquial Finnish is also written down, making it very difficult to reliably estimate the probability of the rare words in new contexts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In a semisupervised setting, it is useful to add a hyperparameter to control the weight of the data likelihood.

“In a semisupervised setting, it is useful to add a hyperparameter to control the weight of the data likelihood [26]:”

• cites [26] [26] O. Kohonen, S. Virpioja, and K. Lagus, "Semi-supervised learning of concatenative morphology," in Proc. Meeting of the ACL Special Interest Group on Computational Morphology and Phonology. Uppsala, Sweden: Association for Computational Linguistics, July 2010, pp. 78–86.Not checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

In a class-based language model, the probability of a word within its class is usually modeled simply as the unigram probability of the word in the training data.

“In a class-based language model, the probability of a word within its class is usually modeled simply as the unigram probability of the word in the training data [15]:”

• cites [15] [15] P. F. Brown, P. V. deSouza, R. L. Mercer, V. J. D. Pietra, and J. C. Lai, "Class-based n-gram models of natural language," Computational Linguistics, vol. 18, no. 4, pp. 467–479, Dec. 1992. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based).

Much of the vocabulary in conversational Finnish text is due to colloquial Finnish pronunciations being written down phonetically.

“Much of the vocabulary in conversational Finnish text is due to colloquial Finnish pronunciations being written down phonetically.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Because all the words can appear in at most one class, multiple standard words can be merged into one class, but this is rare.

“Because all the words can appear in at most one class, multiple standard words can be merged into one class, but this is rare.”

Classified as probably not a substantial claim. May combine two claims.

Finnish and Estonian are highly agglutinative languages, so the number of different word forms that appear in training corpora is huge.

“Finnish and Estonian are highly agglutinative languages, so the number of different word forms that appear in training corpora is huge.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Morfessor is an unsupervised method that uses a statistical model to split words into smaller fragments.

“Morfessor is an unsupervised method that uses a statistical model to split words into smaller fragments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The network tends to learn similar representation for semantically similar words.

“These distributed representations are interesting on their own, because the network tends to learn similar representation for semantically similar words [18].”

• cites [18] [18] T. Mikolov, S. W. tau Yih, and G. Zweig, "Linguistic regularities in continuous space word representations," in Proc. NAACL HLT. Stroudsburg, PA, USA: Association for Computational Linguistics, Jun. 2013, pp. 746–751.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A special token such as <unk> can be used in place of any out-of-shortlist (OOS) words, which is necessary with RNN language models in particular.

“A special token such as <unk> can be used in place of any out-of-shortlist (OOS) words, which is necessary with RNN language models in particular.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The exchange algorithm is greedy so the order in which the words are iterated may affect the result.

“The exchange algorithm is greedy so the order in which the words are iterated may affect the result.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The initialization may also affect whether the optimization will get stuck in a local optimum, and how fast it will converge.

“The initialization may also affect whether the optimization will get stuck in a local optimum, and how fast it will converge.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When annotations are not available, candidate expansions for a non-standard word can be found by comparing its lexical or phonemic form to standard words.

“When annotations are not available, candidate expansions for a non-standard word can be found by comparing its lexical or phonemic form to standard words [21].”

• cites [21] [21] B. Han and T. Baldwin, "Lexical normalisation of short text messages: Makn sens a #twitter," in Proc. ACL HLT. Stroudsburg, PA, USA: Association for Computational Linguistics, Jun. 2011, pp. 368–378.Not checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Especially the subword-based approach should benefit from the capability of modeling long contexts.

“Especially the subword-based approach should benefit from this capability of modeling long contexts.”

May combine two claims.

As high-order n-grams are required to provide enough context information for subword-based modeling, the authors use variable-length n-gram models trained using the VariKN toolkit.

“As high-order n-grams are required to provide enough context information for subword-based modeling, we use variable-length n-gram models trained using the VariKN toolkit”

• cites [28] [28] V. Siivola, T. Hirsimäki, and S. Virpioja, "On growing and pruning Kneser-Ney smoothed n-gram models," IEEE Transactions on Audio, Speech & Language Processing, vol. 15, no. 5, pp. 1617–1624, 2007. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

These fragments often resemble the surface forms of morphemes, the smallest information-bearing units of a language.

“these fragments often resemble the surface forms of morphemes, the smallest information-bearing units of a language”

Classified as probably not a substantial claim.

There are often several ways to write the same word depending on how colloquial the writing style is.

“There are often several ways to write the same word depending on how colloquial the writing style is.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Subword models have been successfully used in Finnish ASR for more than a decade.

“Subword models have been successfully used in Finnish ASR for more than a decade [2].”

• cites [2] [2] T. Hirsimäki, M. Creutz, V. Siivola, M. Kurimo, S. Virpioja, and J. Pylkkönen, "Unlimited vocabulary speech recognition with morph language models applied to Finnish," Computer Speech & Language, vol. 20, no. 4, pp. 515–541, Oct. 2006. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

A balanced binary tree hierarchy for hierarchical softmax, which is log2(N) levels deep, would take logarithmic time.

“in total the hierarchical softmax would take logarithmic time. [37]”

• cites [37] [37] F. Morin and Y. Bengio, "Hierarchical probabilistic neural network lan- guage model," in Proc. International Workshop on Artificial Intelligence and Statistics (AISTATS), R. G. Cowell and Z. Ghahramani, Eds. New Jersey, USA: Society for Artificial Intelligence and Statistics, Jan. 2005, pp. 246–252.Not checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based).

Using an RNN language model in the first pass of large-vocabulary speech recognition is unrealistic.

“Nevertheless, using an RNN language model in the first pass of large-vocabulary speech recognition is unrealistic.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The memory requirement for compiling the FST-based decoding graph can be lowered by reducing the number of contexts in the first-pass n-gram model.

“The memory requirement can be lowered by reducing the number of contexts in the first-pass n-gram model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Hierarchical NNLM can reduce the training time of feedforward networks considerably because different n-grams are used to train each word prediction model.

“Hierarchical NNLM is a class-based model that consists of a neural network that predicts the class and separate neural networks that predict the word inside a class [36]. This can reduce training time of feedforward networks considerably, because different n-grams are used to train each word prediction model.”

• cites [36] [36] H.-K. Kuo, E. Arisoy, A. Emami, and P. Vozila, "Large scale hierarchical neural network language models," in Proc. INTERSPEECH. ISCA, Sep. 2012, pp. 1672–1675. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The NNLM can be combined with a large-vocabulary n-gram model to obtain a probability for every training word.

“The NNLM can be combined with a large-vocabulary n-gram model to obtain a probability for every training word.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Hierarchical softmax speeds up evaluation as well, whereas sampling-based output layers are used only during training and the output is properly normalized using softmax during inference.

“An advantage of hierarchical softmax compared to sampling based output layers is that hierarchical softmax speeds up evaluation as well, while sampling is used only during training and the output is properly normalized using softmax during inference.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It also becomes computationally demanding to normalize the resulting probability distribution correctly using a large n-gram model.

“It also becomes computationally demanding to normalize the resulting probability distribution correctly using a large n-gram model [34].”

• cites [34] [34] J. Park, X. Liu, M. J. F. Gales, and P. C. Woodland, "Improved neural network based language modelling and adaptation," in Proc. INTERSPEECH. ISCA, Sep. 2010, pp. 1041–1044. https://arxiv.org/abs/1511.06909 https://arxiv.org/abs/1301.3781 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 11, NOVEMBER 2017 13 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

When rescoring lattices, the output does not have to be a proper probability distribution.

“When rescoring lattices, the output does not have to be a proper probability distribution.”

May combine two claims.

The exchange algorithm can be stopped any time, so there is no upper limit on the number of classes that can be trained, but the quality of the clustering may suffer if it is stopped early.

“The exchange algorithm can be stopped any time, so there is no upper limit on the number of classes that can be trained, but the quality of the clustering may suffer if it is stopped early.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Memory required to store the weight matrices can be reduced by using a small projection layer and a small layer before the output layer, or by factorizing a large weight into the product of two smaller matrices.

“Memory required to store the weight matrices can be reduced by using a small projection layer and a small layer before the output layer, or factorizing a large weight into the product of two smaller matrices [35].”

• cites [35] [35] T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramab- hadran, "Low-rank matrix factorization for deep neural network training with high-dimensional output targets," in Proc. ICASSP. IEEE, May 2013, pp. 6655–6659. sourceNot checked — no full text of the source

• AssumptionRests on an assumption: Current GPU boards typically have no more than 12 GB of memory.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

An n-gram language model is convenient to integrate with a feedforward NNLM, which is a particular kind of n-gram model itself, but less trivial in the authors' RNN decoder.

“An n-gram language model is convenient to integrate with a feedforward NNLM, which is a particular kind of n-gram model itself, but less trivial in our RNN decoder.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The <unk> probability represents the total probability mass of all OOS words, which can be distributed according to n-gram language model probabilities.

“The <unk> probability represents the total probability mass of all OOS words, which can be distributed according to n-gram language model probabilities.”

May combine two claims.

Perplexities from n-gram and neural network word and class models are comparable to one another because they model the same vocabulary consisting of all the training set words.

“The perplexities from n-gram and neural network word and class models are all comparable to one another, because they model the same vocabulary consisting of all the training set words.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Rescoring a word lattice using an RNN language model is equivalent to rescoring a huge n-best list, unless some approximation is used to limit the dependency of a probability on the earlier context.

“Rescoring a word lattice using an RNN language model is equivalent to rescoring a huge n-best list, unless some approximation is used to limit the dependency of a probability on the earlier context.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Subword models can model unseen words, so the perplexities in subword experiments are higher.

“Subwords can model also unseen words, so the perplexities in subword experiments are higher.”

Classified as probably not a substantial claim.

The worst-case running times of the exchange and Brown algorithms have quadratic dependency on the number of classes.

“The worst case running times of the exchange and Brown algorithms have quadratic dependency on the number of classes.”

May combine two claims.

The shortlist and word class models can predict all training set words, so their perplexities can be compared.

“The shortlist and word class models can predict all training set words, so the perplexities can be compared.”

Classified as probably not a substantial claim.

One reason why word classes are especially beneficial in conversational tasks may be that in the absence of large conversational corpora, most training data is from the Internet.

“One reason why word classes are especially beneficial in the conversational tasks may be that in the absence of large conversational corpora, most of our training data is from the Internet.”

• AssumptionRests on an assumption: Web data is noisy and there are many ways to write the same word.