Japanese and Korean voice search
The DOI matches its Crossref record; 7 of the 8 references it cites were found in a registry or an index.
Abstract
This paper describes challenges and solutions for building a successful voice search system as applied to Japanese and Korean at Google. We describe the techniques used to deal with an infinite vocabulary, how modeling completely in the written domain for language model and dictionary can avoid some system complexity, and how we built dictionaries, language and acoustic models in this framework. We show how to deal with the difficulty of scoring results for multiple script languages because of ambiguities. The development of voice search for these languages led to a significant simplification of the original process to build a system for any new language which in in parts became our default process for internationalization of voice search.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“LogRobust [8], which encodes log templates into semantic vectors using the FastText pre-trained model [31], cannot work well on 2 out of 4 datasets.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [8]
In the citing paper
“Their empirical study on a Microsoft online service system shows that up to 30.3% logs are changed in the latest version.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [8]
In the citing paper
“Zhang et al. [8] indicated that log data is unstable, meaning that new log events often appear due to software evolution at its lifetime.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [8]
In the citing paper
“Another option is to use domain knowledge to reduce the input space of a model — such strategies include patch embeddings [Dosovitskiy et al. 2020] as well as fixed or learned tokenization [Schuster and Nakajima 2012; Sennrich et al. 2016; Kudo and Richardson 2018; van den Oord et al. 2017; Ramesh et al. 2021]. These methods may alleviate some challenges facing generative approaches, but have the”
From On the Opportunities and Risks of Foundation Models (2021) · cites this paper as Schuster and Nakajima 2012
Citations
Cites
Semi-tied covariance matrices for hidden Markov models
Gales, M. J. F. (1999) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.759034 · Cited by 537
Boosted MMI for model and feature-space discriminative training
Povey, D., Kanevsky, D., Kingsbury, B., Ramabhadran, B., Saon, G., & Visweswariah, K. (2008) · 2008 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2008.4518545 · Cited by 341
“Your Word is my Command”: Google Search by Voice: A Case Study
Schalkwyk, J., Beeferman, D., Beaufays, F., Byrne, B., Chelba, C., Cohen, M., Kamvar, M., & Strope, B. (2010) · Advances in Speech Recognition · doi:10.1007/978-1-4419-5951-5_4 · Cited by 232
Building transcribed speech corpora quickly and cheaply for many languages
Hughes, T., Nakajima, K., Ha, L., Vasu, A., Moreno, P. J., & LeBeau, M. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-551 · Cited by 75
Self-citationGoogle Search by Voice: A Case Study
Schalkwyk, J., Beeferman, D., Beaufays, F., Byrne, B., Chelba, C. I., Cohen, M., Garrett, M., & Strope, B. P. (2010) · Cited by 63
Query language modeling for voice search
Chelba, C. I., Schalkwyk, J., Brants, T., Ha, V., Harb, B., Neveitt, W., Parada, C., & Xu, P. (2010) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2010.5700834 · Cited by 28
Search by voice in Mandarin Chinese
Shan, J., Wu, G., Hu, Z., Tang, X., Jansche, M., & Moreno, P. J. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-129 · Cited by 18
Research works printed but not linked: 1
- sung, 0, Deploying Search by Voice in Cantonese, Proc Interspeech 2011
Cited by
Items 201–300 of 933
Small and Practical BERT Models for Sequence LabelingPreprint
Tsai, H., Riesa, J., Johnson, M., Arivazhagan, N., Li, X., & Archer, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00100 · Cited by 40
“To train a multilingual BERT model for our sequence prediction tasks, we add a softmax layer on top of the the first wordpiece (Schuster and Nakajima, 2012) of each token3 and finetune on data”
MethodThe Impact of Domain-Specific Pre-Training on Named Entity Recognition Tasks in Materials SciencePreprint
Walker, N., Trewartha, A., Huo, H., Lee, S.-H., Cruse, K., Dagdelen, J. M., Dunn, A., Persson, K. A., Ceder, G., & Jain, A. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3950755 · Cited by 48
Siino, M., Tinnirello, I., & Cascia, M. L. (2025) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000107 · Cited by 38
Waterworth, D., Sethuvenkatraman, S., & Sheng, Q. Z. (2021) · Advances in Applied Energy · doi:10.1016/j.adapen.2021.100041 · Cited by 37
ConvFiT: Conversational Fine-Tuning of Pretrained Language Models
Vulić, I., Su, P.-H., Coope, S., Gerz, D., Budzianowski, P., Casanueva, I., Mrkšić, N., & Wen, T.-H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.88 · Cited by 43
“The text t is tokenized into subwords (Schuster and Nakajima, 2012) relying on each en-coder’s dedicated tokeniser.”
MethodZhao, L., Gao, W., & Fang, J. (2021) · Applied Sciences · doi:10.3390/app112210915 · Cited by 35
“To overcome the problem of unknown words, the most common practice is subword tokenization with methods such as byte-pair encoding (BPE) [25], word-piece model (WPM) [26], or sentence-piece model (SPM) [27].”
MethodTraining-free Lexical Backdoor Attacks on Language Models
Huang, Y., Zhuo, T. Y., Xu, Q., Hu, H., Yuan, X., & Chen, C. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583348 · Cited by 63
“These three tokenization methods use different strategies to learn subwords in the corpus, where both BPE and WordPiece identify subwords based on frequencies but differ from final decisions of dictionary construction, and UnigramLM solely rely on a probabilistic model instead of occurrences.”
MethodInfluentialLarge language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34
“Tokenization methods, such as Byte Pair Encoding (BPE) (Sennrich et al., 2016) and WordPiece (Schuster & Nakajima, 2012), convert text into discrete tokens, forming a foundation for subsequent processing.”
RAPPPID: towards generalizable protein interaction prediction with AWD-LSTM twin networks
Szymborski, J., & Emad, A. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac429 · Cited by 34
Improving End-to-end Speech Recognition with Pronunciation-assisted Sub-word Modeling
Xu, H., Ding, S., & Watanabe, S. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682494 · Cited by 43
He, J., Nguyen, D. Q., Akhondi, S. A., Druckenbrodt, C., Thorne, C., Hoessel, R., Afzal, Z., Zhai, Z., Fang, B., Yoshikawa, H., et al. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.654438 · Cited by 45
“Popular subword tokenizationmethods includeWordPiece (Schuster and Nakajima, 2012) and Byte Pair Encoding (BPE) (Radford et al.”
RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions
Chiu, C., Narayanan, A., Han, W., Prabhavalkar, R., Zhang, Y., Jaitly, N., Pang, R., Sainath, T. N., Nguyen, P., Cao, L., et al. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383518 · Cited by 46
“The target is represented by a sequence of word piece tokens [25], with a vocabulary size of 4096.”
BackgroundMethodNeural Language Modeling with Visual FeaturesPreprint
Anastasopoulos, A., Kumar, S., & Liao, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.02930 · Cited by 38
“We tokenize the training data using a vocabulary of 66K wordpieces (Schuster and Nakajima, 2012).”
MethodA survey on natural language processing (nlp) and applications in insurancePreprint
Ly, A., Uthayasooriyar, B., & Wang, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00462 · Cited by 42
“A lot of methods cover tokenization, from Stemming [M.F. Porter 1980] to WordPiece [Schuster and Nakajima 2012].”
MethodPUTraceAD: Trace Anomaly Detection with Partial Labels based on GNN and PU Learning
Zhang, K., Zhang, C., Peng, X., & Sha, C. (2022) · 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre55969.2022.00032 · Cited by 33
“To alleviate the problem we follow existing NLP (Natural Language Processing) studies [14] and use the WordPiece (a subword segmentation algorithm) [15] to further split the words to tokens.”
MethodSubword Pooling Makes a Difference
Ács, J., Kádár, Á., & Kornai, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.194 · Cited by 44
Too Much in Common: Shifting of Embeddings in Transformer Language Models and its Implications
Biś, D., Podkorytov, M., & Liu, X. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.403 · Cited by 73
“, 2016) and WordPiece (Schuster and Nakajima, 2012; Wu et al., 2016), suggests that subword units induced by tokenization algorithms exhibit similar frequency imbalance to that of full vocabulary.”
BackgroundMethodInfluentialHybrid model: IndoBERT and long short-term memory for detecting Indonesian hoax news
Yefferson, D. Y., Lawijaya, V., & Girsang, A. S. (2024) · IAES International Journal of Artificial Intelligence · doi:10.11591/ijai.v13.i2.pp1913-1924 · Cited by 32
“With this in mind, WordPiece tokenization ensures that the model can still process and store information from OOV words [24].”
MethodLess is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging
Prabhavalkar, R., He, Y., Rybach, D., Campbell, S. M., Narayanan, A., Strohman, T., & Sainath, T. N. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414212 · Cited by 38
Sub-Character Tokenization for Chinese Pretrained Language Models
Si, C., Zhang, Z., Chen, Y., Qi, F., Wang, X., Liu, Z., Wang, Y., Liu, Q., & Sun, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00560 · Cited by 59
“, 2016), WordPiece (Schuster and Nakajima, 2012) and unigram language model segmentation (Kudo, 2018).”
BackgroundWang, Y.-Q., Shi, Y., Zhang, F., Wu, C., Chan, J., Yeh, C.-F., & Xiao, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414087 · Cited by 38
Hofmann, V., Schuetze, H., & Pierrehumbert, J. B. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.43 · Cited by 85
“BERT/WordPiece again lies in between.”
MethodInfluentialCritical assessment of transformer-based AI models for German clinical notes
Lentzen, M., Madan, S., Lage‐Rupprecht, V., Langnickel, L., Fluck, J., Jacobs, M. D., Mittermaier, M., Witzenrath, M., Brunecker, P., Hofmann‐Apitius, M., et al. (2022) · JAMIA Open · doi:10.1093/jamiaopen/ooac087 · Cited by 37
“For the ELECTRA models, we used our biomedical corpus to generate a new vocabulary for the WordPiece tokenizer.(43) Then, we initialized 2 new ELECTRA models in both the small and base configurations and subsequently trained both with the hyperparameters specified in Supplementary Table B.”
MethodLanguage models for the prediction of SARS-CoV-2 inhibitors
Blanchard, A. E., Gounley, J., Bhowmik, D., Chandra Shekar, M., Lyngaas, I., Gao, S., Yin, J., Tsaris, A., Wang, F., & Glaser, J. (2022) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420221121804 · Cited by 30
Tyagi, S., Nicolis, M., Rohnke, J., Drugman, T., & Lorenzo-Trueba, J. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1411 · Cited by 30
Segura-Bédmar, I., Camino-Perdones, D., & Guerrero‐Aspizua, S. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04810-y · Cited by 30
“The wordpiece tokenizer first splits the text into tokens, and then the rare words are broken into smaller meaningful words (named wordpieces) [64].”
UniASM: Binary code similarity detection without fine-tuning
Gu, Y., Shu, H., Kang, F., & Hu, F. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129646 · Cited by 34
“designed for assembly code and two tokenization methods (Byte-Pair Encoding [74] and Word-Piece [75]) designed for natural language.”
Learning to Segment Inputs for NMT Favors Character-Level ProcessingPreprint
Kreutzer, J., & Sokolov, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.01480 · Cited by 32
“Two subword segmentation methods are presently the state-of-the-art in NMT: the byte-pair encoding (BPE), that starts with a dictionary of single characters and iteratively creates a new entry from the two currently most frequent entries [3, 4], and a similar wordpiece (WP) model [5].”
MethodAccelerating Transformer Inference for Translation via Parallel Decoding
Santilli, A., Severino, S., Postolache, E., Maiorca, V., Mancusi, M., Marin, R., & Rodola, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.689 · Cited by 29
“Source and target sentences are generally tokenized in words or subwords (Kudo and Richardson, 2018; Schuster and Nakajima, 2012; Sennrich et al., 2016; Kudo, 2018); here, we use the subfix notation x = ( x 1 , . . . , x n ) and y = ( y 1 , . . . , y m ) to indicate specific tokens in the sequence.”
MethodThe Evolution of AI: From Classical Machine Learning to Modern Large Language Models
Mundlamuri, R., Gunnam, G. R., Mysari, N. K., & Pujuri, J. (2025) · IEEE Access · doi:10.1109/access.2025.3621344 · Cited by 43
“Variants of BPE, such as WordPiece introduced by Schuster and Nakajima for voice search, were adopted widely, notably by BERT and GPT models [33].”
Learning variable length units for SMT between related languages via Byte Pair Encoding
Kunchukuttan, A., & Bhattacharyya, P. (2017) · Workshop on Subword and Character Level Models in NLP (SCLeM) · doi:10.18653/v1/w17-4102 · Cited by 29
“, 2016), Huffman encoding based units (Chitnis and DeNero, 2015), wordpieces (Schuster and Nakajima, 2012; Wu et al., 2016).”
BackgroundRobust Prediction of Punctuation and Truecasing for Medical ASR
Sunkara, M., Ronanki, S., Dixit, K., Bodapati, S., & Kirchhoff, K. (2020) · doi:10.18653/v1/2020.nlpmc-1.8 · Cited by 29
Comparison of neutrosophic approach to various deep learning models for sentiment analysis
Sharma, M., Kandasamy, I., & Vasantha, W. B. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107058 · Cited by 29
“Some popular tokenization algorithms PLM’s use are Byte-Pair Encoding (BPE) [57], Word-Piece [58], and SentencePiece [59].”
MethodHu, H., Yang, X., Raeesy, Z., Guo, J., Keskin, G., Arsikere, H., Rastrow, A., Stolcke, A., & Maas, R. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414291 · Cited by 38
“The softmax layer consists of 10K output units and is trained to predict word-piece tokens, which are generated using the byte pair encoding algorithm [22].”
MethodWhat do tokens know about their characters and how do they know it?
Kaushal, A., & Mahowald, K. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.179 · Cited by 57
MedBERT: A Pre-trained Language Model for Biomedical Named Entity Recognition
Vasantharajan, C., Tun, K. Z., Ho, T.-N., Jain, S., Rong, T., & Siong, C. E. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9980157 · Cited by 44
Fine-tuning transformers: Vocabulary transfer
Mosin, V., Samenko, I., Kozlovskii, B. M., Tikhonov, A., & Yamshchikov, I. P. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103860 · Cited by 36
“In particular, [18] mention that WordPiece [19] has a small advantage over BPE.”
BackgroundHarnessing the Power of General-Purpose LLMs in Hardware Trojan Design
Kokolakis, G., Moschos, A., & Keromytis, A. D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61486-6_11 · Cited by 39
“Depending on the underlying algorithm, the representation of a token can be a word [4], a sub-word [51, 52, 59] or a character [28].”
BackgroundExploiting Multilingual Corpora Simply and Efficiently in Neural Machine Translation
Dabre, R., Cromierès, F., & Kurohashi, S. (2018) · Journal of Information Processing · doi:10.2197/ipsjjip.26.406 · Cited by 28
Multimodal Semi-Supervised Learning Framework for Punctuation Prediction in Conversational Speech
Sunkara, M., Ronanki, S., Bekal, D., Bodapati, S., & Kirchhoff, K. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3074 · Cited by 31
Improving the Prosody of RNN-Based English Text-To-Speech Synthesis by Incorporating a BERT Model
Kenter, T., Sharma, M. K., & Clark, R. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1430 · Cited by 45
“For each syllable in the utterance it encodes 1) the prosodic features encoded frame by frame (by the blue frame-level RNN in the lower left of the figure), 2) the phone-level features encoded by the phoneme level RNN (in purple, in the lower right), 3) syllable-level features 4) BERT WordPiece embeddings [25], and 5)…”
MethodSellam, T., Pu, A., Chung, H. W., Gehrmann, S., Tan, Q., Freitag, M., Das, D., & Parikh, A. P. (2020) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2020.wmt-1.102 · Cited by 47
Jaume-Santero, F., Bornet, A., Valery, A., Naderi, N., Alvarez, D. V., Proios, D., Yazdani, A., Bournez, C., Fessard, T. C., & Teodoro, D. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c01407 · Cited by 29
Rethinking tokenization: Crafting better tokenizers for large language models
Yang, J. (2024) · International Journal of Chinese Linguistics · doi:10.1075/ijchl.00023.yan · Cited by 37
Multilingual Speech Recognition with Self-Attention Structured Parameterization
Zhu, Y., Haghani, P., Tripathi, A., Ramabhadran, B., Farris, B., Xu, H., Han, L., Sak, H., Leal, I., Gaur, N., et al. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2847 · Cited by 39
“We train this model to output word-piece [32] units in all our experiments.”
MethodPromises and perils of using Transformer-based models for SE research
Xiao, Y., Zuo, X., Lu, X., Dong, J. S., Cao, X., & Beschastnikh, I. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107067 · Cited by 28
Topic level summary generation using BERT induced Abstractive Summarization Model
Ramina, M., Darnay, N., Ludbe, C., & Dhruv, A. (2020) · International Conference on Intelligent Computing and Control Systems (ICICCS) · doi:10.1109/iciccs48265.2020.9120997 · Cited by 33
“In BERT this is done by WordPiece tokenization [11].”
MethodAutomatic Speech Recognition: A Comprehensive Survey
Rista, A., & Kadriu, A. (2020) · SEEU Review · doi:10.2478/seeur-2020-0019 · Cited by 29
“To improve performance, this framework uses the sub-word units as a word-piece model, ranging from graphemes all the way up to entire words (Schuster and Nakajima, 2012).”
MethodInvestigating cross-lingual training for offensive language detection
Pelicon, A., Shekhar, R., Škrlj, B., Purver, M., & Pollak, S. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.559 · Cited by 39
“BERT-based models use WordPiece ( Schuster & Nakajima, 2012 ; Wu et al., 2016 ) to create the vocabulary.”
MethodLarge-Scale ASR Domain Adaptation Using Self- and Semi-Supervised Learning
Hwang, D., Misra, A., Huo, Z., Siddhartha, N., Garg, S., Qiu, D., Sim, K. C., Strohman, T., Beaufays, F., & He, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746719 · Cited by 27
Catelli, R., Bevilacqua, L., Mariniello, N., di Carlo, V. S., Magaldi, M., Fujita, H., Pietro, G. D., & Esposito, M. (2023) · IEEE Access · doi:10.1109/access.2023.3277490 · Cited by 27
“In addition, the use of the WordPieceModel-based tokenizer [45], which divides words into common subwords, allows better handling of the out-of-vocabulary (OOV) word problem, as well as also reducing the size of the vocabulary itself.”
MethodA Comprehensive Analysis of Various Tokenizers for Arabic Large Language Models
Qarah, F., & Alsanoosy, T. (2024) · Applied Sciences · doi:10.3390/app14135696 · Cited by 35
Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., & Çöltekin, Ç. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3950378 · Cited by 27
Williams, E., Rodrigues, P., & Novak, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02431 · Cited by 30
“[22], and was trained on Wikipedia and the BooksCorpus [30].”
MethodDistilling Large Language Models into Tiny and Effective Students using pQRNNPreprint
Kaliamoorthi, P., Siddhant, A., Li, E., & Johnson, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.08890 · Cited by 29
“It uses an embedding table with a wordpiece (Schuster and Nakajima, 2012) vocabulary of 110K resulting in approximately 162 million parameters.”
MethodFinding Generalizable Evidence by Learning to Convince Q&A Models
Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., & Cho, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1244 · Cited by 27
Transformer Based Deliberation for Two-Pass Speech Recognition
Hu, K., Pang, R., Sainath, T. N., & Strohman, T. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383497 · Cited by 26
“The RNN-T decoder is trained to predict 4,096 mixed-case wordpieces [28].”
BackgroundMethodExperience of neural machine translation between Indian languages
Dewangan, S., Alva, S., Joshi, N., & Bhattacharyya, P. (2021) · Machine Translation · doi:10.1007/s10590-021-09263-3 · Cited by 32
“Instead of using words as input and output tokens during translation, Sennrich et al. (2016) found that subword models not only make the translation process simpler but also address the OOV problem2.”
BackgroundMethodPre-trained transformer-based language models for Sundanese
Wongso, W., Lucky, H., & Suhartono, D. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00590-7 · Cited by 29
Features or Spurious Artifacts? Data-centric Baselines for Fair and Robust Hate Speech Detection
Ramponi, A., & Tonelli, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.221 · Cited by 52
“For segmenting texts into tokens, we rely on the training portion of each dataset only (Section 5) and employ the WordPiece (Schuster and Nakajima, 2012) subword tokenizer as used in BERT (Devlin et al.”
MethodKunchukuttan, A., & Bhattacharyya, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08925 · Cited by 26
“Such subwords were first proposed in the context of Neural MT: BPE (Sennrich et al., 2016), wordpieces (Schuster and Nakajima, 2012; Wu et al., 2016), Huffman encodings (Chitnis and DeNero, 2015).”
BackgroundPushing the Limits of Non-Autoregressive Speech Recognition
Ng, E. G., Chiu, C., Zhang, Y., & Chan, W. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-337 · Cited by 30
“All other models use a 1024-token subword vocabulary [29] constructed from the LibriSpeech 960h training set.”
MethodEnhancement of Text Analysis Using Context-Aware Normalization of Social Media Informal Text
Khan, J., & Lee, S. (2021) · Applied Sciences · doi:10.3390/app11178172 · Cited by 25
“The WordPiece algorithms ensure normalization of many OOV words by its segmentation mechanism to allow only the valid vocabulary sequences (characters combination) [77].”
MethodA Brief Overview of Universal Sentence Representation Methods: A Linguistic View
Li, R., Zhao, X. H., & Moens, M. (2022) · ACM Computing Surveys · doi:10.1145/3482853 · Cited by 48
“The second is the sub-word tokenization methods [72, 73, 121, 126] in which a rare word is broken into several pre-learned sub-word units.”
MethodNamed entity recognition in Turkish: A comparative study with detailed error analysis
Ozcelik, O., & Toraman, Ç. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103065 · Cited by 31
Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contexts
Zhang, Y., Zhang, Y., Michalski, M., Jiang, Y., Yu, M., & Han, J. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3570475 · Cited by 25
“[36, 37], and each word piecew j will have an output representation”
BackgroundFilling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach
Lazar, K., Saret, B., Yehudai, A., Horowitz, W., Wasserman, N., & Stanovsky, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.384 · Cited by 25
“To adapt M-BERT to Akkadian, we assign its 99 available free tokens, optimizing for maximum likelihood by the WordPiece tokenization algorithm (Schuster and Nakajima, 2012; Wu et al., 2016).”
MethodProceedings of the 6th Workshop on Asian Translation
(2019) · doi:10.18653/v1/d19-52 · Cited by 25
Analyzing Encoded Concepts in Transformer Language Models
Sajjad, H., Durrani, N., Dalvi, F., Alam, F., Khan, A., & Xu, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.225 · Cited by 24
Badalotti, D., Agrawal, A., Pensato, U., Angelotti, G., & Marcheselli, S. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105626 · Cited by 27
Deception detection in text and its relation to the cultural dimension of individualism/collectivism
Papantoniou, K., Papadakos, P., Patkos, T., Flouris, G., Androutsopoulos, I., & Plexousakis, D. (2021) · Natural Language Engineering · doi:10.1017/s1351324921000152 · Cited by 29
Learning Disentangled Textual Representations via Statistical Measures of Similarity
Colombo, P., Staerman, G., Noiry, N., & Piantanida, P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.187 · Cited by 26
Fast Vocabulary Transfer for Language Model Compression
Gee, L., Zugarini, A., Rigutini, L., & Torroni, P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.41 · Cited by 60
“The tokenizer (Sennrich et al., 2016; Schuster and Nakajima, 2012; Kudo and Richardson, 2018) is a crucial part of modern LMs.”
BackgroundEffective Parallel Corpus Mining using Bilingual Sentence EmbeddingsPreprint
Guo, M., Shen, Q., Yang, Y., Ge, H., Cer, D. M., Ábrego, G. H., Stevens, K., Constant, N., Sung, Y.-H., Strope, B. P., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11906 · Cited by 24
“For each language pair, sentence pairs are segmented using a shared 32,000 wordpiece vocabulary (Schuster and Nakajima, 2012).”
MethodCorpora Generation for Grammatical Error CorrectionPreprint
Lichtarge, J., Alberti, C., Kumar, S., Shazeer, N., Parmar, N., & Tong, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05780 · Cited by 28
“The word tokens are split into subwords using a variant of the byte-pair encoding technique (Sennrich et al., 2016b), described in Schuster and Nakajima (2012).”
Emitting Word Timings with End-to-End Models
Sainath, T. N., Pang, R., Rybach, D., García, B., & Strohman, T. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1059 · Cited by 23
“Both decoders are trained to predict 4,096 word pieces [12], which are derived from a large corpus of text.”
BackgroundA survey on text classification: Practical perspectives on the Italian language
Gasparetto, A., Zangari, A., Marcuzzo, M., & Albarelli, A. (2022) · PLoS ONE · doi:10.1371/journal.pone.0270904 · Cited by 23
“The WordPiece tokenizer [16] was initially developed for Japanese text segmentation problems, and relies on the creation of n -gram-based language models (in the classical sense, as we describe later in the Text representation section) to recognize recurring syllables, prefixes and word segments in…”
JOIST: A Joint Speech and Text Streaming Model for ASR
Sainath, T. N., Prabhavalkar, R., Bapna, A., Zhang, Y., Huo, Z., Chen, Z., Li, B., Wang, W., & Strohman, T. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022774 · Cited by 39
“All models use the Hybrid Au-toregressive Transducer (HAT) factorization [48] to predict 4,096 word pieces [49].”
BackgroundMethodGRiT: A Generative Region-to-Text Transformer for Object Understanding
Wu, J., Wang, J., Yang, Z., Gan, Z., Liu, Z., Yuan, J., & Wang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72989-8_12 · Cited by 23
Weakly Supervised Grammatical Error Correction using Iterative DecodingPreprint
Lichtarge, J., Alberti, C., Kumar, S., Shazeer, N., & Parmar, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01710 · Cited by 27
“2 We use 6 layers for both the encoder and the de-coder, 8 attention heads, a dictionary of 32k word pieces (Schuster and Nakajima, 2012), embedding size d model = 1024 , a position-wise feed forward network at every layer of inner size d ff = 4096 , and Adafactor as optimizer with inverse squared…”
MethodExploring Benefits of Transfer Learning in Neural Machine TranslationPreprint
Kocmi, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.01622 · Cited by 27
“It is similar to BPE and is based on an algorithm developed by Schuster and Nakajima (2012). Wu et al.”
MethodAsynchronous Real-Time Federated Learning for Anomaly Detection in Microservice Cloud Applications
Raeiszadeh, M., Ebrahimzadeh, A., Glitho, R., Eker, J., & Mini, R. A. F. (2025) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2025.3527919 · Cited by 27
Knowledge Transfer from Large-Scale Pretrained Language Models to End-To-End Speech Recognizers
Kubo, Y., Karita, S., & Bacchiani, M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746801 · Cited by 36
Languages Through the Looking Glass of BPE Compression
Gutierrez-Vasques, X., Bentz, C., & Samardżić, T. (2023) · Computational Linguistics · doi:10.1162/coli_a_00489 · Cited by 44
Huber, E., Sauppe, S., Isasi-Isasmendi, A., Bornkessel‐Schlesewsky, I., Merlo, P., & Bickel, B. (2023) · Neurobiology of Language · doi:10.1162/nol_a_00121 · Cited by 34
“Both BERT and RoBERTa use subword tokenisers: WordPiece (Schuster & Nakajima, 2012) for BERT and Byte-Pair Encoding (Sennrich et al., 2016) for RoBERTa.”
MethodHomogeneous tokenizer matters: Homogeneous visual tokenizer for remote sensing image understanding
Shao, R., Zhang, Z., Tao, C., Zhang, Y., Peng, C., & Li, H. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.09.009 · Cited by 25
“The most popular NL tokenizers, such as Word2Vec[13], Byte-Pair En-coding (BPE)[47], WordPiece[48], Unigram[16], and others[14, 15], belong to the category of word tokenizers or subword tokenizers, which use words or sub-words as the basic elements of language.”
MethodAn Effective Non-Autoregressive Model for Spoken Language Understanding
Cheng, L., Jia, W., & Yang, W. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482229 · Cited by 24
“Considering that SNIPS contains much more Out-of-Vocabulary (OOV) tokens, we attribute the improvement of BERT to alleviating the OOV problem by the WordPiece encoding [26].”
MethodKPI-EDGAR: A Novel Dataset and Accompanying Metric for Relation Extraction from Financial Documents
Deußer, T., Ali, S. M., Hillebrand, L., Nurchalifah, D., Jacob, B., Bauckhage, C., & Sifa, R. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00254 · Cited by 22
“Given a WordPiece [44] tokenized input sentence we leverage a pre-trained BERT [27] model to obtain our encoded token embedding.”
MethodProbing and Fine-tuning Reading Comprehension Models for Few-shot Event ExtractionPreprint
Feng, R., Yuan, J., & Zhang, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11325 · Cited by 25
“ment prediction and question answering. BERT [8] model is a masked language model that consists of deep multihead attention layers. Given a pair of input sentences �(1),�(2), BERTfirst runs WordPiece [38] tokenizer to tokenize the both sentences into a sequence of token ids: [�(�) 1 ,...,� (�) � � ],�= 1,2. Then, two…”
The Foundations of Tokenization: Statistical and Computational ConcernsPreprint
Gastaldi, J. L., Terilla, J., Malagutti, L., DuSell, B., Vieira, T., & Cotterell, R. (2024) · arXiv (Cornell University) · doi:10.3929/ethz-b-000709079 · Cited by 39
Chung, P., Fong, C. T., Walters, A. M., Yetişgen, M., & O’Reilly-Shah, V. N. (2023) · BMC Anesthesiology · doi:10.1186/s12871-023-02248-0 · Cited by 25
Numeracy enhances the Literacy of Language Models
Thawani, A., Pujara, J., & Ilievski, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.557 · Cited by 21
“Default* is the usual way that BERT (Devlin et al., 2019) encode numbers: subword tokenization (Schuster and Nakajima, 2012) followed by lookup embeddings.”
BackgroundOverlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages
Patil, V., Talukdar, P. P., & Sarawagi, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.18 · Cited by 52
Cracking double-blind review: Authorship attribution with deep learning
Bauersfeld, L., Romero, Á., Muglikar, M., & Scaramuzza, D. (2023) · PLoS ONE · doi:10.1371/journal.pone.0287611 · Cited by 28
“The tokenizer used in our case is based on WordPiece (Schuster and Nakajima, 2012).”
MethodEmotion quantification and classification using the neutrosophic approach to deep learning
Sharma, M., Kandasamy, I., & Vasantha, W. B. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110896 · Cited by 24
Improving Context-Aware Neural Machine Translation Using Self-Attentive Sentence Embedding
Yun, H., Hwang, Y., & Jung, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6494 · Cited by 21
“For each language pair, we tokenize each text by the wordpiece model (Schuster and Nakajima 2012; Wu et al. 2016) with a vocabulary of about 16,000 tokens.”
MethodTEMP: Taxonomy Expansion with Dynamic Margin Loss through Taxonomy-Paths
Liu, Z., Xu, H., Wen, Y., Jiang, N., Wu, H., & Yuan, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.313 · Cited by 21
“Take the Word-Piece tokenization (Schuster and Nakajima, 2012) used by BERT as an example, to be in line with contextual encoders, the words in taxonomy-path P and the definition sentence S of the last node are concatenated to form the input string as shown in Figure 3.”
BackgroundMethodInfluential(2021) · Educational Data Mining · Cited by 21
Rethinking data-driven networking with foundation models: challenges and opportunities
Le, F., Srivatsa, M., Ganti, R., & Sekar, V. (2022) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3563766.3564109 · Cited by 27
“[68] Mike Schuster and Kaisuke Nakajima.”
MethodInfluential
Metadata
- Authors
- Schuster, M., & Nakajima, K.
- Affiliations
- Google (United States) (US)
- Venue
- 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2012-03
- Pages
- 5149-5152
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2121879602
- Cited by
- 862
- References
- 8
- Citations by year
- 2026: 36 · 2025: 66 · 2024: 90 · 2023: 124 · 2022: 137 · 2021: 155 · 2020: 144 · 2019: 59 · 2018: 36 · 2017: 10 · 2016: 3 · 2013: 2
- FWCI
- 0.49
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- voice search, Japanese language, Korean language, acoustic modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.