Japanese and Korean voice search
The DOI matches its Crossref record; 7 of the 8 references it cites were found in a registry or an index.
Abstract
This paper describes challenges and solutions for building a successful voice search system as applied to Japanese and Korean at Google. We describe the techniques used to deal with an infinite vocabulary, how modeling completely in the written domain for language model and dictionary can avoid some system complexity, and how we built dictionaries, language and acoustic models in this framework. We show how to deal with the difficulty of scoring results for multiple script languages because of ambiguities. The development of voice search for these languages led to a significant simplification of the original process to build a system for any new language which in in parts became our default process for internationalization of voice search.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“LogRobust [8], which encodes log templates into semantic vectors using the FastText pre-trained model [31], cannot work well on 2 out of 4 datasets.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [8]
In the citing paper
“Their empirical study on a Microsoft online service system shows that up to 30.3% logs are changed in the latest version.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [8]
In the citing paper
“Zhang et al. [8] indicated that log data is unstable, meaning that new log events often appear due to software evolution at its lifetime.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [8]
In the citing paper
“Another option is to use domain knowledge to reduce the input space of a model — such strategies include patch embeddings [Dosovitskiy et al. 2020] as well as fixed or learned tokenization [Schuster and Nakajima 2012; Sennrich et al. 2016; Kudo and Richardson 2018; van den Oord et al. 2017; Ramesh et al. 2021]. These methods may alleviate some challenges facing generative approaches, but have the”
From On the Opportunities and Risks of Foundation Models (2021) · cites this paper as Schuster and Nakajima 2012
Citations
Cites
Semi-tied covariance matrices for hidden Markov models
Gales, M. J. F. (1999) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.759034 · Cited by 537
Boosted MMI for model and feature-space discriminative training
Povey, D., Kanevsky, D., Kingsbury, B., Ramabhadran, B., Saon, G., & Visweswariah, K. (2008) · 2008 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2008.4518545 · Cited by 341
“Your Word is my Command”: Google Search by Voice: A Case Study
Schalkwyk, J., Beeferman, D., Beaufays, F., Byrne, B., Chelba, C., Cohen, M., Kamvar, M., & Strope, B. (2010) · Advances in Speech Recognition · doi:10.1007/978-1-4419-5951-5_4 · Cited by 232
Building transcribed speech corpora quickly and cheaply for many languages
Hughes, T., Nakajima, K., Ha, L., Vasu, A., Moreno, P. J., & LeBeau, M. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-551 · Cited by 75
Self-citationGoogle Search by Voice: A Case Study
Schalkwyk, J., Beeferman, D., Beaufays, F., Byrne, B., Chelba, C. I., Cohen, M., Garrett, M., & Strope, B. P. (2010) · Cited by 63
Query language modeling for voice search
Chelba, C. I., Schalkwyk, J., Brants, T., Ha, V., Harb, B., Neveitt, W., Parada, C., & Xu, P. (2010) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2010.5700834 · Cited by 28
Search by voice in Mandarin Chinese
Shan, J., Wu, G., Hu, Z., Tang, X., Jansche, M., & Moreno, P. J. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-129 · Cited by 18
Research works printed but not linked: 1
- sung, 0, Deploying Search by Voice in Cantonese, Proc Interspeech 2011
Cited by
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
“Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google’s phrase-based production system.”
ResultSelf-citationSpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558
“The WPM for Lib-riSpeech 960h is constructed using the training set transcripts.”
MethodInfluentialConvolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369
“Most recent work [51, 58, 17, 42] uses the better performing word pieces [50] for both models.”
MethodA Time Series is Worth 64 Words: Long-term Forecasting with Transformers
Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.14730 · Cited by 548
“In NLP, BERT (Devlin et al., 2018) considers subword-based tokenization (Schuster & Nakajima, 2012) instead of performing character-based tokenization.”
MethodDeep Learning--based Text Classification: A Comprehensive Review
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., & Gao, J. (2022) · ACM Computing Surveys · doi:10.1145/3439726 · Cited by 1,538
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., et al. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00065 · Cited by 1,747
“hemodelwithallmultilingualdataconsistingof multiplelanguagepairsatonce,possiblyafterover-orundersamplingsomeofthedatatoadjustforthe relativeratioofthelanguagedataavailable. Weuseasharedwordpiecemodel [18,21]acrossallthesource andtargetdatausedfortraining,usuallywith32,000wordpieces. Alltrainingiscarriedoutasinourgener…”
BackgroundSelf-citationA Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
“WordPiece [41]: It was introduced in [41] as a novel text segmentation technique for Japanese and Korean languages to improve the language model for voice search systems.”
CamemBERT: a Tasty French Language Model
Martin, L., Muller, B., Ortiz Suárez, P. J., Dupont, Y., Romary, L., de la Clergerie, É., Seddah, D., & Sagot, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.645 · Cited by 754
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models
Chiu, C.-C., Sainath, T. N., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss, R. J., Rao, K., Gonina, E., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462105 · Cited by 1,043
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
Kudo, T. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1007 · Cited by 1,021
“3 Subword segmentations with language model 3.1 Byte-Pair-Encoding (BPE) Byte-Pair-Encoding (BPE) (Sennrich et al., 2016; Schuster and Nakajima, 2012) is a subword segmentation algorithm widely used in many NMT systems 1 .”
BackgroundMethodInfluential(2024) · doi:10.1109/access.2024.3365742 · Cited by 934
“This algorithm has distinct technique for tokenizing from the input and then, applied for the specific tasks [47], [48], [49].”
MethodBERTimbau: Pretrained BERT Models for Brazilian Portuguese
Souza, F., Nogueira, R., & Lotufo, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-61377-8_28 · Cited by 647
Streaming End-to-end Speech Recognition for Mobile Devices
He, Y., Sainath, T. N., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., et al. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682336 · Cited by 601
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation
Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G., Jones, L., Schuster, M., Shazeer, N., Parmar, N., et al. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1008 · Cited by 564
“Each sentence was encoded into a sequence of sub-word units obtained by first tokenizing the sentence with the Moses tokenizer, then splitting tokens into subword units (also known as “wordpieces”) using the approach described in (Schuster and Nakajima, 2012).”
MethodSelf-citationMuRIL: Multilingual Representations for Indian Languages
Khanuja, S., Bansal, D., Mehtani, S., Khosla, S., Dey, A., Gopalan, B., Margam, D. K., Aggarwal, P., Nagipogu, R. T., Dave, S., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.10730 · Cited by 158
“Vocabulary: We learn a cased WordPiece (Schuster and Nakajima, 2012; Wu et al., 2016) vocabulary from the upsampled pre-training data using the wordpiece vocabulary generation”
MethodNeural Machine Translation: A Review
Stahlberg, F. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12007 · Cited by 397
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic
Abdul-Mageed, M., Elmadany, A., & Nagoudi, E. M. B. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.551 · Cited by 397
“Multi-lingual models such as mBERT and XLM-R have 5 K (out of 110 K) and 14 K (out of 250 K) Arabic WordPieces, respectively, in their vocabularies.”
MethodInfluentialRecent Advances in End-to-End Automatic Speech Recognition
Li, J. (2022) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000050 · Cited by 395
Large scale distributed neural network training through online distillation
Anil, R., Pereyra, G., Passos, A., Ormandi, R., Dahl, G. E., & Hinton, G. E. (2018) · arXiv · doi:10.48550/arxiv.1804.03235 · Cited by 129
“We used 256 dimensional input embeddings and a vocabulary of 24006 word pieces (Schuster & Nakajima, 2012), including sentence and paragraph start and end tokens, out of vocabulary (generally non-English characters), and end of document.”
MethodLiT: Zero-Shot Transfer with Locked-image text Tuning
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01759 · Cited by 402
“BERT uses a WordPiece (WP) tokenizer [49, 64], and all others use the SentencePiece (SP) tokenizer [34], a component which we also ablate with the ViT model.”
MethodZampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., & Çöltekin, Ç. (2020) · Proceedings of the Fourteenth Workshop on Semantic Evaluation · doi:10.18653/v1/2020.semeval-1.188 · Cited by 378
Koehn, P. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108608480 · Cited by 236
Deep Learning Based Text Classification: A Comprehensive Review
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Asgari-Chenaghlu, M., & Gao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03705 · Cited by 381
Log-based Anomaly Detection Without Log Parsing
Le, V.-H., & Zhang, H. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678773 · Cited by 361
“In our work, we adopt the WordPiece tokenization [36], [37], which is widely used in many recent language modeling studies [38], [39], [40].”
MethodInfluentialA Survey of Multilingual Neural Machine Translation
Dabre, R., Chu, C., & Kunchukuttan, A. (2021) · ACM Computing Surveys · doi:10.1145/3406095 · Cited by 294
“” To overcome the problem of unknown words, the most common practice involves subword tokenization using methods such as byte-pair encoding (BPE) [131], word-piece model (WPM) [127], or sentence-piece model (SPM) [82].”
MethodGECToR – Grammatical Error Correction: Tag, Not Rewrite
Omelianchuk, K., Atrasevych, V., Chernodub, A., & Skurzhanskyi, O. (2020) · Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2020.bea-1.16 · Cited by 274
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey
Qu, G., Chen, Q., Wei, W., Lin, Z., Chen, X., & Huang, K. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3527641 · Cited by 235
“First, input data, such as words or sentences, are segmented into token sequences through a process called tokenization, using tokenizers like WordPiece [56], [57] or Byte-Pair Encoding [51], [58].”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
Zhang, Y., Qin, J., Park, D., Han, W., Chiu, C., Pang, R., Le, Q. V., & Wu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10504 · Cited by 190
“A 1024-token word-piece-model (WPM) [52] constructed from the transcripts of the LibriSpeech training set is used to tokenize the transcripts.”
MethodBetween words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPPreprint
Mielke, S. J., Alyafeai, Z., Salesky, E., Raffel, C., Dey, M., Gallé, M., Raja, A., Si, C., Lee, W. Y., Sagot, B., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.10508 · Cited by 384
“A very similar technique had been proposed under the name “WordPiece” by Schuster and Nakajima (2012) for Japanese and Korean text (where reliance on space-separated tokens is impossible as text is written without spaces), though it is also used in BERT (Devlin et al., 2018) and other models.”
MethodEnhancing clinical concept extraction with contextual embeddings
Si, Y., Wang, J., Xu, H., & Roberts, K. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz096 · Cited by 322
“One important notable difference between BERT and both ELMo and the traditional word embeddings is that BERT breaks words down into subword tokens, referred to as wordpieces (Schuster and Nakajima, 2012).”
BackgroundA Survey on Text Classification Algorithms: From Text to Predictions
Gasparetto, A., Marcuzzo, M., Zangari, A., & Albarelli, A. (2022) · Information · doi:10.3390/info13020083 · Cited by 253
Transformer-based deep learning models for the sentiment analysis of social media data
Tabinda Kokab, S., Asghar, S., & Naz, S. (2022) · Array · doi:10.1016/j.array.2022.100157 · Cited by 237
BPE-Dropout: Simple and Effective Subword Regularization
Provilkov, I., Emelianenko, D., & Voita, E. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.170 · Cited by 218
End-to-End Speech Recognition: A Survey
Prabhavalkar, R., Hori, T., Sainath, T. N., Schlüter, R., & Watanabe, S. (2024) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3328283 · Cited by 223
“Our description is agnostic to the specific representation used for decomposing the word sequence into labels; popular choices include characters, words, or sub-word sequences (e.g., BPE [32], word-pieces [37]).”
MethodChung, Y.-A., Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang, R., & Wu, Y. (2021) · 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru51503.2021.9688253 · Cited by 293
Song, X., Salcianu, A., Song, Y., Dopson, D., & Zhou, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.160 · Cited by 172
“More recently, subword tokenization techniques have become a near-universal feature of modern NLP models, including BERT (Devlin et al., 2019), GPT-3 (Brown et al., 2020), XLNet (Yang et al., 2019), etc. Common subword tokenization techniques include Byte-Pair Encoding (BPE) (Schus-ter and Nakajima, 2012; Sennrich et…”
BackgroundMethodInfluentialByte Pair Encoding is Suboptimal for Language Model Pretraining
Bostrom, K., & Durrett, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.414 · Cited by 172
Fine-grained Sentiment Classification using BERT
Munikar, M., Shakya, S., & Shrestha, A. (2019) · 2019 Artificial Intelligence for Transforming Business and Society (AITB) · doi:10.1109/aitb48515.2019.8947435 · Cited by 226
“2) Tokenization: We then tokenize the text using the WordPiece tokenizer [14].”
MethodLearning and Evaluating Contextual Embedding of Source Code
Kanade, A., Maniatis, P., Balakrishnan, G., & Shi, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00059 · Cited by 143
“We greedily compress the program vocabulary into a subword vocabulary (Schuster & Nakajima, 2012) using the SubwordTextEncoder from the Tensor2Tensor project (Vaswani et al., 2018), resulting in slightly over 50K tokens.”
MethodAn Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model
Kannan, A., Wu, Y., Nguyen, P., Sainath, T. N., Chen, Z., & Prabhavalkar, R. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462682 · Cited by 253
Improved Noisy Student Training for Automatic Speech Recognition
Park, D. S., Zhang, Y., Jia, Y., Han, W., Chiu, C.-C., Li, B., Wu, Y., & Le, Q. V. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1470 · Cited by 218
Siino, M., Tinnirello, I., & Cascia, M. L. (2023) · Information Systems · doi:10.1016/j.is.2023.102342 · Cited by 284
“Although common forms of tokenization are performed at word-level, several sub-word tokenization strategies are discussed in the literature [28–30].”
BackgroundARBERT & MARBERT: Deep Bidirectional Transformers for ArabicPreprint
Abdul-Mageed, M., Elmadany, A., & Nagoudi, E. M. B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.01785 · Cited by 260
“Multi-lingual models such as mBERT and XLM-R have 5 K (out of 110 K) and 14 K (out of 250 K) Arabic WordPieces, respectively, in their vocabularies.”
MethodInfluentialEnd-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
Synnaeve, G., Xu, Q., Kahn, J., Likhomanenko, T., Grave, É., Pratap, V., Sriram, A., Liptchinsky, V., & Collobert, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08460 · Cited by 150
“All AMs are token-based, outputting 10k word pieces [19], although this is just a choice for simplicity in this comparative study, not the result of a limitation.”
The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language ModelsPreprint
Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., & Habash, N. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06678 · Cited by 583
Croce, D., Castellucci, G., & Basili, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.191 · Cited by 202
“The building block of BERT is the Transformer (Vaswani et al., 2017), an attention-based mechanism that learns contextual relations between words (or sub-words, i.e., word pieces, (Schuster and Nakajima, 2012)) in a text.”
BackgroundScalable Second Order Optimization for Deep LearningPreprint
Anil, R., Gupta, V., Koren, T., Regan, K., & Singer, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09018 · Cited by 237
“The model makes use of a sub-word vocabulary that contains 32K word pieces (Schuster & Nakajima, 2012).”
MethodLingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Shen, J., Nguyen, P., Wu, Y., Chen, Z., Chen, M. X., Ye, J., Kannan, A., Sainath, T. N., Cao, Y. J., Chiu, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08295 · Cited by 165
“Available tokenizers include VocabFileTokenizer which uses a look-up table provided as a file, BpeTokenizer for byte pair encoding [18], and WpmTokenizer for word-piece models [17].”
MethodSelf-citationIconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language ReasoningPreprint
Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Zhou, Y., Liang, X., & Zhu, S. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13214 · Cited by 365
“We use the WordPiece [45] subword tokenizer and the resulting sequence is padded to the maximum length.”
MethodA Survey of Deep Learning Techniques for Neural Machine Translation
Yang, S., Wang, Y., & Chu, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07526 · Cited by 95
“Similarly, Wu et al. used “workpieces” concept to handle the OOV words which were once applied on the Google speech recognition system to solve Japanese/Korean segmentation problem [60] [24].”
MethodA Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency
Sainath, T. N., He, Y., Li, B., Narayanan, A., Pang, R., Bruguier, A., Chang, S.-Y., Li, W., Alvarez, R., Chen, Z., et al. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054188 · Cited by 185
“Since our E2E models directly predict word-pieces [12], we address this by including accented English data from different locales [13].”
BackgroundMethodTrewartha, A., Walker, N., Huo, H., Lee, S., Cruse, K., Dagdelen, J., Dunn, A., Persson, K. A., Ceder, G., & Jain, A. (2022) · Patterns · doi:10.1016/j.patter.2022.100488 · Cited by 163
“(BPE).(81,82) BPE relies on a pre-tokenizer that splits the training data into words.”
BackgroundGmail Smart Compose: Real-Time Assisted Writing
Chen, M. X., Lee, B. N., Bansal, G., Cao, Y., Zhang, S., Lu, J., Tsay, J., Wang, Y., Dai, A. M., Chen, Z., et al. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330723 · Cited by 169
“We use WordPiece [43] models and split the tokens into sub-word units (wordpieces).”
MethodDeep Context: End-to-end Contextual Speech Recognition
Pundak, G., Sainath, T. N., Prabhavalkar, R., Kannan, A., & Zhao, D. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639034 · Cited by 158
Zhang, Y., Park, D. S., Han, W., Qin, J., Gulati, A., Shor, J., Jansen, A., Xu, Y., Huang, Y., Wang, S., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3182537 · Cited by 161
“Two tokenization schemes are used for the transcripts: graphemes or word-piece models (WPMs) [65], the details of”
MethodLeveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation
Jia, Y., Johnson, M., Macherey, W., Weiss, R. J., Cao, Y., Chiu, C.-C., Ari, N., Laurenzo, S., & Wu, Y. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683343 · Cited by 150
BatteryBERT: A Pretrained Language Model for Battery Database Enhancement
Huang, S., & Cole, J. M. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c00035 · Cited by 135
URLTran: Improving Phishing URL Detection Using Transformers
Maneriker, P., Stokes, J. W., Lazo, E. G., Carutasu, D., Tajaddodianfar, F., & Gururajan, A. (2021) · IEEE Military Communications Conference (MILCOM) · doi:10.1109/milcom52596.2021.9653028 · Cited by 191
“The BPE models first break them URL, um , into a sequence of text tokens, TOKm , where the individual tokens may represent entire words or subwords [33, 35, 42].”
MethodLexicon-Based vs. Bert-Based Sentiment Analysis: A Comparative Study in Italian
Catelli, R., Pelosi, S., & Esposito, M. (2022) · Electronics · doi:10.3390/electronics11030374 · Cited by 189
“The tokenization and management of the out-of-vocabulary words is achieved through the WordPieceModel [69,70], which identifies the common sub-words through which the dictionary is built.”
MethodSpecaugment on Large Scale Datasets
Park, D. S., Zhang, Y., Chiu, C.-C., Chen, Y., Li, B., Chan, W., Le, Q. V., & Wu, Y. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053205 · Cited by 140
“The text is tokenized using a Word Piece Model (WPM) [27] of vocabulary size 4k.”
MethodLu, H., Ehwerhemuepha, L., & Rakovski, C. S. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01665-y · Cited by 168
“Regular tokenizing with Keras Tokenizer was performed to convert text into numbers for all models except for BERT which uses a different tokenization technique (WordPiece tokenization) [18, 28].”
MethodMinimum Word Error Rate Training for Attention-Based Sequence-to-Sequence Models
Prabhavalkar, R., Sainath, T. N., Wu, Y., Nguyen, P., Chen, Z., Chiu, C., & Kannan, A. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461809 · Cited by 201
Shallow-Fusion End-to-End Contextual Biasing
Zhao, D. X., Sainath, T. N., Rybach, D., Rondon, P., Bhatia, D. S., Li, B., & Pang, R. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1209 · Cited by 135
“The joint network is fed to a softmax layer, with either 96 units (for graphemes) or 4,096 units (for wordpieces [21]).”
MethodTwo-Pass End-to-End Speech Recognition
Sainath, T. N., Pang, R., Rybach, D., He, Y., Prabhavalkar, R., Li, W., Visontai, M., Liang, Q., Strohman, T., Wu, Y., et al. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1341 · Cited by 134
Language models and Automated Essay ScoringPreprint
Rodriguez, P. U., Jafari, A., & Ormerod, C. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09482 · Cited by 160
“The word embeddings used by BERT are WordPiece embeddings [30], which consist in a tokenization technique in which the words are split into sub-word units.”
MethodA Spelling Correction Model for End-to-end Speech Recognition
Guo, J., Sainath, T. N., & Weiss, R. J. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683745 · Cited by 167
Corpora Generation for Grammatical Error Correction
Lichtarge, J., Alberti, C., Kumar, S., Shazeer, N., Parmar, N., & Tong, S. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1333 · Cited by 163
“The word tokens are split into subwords using a variant of the byte-pair encoding technique (Sennrich et al., 2016b), described in Schuster and Nakajima (2012).”
AMR Parsing as Sequence-to-Graph Transduction
Zhang, S., Ma, X., Duh, K., & Durme, B. V. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1009 · Cited by 180
“As BERT adopts WordPiece tokenization (Schuster and Nakajima, 2012; Wu et al., 2016), one word may correspond to multiple hidden states of BERT.”
MethodYang, Y., Ábrego, G. H., Yuan, S., Guo, M., Shen, Q., Cer, D. M., Sung, Y.-H., Strope, B. P., & Kurzweil, R. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/746 · Cited by 128
Imagination improves Multimodal TranslationPreprint
Elliott, D., & Kádár, Á. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.04350 · Cited by 128
“When we perform experiments with the News Commentary corpus, we first calculate a 17,597 sub-word vocabulary us-ing SentencePiece (Schuster and Nakajima, 2012) over the concatentation of the Multi30K and News Commentary datasets.”
MethodSabet, M. J., Dufter, P., & Schütze, H. (2020) · Open access LMU (Ludwid Maxmilian's Universitat Munchen) · doi:10.5282/ubm/epub.72200 · Cited by 127
Impact of Tokenization on Language Models: An Analysis for Turkish
Toraman, Ç., Yılmaz, E. H., Şahinuç, F., & Ozcelik, O. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3578707 · Cited by 207
Small and Practical BERT Models for Sequence Labeling
Tsai, H., Riesa, J., Johnson, M., Arivazhagan, N., Li, X., & Archer, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1374 · Cited by 113
“To train a multilingual BERT model for our sequence prediction tasks, we add a softmax layer on top of the the first wordpiece (Schuster and Nakajima, 2012) of each token3 and finetune on data”
MethodDeep Learning in Sentiment Analysis: Recent Architectures
Abdullah, T., & Ahmet, A. (2022) · ACM Computing Surveys · doi:10.1145/3548772 · Cited by 134
“WordPiece [135] was first used in voice recognition problems.”
MethodSqueezeBERT: What can computer vision teach NLP about efficient neural networks?
Iandola, F., Shaw, A., Krishna, R., & Keutzer, K. (2020) · Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing · doi:10.18653/v1/2020.sustainlp-1.17 · Cited by 114
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings
Guo, M., Shen, Q., Yang, Y., Ge, H., Cer, D. M., Ábrego, G. H., Stevens, K., Constant, N., Sung, Y.-H., Strope, B. P., et al. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6317 · Cited by 111
“For each language pair, sentence pairs are segmented using a shared 32,000 wordpiece vocabulary (Schuster and Nakajima, 2012).”
MethodRecognizing Long-Form Speech Using Streaming End-to-End Models
Narayanan, A., Prabhavalkar, R., Chiu, C., Rybach, D., Sainath, T. N., & Strohman, T. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003913 · Cited by 154
“We use word-pieces [23] as output units in this work(3).”
BackgroundMethodFeature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover
Fröhling, L., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.443 · Cited by 106
Roudsari, A. H., Afshar, J., Lee, W., & Lee, S. (2021) · Scientometrics · doi:10.1007/s11192-021-04179-4 · Cited by 103
Elliott, D., Frank, S., Barrault, L., Bougares, F., & Specia, L. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4718 · Cited by 239
“The unconstrained models are trained over a concatenation of the Multi30K, News Commentary (Tiedemann, 2012) parallel texts, and MS COCO (Chen et al., 2015) dataset with a joint source-target vocabulary of 17,597 word pieces (Schuster and Nakajima, 2012).”
BackgroundMethodRnn-Transducer with Stateless Prediction Network
Ghodsi, M., Liu, X., Apfel, J., Cabrera, R., & Weinstein, E. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054419 · Cited by 101
“Its target is a sequence of sub-word symbols, represented by yu, which can be graphemes (characters) or wordpieces [14].”
BackgroundA Better and Faster end-to-end Model for Streaming ASR
Li, B., Gulati, A., Yu, J., Sainath, T. N., Chiu, C., Narayanan, A., Chang, S.-Y., Pang, R., He, Y., Qin, J., et al. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413899 · Cited by 145
“All the E2E models are trained to predict 4,096 word pieces [31].”
MethodA Primer on Pretrained Multilingual Language Models
Doddapaneni, S., Ramesh, G., Khapra, M. M., Kunchukuttan, A., & Kumar, P. (2025) · ACM Computing Surveys · doi:10.1145/3727339 · Cited by 108
“source languages get a reasonable share of the total vocabulary used by the model (while training the wordpiece (Schuster and Nakajima, 2012) or sentencepiece model (Kudo and Richardson, 2018b)).”
Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability
Li, J., Zhao, R., Meng, Z., Liu, Y., Wei, W., Parthasarathy, S., Mazalov, V., Wang, Z., He, L., Zhao, S., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3016 · Cited by 98
Multi-granularity Prediction for Scene Text Recognition
Wang, P., Da, C., & Yao, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19815-1_20 · Cited by 132
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
Tay, Y., Tran, V. Q., Ruder, S., Gupta, J., Chung, H. W., Bahri, D., Qin, Z., Baumgartner, S., Yu, C., & Metzler, D. (2021) · arXiv · doi:10.48550/arxiv.2106.12672 · Cited by 92
“State-of-the-art pre-trained language models (Devlin et al., 2019) generally rely on data-driven subword-based tokenization algorithms (Schuster and Nakajima, 2012; Sennrich et al., 2016; Wu et al., 2016; Kudo and Richardson, 2018) while expert-crafted segmentation algorithms are still common for…”
BackgroundNeural Models of Text Normalization for Speech Applications
Zhang, H., Sproat, R., Ng, A. H., Stahlberg, F., Peng, X., Gorman, K., & Roark, B. (2019) · Computational Linguistics · doi:10.1162/coli_a_00349 · Cited by 125
“The choices for T context include (from coarse to fine) words, word pieces (Schuster and Nakajima 2012; Sennrich, Haddow, and Birch 2016; Wu et al. 2016), or even characters.”
BackgroundMethodUrman, A., Ho, J. C., & Katz, S. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256675 · Cited by 102
Advancing Acoustic-to-Word CTC Model
Li, J., Ye, G., Das, A. K., Zhao, R., & Gong, Y. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462017 · Cited by 108
Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine TranslationPreprint
Eriguchi, A., Johnson, M., Fırat, O., Kazawa, H., & Macherey, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.04686 · Cited by 92
“We generated a shared sub-word vocabulary (Sennrich, Haddow, and Birch 2016; Schuster and Nakajima 2012) of 32K units from all source and target training data.”
MethodInformation Retrieval: Advanced Topics and Techniques
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 88
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
Hill, F., Mokrá, S., Wong, N., & Harley, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09382 · Cited by 31
“Ablation experiments isolate the role of WordPiece tokenization [Schuster and Nakajima, 2012] in robustness to typed human instructions.”
BackgroundEntity matching with transformer architectures - a step forward in data integration
Brunner, U., & Stockinger, K. (2020) · Zürcher Hochschule für Angewandte Wissenschaften digital collection (Zurich University of Applied Sciences) · doi:10.21256/zhaw-19637 · Cited by 83
PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS
Ye, J., Zen, H., Shen, J., Zhang, Y., & Wu, Y. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1757 · Cited by 108
Federated Learning of N-Gram Language Models
Chen, M., Suresh, A. T., Mathews, R., Wong, A., Allauzen, C., Beaufays, F., & Riley, M. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1012 · Cited by 57
“To train neural models on morphologically rich languages, subword segments such as byte-pair encodings or word-pieces (Shibata et al., 1999; Schuster and Nakajima, 2012; Kudo, 2018) are typically used.”
MethodThree Strategies to Improve One-to-Many Multilingual Translation
Wang, Y., Zhang, J., Zhai, F., Xu, J., & Zong, C. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1326 · Cited by 77
“We use wordpiece method (Wu et al., 2016; Schuster and Nakajima, 2012) to encode source side sentences and the combination of target side sentences.”
MethodMulti-SimLex: A Large-Scale Evaluation of Multilingual and Crosslingual Lexical Semantic Similarity
Vulić, I., Baker, S., Ponti, E. M., Petti, U., Leviant, I., Wing, K., Majewska, O., Bar, E., Malone, M., Poibeau, T., et al. (2020) · Computational Linguistics · doi:10.1162/coli_a_00391 · Cited by 122
“They receive subword-level tokens as inputs (such as WordPieces (Schuster and Nakajima 2012)) to tackle data sparsity.”
BackgroundDifferentiable Reasoning over a Virtual Knowledge Base
Dhingra, B., Zaheer, M., Balachandran, V., Neubig, G., Salakhutdinov, R., & Cohen, W. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10640 · Cited by 36
“The queries are encoded with a smaller BERT-like model: specifically, they are tokenized with WordPieces (Schuster & Nakajima, 2012), appended to a special [CLS] token, and then passed through a 4 -layer Transformer network (Vaswani et al., 2017) with the same architecture as BERT, producing an output sequence H q .”
Method
Metadata
- Authors
- Schuster, M., & Nakajima, K.
- Affiliations
- Google (United States) (US)
- Venue
- 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2012-03
- Pages
- 5149-5152
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2121879602
- Cited by
- 862
- References
- 8
- Citations by year
- 2026: 36 · 2025: 66 · 2024: 90 · 2023: 124 · 2022: 137 · 2021: 155 · 2020: 144 · 2019: 59 · 2018: 36 · 2017: 10 · 2016: 3 · 2013: 2
- FWCI
- 0.49
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- voice search, Japanese language, Korean language, acoustic modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.