Japanese and Korean voice search
The DOI matches its Crossref record; 7 of the 8 references it cites were found in a registry or an index.
Abstract
This paper describes challenges and solutions for building a successful voice search system as applied to Japanese and Korean at Google. We describe the techniques used to deal with an infinite vocabulary, how modeling completely in the written domain for language model and dictionary can avoid some system complexity, and how we built dictionaries, language and acoustic models in this framework. We show how to deal with the difficulty of scoring results for multiple script languages because of ambiguities. The development of voice search for these languages led to a significant simplification of the original process to build a system for any new language which in in parts became our default process for internationalization of voice search.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“LogRobust [8], which encodes log templates into semantic vectors using the FastText pre-trained model [31], cannot work well on 2 out of 4 datasets.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [8]
In the citing paper
“Their empirical study on a Microsoft online service system shows that up to 30.3% logs are changed in the latest version.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [8]
In the citing paper
“Zhang et al. [8] indicated that log data is unstable, meaning that new log events often appear due to software evolution at its lifetime.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [8]
In the citing paper
“Another option is to use domain knowledge to reduce the input space of a model — such strategies include patch embeddings [Dosovitskiy et al. 2020] as well as fixed or learned tokenization [Schuster and Nakajima 2012; Sennrich et al. 2016; Kudo and Richardson 2018; van den Oord et al. 2017; Ramesh et al. 2021]. These methods may alleviate some challenges facing generative approaches, but have the”
From On the Opportunities and Risks of Foundation Models (2021) · cites this paper as Schuster and Nakajima 2012
Citations
Cites
Semi-tied covariance matrices for hidden Markov models
Gales, M. J. F. (1999) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.759034 · Cited by 537
Boosted MMI for model and feature-space discriminative training
Povey, D., Kanevsky, D., Kingsbury, B., Ramabhadran, B., Saon, G., & Visweswariah, K. (2008) · 2008 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2008.4518545 · Cited by 341
“Your Word is my Command”: Google Search by Voice: A Case Study
Schalkwyk, J., Beeferman, D., Beaufays, F., Byrne, B., Chelba, C., Cohen, M., Kamvar, M., & Strope, B. (2010) · Advances in Speech Recognition · doi:10.1007/978-1-4419-5951-5_4 · Cited by 232
Building transcribed speech corpora quickly and cheaply for many languages
Hughes, T., Nakajima, K., Ha, L., Vasu, A., Moreno, P. J., & LeBeau, M. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-551 · Cited by 75
Self-citationGoogle Search by Voice: A Case Study
Schalkwyk, J., Beeferman, D., Beaufays, F., Byrne, B., Chelba, C. I., Cohen, M., Garrett, M., & Strope, B. P. (2010) · Cited by 63
Query language modeling for voice search
Chelba, C. I., Schalkwyk, J., Brants, T., Ha, V., Harb, B., Neveitt, W., Parada, C., & Xu, P. (2010) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2010.5700834 · Cited by 28
Search by voice in Mandarin Chinese
Shan, J., Wu, G., Hu, Z., Tang, X., Jansche, M., & Moreno, P. J. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-129 · Cited by 18
Research works printed but not linked: 1
- sung, 0, Deploying Search by Voice in Cantonese, Proc Interspeech 2011
Cited by
Items 101–200 of 933
Catelli, R., Gargiulo, F., Casola, V., Pietro, G. D., Fujita, H., & Esposito, M. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106779 · Cited by 82
“Lastly, BERT uses a sub-word vocabulary based on the WordPieceModel segmenter [75], a data-based approach to break down a word into sub-words that is more effective than operating at the word level.”
Deep Graph Learning for Anomalous Citation Detection
Liu, J., Xia, F., Xu, F., Ren, J. F., & Liu, H. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3145092 · Cited by 87
“In the experiment, 15% of WordPiece Token [49] will be randomly masked.”
MethodChappuis, C., Zermatten, V., Lobry, S., Saux, B. L., & Tuia, D. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00143 · Cited by 110
“This is identical to BertTokenizer, which is based on the subword tokenization algorithm WordPiece [26,36].”
MethodAn Overview of Language Models: Recent Developments and Outlook
Wei, C., Wang, Y.-C., Wang, B., & Kuo, C. ‐. J. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000010 · Cited by 86
“[43] WordPiece is another data-driven subword algorithm.”
Deliberation Model Based Two-Pass End-To-End Speech Recognition
Hu, K., Sainath, T. N., Pang, R., & Prabhavalkar, R. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053606 · Cited by 73
“The LAS decoder has a 4,096-dimensional softmax layer to predict the same mixed-case wordpieces [19] as the RNN-T.”
BackgroundMethodBenchmarking Large Language Models for Log Analysis, Security, and Interpretation
Karlsen, E., Luo, X., Zincir-Heywood, N., & Heywood, M. (2024) · Journal of Network and Systems Management · doi:10.1007/s10922-024-09831-x · Cited by 71
Tensorized Embedding Layers for Efficient Model CompressionPreprint
Hrinchuk, O., Khrulkov, V., Mirvakhabova, L., Orlova, E., & Oseledets, I. V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10787 · Cited by 74
Chen, D., Chen, H., Yang, Y., Lin, A., & Yu, Z. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.239 · Cited by 129
Identifying Elements Essential for BERT’s Multilinguality
Dufter, P., & Schütze, H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.358 · Cited by 139
“We create a vocabulary of size 2048 from the Easy-to-Read Bible with the wordpiece tokenizer (Schuster and Nakajima, 2012).”
MethodMulti-task learning in under-resourced Dravidian languages
Hande, A., Hegde, S. U., Priyadharshini, R., Ponnusamy, R., Kumaresan, P. K., Thavareesan, S., & Chakravarthi, B. R. (2022) · Journal of Data Information and Management · doi:10.1007/s42488-022-00070-w · Cited by 72
“Unlike the conventional wordpiece (Schuster and Nakajima 2012) and Unigram tokenizer (Kudo 2018), RoBERTa uses a Byte-level BPE tokenizer (Sennrich et al. 2016) for its tokenization process.”
MethodSabet, M. J., Dufter, P., Yvon, F., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.147 · Cited by 287
Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words
Hofmann, V., Pierrehumbert, J. B., & Schütze, H. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.279 · Cited by 135
“We train two main models on the introduced binary classification task: BERT with the standard WordPiece segmentation ( s w ) and BERT using the derivational segmentation ( s d ), a model that we refer to as DelBERT ( De rivation l everaging BERT ).”
BackgroundMethodInfluentialConstructing and analyzing domain-specific language model for financial text mining
Suzuki, M., Sakaji, H., Hirano, M., & Izumi, K. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103194 · Cited by 80
Steel design based on a large language model
Tian, S., Jiang, X. F., Wang, W., Jing, Z., Zhang, C., Zhang, C., Lookman, T., & Su, Y. (2024) · Acta Materialia · doi:10.1016/j.actamat.2024.120663 · Cited by 74
Foundation Models in Agriculture: A Comprehensive Review
Yin, S., Xi, Y., Zhang, X., Sun, C., & Mao, Q. (2025) · Agriculture · doi:10.3390/agriculture15080847 · Cited by 76
Deep Learning or Classical Machine Learning? An Empirical Study on Log-Based Anomaly Detection
Yu, B., Yao, J., Fu, Q., Zhong, Z., Xie, H., Wu, Y., Ma, Y., & He, P. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623308 · Cited by 65
“Then it uses WordPiece tokenization [70, 79] to handle the OOV words, and uses a pre-trained BERT [19] to obtain the semantic meaning of log messages.”
MethodBiomedical Event Extraction as Sequence Labeling
Ramponi, A., van der Goot, R., Lombardo, R., & Plank, B. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.431 · Cited by 81
“The first WordPiece (Schuster and Naka-jima, 2012) of each token x i is used for prediction, where the contextual hidden representation e i of the token x i is encoded with layer-wise attention over the BERT layers, similarly to (Peters et al., 2018; Kondratyuk and Straka, 2019).”
MethodFine-tuning Pretrained Multilingual BERT Model for Indonesian Aspect-based Sentiment Analysis
Azhar, A. N., & Khodra, M. L. (2020) · International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA) · doi:10.1109/icaicta49861.2020.9428882 · Cited by 64
“…difference between traditional tokenization process and BERT tokenization process is that BERT tokenization does not only perform basic tokenization steps (lower-casing, whitespace removal, etc.) but also uses WordPiece (Wu, et al., 2016; Schuster & Nakajima, 2012) tokenizer to tokenize the text.”
MethodContextual semantic embeddings for ontology subsumption prediction
Chen, J., He, Y., Geng, Y., Jiménez-Ruiz, E., Dong, H., & Horrocks, I. R. (2023) · World Wide Web · doi:10.1007/s11280-023-01169-9 · Cited by 88
“In pre-training, each input is a sequence composed of a [CLS] special token, two sentences (denoted as A and B), each of which is tokenized e.g., into sub-words by WordPiece [38], and a [SEP] special token that separates A and B.”
BackgroundInfluentialLeveraging orthographic information to improve machine translation of under-resourced languages
Chakravarthi, A. (2020) · Arrow@dit (Dublin Institute of Technology) · doi:10.13025/17483 · Cited by 76
HeBERT and HebEMO: A Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition
Chriqui, A., & Yahav, I. (2022) · INFORMS Journal on Data Science · doi:10.1287/ijds.2022.0016 · Cited by 85
“Common granularity levels include characters, n-gram-based sub-words (using WordPiece algorithm (Schuster and Nakajima 2012)), morpheme-based sub-words, and full words (see Figure 1 for the differences between the approaches).”
MethodNodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge GraphsPreprint
Galkin, M., Denis, E., Wu, J., & Hamilton, W. L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12144 · Cited by 114
“Subword-powered algorithms such as fastText (Bojanowski et al., 2017), Byte-Pair Encoding (Sennrich et al., 2016), and WordPiece (Schuster & Nakajima, 2012) became a standard step in preprocessing pipelines of large language models and allowed to construct fixed-size token vocabularies, e.g., BERT (Devlin et al., 2019)…”
MethodEfficient Transformers with Dynamic Token Pooling
Nawrot, P., Chorowski, J., Łańcucki, A., & Ponti, E. M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.353 · Cited by 101
“Widespread algorithms for extracting variable-length boundaries for text are subword tokenizers, including Unigram (Kudo, 2018), Byte Pair Encoding (BPE; Sennrich et al., 2016), and Word-Piece (Schuster and Nakajima, 2012).”
MethodSelf-Training for End-to-End Speech Translation
Pino, J., Xu, Q., Ma, X., Dousti, M. J., & Tang, Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2938 · Cited by 93
Vogelsang, A., & Fischbach, J. (2025) · Springer eBooks · doi:10.1007/978-3-031-73143-3_16 · Cited by 61
ValueNet: A Natural Language-to-SQL System that Learns from Database Information
Brunner, U., & Stockinger, K. (2021) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde51399.2021.00220 · Cited by 95
“Each value token is further tokenized in word pieces using the WordPiece [29] segmentation algorithm.”
MethodProcessing South Asian Languages Written in the Latin Script: the Dakshina DatasetPreprint
Roark, B., Wolf-Sonkin, L., Kirov, C., Mielke, S. J., Johny, C., Demirşahin, İ., & Hall, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.01176 · Cited by 88
“Instead of single character sub-word tokens, we make use of a dictionary of 32k “word pieces” (over both input and output vocabularies), using the approach outlined in Schuster and Nakajima (2012).”
MethodCascaded Encoders for Unifying Streaming and Non-Streaming ASR
Narayanan, A., Sainath, T. N., Pang, R., Yu, J., Chiu, C., Prabhavalkar, R., Variani, E., & Strohman, T. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414607 · Cited by 87
“As output representation, we use a word-piece model with a vocabulary size of 4096 [33], and an embedding size of 128.”
MethodSentence-Level Fluency Evaluation: References Help, But Can Be Spared!
Kann, K., Rothe, S., & Filippova, K. (2018) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k18-1031 · Cited by 93
“WordPiece models are estimated using a data-driven approach which maximizes the LM likelihood of the training corpus as described in Wu et al. (2016) and Schuster and Nakajima (2012).”
MethodImproving Multilingual Models with Language-Clustered Vocabularies
Chung, H. W., Garrette, D., Tan, K. C., & Riesa, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.367 · Cited by 93
Which *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 76
Evaluating Various Tokenizers for Arabic Text Classification
Alyafeai, Z., Al-shaibani, M. S., Ghaleb, M., & Ahmad, I. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-10990-8 · Cited by 77
“The most prominent tokenizers in the literature are byte pair encoding BPE [52], WordPiece [51], SentecePiece [34], Huffman encoding [22], and orthographic syllable [35].”
BackgroundHow (Non-)Optimal is the Lexicon?
Pimentel, T., Nikkarinen, I., Mahowald, K., Cotterell, R., & Blasí, D. E. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.350 · Cited by 61
AVA: A Financial Service Chatbot Based on Deep Bidirectional Transformers
Yu, S., Chen, Y., & Zaidi, H. A. (2021) · Frontiers in Applied Mathematics and Statistics · doi:10.3389/fams.2021.604842 · Cited by 60
“At representation layer, Word-piece model(Schuster & Nakajima, 2012) and Byte-Pair-Encoding(BPE) model(Gage, 1994; Sennrich et al., 2016) are common techniques to segment words into smaller units, thus similarities at sub-word level can be captured by NLP models and generalized on…”
Zangari, A., Marcuzzo, M., Schiavinato, M., Gasparetto, A., & Albarelli, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119984 · Cited by 75
“In the case of LMs like BERT, the tokenizer utilized was the one associated with the original work ( i.e. , WordPiece for BERT) (Schuster and Nakajima, 2012).”
MethodHerBERT: Efficiently Pretrained Transformer-based Language Model for PolishPreprint
Mroczkowski, R., Rybak, P., Wróblewska, A., & Gawlik, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.01735 · Cited by 146
“The original BERT model used WordPiece implementation (Schuster and Nakajima, 2012), which was later replaced by SentencePiece (Kudo and Richardson, 2018).”
MethodEfficient Minimum Word Error Rate Training of RNN-Transducer for End-to-End Speech Recognition
Guo, J., Tiwari, G., Droppo, J., Segbroeck, M. V., Huang, C.-W., Stolcke, A., & Maas, R. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1557 · Cited by 64
Nayak, A., Timmapathini, H., Ponnalagu, K., & Venkoparao, V. G. (2020) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2020.insights-1.1 · Cited by 55
“BERT relies on the WordPiece algorithm (Schuster and Nakajima, 2012) to create the vocabulary, that chooses those sub-word units for the vocabu-”
MethodErdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
“Commonly used choices for units are words, characters, or subword units extracted by methods such as WordPiece (Schuster & Nakajima, 2012), Byte-Pair Encodings (Sennrich & Haddow, 2016) and SentencePiece (Kudo & Richardson, 2018).”
MethodCEQE: Contextualized Embeddings for Query Expansion
Naseri, S., Dalton, J., Yates, A., & Allan, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_31 · Cited by 54
KPI-BERT: A Joint Named Entity Recognition and Relation Extraction Model for Financial Reports
Hillebrand, L., Deußer, T., Dilmaghani, T., Kliem, B., Loitz, R., Bauckhage, C., & Sifa, R. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956191 · Cited by 51
“Given a WordPiece [19] tokenized input sentence of n subwords we use a pre-trained BERT [4] model to obtain a sequence of n+1 encoded token embeddings, (c, t1, t2, .”
MethodImproving the Performance of Online Neural Transducer Models
Sainath, T. N., Chiu, C., Prabhavalkar, R., Kannan, A., Wu, Y., Nguyen, P., & Chen, Z. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462366 · Cited by 51
The SIGMORPHON 2020 Shared Task on Multilingual Grapheme-to-Phoneme Conversion
Gorman, K., Ashby, L. F. E., Goyzueta, A., McCarthy, A. D., Wu, S., & You, D. (2020) · SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology (SIGMORPHON) · doi:10.18653/v1/2020.sigmorphon-1.2 · Cited by 51
“Finally we note that submissions make use of unsupervised tokenization techniques such as byte-pair encoding (Schuster and Nakajima 2012).”
MethodCruz-Benito, J., Vishwakarma, S., Martin-Fernandez, F., & Faro, I. (2021) · AI · doi:10.3390/ai2010001 · Cited by 43
On the Effectiveness of Transfer Learning for Code Search
Salza, P., Schwizer, C., Gu, J., & Gall, H. C. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3192755 · Cited by 43
“WORDPIECE [40] is one of the most used subword-based tokenization algorithms, which increased its TABLE 1 Size of the Pre-Training Datasets Fig.”
MethodEffect of tokenization on transformers for biological sequences
Dotan, E., Jaschek, G., Pupko, T., & Belinkov, Y. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae196 · Cited by 49
“To this end, we trained BPE, WordPiece, and Unigram tokenizers on samples of proteins from the 2 : 2 billion protein sequences of the BFD dataset (Steinegger and S € oding 2018).”
BackgroundMethodInfluentialSAMU-XLSR: Semantically-Aligned Multimodal Utterance-Level Cross-Lingual Speech Representation
Khurana, S., Laurent, A., & Glass, J. R. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3192714 · Cited by 84
“The LaBSE’s transformer encoder takes as input text that is tokenized into “wordpieces” [26], [27] and outputs a sequence of contextual token embedding W ∈ RL×768.”
BackgroundDarijaBERT: a step forward in NLP for the written Moroccan dialect
Gaanoun, K., Naira, A. M., Allak, A., & Benelallam, I. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-023-00498-2 · Cited by 49
“A WordPiece tokenizer [17] was utilized to generate 80k, 110k, and 160k tokens for DarijaBERT, DarijaBERT-arabizi, and DarijaBERTmix, respectively.”
Hierarchical Text Classification and Its Foundations: A Review of Current Research
Zangari, A., Marcuzzo, M., Rizzo, M., Giudice, L., Albarelli, A., & Gasparetto, A. (2024) · Electronics · doi:10.3390/electronics13071199 · Cited by 68
WikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and Discourse
Faruqui, M., Pavlick, E., Tenney, I., & Das, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1028 · Cited by 67
“We use a two-layer bidirectional encoder using the same 300-dimensional FastText embeddings as in § 6.1, and a sequence decoder with attention (Bahdanau et al., 2015) using a learned wordpiece model (Schuster and Nakajima, 2012) with a vocabulary of 16,000.”
MethodInfluentialConfidence Estimation for Attention-Based Sequence-to-Sequence Models for Speech Recognition
Li, Q., Qiu, D., Zhang, Y., Li, B., He, Y., Woodland, P. C., Cao, L., & Strohman, T. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414920 · Cited by 81
Multi-view Subword Regularization
Wang, X., Ruder, S., & Neubig, G. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.40 · Cited by 74
Adapting vs. Pre-training Language Models for Historical Languages
Manjavacas, E., & Fonteyn, L. (2022) · Journal of Data Mining & Digital Humanities · doi:10.46298/jdmdh.9152 · Cited by 77
“…language model implementations rely on tokenization procedures – like Byte-Pair-Encoding (BPE) [Gage, 1994, Sennrich et al., 2016] or WordPiece [Schuster and Nakajima, 2012] – that break down character sequences into so-called sub-word tokens, optimizing a particular information-theoretic…”
InfluentialSanskrit Word Segmentation Using Character-level Recurrent and Convolutional Neural Networks
Hellwig, O., & Nehrdich, S. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1295 · Cited by 85
“…the output of the softmax in a greedy fashion without considering interactions between adjacent output classes. seq2seq We retrain the model described in Reddy et al. (2018) with our data after pre-processing them with the unsupervised text to-kenizer sentencepiece (Schuster and Nakajima, 2012).”
MethodSurvey of Neural Text Representation Models
Babić, K., Martinčić-Ipšić, S., & Meštrović, A. (2020) · Information · doi:10.3390/info11110511 · Cited by 54
“SP (SentencePiece) [52] and WP (WordPiece) [53] are used for attention models as well (e.”
MethodEfficient Domain Adaptation of Language Models via Adaptive Tokenization
Sachidananda, V., Kessler, J. S., & Lai, Y.-A. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.16 · Cited by 77
Idiomatic Expression Identification using Semantic Compatibility
Zeng, Z., & Bhat, S. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00442 · Cited by 78
“In our implementation, the tokenizer for the language model uses the WordPiece algorithm (Schuster and Nakajima, 2012) prominently used in BERT (Devlin et al.”
MethodA Transformer-Based Neural Machine Translation Model for Arabic Dialects That Utilizes Subword Units
Baniata, L. H., Ampomah, I. K. E., & Park, S. (2021) · Sensors · doi:10.3390/s21196509 · Cited by 53
“This study uses the WordPiece Model (WPM) implementation, which was originally used by Google to tackle a Japanese–Korean segmentation challenge [41].”
MethodAn Empirical Study of Language Relatedness for Transfer Learning in Neural Machine Translation
Dabre, R., Nakagawa, T., & Kazawa, H. (2017) · Institutional Repositories DataBase (IRDB) · Cited by 46
A Deep Transfer Learning Approach to Modeling Teacher Discourse in the Classroom
Jensen, E., Pugh, S. L., & D’Mello, S. K. (2021) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3448139.3448168 · Cited by 61
“Unlike the bag of n-grams approach used for the Random Forest models, BERT processes the automatically transcribed utterances using WordPiece tokenization [63].”
MethodInvestigating the Effectiveness of BPE: The Power of Shorter Sequences
Gallé, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1141 · Cited by 111
“, 2016), although a similar technique had been used previously for the creation of dictionaries and language models for voice search (Schuster and Nakajima, 2012).”
MethodIntroduction to Neural Transfer Learning With Transformers for Social Science Text Analysis
Wankmüller, S. (2022) · Sociological Methods & Research · doi:10.1177/00491241221134527 · Cited by 46
“In the WordPiece variant of BPE, the algorithm merges at each step the character pair that, when merged, results in the highest increase in the likelihood of the training corpus compared to all other pairs (Schuster & Nakajima 2012, p. 5150).”
Data Weighted Training Strategies for Grammatical Error Correction
Lichtarge, J., Alberti, C., & Kumar, S. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00336 · Cited by 59
“We use a 32k word piece dictionary (Schuster and Nakajima, 2012).”
MethodHybrid CTC-Attention based End-to-End Speech Recognition using Subword Units
Xiao, Z., Ou, Z., Chu, W., & Lin, H. Z. (2018) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp.2018.8706675 · Cited by 44
How Domain Terminology Affects Meeting Summarization Performance
Koay, J. J., Roustai, A., Dai, X., Burns, D., Kerrigan, A., & Liu, F. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.499 · Cited by 36
End-to-end neural systems for automatic children speech recognition: An empirical study
Shivakumar, P. G., & Narayanan, S. S. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101289 · Cited by 71
The Future of Computational Linguistics: On Beyond Alchemy
Church, K., & Liberman, M. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.625341 · Cited by 54
“Word pieces are used for a variety of applications: speech (Schuster and Nakajima, 2012), translation (Wu et al., 2016), as well as tasks in the GLUE benchmark such as: sentiment, paraphrase and coreference.”
MethodHofstätter, S., Khattab, O., Althammer, S., Sertkan, M., & Hanbury, A. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557367 · Cited by 67
“The BERT tokenizer first splits full text on whitespace and punctuation characters and then uses the WordPiece algorithm [44] to split words to sub-word tokens in a reduced vocabulary.”
MethodOn the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
Irie, K., Prabhavalkar, R., Kannan, A., Bruguier, A., Rybach, D., & Nguyen, P. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2277 · Cited by 53
“We train word piece models [11] of size 16K (16,384) on each training subset.”
BackgroundMethodMulti-task learning for toxic comment classification and rationale extraction
Nelatoori, K. B., & Bindu, K. H. (2022) · Journal of Intelligent Information Systems · doi:10.1007/s10844-022-00726-4 · Cited by 45
“, BERT, XLM-Roberta), each input sequence is tokenized using the WordPiece tokenizer (Schuster and Nakajima, 2012).”
MethodAnalysis of Deep Learning Model Combinations and Tokenization Approaches in Sentiment Classification
Erkan, A., & Güngör, T. (2023) · IEEE Access · doi:10.1109/access.2023.3337354 · Cited by 52
“…for eleven different tokenization schemes, which are surface form, preprocessed form, surface form with stopwords eliminated, stem, lemma, morphemes, byte pair encoding (BPE) [10], WordPiece [11], unigram language model (ULM) [12], syllables (for Turkish) and partial surface form (for Turkish).”
BackgroundMethodCompression of End-to-End Models
Pang, R., Sainath, T. N., Prabhavalkar, R., Gupta, S., Wu, Y., Zhang, S., & Chiu, C. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1025 · Cited by 42
“All networks are trained to predict 4,096 word pieces [31] which are derived using a large corpora of text transcripts.”
MethodLow Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models
Wolfe, R., & Caliskan, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.41 · Cited by 70
“In the context of studies of race, Sen and Wasow (2016) usefully define an exposure study as one which uses "a cue or signal that generates some reaction," and note that "names often act as a proxy for for traits associated with racial or ethnic groups."”
Background(2020) · doi:10.18653/v1/2020.conll-1.17 · Cited by 41
“For bert-base , the representation of an input sequence consists in a 12-layer activation network with 768-dimensional vectors for each input token (where tokens correspond to sub-word WordPieces [Schuster and Nakajima, 2012]) at each of 12 layers.”
MethodAn Evaluation of Neural Machine Translation Models on Historical Spelling NormalizationPreprint
Tang, G., Cap, F., Pettersson, E., & Nivre, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.05210 · Cited by 58
“Wu et al. (2016) apply the wordpiece model (Schuster and Nakajima, 2012) to segmentation.”
MethodNatural Language Processing Methods for Language Modeling
Nemeskey, D. M. (2021) · Eötvös Loránd Tudományegyetem · doi:10.15476/elte.2020.066 · Cited by 85
Ando, K., Okumura, T., Komachi, M., Horiguchi, H., & Matsumoto, Y. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000158 · Cited by 49
A Compact and Language-Sensitive Multilingual Translation Method
Wang, Y., Zhou, L., Zhang, J., Zhai, F., Xu, J., & Zong, C. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1117 · Cited by 40
“We use wordpiece method (Wu et al., 2016; Schuster and Nakajima, 2012) encode the combination of both source side sentences and target side sentences.”
MethodA Simple But Effective Bert Model for Dialog State Tracking on Resource-Limited Systems
Lai, T., Tran, Q. H., Bui, T., & Kihara, D. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053975 · Cited by 40
“For each sentence, we use the WordPiece tokenizer [20, 13] to tokenize the sentence into a sequence of tokens.”
MethodDualSC: Automatic Generation and Summarization of Shellcode via Transformer and Dual Learning
Yang, G., Chen, X., Zhou, Y., & Yu, C. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00052 · Cited by 45
Pre-trained Contextual Embedding of Source CodePreprint
Kanade, A., Maniatis, P., Balakrishnan, G., & Shi, K. (2019) · arXiv (Cornell University) · Cited by 40
A Machine Learning Method for Prediction of Stock Market Using Real-Time Twitter Data
Albahli, S., Irtaza, A., Nazir, T., Mehmood, A., Alkhalifah, A., & Albattah, W. (2022) · Electronics · doi:10.3390/electronics11203414 · Cited by 39
“There are multiple solutions to solve this problem, including tokenization, smoothing technique, and semantic representations [42,43].”
MethodDisaster assessment from social media using multimodal deep learning
Shetty, N. P., Bijalwan, Y., Chaudhari, P., Shetty, J., & Muniyal, B. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19818-0 · Cited by 39
“The input text X t is first tokenized [20, 21] and then encoded using the different variants of the BERT model, namely BERT [22], RoBERTa [23], and BERTweet [24].”
Large language models for bioinformatics
Ruan, W., Lyu, Y., Zhang, J., Cai, J., Shu, P., Yang, G., Lu, Y., Gao, S., Wang, Y., Wang, P., et al. (2025) · Quantitative Biology · doi:10.1002/qub2.70014 · Cited by 48
“Popular tokenization algorithms, such as Byte ‐ Pair Encoding (BPE) [79], WordPiece [80], and Unigram [81], are widely used.”
ECLARE: Extreme Classification with Label Graph Correlations
Mittal, A., Sachdeva, N., Agrawal, S., Agarwal, S., Kar, P., & Varma, M. (2021) · The Web Conference · doi:10.1145/3442381.3449815 · Cited by 73
“ECLARE used the word piece tokenizer [39] to create a shared vocabulary for documents and labels.”
MethodBERT models for Brazilian Portuguese: Pretraining, evaluation and tokenization analysis
de Souza, F. C., Nogueira, R. F., & Lotufo, R. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110901 · Cited by 50
Predicting life satisfaction using machine learning and explainable AI
Khan, A. E., Hasan, M. J., Anjum, H., Mohammed, N., & Momen, S. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e31158 · Cited by 30
“The WordPiece [52] tokenizers available for each model were used to tokenize the texts for processing by the language models and use the first 512 tokens due to the input limitation of BERT-like models.”
An Investigation of Phone-Based Subword Units for End-to-End Speech Recognition
Wang, W., Wang, G., Bhatnagar, A., Zhou, Y., Xiong, C., & Socher, R. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1873 · Cited by 46
“However, there has been a surge of interest in using character and character-based subwords, such as byte pair encoding (BPE, [3]) and word-pieces [4], in modern end-to-end systems [5, 6, 7, 8, 9, 10, 11, 12, 13, 14].”
BackgroundCatelli, R., Gargiulo, F., Casola, V., Pietro, G. D., Fujita, H., & Esposito, M. (2021) · IEEE Access · doi:10.1109/access.2021.3054479 · Cited by 40
“A similar approach has been used by BERT, whose tokenizer is based on WordPieceModel segmenter [47] which always works on a sub-word level.”
MethodAutomating Genetic Algorithm Mutations for Molecules Using a Masked Language Model
Blanchard, A. E., Chandrashekar, M., Gao, S. B., Gounley, J. P., Lyngaas, I., Gläser, J., & Bhowmik, D. (2022) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2022.3144045 · Cited by 41
Pseudo Label Is Better Than Human Label
Hwang, D., Sim, K. C., Huo, Z., & Strohman, T. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-11034 · Cited by 54
Aviation-BERT: A Preliminary Aviation-Specific Natural Language Model
Chandra, C., Jing, X., Bendarkar, M. V., Sawant, K., Elias, L. R., Kirby, M. R., & Mavris, D. N. (2023) · AIAA AVIATION Forum · doi:10.2514/6.2023-3436 · Cited by 37
“WordPiece tokenization [30, 31], an adaptation of Byte-Pair Encoding (BPE) [32], is used to develop the vocabulary for pre-trained BERT-Base.”
MethodA Survey of Orthographic Information in Machine Translation
Chakravarthi, B. R., Rani, P., Arčan, M., & McCrae, J. P. (2021) · SN Computer Science · doi:10.1007/s42979-021-00723-4 · Cited by 43
“, 2019) which has been shown to deal with multiple script languages ambiguities (Schuster and Nakajima, 2012; Wu et al., 2016).”
BackgroundHrinchuk, O., Khrulkov, V., Mirvakhabova, L., Orlova, E., & Oseledets, I. V. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.436 · Cited by 57
“However, both of these methods rely on the assumption of power-law distribution of tokens and are not efficient when dealing with other popular tokeniza-tions, such as wordpieces (Schuster and Nakajima, 2012; Wu et al., 2016) or BPEs (Sennrich et al., 2015).”
BackgroundThe SIGMORPHON 2022 Shared Task on Morpheme Segmentation
Batsuren, K., Bella, G., Arora, A., Martinović, V., Gorman, K., Žabokrtský, Z., Ganbold, A., Dohnalová, Š., Ševčíková, M., Pelegrinová, K., et al. (2022) · SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology (SIGMORPHON) · doi:10.18653/v1/2022.sigmorphon-1.11 · Cited by 79
Cross lingual transfer learning for sentiment analysis of Italian TripAdvisor reviews
Catelli, R., Bevilacqua, L., Mariniello, N., di Carlo, V. S., Magaldi, M., Fujita, H., Pietro, G. D., & Esposito, M. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118246 · Cited by 39
Enhancing BERT for Lexical Normalization
Müller, B., Sagot, B., & Seddah, D. (2019) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/d19-5539 · Cited by 50
“BERT takes as input sub-word units in the form of WordPiece tokens originally introduced in Schuster and Nakajima (2012).”
BackgroundMethodFaster, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces
Zhang, F., Wang, Y.-Q., Zhang, X., Liu, C., Saraf, Y., & Zweig, G. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1995 · Cited by 37
“Since CTC training does not require alignment labels per frame, graphemes [18], word-pieces (WP) [13, 28–31] or even whole words [32] can be directly modeled.”
BackgroundMind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding
Tan, S., Joty, S., Varshney, L. R., & Kan, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.455 · Cited by 53
pNLP-Mixer: an Efficient all-MLP Architecture for Language
Fusco, F., Pascual, D., Staar, P., & Antognini, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.6 · Cited by 40
Metadata
- Authors
- Schuster, M., & Nakajima, K.
- Affiliations
- Google (United States) (US)
- Venue
- 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2012-03
- Pages
- 5149-5152
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2121879602
- Cited by
- 862
- References
- 8
- Citations by year
- 2026: 36 · 2025: 66 · 2024: 90 · 2023: 124 · 2022: 137 · 2021: 155 · 2020: 144 · 2019: 59 · 2018: 36 · 2017: 10 · 2016: 3 · 2013: 2
- FWCI
- 0.49
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- voice search, Japanese language, Korean language, acoustic modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.