The DOI matches its Crossref record; 2 of the 19 references it cites were found in a registry or an index, and 17 are not checked yet.
Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models
Open accessAbstract
In this work, we show a fundamental limitation in vocabulary adaptation approaches that use Byte-Pair Encoding (BPE) tokenization scheme for fine-tuning pretrained language models (PLMs) to expert domains.Current approaches trivially append the target domainspecific vocabulary (V DOMAIN ) at the end of the PLM vocabulary.This approach leads to a lower priority score and causes sub-optimal tokenization in BPE that iteratively uses merge rules to tokenize a given text.To mitigate this issue, we propose ADAPTBPE where the BPE tokenization initialization phase is modified to first perform the longest string matching on the added (target) vocabulary before tokenizing at the character level.We perform an extensive evaluation of ADAPTBPE versus the standard BPE over various classification and summarization tasks; ADAPTBPE improves by 3.57% (in terms of accuracy) and 1.87% (in terms of Rouge-L), respectively.ADAPTBPE for MED-VOC works particularly well when reference summaries have high OOV concentration or are longer in length.We also conduct a human evaluation, revealing that ADAPTBPE generates more relevant and more faithful summaries as compared to MEDVOC.We make our codebase publicly available at https://github.com/ gb-kgp/adaptbpe.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In byte-level BPE, merge rules are applied greedily in rank order; when a new token is inserted with a heuristic split, an earlier-ranking merge may fire first, rendering the inserted token strict-unreachable.
“in byte-level BPE, merge rules are applied greedily in rank order; when a new token is inserted with a heuristic split, an earlier-ranking merge may fire first, rendering the inserted token strict-unreachable-a failure mode termed ill-tokenization (Balde et al., 2024).”
From Writing-System-Level Tokenizer Adaptation for Byte-Level BPE (2026) · cites this paper as Balde et al., 2024
Section 3 says standard BPE selects the applicable merge with the least rank, and the Introduction and ADAPTBPE Tokenization Scheme section say existing higher-priority merges can prevent added vocabulary from forming, a failure the paper calls ill-tokenization. The copy does not describe insertion with a heuristic split or establish that an inserted token is strictly unreachable; it describes character-level initialization rather than specifically byte-level BPE. (LLM, conf=0.84)
Experimental — accuracy not yet measuredBalde et al. (2024) report the same failure mode in domain vocabulary adaptation.
“Balde et al. (2024) report the same failure mode in domain vocabulary adaptation”
From Writing-System-Level Tokenizer Adaptation for Byte-Level BPE (2026) · cites this paper as Balde et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 11
- Ting Chen, Simon Kornblith, et al. 2020. A simple framework for contrastive learning of visual repre- sentations. In International conference on machine learning, pages 1597-1607. PMLR. · DOI
- Matthias Gallé. 2019. Investigating the effectiveness of BPE: The power of shorter sequences. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 1375-1381. · DOI
- Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81.
- Yinhan Liu, Myle Ott, et al. 2019. RoBERTa: A ro- bustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. · arXiv
- Diego Mollá and María Elena Santiago-Martínez. 2011. Development of a corpus for evidence based medicine summarisation. In Proceedings of the Aus- tralasian Language Technology Association Work- shop, pages 86-94.
- Craig W. Schmidt, Varshini Reddy, et al. 2024. To- kenization is more than compression. Preprint, arXiv:2402.18376. · arXiv
- Luca Soldaini and Nazli Goharian. 2016. Quickumls: a fast, unsupervised approach for medical concept extraction. In MedIR workshop, SIGIR, pages 1-4.
- Hugo Touvron, Louis Martin, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288. · arXiv
- Yonghui Wu, Mike Schuster, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. Preprint, arXiv:1609.08144. · arXiv
- Shweta Yadav, Deepak Gupta, et al. 2022. Chq-summ: A dataset for consumer healthcare question summa- rization. arXiv preprint arXiv:2206.06581. · arXiv
- Vilém Zouhar, Clara Meister, et al. 2023. A formal perspective on byte-pair encoding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 598-614. · DOI
Cited by
Sorted by citations
Can large language models effectively reason about adverse weather conditions?
Zafarmomen, N., & Samadi, V. S. (2025) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2025.106421 · Cited by 23
Xiao, S., Gu, H., Shen, D., Niu, Z., Xiao, J., & Yu, F. (2025) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2025.104935 · Cited by 3
He, Z., Sun, Y., Li, H., Sun, C., Yang, X., Chen, H., Liao, M., & Bo, X. (2026) · Genome biology · doi:10.1186/s13059-026-03948-9 · Cited by 2
Writing-System-Level Tokenizer Adaptation for Byte-Level BPE
Didenko, B. (2026) · arXiv · doi:10.48550/arxiv.2608.00582
Keerthika, J., Gayathri, K., Dharshika, S., & Charubala, B. (2025) · International Conference on Intelligent Innovations in Engineering and Technology (ICIIET) · doi:10.1109/iciiet65921.2025.11378729
Research on Hybrid Word Segmentation Strategy for Tele-LLMs Integrating Domain Knowledge
Du, Q., & Wu, L. (2025) · IEEE International Conference on Communication Technology (ICCT) · doi:10.1109/icct67417.2025.11373994
Subword Tokenization for Agglutinative Low-Resource Languages: A Case Study on Crimean Tatar
Omer, E., & Mocanu, M. (2025) · IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) · doi:10.1109/iccp68926.2025.11427162
Metadata
- Authors
- Balde, G., Roy, S., Mondal, M., & Ganguly, N.
- Affiliations
- Indian Institute of Technology Kharagpur (IN)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2024
- Published
- 2024
- Pages
- 14724-14733
- OpenAlex
- W4404780767
- Cited by
- 6
- References
- 0
- Citations by year
- 2026: 1 · 2025: 5
- FWCI
- 1.73
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- byte pair encoding, vocabulary adaptation, domain-specific terminology, pre-trained language models, fine-tuning, out-of-vocabulary words, medical text summarization, text classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
