The DOI matches its DataCite record; 13 of the 13 references it cites were found in a registry or an index.
WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Models
Open accessAbstract
Large-scale auto-regressive language models pretrained on massive text have demonstrated their impressive ability to perform new natural language tasks with only a few text examples, without the need for fine-tuning. Recent studies further show that such a few-shot learning ability can be extended to the text-image setting by training an encoder to encode the images into embeddings functioning like the text embeddings of the language model. Interested in exploring the possibility of transferring the few-shot learning ability to the audio-text setting, we propose a novel speech understanding framework, WavPrompt, where we finetune a wav2vec model to generate a sequence of audio embeddings understood by the language model. We show that WavPrompt is a few-shot learner that can perform speech understanding tasks better than a naive text baseline. We conduct detailed ablation studies on different components and hyperparameters to empirically identify the best model configuration. In addition, we conduct a non-speech understanding experiment to show WavPrompt can extract more information than just the transcriptions. Code is available at https://github.com/Hertin/WavPrompt
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Other recent audio LLMs primarily focus on only speech.
“Other recent audio LLMs [11,12,13,14] primarily focus on only speech.”
From Joint Audio and Speech Understanding (2023) · cites this paper as [11]
Section 3.1 (page not available) lists four speech datasets, supporting the characterization that WAVPROMPT primarily focuses on speech. But it also lists a non-speech dataset, and Section 3.5 reports a non-speech sound-classification experiment. The claim that it focuses on *only* speech is therefore not supported. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 6
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhari- wal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Lan- guage models are few-shot learners," Advances in neural informa- tion processing systems, vol. 33, pp. 1877-1901, 2020.
- Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh, "Calibrate before use: Improving few-shot performance of language mod- els," in International Conference on Machine Learning. PMLR, 2021, pp. 12 697-12 706.
- M. Tsimpoukelli, J. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill, "Multimodal few-shot learning with frozen language mod- els," Advances in Neural Information Processing Systems, vol. 34, 2021.
- A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations," Advances in Neural Information Processing Systems, vol. 33, pp. 12 449-12 460, 2020.
- W.-N. Hsu, D. F. Harwath, C. Song, and J. R. Glass, "Text- free image-to-speech synthesis using learned segmental units," in ACL, 2021.
Cited by
Sorted by citations
Yang, C.-H. H., Gu, Y., Liu, Y.-C., Ghosh, S., Bulyko, I., & Stolcke, A. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389673 · Cited by 64
A Survey on Speech Large Language Models for Understanding
Peng, J., Wang, Y., Li, B., Guo, Y., Wang, H., Fang, Y., Xi, Y., Li, H., Li, X., Zhang, K., et al. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3640535 · Cited by 92
SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks
Chang, K.-W., Wu, H., Wang, Y.-K., Wu, Y.-K., Shen, H., Tseng, W.-C., Kang, I.-T., Li, S.-W. D., & Lee, H.-Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3436618 · Cited by 8
M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper
Zhou, J., Zhao, S., He, J., Wang, H., Zeng, W., Chen, Y., Sun, H., Kong, A., & Qin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888086 · Cited by 6
Toward a Unified Perspective on Parameter-Efficient Fine Tuning for Speaker Verification
Li, Z., Mak, M., Pilancı, M., Lee, H.-Y., Gan, C.-X., Sheng, J., & Meng, H. M. L. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3682068 · Cited by 5
Zero-resource Speech Translation and Recognition with LLMs
Mundnich, K., Niu, X., Mathur, P., Ronanki, S., Houston, B., Elluru, V. R., Das, N., Hou, Z., Huybrechts, G., Bhatia, A., et al. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890008 · Cited by 3
Can RAG-Driven Enhancements Amplify Audio LLMs for Low-Resource Languages?
Dutta, B., Ranjan, R., Jain, A., Singh, R., & Mayank Vatsa, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889964 · Cited by 3
Enhancing Far-Field Speech Recognition with Mixer: A Novel Data Augmentation Approach
Niu, T., Chen, Y., Qu, D., & Hu, H. (2025) · Applied Sciences · doi:10.3390/app15074073 · Cited by 3
He, M., & Garner, P. N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.291 · Cited by 2
Robust End-to-End Spoken Language Understanding in Low-Resource Settings
Piñeiro-Martín, A., Mateo, C. G., Docío-Fernández, L., & López-Pérez, M. D. C. (2026) · IEEE Access · doi:10.1109/access.2026.3660257 · Cited by 1
Hot-Fixing Wake Word Recognition for End-to-End ASR Via Neural Model Reprogramming
Ku, P.-J., Chen, I., Yang, C.-H. H., Raju, A., Dheram, P., Ghahremani, P., King, B., Liu, J., Ren, R., & Nidadavolu, P. S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448320 · Cited by 2
Meta-Prompt: Boosting Whisper's Performance in Low-Resource Speech Recognition
Chen, Y., Niu, T., Zhang, H., Zhang, W., & Qu, D. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3484328 · Cited by 1
Bidirectional Representations for Low-Resource Spoken Language Understanding
Meeus, Q., Moens, M., & hamme, H. V. (2023) · Applied Sciences · doi:10.3390/app132011291 · Cited by 2
Joint Audio and Speech Understanding
Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., & Glass, J. (2023) · arXiv · doi:10.48550/arxiv.2309.14405 · Cited by 67
Mamba-Based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition
Masuyama, Y., Miyazaki, K., & Murata, M. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832186 · Cited by 2
Speech in-context learning of paralinguistic tasks
Wong, J. H. M., Huzaifah, M., Chen, N. F., & Aw, A. T. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434787
Metadata
- Authors
- Gao, H., Ni, J., Qian, K., Zhang, Y., Chang, S., & Hasegawa-Johnson, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 2738-2742
- ISSN
- 2308-457X (ISSN-L)
- arXiv
- 2203.15863
- OpenAlex
- W4297841687
- Cited by
- 16
- References
- 0
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 4 · 2023: 4
- FWCI
- 1.94
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- public-domain
- arxiv.org/licenses/nonexclusive-distrib/1.0/
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- Center for Cognitive Computing Systems Research
- University of Illinois at Urbana-Champaign
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- few-shot learning, spoken language understanding, frozen language models, prompt-based fine-tuning, speech recognition, low-resource languages
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
