The DOI matches its DataCite record.
SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks
Open accessAbstract
Prompt tuning is a technology that tunes a small set of parameters to steer a pre-trained language model (LM) to directly generate the output for downstream tasks. Recently, prompt tuning has demonstrated its storage and computation efficiency in both natural language processing (NLP) and speech processing fields. These advantages have also revealed prompt tuning as a candidate approach to serving pre-trained LM for multiple tasks in a unified manner. For speech processing, SpeechPrompt shows its high parameter efficiency and competitive performance on a few speech classification tasks. However, whether SpeechPrompt is capable of serving a large number of tasks is unanswered. In this work, we propose SpeechPrompt v2, a prompt tuning framework capable of performing a wide variety of speech classification tasks, covering multiple languages and prosody-related tasks. The experiment result shows that SpeechPrompt v2 achieves performance on par with prior works with less than 0.15M trainable parameters in a unified framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Other recent audio LLMs primarily focus on only speech.
“Other recent audio LLMs [11,12,13,14] primarily focus on only speech.”
From Joint Audio and Speech Understanding (2023) · cites this paper as [13]
The paper focuses primarily on speech classification, as described in Section 5.1. But Section 4.1 also evaluates classification of non-speech environmental sounds, so the claim that its focus is “only speech” is too absolute. Page numbers are unavailable in our copy. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Still being checked.
Research works printed but not linked: 29
- A.Mohamed, H.-y.Lee, L.Borgholt, J. D.Havtorn, J.Edin, C.Igel, K.Kirchhoff, S.-W.Li, K.Livescu, L.Maaløe, et al., "Self-supervised speech representation learning: A review," arXiv preprint arXiv:2205.10643, 2022. · arXiv
- P.Liu, W.Yuan, J.Fu, Z.Jiang, H.Hayashi, and G.Neubig, "Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing," arXiv preprint arXiv:2107.13586, 2021. · arXiv
- X. L.Li and P.Liang, "Prefix-tuning: Optimizing con- tinuous prompts for generation," in ACL/IJCNLP (1). 2021, pp. 4582-4597, Association for Computational Linguistics.
- K.Lakhotia, E.Kharitonov, W.-N.Hsu, Y.Adi, A.Polyak, B.Bolte, T.-A.Nguyen, J.Copet, A.Baevski, A.Mohamed, et al., "On generative spoken language modeling from raw audio," Transactions of the As- sociation for Computational Linguistics, vol. 9, pp. 1336-1354, 2021.
- E.Kharitonov, A.Lee, A.Polyak, Y.Adi, J.Copet, K.Lakhotia, T. A.Nguyen, M.Rivière, A.Mohamed, E.Dupoux, and W.Hsu, "Text-free prosody-aware gen- erative spoken language modeling," in ACL (1), 2022, pp. 8666-8681.
- H.Gao, J.Ni, K.Qian, Y.Zhang, S.Chang, and M.Hasegawa-Johnson, "Wavprompt: Towards few-shot spoken language understanding with frozen language models," in INTERSPEECH. 2022, pp. 2738-2742, ISCA.
- C. H.Yang, Y.Tsai, and P.Chen, "Voice2series: Re- programming acoustic models for time series classifi- cation," in ICML. 2021, vol. 139 of Proceedings of Ma- chine Learning Research, pp. 11808-11819, PMLR.
- H.Yen, P.Ku, C. H.Yang, H.Hu, S. M.Siniscalchi, P.Chen, and Y.Tsao, "A study of low-resource speech commands recognition based on adversarial reprogram- ming," CoRR, vol. abs/2110.03894, 2021. · arXiv
- R.Vygon and N.Mikhaylovskiy, "Learning efficient representations for keyword spotting with triplet loss," in International Conference on Speech and Computer, 2021, pp. 773-785.
- Y.Tian and P. J.Gorinski, "Improving end-to-end speech-to-intent classification with reptile," in INTER- SPEECH. 2020, pp. 891-895, ISCA.
- D.Bermuth, A.Poeppel, and W.Reif, "Finstreder: Sim- ple and fast spoken language understanding with finite state transducers using modern speech-to-text models," arXiv preprint arXiv:2206.14589, 2022. · arXiv
- J.Shor, A.Jansen, W.Han, D.Park, and Y.Zhang, "Uni- versal paralinguistic speech representations using self- supervised conformers," in ICASSP, 2022, pp. 3169- 3173.
- A.Ahamad, A.Anand, and P.Bhargava, "Accentdb: A database of non-native english accents to assist neural speech recognition," in LREC. 2020, pp. 5351-5358, European Language Resources Association.
- S.Castro, D.Hazarika, V.Pérez-Rosas, R.Zimmermann, R.Mihalcea, and S.Poria, "Towards multimodal sarcasm detection (an obviously perfect paper)," in Proceed- ings of the 57th Conference of ACL, 2019, pp. 4619- 4629.
- A.Ray, S.Mishra, A.Nunna, and P.Bhattacharyya, "A multimodal corpus for emotion recognition in sarcasm," in Proceedings of the Thirteenth LREC, 2022, pp. 6992- 7003.
- K.Hechmi, T. N.Trong, V.Hautamäki, and T.Kinnunen, "Voxceleb enrichment for age and gender recognition," in ASRU, 2021, pp. 687-693.
- F.Jia, S.Majumdar, and B.Ginsburg, "Marblenet: Deep 1d time-channel separable convolutional neural network for voice activity detection," in ICASSP, 2021, pp. 6818-6822.
- K.Chen, X.Du, B.Zhu, Z.Ma, T.Berg-Kirkpatrick, and S.Dubnov, "Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection," in ICASSP, 2022, pp. 646-650.
- P.Warden, "Speech commands: A dataset for limited-vocabulary speech recognition," arXiv preprint arXiv:1804.03209, 2018. · arXiv
- V.Renkens, S.Janssens, B.Ons, J. F.Gemmeke, et al., "Acquisition of ordinal words using weakly supervised nmf," in SLT, 2014, pp. 30-35.
- A.Kolesau and D. Šešok, "Unsupervised pre-training for voice activation," Applied Sciences, vol. 10, no. 23, pp. 8643, 2020.
- L.Lugosch, M.Ravanelli, P.Ignoto, V. S.Tomar, and Y.Bengio, "Speech model pre-training for end-to-end spoken language understanding," in INTERSPEECH. 2019, pp. 814-818, ISCA.
- A.Nautsch, X.Wang, N.Evans, T. H.Kinnunen, V.Vestman, M.Todisco, H.Delgado, M.Sahidullah, J.Yamagishi, and K. A.Lee, "Asvspoof 2019: spoofing countermeasures for the detection of synthesized, converted and replayed speech," IEEE Trans. Biom. Behav. Identity Sci., vol. 3, no. 2, pp. 252-265, 2021.
- C.Busso, M.Bulut, C.-C.Lee, A.Kazemzadeh, E.Mower, S.Kim, J. N.Chang, S.Lee, and S. S.Narayanan, "Iemo- cap: Interactive emotional dyadic motion capture database," Language resources and evaluation, vol. 42, no. 4, pp. 335-359, 2008.
- E.Fonseca, J.Pons, X.Favory, F.Font, D.Bogdanov, A.Ferraro, S.Oramas, A.Porter, and X.Serra, "Freesound datasets: A platform for the creation of open audio datasets," in ISMIR, 2017, pp. 486-493.
- K. J.Piczak, "Esc: Dataset for environmental sound classification," in Proceedings of the 23rd ACM interna- tional conference on Multimedia, 2015, pp. 1015-1018.
- W.Hsu, B.Bolte, Y. H.Tsai, K.Lakhotia, R.Salakhutdinov, and A.Mohamed, "Hubert: Self- supervised speech representation learning by masked prediction of hidden units," IEEE ACM Trans. Audio Speech Lang. Process., vol. 29, pp. 3451-3460, 2021.
- C.Liu, H.Lee, and W.Yih, "Structured prompt tuning," CoRR, vol. abs/2205.12309, 2022. · arXiv
- S. M.Lundberg and S.-I.Lee, "A unified approach to interpreting model predictions," Advances in neural in- formation processing systems, vol. 30, 2017.
Cited by
Sorted by citations
Towards audio language modeling -- an overviewPreprint
Wu, H., Chen, X., Lin, Y., Chang, K.-W., Chung, H.-L., Liu, A. H., & Lee, H.-Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.13236 · Cited by 69
Self-citationLow-Rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition
Yu, Y., Yang, C.-H. H., Kolehmainen, J., Shivakumar, P. G., Gu, Y., Ren, S. R. R., Luo, Q., Gourav, A., Chen, I., Liu, Y.-C., et al. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389632 · Cited by 53
Huang, C., Lu, K.-H., Wang, S.-H., Hsiao, C.-Y., Kuan, C.-Y., Wu, H., Arora, S., Chang, K., Shi, J., Peng, Y., et al. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448257 · Cited by 52
Self-citationWhispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition
Radhakrishnan, S., Yang, C.-H. H., Khan, S. A., Kumar, R., Kiani, N. A., Gómez-Cabrero, D., & Tegner, J. N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.618 · Cited by 48
HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models
Chen, C., Hu, Y., Yang, C.-H. H., Siniscalchi, S. M., Chen, P., & Chng, E. S. (2023) · neural information processing systems · doi:10.52202/075280-1375 · Cited by 45
Gong, Y., Luo, H., Liu, A. H., Karlinsky, L., & Glass, J. (2023) · arXiv · doi:10.48550/arxiv.2305.10790 · Cited by 43
Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition
Hono, Y., Mitsuda, K., Zhao, T., Mitsui, K., Wakatsuki, T., & Sawada, K. B. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.787 · Cited by 32
Extending Whisper with Prompt Tuning to Target-Speaker ASR
Ma, H., Peng, Z., Shao, M., Li, J., & Liu, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447492 · Cited by 26
Decoder-only Architecture for Streaming End-to-end Speech Recognition
Tsunoo, E., Futami, H., Kashiwagi, Y., Arora, S., & Watanabe, S. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-705 · Cited by 18
Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data AugmentationPreprint
Tsunoo, E., Futami, H., Kashiwagi, Y., Arora, S., & Watanabe, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.08876 · Cited by 13
Everson, K., Gu, Y., Yang, H., Shivakumar, P. G., Lin, G.-T., Kolehmainen, J., Bulyko, I., Gandhe, A., Ghosh, S., Hamza, W., et al. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447938 · Cited by 12
Self-citationLi, Z., Mak, M., & Meng, H. M. L. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447795 · Cited by 11
Hsiao, C.-Y., Lu, K.-H., Fu, Y.-K., Lin, G.-T., Hung, H.-T., & Lee, H.-Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.10065 · Cited by 8
Self-citationPrompting and Adapter Tuning For Self-Supervised Encoder-Decoder Speech Model
Chang, K., Chen, M.-H., Lin, Y., Hsu, J. N., Huang, P. K.-M., Huang, C., Li, S.-W. D., & Lee, H.-Y. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389731 · Cited by 7
Self-citationSRAG: Speech Retrieval Augmented Generation for Spoken Language Understanding
Yang, H., Zhang, M., Wei, D., & Guo, J. (2024) · IEEE International Conference on Control, Electronics and Computer Technology (ICCECT) · doi:10.1109/iccect60629.2024.10546001 · Cited by 7
Improving Multilingual and Code-Switching ASR Using Large Language Model Generated Text
Hu, K., Sainath, T. N., Li, B., Zhang, Y., Cheng, Y., Wang, T., Zhang, Y., & Liu, F. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389644 · Cited by 7
IRAG: Iterative Retrieval Augmented Generation for SLU
Yang, H., Zhang, M., & Wei, D. (2024) · IEEE International Colloquium on Signal Processing & Its Applications (CSPA) · doi:10.1109/cspa60979.2024.10525270 · Cited by 5
SRAG: Speech Retrieval Augmented Generation for Spoken Language Understanding
Yang, H., Zhang, M., & Wei, D. (2023) · International Seminar on Research of Information Technology and Intelligent Systems (ISRITI) · doi:10.1109/isriti60336.2023.10467285 · Cited by 5
Prompting Whisper for QA-driven Zero-shot End-to-end Spoken Language Understanding
Li, M., Keizer, S., & Doddipatla, R. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-26 · Cited by 4
Wang, Y., Jia, Q., Xu, C., Chen, F., Liu, Y., Zhang, H., Jin, L., Liu, L., & Wang, Z. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i31.39874 · Cited by 3
Self-citationMun, J., Kim, S., & Chung, M. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-131 · Cited by 3
LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs
Mousavi, P., Gupta, S., Subakan, C., & Ravanelli, M. (2025) · Interspeech (USB) · doi:10.21437/interspeech.2025-2218 · Cited by 2
Hot-Fixing Wake Word Recognition for End-to-End ASR Via Neural Model Reprogramming
Ku, P.-J., Chen, I., Yang, C.-H. H., Raju, A., Dheram, P., Ghahremani, P., King, B., Liu, J., Ren, R., & Nidadavolu, P. S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448320 · Cited by 2
TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR
Kumar, S., Madikeri, S. R., Gomez, J. P. Z., Thorbecke, I., Villatoro-Tello, E., Burdisso, S., Motlíček, P., S, K. P. D., & Ganapathiraju, A. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1167 · Cited by 2
What do Speech Foundation Models Learn? Analysis and ApplicationsPreprint
Pasad, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.12255 · Cited by 2
Audio-Visual Intelligence in Large Foundation ModelsPreprint
Qin, Y., liu, K., Wu, S., Wang, K., Deng, S., Tian, Y., Xiao, J., Xing, Y., Ma, Y., Li, B., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.04045 · Cited by 2
MPF: A multi-level perceiving framework for multimodal sarcasm detection
Zhuang, X., Li, Z., Zhou, F., Zhang, C., & Ma, H. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104436 · Cited by 1
Differentially Private Adapters for Parameter Efficient Acoustic Modeling
Ho, C.-W., Yang, C.-H. H., & Siniscalchi, S. M. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-551 · Cited by 1
Are Soft Prompts Good Zero-Shot Learners for Speech Recognition?
Ng, D., Zhang, C., Zhang, R., Ma, Y., Ritter-Gutierrez, F., Nguyen, T. H., Ni, C., Zhao, S., Chng, E. S., & Ma, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447746 · Cited by 1
Joint Audio and Speech Understanding
Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., & Glass, J. (2023) · arXiv · doi:10.48550/arxiv.2309.14405 · Cited by 67
“Other recent audio LLMs [11, 12, 13, 14] primarily focus on only speech.”
BackgroundCPPF: A contextual and post-processing-free model for automatic speech recognitionPreprint
Zhang, L., Tian, Z., Chen, X., Sun, J., Xiang, H., Ding, K., & Wan, G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.07413
Shome, D., Ghahjaverestan, N. M., & Etemad, A. (2024) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3678957.3685722
Low-Resource Speech Recognition and Understanding for Challenging Applications
Zuluaga Gomez, J. P. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.5075/epfl-thesis-10616
Maximal speaker separability via robust speaker representation learning
(219258), Z. L. (n.d.) · PolyU Institutional Research Archive (Hong Kong Polytechnic University)
Fang, T., & Yang, G. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3795843
Metadata
- Authors
- Chang, K.-W., Wang, Y.-K., Shen, H., Kang, I.-T., Tseng, W.-C., Li, S.-W., & Lee, H.-Y.
- Affiliations
- National Taiwan University (TW)
- Pennsylvania State University (US)
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2303.00733
- OpenAlex
- W4322825254
- Cited by
- 15
- References
- 0
- Citations by year
- 2024: 8 · 2023: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- prompt tuning, speech classification, pre-trained language models, parameter efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
