CoChat
Sign Up
PreprintCited by 6Speech Recognition and SynthesisSpeech and Audio ProcessingSpeech and dialogue systems

SpeechNet: A Universal Modularized Model for Speech Processing Tasks

Open access
Chen, Yi-Chen, Chi, Po-Han, Yang, Shu-wen, Chang, Kai-Wei, Lin, Jheng-hao, Huang, Sung-Feng, Liu, Da-Rong, Liu, Chi-Liang +2 more·2021·arXiv
DOI 10.48550/arxiv.2105.03070arXiv 2105.03070
Open this paper in CoChat — free accountOpen PDF

Abstract

There is a wide variety of speech processing tasks ranging from extracting content information from speech signals to generating speech signals. For different tasks, model networks are usually designed and tuned separately. If a universal model can perform multiple speech processing tasks, some tasks might be improved with the related abilities learned from other tasks. The multi-task learning of a wide variety of speech processing tasks with a universal model has not been studied. This paper proposes a universal modularized model, SpeechNet, which treats all speech processing tasks into a speech/text input and speech/text output format. We select five essential speech processing tasks for multi-task learning experiments with SpeechNet. We show that SpeechNet learns all of the above tasks, and we further analyze which tasks can be improved by other tasks. SpeechNet is modularized and flexible for incorporating more modules, tasks, or training approaches in the future. We release the code and experimental settings to facilitate the research of modularized universal models and multi-task learning of speech processing tasks.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2105.03070

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing representative audio-language multi-task models (SpeechNet, SpeechT5, VIOLA, Whisper, and Pengi) are limited to processing specific audio types, such as human speech or natural sounds.

    “Existing representative audio-language multi-tasks language models, such as SpeechNet (Chen et al., 2021), SpeechT5 (Ao et al., 2021), VIOLA (Wang et al., 2023d), Whisper (Radford et al., 2023) and Pengi (Deshmukh et al., 2023) are limited to processing specific audio types, such as human speech or natural sounds.”

    From Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models (2023) · cites this paper as Chen et al., 2021

Cites

Flagged first, then by citations

Cited by

Sorted by citations

  • Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models

    Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2311.07919 · Cited by 61 · see citing works

    “Existing representative audio-language multi-tasks language models, such as SpeechNet (Chen et al., 2021), SpeechT5 (Ao et al., 2021), VIOLA (Wang et al., 2023d), Whisper (Radford et al., 2023) and Pengi (Deshmukh et al., 2023) are limited to processing specific audio types, such as human speech or natural sounds.”
    BackgroundMethodInfluential

  • Qwen2-Audio Technical Report

    Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.10759 · Cited by 8 · see citing works

    “…Qwen2-Audio Figure 1: Performance of Qwen2-Audio, Qwen-Audio and previous top-tiers from LALMs such as SpeechT5 (Ao et al., 2021), SpeechNet (Chen et al., 2021), SpeechLLaMA (Wu et al., 2023a), SALMONN (Tang et al., 2024), Whisper (Radford et al., 2023) Pengi (Deshmukh et al., 2023), and…”
    BackgroundSelf-citation

Authors
Chen, Y.-C., Chi, P.-H., Yang, S.-W., Chang, K.-W., Lin, J.-H., Huang, S.-F., Liu, D.-R., Liu, C.-L., Lee, C.-K., & Lee, H.-Y.
Affiliations
  • National Taiwan University (TW)
  • Nvidia (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2105.03070
arXiv
2105.03070
OpenAlex
W3161940574
Cited by
6
References
5
Citations by year
2024: 2 · 2023: 1 · 2022: 2 · 2021: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
multi-task learning, speech recognition, speech synthesis, speech-to-text, text-to-speech

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review