The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Conv-TasNet: Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation
Open accessAbstract
Single-channel, speaker-independent speech separation methods have recently seen great progress. However, the accuracy, latency, and computational cost of such methods remain insufficient. The majority of the previous methods have formulated the separation problem through the time-frequency representation of the mixed signal, which has several drawbacks, including the decoupling of the phase and magnitude of the signal, the suboptimality of time-frequency representation for speech separation, and the long latency of the entire system. To address these shortcomings, we propose a fully-convolutional time-domain audio separation network (Conv-TasNet), a deep learning framework for end-to-end time-domain speech separation. Conv-TasNet uses a linear encoder to generate a representation of the speech waveform optimized for separating individual speakers. Speaker separation is achieved by applying a set of weighting functions (masks) to the encoder output. The modified encoder representations are then inverted back to the waveforms using a linear decoder. The masks are found using a temporal convolutional network (TCN) consisting of stacked 1-D dilated convolutional blocks, which allows the network to model the long-term dependencies of the speech signal while maintaining a small model size. The proposed Conv-TasNet system significantly outperforms previous time-frequency masking methods in separating two- and three-speaker mixtures. Additionally, Conv-TasNet surpasses several ideal time-frequency magnitude masks in two-speaker speech separation as evaluated by both objective distortion measures and subjective quality assessment by human listeners. Finally, Conv-TasNet has a significantly smaller model size and a much shorter minimum latency, making it a suitable solution for both offline and real-time speech separation applications. This study therefore represents a major step toward the realization of speech separation systems for real-world speech processing technologies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
Bai, S., Kolter, J. Z., & Koltun, V. (2018) · arXiv · doi:10.48550/arxiv.1803.01271 · Cited by 4,655
A Statistical Method for Evaluating Systematic Relationships
Sokal, R. R., & Michener, C. D. (1958) · Biodiversity Heritage Library (Smithsonian Institution) · doi:10.5281/zenodo.16435756 · Cited by 4,416
Rix, A. W., Beerends, J. G., Hollier, M. P., & Hekstra, A. P. (2002) · 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221) · doi:10.1109/icassp.2001.941023 · Cited by 3,149
Performance measurement in blind audio source separation
Vincent, E., Gribonval, R., & Fevotte, C. (2006) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/tsa.2005.858005 · Cited by 2,889
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238 · cited 3×
“Motivated by the temporal convolutional network (TCN) [23], [24], [25], we propose a fully convolutional separation module that consists of stacked 1-D dilated convolutional blocks, as shown in Figure 1 B.”
MethodInfluentialSignal estimation from modified short-time Fourier transform
Griffin, D., & Lim, J. (1984) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1984.1164317 · Cited by 2,095
Supervised Speech Separation Based on Deep Learning: An Overview
Wang, D., & Chen, J. (2018) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2018.2842159 · Cited by 1,524
A Regression Approach to Speech Enhancement Based on Deep Neural Networks
Xu, Y., Du, J., Dai, L.-R., & Lee, C.-H. (2015) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2014.2364452 · Cited by 1,366
Deep clustering: Discriminative embeddings for segmentation and separation
Hershey, J. R., Chen, Z., Le Roux, J., & Watanabe, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7471631 · Cited by 1,354
Convex and Semi-Nonnegative Matrix Factorizations
Ding, C. H. Q., Li, T., & Jordan, M. I. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.277 · Cited by 1,317
SEGAN: Speech Enhancement Generative Adversarial Network
Pascual, S., Bonafonte, A., & Serrà, J. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1428 · Cited by 1,297
SURVEY OF INDEPENDENT COMPONENT ANALYSIS
Hyvärinen, A. (1999) · Cited by 1,120
On Training Targets for Supervised Speech Separation
Wang, Y., Narayanan, A., & Wang, D. (2014) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2014.2352935 · Cited by 1,067
An Experimental Study on Speech Enhancement Based on Deep Neural Networks
Xu, Y., Du, J., Dai, L.-R., & Lee, C.-H. (2014) · IEEE Signal Processing Letters · doi:10.1109/lsp.2013.2291240 · Cited by 924
Temporal Convolutional Networks: A Unified Approach to Action Segmentation
Lea, C., Vidal, R., Reiter, A., & Hager, G. D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_7 · Cited by 924
Permutation invariant training of deep models for speaker-independent multi-talker speech separation
Yu, D., Kolbaek, M., Tan, Z.-H., & Jensen, J. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952154 · Cited by 839
Speech enhancement based on deep denoising autoencoder
Lu, X., Tsao, Y., Matsuda, S., & Hori, C. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-130 · Cited by 834
Kolbaek, M., Yu, D., Tan, Z.-H., & Jensen, J. (2017) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2017.2726762 · Cited by 820
Blind Source Separation by Sparse Decomposition in a Signal Dictionary
Zibulevsky, M., & Pearlmutter, B. A. (2001) · Neural Computation · doi:10.1162/089976601300014385 · Cited by 766
TaSNet: Time-Domain Audio Separation Network for Real-Time, Single-Channel Speech Separation
Luo, Y., & Mesgarani, N. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462116 · Cited by 686
Self-citationPhase-sensitive and recognition-boosted speech separation using deep recurrent neural networks
Erdogan, H., Hershey, J. R., Watanabe, S., & Le Roux, J. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178061 · Cited by 646
A Consolidated Perspective on Multimicrophone Speech Enhancement and Source Separation
Gannot, S., Vincent, E., Markovich-Golan, S., & Ozerov, A. (2017) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2016.2647702 · Cited by 563
On Ideal Binary Mask As the Computational Goal of Auditory Scene Analysis
Wang, D. (2006) · Speech Separation by Humans and Machines · doi:10.1007/0-387-22794-6_12 · Cited by 556
A novel scheme for speaker recognition using a phonetically-aware deep neural network
Lei, Y., Scheffer, N., Ferrer, L., & McLaren, M. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853887 · Cited by 496
Tonotopic Organization of the Human Auditory Cortex
Romani, G. L., Williamson, S. J., & Kaufman, L. (1982) · Science · doi:10.1126/science.7079770 · Cited by 486
(2007) · Signals and Communication Technology · doi:10.1007/978-1-4020-6479-1 · Cited by 427
Blind source separation of more sources than mixtures using overcomplete representations
Lee, T.-W., Lewicki, M. S., Girolami, M., & Sejnowski, T. J. (1999) · IEEE Signal Processing Letters · doi:10.1109/97.752062 · Cited by 405
Deep attractor network for single-microphone speaker separation
Chen, Z., Luo, Y., & Mesgarani, N. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952155 · Cited by 397
Self-citationBlind Source Separation and Independent Component Analysis: A Review
Lee, S.-Y. (2005) · International Conference on Neural Information Processing · Cited by 374
Singing Voice Separation With Deep U-Net Convolutional Networks.
Jansson, A., Humphrey, E. J., Montecchio, N., Bittner, R., Kumar, A., & Weyde, T. (2017) · City Research Online (City University London) · doi:10.5281/zenodo.1414933 · Cited by 319
Fu, S.-W., Wang, T.-W., Tsao, Y., Lu, X., & Kawai, H. (2018) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2018.2821903 · Cited by 314
Darwin, C. J., Brungart, D. S., & Simpson, B. D. (2003) · The Journal of the Acoustical Society of America · doi:10.1121/1.1616924 · Cited by 302
Independent Vector Analysis: An Extension of ICA to Multivariate Components
Kim, T., Eltoft, T., & Lee, T.-W. (2006) · Lecture Notes in Computer Science · doi:10.1007/11679363_21 · Cited by 284
Speaker-Independent Speech Separation With Deep Attractor Network
Luo, Y., Chen, Z., & Mesgarani, N. (2018) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2018.2795749 · Cited by 260
Self-citationTonotopic Organization of the Auditory Cortex: Pitch Versus Frequency Representation
Pantev, C., Hoke, M., Lütkenhöner, B., & Lehnertz, K. (1989) · Science · doi:10.1126/science.2814476 · Cited by 247
Cepstral analysis synthesis on the mel frequency scale
Imai, S. (2005) · ICASSP '83. IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1983.1172250 · Cited by 239
Depthwise Separable Convolutions for Neural Machine Translation
Kaiser, Ł., Gomez, A. N., & Chollet, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.03059 · Cited by 234
A Novel Scheme for Speaker Recognition Using a Phonetically-Aware Deep Neural Network
Lei, Y., Scheffer, N., Ferrer, L., & McLaren, M. (2014) · doi:10.21236/ada613971 · Cited by 206
Alternative Objective Functions for Deep Clustering
Wang, Z.-Q., Roux, J. L., & Hershey, J. R. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462507 · Cited by 194
Deep clustering and conventional networks for music separation: Stronger together
Luo, Y., Chen, Z., Hershey, J. R., Le Roux, J., & Mesgarani, N. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952118 · Cited by 168
Self-citationSuper-human multi-talker speech recognition: A graphical modeling approach
Hershey, J. R., Rennie, S. J., Olsen, P. A., & Kristjansson, T. T. (2010) · Computer Speech & Language · doi:10.1016/j.csl.2008.11.001 · Cited by 166
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction
Wang, Z.-Q., Roux, J. L., Wang, D., & Hershey, J. R. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1629 · Cited by 150
On the optimality of ideal binary time–frequency masks
Li, Y., & Wang, D. (2008) · Speech Communication · doi:10.1016/j.specom.2008.09.001 · Cited by 144
Advances in deep neural network approaches to speaker recognition
McLaren, M., Lei, Y., & Ferrer, L. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178885 · Cited by 134
Deep Neural Networks for Single-Channel Multi-Talker Speech Recognition
Weng, C., Yu, D., Seltzer, M. L., & Droppo, J. (2015) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2015.2444659 · Cited by 90
Single-channel multi-talker speech recognition with permutation invariant training
Qian, Y., Chang, X., & Yu, D. (2018) · Speech Communication · doi:10.1016/j.specom.2018.09.003 · Cited by 88
Xu, C., Rao, W., Xiao, X., et al. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462471 · Cited by 87
Wang, F.-Y., Chi, C., Chan, T., & Wang, Y. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.72 · Cited by 82
Roux, J. L., Ono, N., & Sagayama, S. (2008) · Cited by 80
End-To-End Source Separation With Adaptive Front-Ends
Venkataramani, S., Casebeer, J., & Smaragdis, P. (2018) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/acssc.2018.8645535 · Cited by 80
Real-time Single-channel Dereverberation and Separation with Time-domain Audio Separation Network
Luo, Y., & Mesgarani, N. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2290 · Cited by 78
Self-citationCracking the cocktail party problem by multi-beam deep attractor network
Chen, Z., Li, J., Xiao, X., Yoshioka, T., Wang, H., Wang, Z., & Gong, Y. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268969 · Cited by 40
CBLDNN-Based Speaker-Independent Speech Separation Via Generative Adversarial Training
Li, C., Zhu, L., Xu, S., Gao, P., & Xu, B. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462505 · Cited by 37
Cited by
SDR – Half-baked or Well Done?
Roux, J. L., Wisdom, S., Erdogan, H., & Hershey, J. R. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683855 · Cited by 1,146
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation
Luo, Y., Chen, Z., & Yoshioka, T. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054266 · Cited by 780
Self-citationDCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement
Hu, Y., Liu, Y., Lv, S., Xing, M., Zhang, S., Fu, Y., Wu, J., Zhang, B., & Xie, L. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2537 · Cited by 737
Attention Is All You Need In Speech Separation
Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., & Zhong, J. (2021) · ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp39728.2021.9413901 · Cited by 635
Real Time Speech Enhancement in the Waveform Domain
Défossez, A., Synnaeve, G., & Adi, Y. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2409 · Cited by 537
Metadata
- Authors
- Luo, Y., & Mesgarani, N.
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2019-08
- Volume
- 27
- Issue
- 8
- Pages
- 1256-1266
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W2952218014
- Cited by
- 2,042
- References
- 61
- FWCI
- 137.85
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.