The DOI matches its DataCite record; 55 of the 55 references it cites were found in a registry or an index.
MelNet: A Generative Model for Audio in the Frequency Domain
Open accessAbstract
Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales that time-domain models have yet to achieve. We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis---showing improvements over previous approaches in both density estimates and human judgments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“MelNet (Vasquez and Lewis, 2019), which has used a 140 h single speaker dataset, as well as VoxCeleb2 (Chung et al., 2018) multi-speaker dataset, has a better performance than the previous works.”
From Audio deepfakes: A survey (2023) · cites this paper as Vasquez and Lewis, 2019
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
Progressive Growing of GANs for Improved Quality, Stability, and Variation
Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017) · arXiv · doi:10.48550/arxiv.1710.10196 · Cited by 4,727
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
Signal estimation from modified short-time Fourier transform
Griffin, D. W., & Lim, J. S. (1984) · IEEE Transactions on Acoustics Speech and Signal Processing · doi:10.1109/tassp.1984.1164317 · Cited by 2,085
Conditional Image Generation with PixelCNN DecodersPreprint
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 2,032
Pixel Recurrent Neural NetworksPreprint
van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.06759 · Cited by 1,793
(1994) · Aston Publications Explorer (Aston University) · Cited by 1,245
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A., & Tran, D. (2018) · arXiv · doi:10.48550/arxiv.1802.05751 · Cited by 1,085
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063
“Waveglow: a flow-based generative network for speech synthesis,”
Prenger, R., Valle, R., & Catanzaro, B. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683143 · Cited by 1,004
Efficient Neural Audio SynthesisPreprint
Kalchbrenner, N., Elsen, E., Simonyan, K., Noury, S., Casagrande, N., Lockhart, E., Stimberg, F., van den Oord, A., Dieleman, S., & Kavukcuoglu, K. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1802.08435 · Cited by 948
Offline Arabic Handwriting Recognition with Multidimensional Recurrent Neural Networks
Graves, A. (2012) · Springer eBooks · doi:10.1007/978-1-4471-4072-6_12 · Cited by 718
Music TransformerPreprint
Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A. M., Hoffman, M. D., Dinculescu, M., & Eck, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.04281 · Cited by 595
Salimans, T., Karpathy, A., Chen, X., & Kingma, D. P. (2017) · arXiv · doi:10.48550/arxiv.1701.05517 · Cited by 584
SampleRNN: An Unconditional End-to-End Neural Audio Generation ModelPreprint
Mehri, S., Kumar, K., Gulrajani, I., Kumar, R., Jain, S., Sotelo, J., Courville, A. C., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07837 · Cited by 551
Adversarial Audio SynthesisPreprint
Donahue, C., McAuley, J., & Puckette, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04208 · Cited by 528
Deep Voice: Real-time Neural Text-to-SpeechPreprint
Arık, S. Ö., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J. J., Ng, A. Y., Raiman, J., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.07825 · Cited by 515
Parallel WaveNet: Fast High-Fidelity Speech SynthesisPreprint
van den Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., van den Driessche, G., Lockhart, E., Cobo, L. C., Stimberg, F., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10433 · Cited by 384
Grid Long Short-Term MemoryPreprint
Kalchbrenner, N., Danihelka, I., & Graves, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.01526 · Cited by 382
GANSynth: Adversarial Neural Audio SynthesisPreprint
Engel, J., Agrawal, K. K., Chen, S., Gulrajani, I., Donahue, C., & Roberts, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08710 · Cited by 378
Music Transformer: Generating Music with Long-Term Structure
Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A., Hoffman, M., Dinculescu, M., & Douglas Eck, C. (2019) · International Conference on Learning Representations · Cited by 330
PixelSNAIL: An Improved Autoregressive Generative ModelPreprint
Chen, X., Mishra, N., Rohaninejad, M., & Abbeel, P. (2017) · International Conference on Machine Learning · doi:10.48550/arxiv.1712.09763 · Cited by 319
ClariNet: Parallel Wave Generation in End-to-End Text-to-SpeechPreprint
Ping, W., Peng, K., & Chen, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.07281 · Cited by 315
(2017) · International Conference on Learning Representations · Cited by 314
Multi-dimensional Recurrent Neural Networks
Graves, A., Fernández, S., & Schmidhuber, J. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74690-4_56 · Cited by 303
Enabling Factorized Piano Music Modeling and Generation with the MAESTRO DatasetPreprint
Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.-Z. A., Dieleman, S., Elsen, E., Engel, J., & Douglas Eck, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12247 · Cited by 301
ReNet: A Recurrent Neural Network Based Alternative to Convolutional NetworksPreprint
Visin, F., Kastner, K., Cho, K., Matteucci, M., Courville, A. C., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00393 · Cited by 285
TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation
Hernandez, F., Nguyen, V., Ghannay, S., Tomashenko, N., & Estève, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99579-3_21 · Cited by 280
Pixel Recursive Super Resolution
Dahl, R., Norouzi, M., & Shlens, J. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.581 · Cited by 273
Parallel WaveNet: Fast High-Fidelity Speech SynthesisPreprint
van den Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., van den Driessche, G., Lockhart, E., Cobo, L. C., Stimberg, F., et al. (2017) · arXiv (Cornell University) · Cited by 262
Hierarchical Generative Modeling for Controllable Speech SynthesisPreprint
Hsu, W.-N., Zhang, Y., Weiss, R. J., Zen, H., Wu, Y., Wang, Y., Cao, Y. J., Ye, J., Chen, Z., Shen, J., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.07217 · Cited by 165
Generative Image Modeling Using Spatial LSTMsPreprint
Theis, L., & Bethge, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03478 · Cited by 160
Parallel Multiscale Autoregressive Density EstimationPreprint
Reed, S., van den Oord, A., Kalchbrenner, N., Colmenarejo, S. G., Wang, Z., Belov, D., & de Freitas, N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.03664 · Cited by 149
Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder
Akuzawa, K., Iwasawa, Y., & Matsuo, Y. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1113 · Cited by 147
Sample Efficient Adaptive Text-to-SpeechPreprint
Chen, Y., Assael, Y., Shillingford, B., Budden, D., Reed, S., Zen, H., Wang, Q., Cobo, L. C., Trask, A., Laurie, B., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10460 · Cited by 139
The challenge of realistic music generation: modelling raw audio at scalePreprint
Dieleman, S., van den Oord, A., & Simonyan, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10474 · Cited by 135
Tacotron: Towards End-to-End Speech SynthesisPreprint
Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Samy Bengio, A., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.10135 · Cited by 123
Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional UpscalingPreprint
Menick, J., & Kalchbrenner, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.01608 · Cited by 122
Fast Spectrogram Inversion Using Multi-Head Convolutional Neural Networks
Arık, S. Ö., Jun, H., & Diamos, G. (2018) · IEEE Signal Processing Letters · doi:10.1109/lsp.2018.2880284 · Cited by 121
Synthesizing Audio with Generative Adversarial Networks
Donahue, C., McAuley, J., & Puckette, M. (2018) · Cited by 100
Modeling Time-Frequency Patterns with LSTM vs. Convolutional Architectures for LVCSR Tasks
Sainath, T. N., & Li, B. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-84 · Cited by 57
Exploring multidimensional lstms for large vocabulary ASR
Li, J., Mohamed, A., Zweig, G., & Gong, Y. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472617 · Cited by 55
Conditioning Deep Generative Raw Audio Models for Structured Automatic MusicPreprint
Manzelli, R., Thakkar, V., Siahkamari, A., & Kulis, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09905 · Cited by 35
Inversion of auditory spectrograms, traditional spectrograms, and other envelope representations
Decorsière, R. J. B., Søndergaard, P. L., MacDonald, E., & Dau, T. (2014) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2014.2367821 · Cited by 24
Expressive Speech Synthesis via Modeling Expressions with Variational AutoencoderPreprint
Akuzawa, K., Iwasawa, Y., & Matsuo, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.02135 · Cited by 14
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · arXiv · doi:10.48550/arxiv.1806.05622
TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation
Hernandez, F., Nguyen, V., Ghannay, S., Tomashenko, N., & Estève, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04699
Research works printed but not linked: 4
- [14][14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems , pages 2672-2680, 2014. -
- [22][22] Sepp Hochreiter and J¨ urgen Schmidhuber. Long shortterm memory. Neural computation , 9(8):1735-1780, 1997. -
- [31][31] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems , pages 10236-10245, 2018. -
- [46][46] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 4(2):26-31, 2012. -
Cited by
Sorted by citations
Jukebox: A Generative Model for MusicPreprint
Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., & Sutskever, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00341 · Cited by 1,101
WaveGrad: Estimating Gradients for Waveform GenerationPreprint
Chen, N., Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., & Chan, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.00713 · Cited by 603
A survey on neural speech synthesis
Tan, X., Qin, T., Soong, F., & Liu, T.-Y. (2021) · arXiv · doi:10.48550/arxiv.2106.15561 · Cited by 216
SuperNNova: an open-source framework for Bayesian, neural network-based supernova classification
Möller, A., & de Boissière, T. (2020) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stz3312 · Cited by 191
Deepfake generation and detection: Case study and challenges
Patel, Y., Tanwar, S., Gupta, R., Bhattacharya, P., Davidson, I. E., Nyameko, R., Aluvala, S., & Vimal, V. (2023) · IEEE Access · doi:10.1109/access.2023.3342107 · Cited by 186
Metadata
- Authors
- Vasquez, S., & Lewis, M.
- ORCID
- Lewis, Mike
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1906.01083
- OpenAlex
- W2948211236
- Cited by
- 107
- References
- 50
- Citations by year
- 2025: 5 · 2024: 5 · 2023: 14 · 2022: 7 · 2021: 35 · 2020: 34 · 2019: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- audio generation, spectrogram, time-frequency representation, long-range dependencies, music generation, text-to-speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
