Extreme Audio Time Stretching Using Neural Synthesis

TL;DR

The paper proposes a hybrid time-scale modification method for extreme stretching of environmental sounds, combining two-stage sines-transients-noise (STN) decomposition with a WaveNet synthesizer that resynthesizes the noise component. In a subjective listening test at TSM factors 4 and 8, the proposed method was often rated superior to four comparison algorithms.

Key findings

• The proposed method outperformed fuzzy phase vocoder (FPV), harmonic-percussive TSM (HP), identity phase locking (IPL), and WSOLA under both tested time-stretching conditions for all samples except “Rooster.”
• Among three local conditioning representations tested for the WaveNet noise synthesizer—spectrogram, mel-spectrogram, and Constant-Q Transform (CQT) spectrogram—the CQT spectrogram produced the highest quality results.
• The method is stereo compatible: STN decomposition is transparent with respect to the stereo field, and stereo balance and phase coherence between left and right channels are preserved when components are recomposed.
• Training used approximately 4 hours of audio, 1,900,000 iterations, and about 200 hours on a GPU; the CQT conditioning used a 256-sample hop at 44.1 kHz, 32.7 Hz minimum frequency, Nyquist maximum, and 48 bins per octave, giving 451 frequency bins.

Methods

• The pipeline decomposes input audio into sines, transients, and noise using two-stage soft spectral masks. Sines are processed with a phase vocoder using identity phase locking, transients are preserved and relocated onto the new time axis, and noise is neurally resynthesized at the desired TSM factor.
• The WaveNet synthesizer is autoregressively trained at 44.1 kHz to generate raw 16-bit audio using a 10-component Mixture-of-Logistic distributions sampling method, with CQT spectrogram local conditioning and beam-search at inference to remove spurious impulse events.
• Evaluation was a formal blind listening test with 14 experienced listeners, six audio samples, TSM factors α = 4 and α = 8, 24 trials total, and 72 audio samples under investigation; ratings were collected on a 0–100 scale and analyzed as Mean Opinion Scores with 95% confidence intervals.

Limitations

• The paper does not include a dedicated limitations section. It states future work involves redesigning the neural synthesis for standard time-stretching factors and using a larger dataset of more relevant sounds.
• Evident from the text: the listening test used only 14 listeners and six short environmental samples (approximately 2 s each), and it evaluated only extreme factors of 4 and 8, not standard TSM factors.
• Evident from the text: the proposed method is designed and trained for environmental sounds, and the “Rooster” sample was an exception where the proposed method did not clearly outperform the others; participants commented that its noisy nature and lack of sharp transients made expectations for a stretched version difficult.