Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 2Music and Audio ProcessingSpeech and Audio ProcessingSpeech Recognition and Synthesis

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; none of the 13 sources its claims cite is checked for support yet.

Extreme Audio Time Stretching Using Neural Synthesis

Open access
Fierro, Leonardo, Wright, Alec, Välimäki, Vesa, Hämäläinen, Matti·2023·ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
DOI 10.1109/icassp49357.2023.10094738
Open this paper in CoChat — free accountOpen PDF

Abstract

A deep neural network solution for time-scale modification (TSM) focused on large stretching factors is proposed, targeting environmental sounds. Traditional TSM artifacts such as transient smearing, loss of presence, and phasiness are heavily accentuated and cause poor audio quality when the TSM factor is four or larger. The weakness of established TSM methods, often based on a phase vocoder structure, lies in the poor description and scaling of the transient and noise components, or nuances, of a sound. Our novel solution combines a sines-transients-noise decomposition with an independent WaveNet synthesizer to provide a better description of the noise component and an improve sound quality for large stretching factors. Results of a subjective listening test against four other TSM algorithms are reported, showing the proposed method to be often superior. The proposed method is stereo compatible and has a wide range of applications related to the slow motion of media content.

Overview

CoChat Summary

TL;DR

The paper proposes a hybrid time-scale modification method for extreme stretching of environmental sounds, combining two-stage sines-transients-noise (STN) decomposition with a WaveNet synthesizer that resynthesizes the noise component. In a subjective listening test at TSM factors 4 and 8, the proposed method was often rated superior to four comparison algorithms.

Key findings

  • The proposed method outperformed fuzzy phase vocoder (FPV), harmonic-percussive TSM (HP), identity phase locking (IPL), and WSOLA under both tested time-stretching conditions for all samples except “Rooster.”
  • Among three local conditioning representations tested for the WaveNet noise synthesizer—spectrogram, mel-spectrogram, and Constant-Q Transform (CQT) spectrogram—the CQT spectrogram produced the highest quality results.
  • The method is stereo compatible: STN decomposition is transparent with respect to the stereo field, and stereo balance and phase coherence between left and right channels are preserved when components are recomposed.
  • Training used approximately 4 hours of audio, 1,900,000 iterations, and about 200 hours on a GPU; the CQT conditioning used a 256-sample hop at 44.1 kHz, 32.7 Hz minimum frequency, Nyquist maximum, and 48 bins per octave, giving 451 frequency bins.

Methods

  • The pipeline decomposes input audio into sines, transients, and noise using two-stage soft spectral masks. Sines are processed with a phase vocoder using identity phase locking, transients are preserved and relocated onto the new time axis, and noise is neurally resynthesized at the desired TSM factor.
  • The WaveNet synthesizer is autoregressively trained at 44.1 kHz to generate raw 16-bit audio using a 10-component Mixture-of-Logistic distributions sampling method, with CQT spectrogram local conditioning and beam-search at inference to remove spurious impulse events.
  • Evaluation was a formal blind listening test with 14 experienced listeners, six audio samples, TSM factors α = 4 and α = 8, 24 trials total, and 72 audio samples under investigation; ratings were collected on a 0–100 scale and analyzed as Mean Opinion Scores with 95% confidence intervals.

Limitations

  • The paper does not include a dedicated limitations section. It states future work involves redesigning the neural synthesis for standard time-stretching factors and using a larger dataset of more relevant sounds.
  • Evident from the text: the listening test used only 14 listeners and six short environmental samples (approximately 2 s each), and it evaluated only extreme factors of 4 and 8, not standard TSM factors.
  • Evident from the text: the proposed method is designed and trained for environmental sounds, and the “Rooster” sample was an exception where the proposed method did not clearly outperform the others; participants commented that its noisy nature and lack of sharp transients made expectations for a stretched version difficult.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/icassp49357.2023.10094738

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Functions for determining soft spectral masks for STN separation, as described in [15].
Fig. 1 Functions for determining soft spectral masks for STN separation, as described in [15].CC-BY · From the PDF
Fig. 2 STN decomposition of a castanet and violin mixture.
Fig. 2 STN decomposition of a castanet and violin mixture.CC-BY · From the PDF
Fig. 3 Block diagram of the TSM pipeline, for a single-channel signal. A thicker line indicates a time-stretched signal.
Fig. 3 Block diagram of the TSM pipeline, for a single-channel signal. A thicker line indicates a time-stretched signal.CC-BY · From the PDF
Fig. 4 (Left) Soda sound and (right) its four-times stretched version processed via the proposed model.
Fig. 4 (Left) Soda sound and (right) its four-times stretched version processed via the proposed model.CC-BY · From the PDF
Fig. 5 Barplots relative to the results of the subjective listening test, showing the average mean opinion scores and 95% confidence intervals of the data for TSM factors (a) α = 4 and (b) α = 8. The proposed method outperforms the opposing algorithms in both cases for most of the samples under test.
Fig. 5 Barplots relative to the results of the subjective listening test, showing the average mean opinion scores and 95% confidence intervals of the data for TSM factors (a) α = 4 and (b) α = 8. The proposed method outperforms the opposing algorithms in both cases for most of the samples under test.CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 1 Not valid · 8 Not checked · 13 Own results and interpretations

5 more claims

Create a free account to see all 22 claims and to verify the 12 references still open.

See all claims — free account
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • It may state two separate claims as one. 7
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 7

Create a free account to see these claims.

Create a free account to see these assumptions.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 2

  1. Moinet, 2013, Slowdio: Audio Time-Scaling for Slow Motion Sports Videos
  2. 2015, Algorithms to measure audio programme loudness and true-peak audio level

Cited by

See all 2

Related

See all

Metadata

Authors
Fierro, L., Wright, A., Välimäki, V., & Hämäläinen, M.
Affiliations
  • Aalto University (FI)
  • Nokia (Finland) (FI)
ORCID
Venue
ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Publisher
IEEE
Published
2023-06-04
Pages
1-5
ISSN
1520-6149 (ISSN-L)
DOI
10.1109/icassp49357.2023.10094738
OpenAlex
W4375869273
Cited by
2
References
33
Citations by year
2025: 1 · 2024: 1
FWCI
0.35
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Aaltodoc (Aalto University) (submittedVersion, other-oa)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Nokia
Topics
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
time-scale modification, environmental sounds, WaveNet, neural audio synthesis, stereo audio

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.