A time-scale modification dataset with subjective quality labels

TL;DR

This paper presents a labeled dataset for time-scale modification (TSM) quality, with 88 source files processed by six methods at 10 time scales for training and 20 source files processed by three additional methods at four time scales for testing. It collects 42,529 subjective ratings and shows that existing objective measures correlate poorly with subjective quality, while a retrained PEAQ-based measure approaches session-level RMSE and PCC.

Key findings

• The dataset contains 5,280 training processed files (88 sources × six methods × 10 time scales) and 240 testing files (20 sources × three additional methods × four time scales); 42,529 ratings were collected from 263 participants across 633 sessions, including 10,354 laboratory ratings.
• After outlier removal (45 sessions, 2,102 ratings, 4.94%) and normalization, session reliability improved: mean RMSE 0.771→0.682 and mean PCC 0.791→0.799; interrater reliability G(q,k) was 0.871 before and 0.909 after normalization.
• Expert and non-expert listeners were equivalent on RMSE/PCC; participants with and without hearing issues showed equivalent PCC but not RMSE; remote and laboratory modalities showed equivalent PCC but not RMSE. Age correlated 0.108 with RMSE and −0.001 with PCC.
• Published objective measures correlated poorly with MOS: SER 0.3707 and D_M 0.1574. A retrained PEAQ network achieved best D = 0.731, RMSE L = 0.668, and PCC q = 0.719, placing it at the 11th and 17th percentiles of subjective sessions.

Methods

• Source material: 34 musical, 37 solo instrument, and 37 voice files (total playback 6 min 42 s; mean duration 3.7 s, SD 1.6 s), mono, 16-bit, 44.1 kHz, normalized before TSM; includes speech, solo harmonic and percussive instruments, sound effects, and a range of music genres.
• TSM processing: training set used six methods (PV, IPL, WSOLA, FESOLA, HPTSM, uTVS) at 10 time scales; testing set used Elastique, FuzzyPV, and NMFTSM at four time scales; an additional evaluation set used all methods at 20 time scales in the range 0.22 < b < 2.2.
• Subjective testing: laboratory MATLAB GUI and remote WAET browser tests, reference-processed pairs, continuous 1–5 quality scale, ITU-R BS.1248-1/BS.1284-style normalization; analysis used MOS, RMSE, PCC, outlier removal, TOST equivalence tests, interrater reliability, and comparison/retraining of objective measures.

Limitations

• The paper states that an error in the uTVS implementation introduced spectral discontinuities for 0.9 < b < 1.1 on some files and these were not removed from the training dataset; it was rectified before the evaluation subset. It also states testing-subset means are noisier due to fewer files and nonuniform processing difficulty, and direct training/testing method comparison was not appropriate.
• The paper states that only 33 sessions reported any hearing issue versus 554 with no issue, increasing standard error; RMSE equivalence for hearing issues and testing modality required widening the equivalence interval to 69.32% and 68.14%, respectively. It also notes some files had not yet converged in MOS standard deviation, though this was a small subset.
• Evident from the text: the objective measure is preliminary and its output only moves away from the mean for b < 0.75 and b = 1; source material was collated from the authors’ previous projects and short excerpts (mean 3.7 s), which may limit generalization beyond this dataset.