TIMIT-TTS: A Text-to-Speech Dataset for Multimodal Synthetic Media Detection
Open accessAbstract
With the rapid development of deep learning techniques, the generation and counterfeiting of multimedia material has become increasingly simple. Current technology enables the creation of videos where both the visual and audio contents are falsified. While the multimedia forensics community has begun to address this threat by developing fake media detectors. However, the vast majority existing forensic techniques only analyze one modality at a time. This is an important limitation when authenticating manipulated videos, because sophisticated forgeries may be difficult to detect without exploiting cross-modal inconsistencies (e.g., across the audio and visual tracks). One important reason for the lack of multimodal detectors is a similar lack of research datasets containing multimodal forgeries. Existing datasets typically contain only one falsified modality, such as deepfaked videos with authentic audio tracks, or synthetic audio with no associated video. Currently, datasets are needed that can be used to develop, train, and test these forensic algorithms. In this paper, we propose a new audio-visual deepfake dataset containing multimodal video forgeries. We present a general pipeline for synthesizing deepfake speech content from a given video, facilitating the creation of counterfeit multimodal material. The proposed method uses Text-to-Speech (TTS) and Dynamic Time Warping (DTW) techniques to achieve realistic speech tracks. We use this pipeline to generate and release TIMIT-TTS, a synthetic speech dataset containing the most cutting-edge methods in the TTS field. This can be used as a standalone audio dataset, or combined with DeepfakeTIMIT and VidTIMIT video datasets to perform multimodal research. Finally, we present numerous experiments to benchmark the proposed dataset in both monomodal (i.e., audio) and multimodal (i.e., audio and video) conditions. This highlights the need for multimodal forensic detectors and more multimodal deepfake data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
TIMIT-TTS is a synthetic speech dataset generated using 12 different TTS methods, based on the VidTIMIT corpus.
“TIMIT-TTS [42] is a synthetic speech dataset generated using 12 different TTS methods, based on the VidTIMIT corpus.”
From Deepfake Media Forensics: Status and Future Challenges (2025) · cites this paper as [42]
Visual-only deepfake detectors can be deceived by audio deepfakes, and audio-only detectors are vulnerable to deepfakes that manipulate visual content.
“Visual-only detectors, for example, can be deceived by audio deepfakes, while audio-only detectors are vulnerable to deepfakes that manipulate visual content [6].”
From A Robust Approach to Multimodal Deepfake Detection (2023) · cites this paper as [6]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 62 · Cited by 65
Research works printed but not linked: 11
- H. Khalid, S. Tariq, M. Kim, and S. Simon Woo, ''FakeAVCeleb: A novel audio-video multimodal deepfake dataset,'' in Proc. Conf. Neural Inf. Process. Syst. Datasets Benchmarks Track (Round), 2021.
- A. V. D. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu, ''WaveNet: A gen- erative model for raw audio,'' in Proc. SSW, vol. 125, 2016, p. 2.
- Mach. Learn. Cybern. (ICMLC), Jul. 2011, pp. 1708-1713.
- J. Kim, S. Kim, J. Kong, and S. Yoon, ''Glow-TTS: A generative flow for text-to-speech via monotonic alignment search,'' in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 8067-8077.
- J. Kim, J. Kong, and J. Son, ''Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,'' in Proc. Int. Conf. Mach. Learn., 2021, pp. 5530-5540.
- N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. Oord, S. Dieleman, and K. Kavukcuoglu, ''Efficient neural audio synthesis,'' in Proc. Int. Conf. Mach. Learn., 2018, pp. 2410-2419.
- C. Sanderson and B. C. Lovell, ''Multi-region probabilistic histograms for robust and scalable identity inference,'' in Proc. Int. Conf. Biometrics, 2009, pp. 199-208.
- M. Tan and Q. Le, ''EfficientNet: Rethinking model scaling for con- volutional neural networks,'' in Proc. Int. Conf. Mach. Learn., 2019, pp. 6105-6114.
- DAVIDE SALVI (Student Member, IEEE) was born in Bergamo, Italy, in 1995. He received the M.Sc. degree in music and acoustic engineering from Politecnico di Milano, Italy, in 2020, where he is currently pursuing the Ph.D. degree with the Image and Sound Processing Laboratory (ISPL), Depart- ment of Electronics, Information and Bioengineer- ing (DEIB). His research interest includes signal processing for multimedia forensics applications.
- BRIAN HOSLER (Student Member, IEEE) received the B.S. degree in electrical engineer- ing from Drexel University, Philadelphia, PA, USA, in 2018, where he is currently pursuing the Ph.D. degree with the Department of Electrical and Computer Engineering. From 2014 to 2016, he was involved in the fields of 2D nanoma- terials with the Drexel Nanomaterials Institute. From 2017 to 2018, he was an Engineer with BMW Manufacturing, Greenville, SC, USA. He is currently a Research Assistant with the Multimedia and Information Security Laboratory (MISL), Drexel University, where he is conducting research on video and multimedia forensics. His current research interests include signal processing and machine learning.
- PAOLO BESTAGINI (Member, IEEE) was born in Novara, Italy, in 1986. He received the M.Sc. degree in telecommunications engineering and the Ph.D. degree in information technology from Politecnico di Milano, Italy, in 2010 and 2014, respectively. He is currently an Assistant Pro- fessor with the Image and Sound Processing Laboratory (ISPL), Department of Electronics, Information and Bioengineering (DEIB), Politec- nico di Milano. He has been a Scientific Inves- tigator for the European projects SCENIC and REWIND coordinated by Politecnico di Milano and a Co-Principal Investigator for the DARPA-funded MediFor Project. He is a Co-Principal Investigator for the DARPA-funded SemaFor Project. His research interests include multimedia forensics and acoustic signal processing for microphone arrays. He is elected as a member of the IEEE Information Forensics and Security Technical Committee for the second time. He serves as an Associate Editor for the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY (TCSVT) and the Journal of Visual Communication and Image Representation (JVCI) (Elsevier). He is Co-Organizer of the IEEE Signal Processing Cup, in 2018 and 2022.
- Authors
- Salvi, D., Hosler, B., Bestagini, P., Stamm, M. C., & Tubaro, S.
- Affiliations
- Politecnico di Milano (IT)
- Drexel University (US)
- Venue
- IEEE Access
- Published
- 2023
- Volume
- 11
- Pages
- 50851-50866
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4381708595
- Cited by
- 65
- References
- 62
- Citations by year
- 2026: 16 · 2025: 31 · 2024: 12 · 2023: 6
- FWCI
- 5.91
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, public-domain)
- arXiv (Cornell University) (submittedVersion)
- Virtual Community of Pathological Anatomy (University of Castilla La Mancha) (submittedVersion, public-domain)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Army Research Office and was accomplished under Cooperative (W911NF-20-2-0111)
- Defense Advanced Research Projects Agency (DARPA) and the Air Force Research Laboratory (FA8750-20-2-1004, HR001120C0126)
- National Science Foundation (1553610)
- U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon
- PREMIER Project, funded by the Italian Ministry of Education, University, and Research within the PRIN 2017 Program
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- text-to-speech, dynamic time warping, video manipulation detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
