The DOI matches its Crossref record.
TCD-TIMIT: An Audio-Visual Corpus of Continuous Speech
Abstract
Automatic audio-visual speech recognition currently lags behind its audio-only counterpart in terms of major progress. One of the reasons commonly cited by researchers is the scarcity of suitable research corpora. This paper details the creation of a new corpus designed for continuous audio-visual speech recognition research . TCD-TIMIT consists of high-quality audio and video footage of 62 speakers reading a total of 6913 phonetically rich sentences. Three of the speakers are professionally-trained lipspeakers, recorded to test the hypothesis that lipspeakers may have an advantage over regular speakers in automatic visual speech recognition systems. Video footage was recorded from two angles: straight on, and at 30°. The paper outlines the recording of footage, and the required post-processing to yield video and audio clips for each sentence. Audio, visual, and joint audio-visual baseline experiments are reported. Separate experiments were run on the lipspeaker and non-lipspeaker data, and the results compared. Visual and audio-visual baseline results on the non-lipspeakers were low overall. Results on the lipspeakers were found to be significantly higher. It is hoped that as a publicly available database, TCD-TIMIT will now help further state of the art in audio-visual speech recognition research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 54
- hosom, 2000, Automatic time alignment of phonemes using acoustic-phonetic information
- vertanen, 2006, ?Baseline WSJ acoustic models for HTK and Sphinx Training recipes and recognition experiments ?
- 10.1109/MMSP.2006.285261 · DOI
- saitoh, 2010, A study of influence of word lip reading by change of frame rate, Proc AVSP-2010, 131
- garofolo, 1993, DARPA TIMIT Acoustic Phonetic Continuous Speech Corpus CDROM
- burnham, 2009, A blueprint for a comprehensive Australian English auditory-visual speech corpus, Sel Proc 2008 HCSNet Workshop Designing Aust Nat Corpus, 96
- sj�lander, 2003, An hmm-based system for automatic segmentation and alignment of speech, Proc Fonetik, 93
- 10.1016/S0167-6393(98)00031-4 · DOI
- 10.1016/0167-6393(93)90083-W · DOI · Paper page
- 2010, Algorithms to Measure Audio Programme Loudness and True-peak Audio Level
- galatas, 2012, Audiovisual speech recognition incorporating facial depth information captured by the Kinect, Proc 20th Eur Signal Process Conf, 2714
- 10.1109/ICME.2012.192 · DOI
- 10.1016/j.patrec.2011.06.011 · DOI
- 10.1109/JPROC.2003.817150 · DOI · Paper page
- petajan, 1984, Automatic lipreading to enhance speech recognition (speech reading)
- 10.1121/1.2229005 · DOI · Paper page
- sanderson, 2008, Biometric Person Recognition Face Speech and Fusion
- 10.1109/ICME.2001.1237849 · doi:10.1109/icme.2001.1237849
- movellan, 1995, Advances in neural information processing systems, 7, 851
- fox, 2005, Audio-and Video-Based Biometric Person Authentication
- 10.1109/ICASSP.2007.366941 · doi:10.1109/icassp.2007.366941
- messer, 1999, XM2VTSDB: The extended M2VTS database, Proc 2nd Int Conf Audio- Video-Based Biometric Person Authentication, 72
- 10.1109/ICASSP.2004.1326155 · doi:10.1109/icassp.2004.1326155
- 10.1145/2141622.2141646 · doi:10.1145/2141622.2141646
- 10.1109/ICASSP.2014.6854158 · DOI
- 10.1109/ICASSP.2001.940793 · DOI
- 10.1109/TASL.2008.2011515 · DOI
- lamel, 1986, Speech database development: Design and analysis of the acoustic-phonetic corpus, Proc DARPA Speech Recog Workshop, 100
- driel, 2009, Building a visual speech recognizer
- 10.1121/1.2935783 · DOI · Paper page
- 10.2991/jcis.2008.61 · doi:10.2991/jcis.2008.61
- chen, 2001, Audiovisual speech processing, IEEE Signal Process Mag, 18, 9, 10.1109/79.911195 · DOI · Paper page
- 10.1109/34.982900 · DOI · Paper page
- 10.1145/1027933.1027972 · DOI
- kleinschmidt, 2007, A continuous speech recognition evaluation protocol for the AVICAR database, Proc Int Conf Signal Process Commun, 339
- bailly-baillire, 2003, The BANCA database and evaluation protocol, Proc Int Conf Audio- Video-Based Biometric Person Authentication, 625, 10.1007/3-540-44887-X_74 · DOI · Paper page
- 10.1109/ICASSP.2002.1006168 · doi:10.1109/icassp.2002.1006168
- chibelushi, 1996, design issues for a digital audio-visual integrated database, IEE Colloquium on Integrated Audio-Visual Processing for Recognition Synthesis and Communication (Ref No 1996/213), 7/1
- goecke, 2004, A detailed description of the Avozes data corpus, Proc 10th Australian Int Conf Speech Sci Technol, 486
- 10.1109/6046.985551 · DOI · Paper page
- gan, 2012, Bimodal speech recognition
- sui, 2012, A 3D audio-visual corpus for speech recognition, Proc SSST, 125
- 10.1109/ICIP.2010.5650963 · doi:10.1109/icip.2010.5650963
- cappelletta, 2012, What is a viseme? Exploring the visual side of automatic speech recognition
- zhang, 2002, Automatic speechreading with applications to human-computer interfaces, EURASIP J Appl Signal Process, 2002, 1228, 10.1155/S1110865702206137 · DOI
- chitu, 2007, Building a data corpus for audio-visual speech recognition, Proc Euromedia 2007, 88
- 10.1155/2008/810362 · DOI
- heckmann, 2002, DCT-based video features for audio-visual speech recognition, Proc INTERSPEECH, 1925
- neti, 2000, ?Audio-visual speech recognition ?
- scanlon, 2003, Visual feature analysis for automatic speechreading, Proc Int Conf Audio-Visual Speech Process, 127
- bliss, 1972, Languages in contact: Some problems of hiberno-english, Proc Royal Irish Academy Section C Archaeol Celtic Studies History Linguistics Literature, 63
- 10.1109/29.46546 · DOI · Paper page
- young, 2006, The HTK Book Version 3 4
- jeffers, 1971, Speechreading (Lipreading)
Cited by
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., & Rubinstein, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201357 · Cited by 620
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss
Chen, L., Maddox, R. K., Duan, Z., & Xu, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00802 · Cited by 437
“We evaluate our model along with state-of-the-art methods on several popular datasets (e.g., GRID [6], LRW [5], VoxCeleb [24] and TCD [13]).”
MethodA review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389
Metadata
- Authors
- Harte, N., & Gillen, E.
- Venue
- IEEE Transactions on Multimedia
- Published
- 2015-05
- Volume
- 17
- Issue
- 5
- Pages
- 603-615
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2029199293
- Cited by
- 250
- References
- 0
- FWCI
- 16.81
- Type
- journal article
- Language
- en
- Open access
- closed
- Funders
- Science Foundation Ireland (07/EN/E007)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: TCD-TIMIT: An Audio-Visual Corpus of Continuous Speech. https://app.cochat.ai/papers/10.1109/tmm.2015.2407694
- The paper
- Harte, N., & Gillen, E. (2015). TCD-TIMIT: An Audio-Visual Corpus of Continuous Speech. IEEE Transactions on Multimedia. https://doi.org/10.1109/tmm.2015.2407694
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.