From Birdsong to Human Speech Recognition: Bayesian Inference on a Hierarchy of Nonlinear Dynamical Systems
Open accessThe DOI matches its Crossref record; 118 of the 119 references it cites were found in a registry or an index.
Abstract
Our knowledge about the computational mechanisms underlying human learning and recognition of sound sequences, especially speech, is still very limited. One difficulty in deciphering the exact means by which humans recognize speech is that there are scarce experimental findings at a neuronal, microscopic level. Here, we show that our neuronal-computational understanding of speech learning and recognition may be vastly improved by looking at an animal model, i.e., the songbird, which faces the same challenge as humans: to learn and decode complex auditory input, in an online fashion. Motivated by striking similarities between the human and songbird neural recognition systems at the macroscopic level, we assumed that the human brain uses the same computational principles at a microscopic level and translated a birdsong model into a novel human sound learning and recognition model with an emphasis on speech. We show that the resulting Bayesian model with a hierarchy of nonlinear dynamical systems can learn speech samples such as words rapidly and recognize them robustly, even in adverse conditions. In addition, we show that recognition can be performed even when words are spoken by different speakers and with different accents-an everyday situation in which current state-of-the-art speech recognition models often fail. The model can also be used to qualitatively explain behavioral data on human speech learning and derive predictions for future experiments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Summary of the hierarchical model of speech learning and recognition. The core of the model is equivalent to the core of the birdsong model [14]. The Equations 1 and 2 on the right side generate the dynamics shown on the left side, and are described in the Model section (see also Table 1 for the meaning of parameters). Speech sounds, i.e., sound waves, enter the model through the cochlear level. The output is a cochleagram (shown for the speech stimulus “zero”), which is a type of frequency-time diagram. There are 86 channels, which represent the firing rate (warm colors for high firing rate and cold colors for low firing rate) of the neuronal ensembles that encode lower frequencies as the channel number increases. We decrease the dimension of this input to six dimensions by averaging every 14 channels (see the color coding to the right of the cochleagram and also see Model). After this cochlear processing, activity is fed forward into the two-level hierarchical model. This input is encoded by the activity of the first level network (shown with the same color coding on the right), which is in turn encoded by activity at the second level (no color coding at this level, different colors represent different neuronal ensembles). From the generative model shown here (core model), we derived a recognition model (for mathematical details see Model).](https://figures.cochat.ai/paper-figures/58ef7d2bc22540ba93cd42ebc8052438/from-birdsong-to-human-speech-recognition-bayesian-inference-on-a-hierarchy-of-figure-1.jpg)
![Figure 2 Schematic structure of an agent and a module. A) An agent consists of several modules, where each module contains an instance of the model shown in Figure 1 and has learned to recognize a single word. Sensory input is recognized by all modules concurrently and each module experiences prediction error during recognition. A module can be considered as a sophisticated dynamic, Bayes-optimal template matcher which produces less prediction error if the stimulus matches better to the module's learned word. A minimum operator performs classification by selecting the module with the least amount of prediction error during recognition. B) At each level in a module, causal and hidden states ( and , respectively) try to minimize the precision-weighted prediction errors ( and ) by exchanging messages. Predictions are transferred from second level to the first and prediction error is propagated back from the first to the second level (see section Model: Learning and Recognition for more details). Adapted from [110].](https://figures.cochat.ai/paper-figures/2299d43c007840e8aa9fdb6d1242e1b3/from-birdsong-to-human-speech-recognition-bayesian-inference-on-a-hierarchy-of-figure-2.jpg)





![Figure 8 Qualitative modeling of experimental results in second language learning. A) The behavioral results of an experiment [67] for the recognition of English words by three groups of native speakers of Italian who differed in their age of arrival in Canada: Early, Mid and Late arrival groups, also compared to a native English speaker (NE) group. Participants were asked to repeat as many words as possible after they heard an English sentence. Sentences were presented at different signal-to-noise ratios given in decibels (dB). Adapted from [67]. B) The results of the learning and recognition simulations where we used the same speech samples as in the Word Recognition Task. The different age of arrival was modeled with different precision ratios at the first level. Recognition accuracy is measured in terms of normalized, total causal prediction errors during recognition relative to a baseline condition of −30 dB noise, i.e., recognition accuracy = 100*[(baseline prediction error-test prediction error)/baseline prediction error]. Note that we used different signal-to-noise ratios than the original experiment because best recognition results with our model were obtained at 30 dB, which corresponds to almost ideal recognition results in humans around 12 dB, and we scaled the remaining ratios accordingly. Each symbol represents the average recognition accuracy obtained from 10 digits where the stimulus was masked with noise at given signal-to-noise ratios.](https://figures.cochat.ai/paper-figures/6c07759999474ff38655aad873faf467/from-birdsong-to-human-speech-recognition-bayesian-inference-on-a-hierarchy-of-figure-8.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Because consequences follow causes, the generative model acquires a temporal depth [166, 155].”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [166]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Neural networks and physical systems with emergent collective computational abilities.
Hopfield, J. J. (1982) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.79.8.2554 · Cited by 19,270 · cited 1×
“Here, we use a Hopfield network [32] to implement such dynamics.”
MethodFriston, K. J., Harrison, L., & Penny, W. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00202-7 · Cited by 4,583 · cited 1×
“This may be done using either a voxel-wise regressor-based approach, or a network analysis (Dynamic Causal Modeling [99,100]).”
MethodA theory of cortical responses.
Friston, K. (2005) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2005.1622 · Cited by 4,578 · cited 1×
“As in the brain, learning changes parameters , such as the synaptic connectivity, of the modules relatively slowly, whereas recognition is based on rapidly changing states of the system, such as the membrane potentials and firing rate [45,46].”
BackgroundSome Experiments on the Recognition of Speech, with One and with Two Ears
Cherry, E. C. (1953) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907229 · Cited by 4,549
Maass, W., Natschläger, T., & Markram, H. (2002) · Neural Computation · doi:10.1162/089976602760407955 · Cited by 4,025 · cited 1×
“Recently, so called reservoir computing techniques using recurrent neural networks have been used for speech recognition [18,51,89–92] and provide excellent recognition results.”
BackgroundA distributed, developmental model of word recognition and naming.
Seidenberg, M. S., & McClelland, J. L. (1989) · Psychological Review · doi:10.1037/0033-295x.96.4.523 · Cited by 3,909 · cited 1×
“Many other computational speech and word recognition models have been proposed that are both neurobiologically plausible and can explain experimental results [19,20,81–87].”
BackgroundThe TRACE model of speech perception
McClelland, J. L., & Elman, J. L. (1986) · Cognitive Psychology · doi:10.1016/0010-0285(86)90015-0 · Cited by 2,956 · cited 3×
“Our model has also several differences from the influential models such as TRACE [19] and Shortlist [20,21] and we provide a more detailed comparison in the Discussion.”
BackgroundMethodAcoustic Modeling Using Deep Belief Networks
Mohamed, A.-R., Dahl, G. E., & Hinton, G. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,727 · cited 1×
“the hidden Markov model and, very recently, deep belief networks and liquid state machines [16–18].”
BackgroundBirdsong and human speech: common themes and mechanisms
Doupe, A. J., & Kuhl, P. K. (1999) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.22.1.567 · Cited by 1,633 · cited 4×
“The similarities include anatomical and functional features that characterize the pathways for vocal production, auditory processing and learning [1,2,7].”
BackgroundInfluentialPredictive coding under the free-energy principle
Friston, K., & Kiebel, S. (2009) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0300 · Cited by 1,579 · cited 1×
“The birdsong model described in [14] performs a Bayesian version of dynamical, predictive coding based on an internal generative model of how birdsong is produced [15].”
MethodSelf-citation
Research works printed but not linked: 2
- Flege J, MacKay I, Imai S (2010) What accounts for “age” effects on overall degree foreign accent? Wrembel, M Kul and K DziubalskaKolaczyk (eds) Achievements and Perspectives in SLA of Speech: New Sounds 2: 65.
- Friston K (2008) Hierarchical models in the brain. PLoS Comput Biol 4: e1000211. 18989391 PMC2570625 · DOI · Paper page
Cited by
Neuromorphic computing with nanoscale spintronic oscillators
Torrejon, J., Riou, M., Araujo, F. A., Tsunegi, S., Khalsa, G., Querlioz, D., Bortolotti, P., Cros, V., Yakushiji, K., Fukushima, A., et al. (2017) · Nature · doi:10.1038/nature23011 · Cited by 1,374
The Encoding of Speech Sounds in the Superior Temporal Gyrus
Yi, H. G., Leonard, M. K., & Chang, E. F. (2019) · Neuron · doi:10.1016/j.neuron.2019.04.023 · Cited by 417
“, forward transition probabilities for prediction) (Blank and Davis, 2016; Friston, 2005; Kiebel et al., 2009; Yildiz et al., 2013) and Hebbian learning processes for object recognition (Dan and Poo, 2004).”
BackgroundActive inference, communication and hermeneutics
Friston, K. J., & Frith, C. D. (2015) · Cortex · doi:10.1016/j.cortex.2015.03.025 · Cited by 266
“Using free energy minimisation, Yildiz, von Kriegstein, and Kiebel (2013) have shown that a hierarchy of nonlinear dynamical systems can learn speech samples rapidly and recognize them robustly, even in adverse conditions.”
BackgroundSpeech encoding by coupled cortical theta and gamma oscillations
Hyafil, A., Fontolan, L., Kabdebon, C., Gutkin, B., & Giraud, A.-L. (2015) · eLife · doi:10.7554/elife.06213 · Cited by 235
Ackermann, H., Hage, S. R., & Ziegler, W. (2014) · Behavioral and Brain Sciences · doi:10.1017/s0140525x13003099 · Cited by 215
Metadata
- Authors
- Yildiz, I. B., von Kriegstein, K., & Kiebel, S. J.
- Affiliations
- École Normale Supérieure - PSL (FR)
- Max Planck Institute for Human Cognitive and Brain Sciences (DE)
- Corresponding author
- Yildiz, Izzet B.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-09-12
- Volume
- 9
- Issue
- 9
- Pages
- e1003219
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC3772045
- OpenAlex
- W2163460643
- Cited by
- 55
- References
- 119
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 5 · 2023: 6 · 2022: 1 · 2021: 8 · 2020: 2 · 2019: 3 · 2018: 2 · 2017: 9 · 2016: 3 · 2015: 10 · 2014: 4
- FWCI
- 6.26
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- Funders
- Max-Planck-Institut für Kognitions- und Neurowissenschaften
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Life Sciences › Agricultural and Biological Sciences › Ecology, Evolution, Behavior and Systematics › Animal Vocal Communication and Behavior
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Keywords
- speech recognition, speech learning, bird song, songbirds, Bayesian inference, online learning
- MeSH terms
- Animals, Animal Communication, Bayes Theorem, Birds, Humans, Speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.