Listening to variation: Human and Whisper responses to grammatical gender agreement
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; none of the 56 sources its claims cite is checked for support yet.
Abstract
Automatic speech recognition (ASR) systems are increasingly used as transcription tools in psycholinguistic research. The present study examines whether Whisper largev3 parallels human sensitivity to morphosyntactic variability in speech production. We focus on Spanish grammatical gender agreement in regular and dual-gendered nouns (DGNs). DGNs are feminine nouns that begin with stressed /a/ and often take masculine determiners (el agua), creating ambiguity between prescriptive rules and real-world usage. Using a controlled sentence-repetition paradigm, we compared native Spanish speakers and Whisper large-v3 in repetition/transcription accuracy and correction behavior when processing determiner-noun gender mismatches. Both humans and Whisper corrected prescriptively ungrammatical input for regular nouns. For DGNs, humans exhibited determiner-specific variability in correction shaped by collocational frequency, whereas Whisper defaulted to faithful transcription regardless of grammaticality or collocational frequency. These results indicate that Whisper behaves as though it favors prescriptively dominant forms in grammatically stable contexts, but shows little adjustment in the absence of reliable distributional cues. Confidence-score and n-best analyses suggested that, in rare DGN correction cases, the prescriptively incorrect determiner produced by humans often remained available among alternative hypotheses, but received a lower decoding score than the prescriptively correct form. Although Whisper can serve as an efficient transcription tool, high transcription accuracy does not entail human-like language processing. Structured human verification and linguistically informed evaluation remain essential when Whisper is used to transcribe or analyze morphosyntactic variation. More broadly, structured grammatical variability provides a testbed for improving Whisper evaluation and developing computational models that better align with human spokenlanguage behavior.
Overview
CoChat Summary
TL;DR
In a Spanish sentence-repetition task, human listeners and Whisper large-v3 both corrected gender-agreement violations for regular nouns, but only humans showed graded, usage-sensitive correction for dual-gendered nouns; Whisper largely transcribed DGN violations verbatim. The study argues Whisper is not a neutral transcription tool and should be supplemented with confidence/n-best diagnostics and human oversight in psycholinguistic research.
Key findings
- Humans showed a strong grammaticality effect for regular nouns: determiner accuracy was near ceiling on correct trials but dropped on incorrect trials (definite/indefinite: .998 vs. .466; demonstrative/possessive: .995 vs. .406). For dual-gendered nouns (DGNs), the effect was reduced and nearly absent with demonstratives/possessives (.821 correct vs. .826 incorrect). Mixed-effects model: grammaticality β = −4.06, noun type β = −1.03, grammaticality × noun type β = 5.62.
- Whisper large-v3 showed a weaker grammaticality effect for regular nouns (incorrect definite/indefinite: M = .596; demonstrative/possessive: M = .451) and no sensitivity to DGN violations (incorrect definite/indefinite: M = .964; demonstrative/possessive: M = .972). Mixed-effects model: grammaticality β = −4.34, grammaticality × noun type β = 8.92.
- Human correction rates were lower when corpus-based non-prescriptive determiner-noun usage was more frequent (β = −.12, p < .001; adjusted R² = .42), and DGNs had lower overall correction rates than regular nouns (β = −.17, p < .001). Whisper also showed an overall usage effect (β = −.17, p < .001; adjusted R² = .39), but this was driven only by regular nouns (β = −.32, p < .001), not DGNs (β = −.02, p = .822).
- Whisper determiner confidence was lower for regular-noun trials (M = .815, SD = .184) than DGN trials (M = .919, SD = .125). DGN corrections were rare (N = 35); in 29/35 trials both faithful and corrected forms appeared among top-5 beam hypotheses, and the corrected form had the higher score in 23/29 trials (M score difference = .038, SD = .063).
Methods
- Forty-one monolingual Spanish speakers from the University of Granada participated; three were excluded for low sentence-repetition accuracy, leaving 38 analyzed participants (valid N = 37 for demographics). Each completed 224 sentence-repetition trials, yielding 8,512 recordings.
- Stimuli followed a 2 (Noun Type: Regular vs. Dual-Gendered) × 2 (Grammaticality: Correct vs. Incorrect) design, with determiner types balanced; stimuli were curated from real-world tweets and recorded by a female native Andalusian Spanish speaker. In each trial, participants heard a sentence after a fixation/ear cue, had a 3000 ms listening window, then repeated aloud after a mouth cue until button press or 20000 ms.
- Whisper large-v3 was selected after benchmarking base, small, large-v3, and turbo models. Human coding used WER plus manual crosschecks; final valid human trials were 7,654 (91.7%), and valid Whisper trials were 5,699 (97.2% of 5,862 trials with verbatim target determiners). Analyses used mixed-effects logistic regressions for accuracy, linear regressions of noun-level correction rates on corpus non-prescriptive usage frequency from the Corpus del Español-Web/Dialects, and Whisper confidence scores plus n-best beam-search hypotheses (beam_size = 5).
Limitations
- Stated: grammatical gender distinctions in rapid or reduced speech are acoustically subtle; controlled laboratory recordings from a single trained speaker limit generalizability to spontaneous speech; the university sample limits broader demographic and dialectal generalization.
- Stated: Whisper failed to recover some target noun phrases, and these exclusions were nonrandom, concentrated in prescriptively correct DGN forms and prescriptively incorrect regular-noun forms; this may make the retained Whisper dataset less representative and likely underestimates Whisper’s DGN insensitivity.
- Stated: the confidence-score analysis was exploratory and based on a selected subset; conclusions are limited to observable input-output behavior from one ASR model family, with primary analyses focused on large-v3 and not other ASR systems; Whisper’s internal representations are not directly interpretable. Evident from the text: no formal inter-annotator agreement statistics were computed for the full dataset because coding was consensus-based, and DGN correction cases were rare (N = 35), making outcome-based confidence comparisons exploratory.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

![Figure 1 Non-prescriptive grammatical gender usage 3 as a function of log collocational frequency for regular nouns (orange) and dual-gendered nouns (green). Each point represents a noun extracted from the Corpus del Español-Web/Dialects[20]. Although prescriptive forms dominate overall, DGNs demonstrate higher rates of non-prescriptive usage than regular nouns.](https://figures.cochat.ai/paper-figures/f5a45906c8044763a36bd3cd82d6c013/listening-to-variation-human-and-whisper-responses-to-grammatical-gender-figure-1.png)






Verification
Claims the paper makes
0 Valid · 1 Not valid · 42 Not checked · 171 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Figure 7, regular nouns showed similar determiner confidence scores across faithful-transcription and speech-error-correction outcomes.”
§ Listening to Variation: Human and Whisper Responses to Grammatical Gender Agreement > 3. Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Grammaticality effects were much stronger for definites/indefinites than for demonstratives/possessives.”
§ Listening to Variation: Human and Whisper Responses to Grammatical Gender Agreement > 3. Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was also a significant main effect of noun type (β = -1.03,SE = .50,z = -2.07,p = .038),with lower determiner repetition accuracy for DGNs than regular nouns, even when collapsing across grammaticality and determiner type.”
§ Listening to Variation: Human and Whisper Responses to Grammatical Gender Agreement > 3. Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Whisper, in contrast, showed almost no sensitivity to incorrect gender agreement for DGNs.”
§ Listening to Variation: Human and Whisper Responses to Grammatical Gender Agreement > 4.2 RQ2:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The model confirmed a strong influence of real-world usage patterns on Whisper correction behavior (adjusted R² = .39,F(3, 330) = 70.4,p < .001).”
§ Listening to Variation: Human and Whisper Responses to Grammatical Gender Agreement > 3. Results
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
107 more claims
Create a free account to see all 214 claims and to verify the 47 references still open.
- It may state two separate claims as one. 71
- Probably not a substantial claim. 55
- The quote only partly states this claim. 1
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = -7.94reported p < .001 · recomputed p < .001
“,z = -7.94,p < .001”
- Consistentz = -2.07reported p = .038 · recomputed p = .038
“,z = -2.07,p = .038”
- Consistentz = 2.01reported p = .045 · recomputed p = .044
“,z = 2.01, p = .045”
- Consistentz = 5.44reported p < .001 · recomputed p < .001
“z = 5.44, p < .001”
- Consistentz = -3.09reported p = .002 · recomputed p = .002
“,z = -3.09,p = .002”
8 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Listening to Variation: Human and Whisper Responses to Grammatical Gender Agreement > 4.1 RQ1:
“However, unlike human participants, Whisper did not show graded variability associated with corpus collocational frequency.”
§ Listening to Variation: Human and Whisper Responses to Grammatical Gender Agreement > 3. Results
“Simple-slope analyses confirmed that corpus-based non-prescriptive usage strongly predicted reduced speech error correction rates only for regular nouns (β = -.32,95% CI [-.40, -.24], t = -8.14, p < .001)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Listening to Variation: Human and Whisper Responses to Grammatical Gender Agreement > 1.2 The Present Study
“In Lombardi and Potter [26], participants were instructed to recall sentences as accurately as possible after a brief intervening task.”
§ Listening to Variation: Human and Whisper Responses to Grammatical Gender Agreement > References
“26. J.E. Tree, P.J. Meijer, Building syntactic structure in speaking, Journal of Psycholinguistic Research.”
Citations
Cites
Speaking: From Intention to Articulation
Levelt, W. J. M. (1989) · doi:10.7551/mitpress/6393.001.0001 · Cited by 4,804
Speaking : From intention to articulation
Levelt, W. J. M. (1989) · Cited by 3,059
Robust Speech Recognition via Large-Scale Weak Supervision
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022) · arXiv · doi:10.48550/arxiv.2212.04356 · Cited by 1,953
Good-Enough Representations in Language Comprehension
Ferreira, F., Bailey, K. G. D., & Ferraro, V. (2002) · Current Directions in Psychological Science · doi:10.1111/1467-8721.00158 · Cited by 1,147
The Analysis of Sentence Production
Garrett, M. F. (1975) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60270-4 · Cited by 976
The ‘Good Enough’ Approach to Language Comprehension
Ferreira, F., & Patson, N. D. (2007) · Language and Linguistics Compass · doi:10.1111/j.1749-818x.2007.00007.x · Cited by 730
Confidence measures for speech recognition: A survey
Jiang, H. (2005) · Speech Communication · doi:10.1016/j.specom.2004.12.004 · Cited by 358
The regeneration of syntax in short term memory
Lombardi, L., & Potter, M. C. (1992) · Journal of Memory and Language · doi:10.1016/0749-596x(92)90036-w · Cited by 197
MONTRUL, S., DAVIDSON, J., DE LA FUENTE, I., & FOOTE, R. (2014) · Bilingualism: Language and Cognition · doi:10.1017/s1366728913000114 · Cited by 100
Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction
Bryant, C., Felice, M., & Briscoe, T. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1074 · Cited by 100
Research works printed but not linked: 2
- Curzan, 2013, Highly irregular, Mich. Today
- Davies, 2016, Corpus del Español: web/Dialects Corpus, Corpus Esp.
Cited by
Still being checked.
Metadata
- Authors
- Shen, Y., Mulík, S., Bajo, M. T., Dussias, P. E., & Beatty-Martínez, A. L.
- Affiliations
- University of California San Diego (US)
- Corresponding author
- Shen, Yumeng
- Venue
- Computer Speech & Language
- Publisher
- Elsevier BV
- Published
- 2027-02
- Volume
- 102
- Pages
- 102038
- ISSN
- 0885-2308 (ISSN-L)
- OpenAlex
- W7171678906
- Semantic Scholar
- e04dcd4ef7a3c652046b82bf45e1f2c220b0873c
- Cited by
- 0
- References
- 33
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
- Locations
- Computer Speech & Language (publishedVersion, cc-by-nc-nd)
- SSRN Electronic Journal (submittedVersion)
- Institutional Repository of the University of Granada (University of Granada) (publishedVersion, cc-by-nc-nd)
- Funders
- Ministerio de Economía y Competitividad
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- gender agreement, speech recognition, transcription accuracy, confidence scores, N-best decoding, spoken language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Listening to variation: Human and Whisper responses to grammatical gender agreement. https://app.cochat.ai/papers/10.1016/j.csl.2026.102038
- The paper
- Shen, Y., Mulík, S., Bajo, M. T., Dussias, P. E., & Beatty-Martínez, A. L. (2027). Listening to variation: Human and Whisper responses to grammatical gender agreement. Computer Speech & Language. https://doi.org/10.1016/j.csl.2026.102038
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.