CoChat

Listening to variation: Human and Whisper responses to grammatical gender agreement

Open access
Shen, Yumeng, Mulík, Stanislav, Bajo, María Teresa, Dussias, Paola E., Beatty-Martínez, Anne L. · 2027 · Computer Speech & Language

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; none of the 56 sources its claims cite is checked for support yet.

Abstract

Automatic speech recognition (ASR) systems are increasingly used as transcription tools in psycholinguistic research. The present study examines whether Whisper largev3 parallels human sensitivity to morphosyntactic variability in speech production. We focus on Spanish grammatical gender agreement in regular and dual-gendered nouns (DGNs). DGNs are feminine nouns that begin with stressed /a/ and often take masculine determiners (el agua), creating ambiguity between prescriptive rules and real-world usage. Using a controlled sentence-repetition paradigm, we compared native Spanish speakers and Whisper large-v3 in repetition/transcription accuracy and correction behavior when processing determiner-noun gender mismatches. Both humans and Whisper corrected prescriptively ungrammatical input for regular nouns. For DGNs, humans exhibited determiner-specific variability in correction shaped by collocational frequency, whereas Whisper defaulted to faithful transcription regardless of grammaticality or collocational frequency. These results indicate that Whisper behaves as though it favors prescriptively dominant forms in grammatically stable contexts, but shows little adjustment in the absence of reliable distributional cues. Confidence-score and n-best analyses suggested that, in rare DGN correction cases, the prescriptively incorrect determiner produced by humans often remained available among alternative hypotheses, but received a lower decoding score than the prescriptively correct form. Although Whisper can serve as an efficient transcription tool, high transcription accuracy does not entail human-like language processing. Structured human verification and linguistically informed evaluation remain essential when Whisper is used to transcribe or analyze morphosyntactic variation. More broadly, structured grammatical variability provides a testbed for improving Whisper evaluation and developing computational models that better align with human spokenlanguage behavior.

Overview

CoChat Summary

TL;DR

In a Spanish sentence-repetition task, human listeners and Whisper large-v3 both corrected gender-agreement violations for regular nouns, but only humans showed graded, usage-sensitive correction for dual-gendered nouns; Whisper largely transcribed DGN violations verbatim. The study argues Whisper is not a neutral transcription tool and should be supplemented with confidence/n-best diagnostics and human oversight in psycholinguistic research.

Key findings

  • Humans showed a strong grammaticality effect for regular nouns: determiner accuracy was near ceiling on correct trials but dropped on incorrect trials (definite/indefinite: .998 vs. .466; demonstrative/possessive: .995 vs. .406). For dual-gendered nouns (DGNs), the effect was reduced and nearly absent with demonstratives/possessives (.821 correct vs. .826 incorrect). Mixed-effects model: grammaticality β = −4.06, noun type β = −1.03, grammaticality × noun type β = 5.62.
  • Whisper large-v3 showed a weaker grammaticality effect for regular nouns (incorrect definite/indefinite: M = .596; demonstrative/possessive: M = .451) and no sensitivity to DGN violations (incorrect definite/indefinite: M = .964; demonstrative/possessive: M = .972). Mixed-effects model: grammaticality β = −4.34, grammaticality × noun type β = 8.92.
  • Human correction rates were lower when corpus-based non-prescriptive determiner-noun usage was more frequent (β = −.12, p < .001; adjusted R² = .42), and DGNs had lower overall correction rates than regular nouns (β = −.17, p < .001). Whisper also showed an overall usage effect (β = −.17, p < .001; adjusted R² = .39), but this was driven only by regular nouns (β = −.32, p < .001), not DGNs (β = −.02, p = .822).
  • Whisper determiner confidence was lower for regular-noun trials (M = .815, SD = .184) than DGN trials (M = .919, SD = .125). DGN corrections were rare (N = 35); in 29/35 trials both faithful and corrected forms appeared among top-5 beam hypotheses, and the corrected form had the higher score in 23/29 trials (M score difference = .038, SD = .063).

Methods

  • Forty-one monolingual Spanish speakers from the University of Granada participated; three were excluded for low sentence-repetition accuracy, leaving 38 analyzed participants (valid N = 37 for demographics). Each completed 224 sentence-repetition trials, yielding 8,512 recordings.
  • Stimuli followed a 2 (Noun Type: Regular vs. Dual-Gendered) × 2 (Grammaticality: Correct vs. Incorrect) design, with determiner types balanced; stimuli were curated from real-world tweets and recorded by a female native Andalusian Spanish speaker. In each trial, participants heard a sentence after a fixation/ear cue, had a 3000 ms listening window, then repeated aloud after a mouth cue until button press or 20000 ms.
  • Whisper large-v3 was selected after benchmarking base, small, large-v3, and turbo models. Human coding used WER plus manual crosschecks; final valid human trials were 7,654 (91.7%), and valid Whisper trials were 5,699 (97.2% of 5,862 trials with verbatim target determiners). Analyses used mixed-effects logistic regressions for accuracy, linear regressions of noun-level correction rates on corpus non-prescriptive usage frequency from the Corpus del Español-Web/Dialects, and Whisper confidence scores plus n-best beam-search hypotheses (beam_size = 5).

Limitations

  • Stated: grammatical gender distinctions in rapid or reduced speech are acoustically subtle; controlled laboratory recordings from a single trained speaker limit generalizability to spontaneous speech; the university sample limits broader demographic and dialectal generalization.
  • Stated: Whisper failed to recover some target noun phrases, and these exclusions were nonrandom, concentrated in prescriptively correct DGN forms and prescriptively incorrect regular-noun forms; this may make the retained Whisper dataset less representative and likely underestimates Whisper’s DGN insensitivity.
  • Stated: the confidence-score analysis was exploratory and based on a selected subset; conclusions are limited to observable input-output behavior from one ASR model family, with primary analyses focused on large-v3 and not other ASR systems; Whisper’s internal representations are not directly interpretable. Evident from the text: no formal inter-annotator agreement statistics were computed for the full dataset because coding was consensus-based, and DGN correction cases were rare (N = 35), making outcome-based confidence comparisons exploratory.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.csl.2026.102038

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Figures

Image 1
CC-BY-NC-ND · From the PDF
Figure 1 Non-prescriptive grammatical gender usage 3 as a function of log collocational frequency for regular nouns (orange) and dual-gendered nouns (green). Each point represents a noun extracted from the Corpus del Español-Web/Dialects[20]. Although prescriptive forms dominate overall, DGNs demonstrate higher rates of non-prescriptive usage than regular nouns.
Figure 1 Non-prescriptive grammatical gender usage 3 as a function of log collocational frequency for regular nouns (orange) and dual-gendered nouns (green). Each point represents a noun extracted from the Corpus del Español-Web/Dialects[20]. Although prescriptive forms dominate overall, DGNs demonstrate higher rates of non-prescriptive usage than regular nouns.CC-BY-NC-ND · From the PDF
Figure 2 An overview of the experimental workflow. Experimental conditions manipulated noun type, grammaticality, and determiner context in target determiner-noun phrases. Participants completed a
Figure 2 An overview of the experimental workflow. Experimental conditions manipulated noun type, grammaticality, and determiner context in target determiner-noun phrases. Participants completed aCC-BY-NC-ND · From the PDF
Figure 3 A flowchart of Whisper transcription pipeline and coding procedure. Trials were sequentially screened at Step 3.a.; trials were excluded if participants failed to produce the target noun verbatim (see Table 3, Example 1) or failed to preserve the same determiner type (see Table 3, Example 2). Trials were then screened at Step 3.b.; trials in which participants failed to produce the target determiner verbatim (see Table 3, Examples 3 and 5) were excluded from Whisper transcription analyses but retained for human repetition/speech error correction analyses. Finally, trials were excluded at Step 3.c. if Whisper failed to transcribe the target noun phrase (see Table 3, Example 4).
Figure 3 A flowchart of Whisper transcription pipeline and coding procedure. Trials were sequentially screened at Step 3.a.; trials were excluded if participants failed to produce the target noun verbatim (see Table 3, Example 1) or failed to preserve the same determiner type (see Table 3, Example 2). Trials were then screened at Step 3.b.; trials in which participants failed to produce the target determiner verbatim (see Table 3, Examples 3 and 5) were excluded from Whisper transcription analyses but retained for human repetition/speech error correction analyses. Finally, trials were excluded at Step 3.c. if Whisper failed to transcribe the target noun phrase (see Table 3, Example 4).CC-BY-NC-ND · From the PDF
Figure 4 Participant-wise Spearman rank correlations ( 𝜌 ) between Whisper and human condition-level accuracy for determiners (left) and nouns (right). Points represent individual participants; error bars show ± 1 standard error; asterisks denote two-sided Wilcoxon signed-rank tests against zero (*** p <.001; ** p <.01; * p <.05).
Figure 4 Participant-wise Spearman rank correlations ( 𝜌 ) between Whisper and human condition-level accuracy for determiners (left) and nouns (right). Points represent individual participants; error bars show ± 1 standard error; asterisks denote two-sided Wilcoxon signed-rank tests against zero (*** p <.001; ** p <.01; * p <.05).CC-BY-NC-ND · From the PDF
Figure 5 Determiner repetition accuracy for regular nouns (left panel) and DGNs (right panel) as a function of grammaticality (correct vs. incorrect) and determiner type (definite/indefinite vs. demonstrative/possessive) for human repetition (top) and Whisper large-v3 transcription (bottom). Bars
Figure 5 Determiner repetition accuracy for regular nouns (left panel) and DGNs (right panel) as a function of grammaticality (correct vs. incorrect) and determiner type (definite/indefinite vs. demonstrative/possessive) for human repetition (top) and Whisper large-v3 transcription (bottom). BarsCC-BY-NC-ND · From the PDF
Figure 6 Relationship between corpus-based non-prescriptive grammatical gender frequency (log-scaled) and speech error correction rates for each determiner-noun phrase for human (left panel) and Whisper large-v3 (right panel). The y-axis is reversed for visual alignment with Figure 5, such that 1.00 indicates consistent speech error correction and 0.00 indicates verbatim repetition by humans or faithful
Figure 6 Relationship between corpus-based non-prescriptive grammatical gender frequency (log-scaled) and speech error correction rates for each determiner-noun phrase for human (left panel) and Whisper large-v3 (right panel). The y-axis is reversed for visual alignment with Figure 5, such that 1.00 indicates consistent speech error correction and 0.00 indicates verbatim repetition by humans or faithfulCC-BY-NC-ND · From the PDF
Figure 7 Whisper large-v3 determiner confidence scores by noun type and transcription outcome. All plotted trials had prescriptively incorrect input determiners. Bars show mean Whisper largev3 determiner confidence scores, with 95% confidence intervals; points represent trial-level determiner confidence values. The left panel shows regular nouns (orange), and the right panel shows DGNs (green).
Figure 7 Whisper large-v3 determiner confidence scores by noun type and transcription outcome. All plotted trials had prescriptively incorrect input determiners. Bars show mean Whisper largev3 determiner confidence scores, with 95% confidence intervals; points represent trial-level determiner confidence values. The left panel shows regular nouns (orange), and the right panel shows DGNs (green).CC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

0 Valid · 1 Not valid · 42 Not checked · 171 Own results and interpretations

107 more claims

Create a free account to see all 214 claims and to verify the 47 references still open.

See all claims — free account
  • It may state two separate claims as one. 71
  • Probably not a substantial claim. 55
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentz = -7.94reported p < .001 · recomputed p < .001

    “,z = -7.94,p < .001”

  • Consistentz = -2.07reported p = .038 · recomputed p = .038

    “,z = -2.07,p = .038”

  • Consistentz = 2.01reported p = .045 · recomputed p = .044

    “,z = 2.01, p = .045”

  • Consistentz = 5.44reported p < .001 · recomputed p < .001

    “z = 5.44, p < .001”

  • Consistentz = -3.09reported p = .002 · recomputed p = .002

    “,z = -3.09,p = .002”

5 of 13 · the rest aren’t loaded

8 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 38 references

Research works printed but not linked: 2

  1. Curzan, 2013, Highly irregular, Mich. Today
  2. Davies, 2016, Corpus del Español: web/Dialects Corpus, Corpus Esp.

Cited by

Still being checked.

Metadata

Authors
Shen, Y., Mulík, S., Bajo, M. T., Dussias, P. E., & Beatty-Martínez, A. L.
Affiliations
University of California San Diego (US)
Corresponding author
Shen, Yumeng
Venue
Computer Speech & Language
Publisher
Elsevier BV
Published
2027-02
Volume
102
Pages
102038
ISSN
0885-2308 (ISSN-L)
OpenAlex
W7171678906
Cited by
0
References
33
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Computer Speech & Language (publishedVersion, cc-by-nc-nd)
  • SSRN Electronic Journal (submittedVersion)
  • Institutional Repository of the University of Granada (University of Granada) (publishedVersion, cc-by-nc-nd)
Funders
  • Ministerio de Economía y Competitividad
  • National Science Foundation
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
gender agreement, speech recognition, transcription accuracy, confidence scores, N-best decoding, spoken language processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Listening to variation: Human and Whisper responses to grammatical gender agreement. https://app.cochat.ai/papers/10.1016/j.csl.2026.102038
The paper
Shen, Y., Mulík, S., Bajo, M. T., Dussias, P. E., & Beatty-Martínez, A. L. (2027). Listening to variation: Human and Whisper responses to grammatical gender agreement. Computer Speech & Language. https://doi.org/10.1016/j.csl.2026.102038

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.