CoChat

The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; none of the 55 sources its claims cite is checked for support yet.

Imperfect Captions Paired to Audiovisual Stimuli Improve Speech Recognition but Do Not Reduce Listening-Related Fatigue

Open access
Esser, Katie, Picou, Erin M., Hornsby, Benjamin W. Y. · 2025 · American Journal of Audiology

Abstract

Purpose: Understanding speech in challenging environments can be fatiguing for individuals with and without hearing loss. Empirical research examining interventions to reduce such fatigue, however, is limited. Our study investigated the effects of imperfect captions, such as those created using automated speech recognition systems, on listening-related fatigue during a challenging speech task. Method: Twenty-two adults (aged 18–63 years) with essentially normal hearing completed a sustained dual task designed to induce listening-related fatigue. The primary task was audiovisual sentence recognition in quiet, with speech levels individually adjusted for ~70% correct performance (without captions). The secondary task was response time (RT) to topic words presented during the speech task. Participants completed the dual task with or without captions. Subjective fatigue ratings were obtained before, during, and after the dual task. Vigilant attention was measured via the secondary task and before and after the dual task using visual stimuli (RTs to a visual marker presented at random intervals). Results: Subjective fatigue ratings increased significantly over the course of the dual task. However, this increase was larger for the group who had captions, even though they had better sentence recognition overall. Evidence of behavioral fatigue (slowed RTs over time) was also present but only for those in the Caption group. Anecdotal reports from study participants suggest that the increased fatigue for the Caption group was related to the cognitive challenge of combining information from the time-locked audiovisual cues and the time-delayed, and imperfect, text captions. Conclusions: Even though they were time-delayed and contained inaccuracies, captions improved speech recognition. However, this benefit was accompanied by greater increases in subjective and behavioral fatigue. Therefore, in some conditions imperfect captions can have negative consequences for listening-related fatigue. Further research is needed to determine whether this pattern holds in different circumstances, such as with audio-only stimuli. Supplemental Material: https://doi.org/10.23641/asha.29646482

Overview

CoChat Summary

TL;DR

This study tested whether imperfect captions paired with audiovisual speech reduce listening-related fatigue in 22 adults with essentially normal hearing during a sustained dual task. Captions improved sentence recognition but were associated with greater subjective and behavioral fatigue, not less.

Key findings

  • Sentence recognition was better for the Caption group than the No Caption group (77% vs. 63% average across blocks; main effect of group: F(1, 20) = 5.30, p = .032, partial η² = .21), with a significant Group × Block interaction; the Caption group was better at Block 1 and Block 2 (15% and 13% better; p = .014, .037) but not Block 3 (11% better; p = .07).
  • Subjective fatigue increased over time for both groups, but more for the Caption group: estimated marginal mean increased by 36.36 points from Block 0 to Block 3 in the Caption group versus 20 points in the No Caption group (Group × Block: F(3, 60) = 2.89, p < .01, partial η² = .13).
  • Topic-word RTs increased significantly from Block 2 to Block 3 only for the Caption group (55 ms, p = .047; No Caption group: −1 ms, p = .99), suggesting behavioral fatigue in the Caption group. PVT RTs increased from pre- to post-dual task for both groups (F(1, 4423.8) = 28.94, p < .001), with no significant Group × Time interaction (F(1, 20) = 0.01, p = .921).
  • Posttask quiz scores were higher for the Caption group overall (66% vs. 56%), but only for questions based on information presented early in the task, t(526) = −2.24, p = .026; later-block quiz questions were more likely correct for both groups.

Methods

  • Twenty-two adults aged 18–63 years (18 women; M = 36 years) with normal to near-normal hearing (mean binaural three-frequency PTA = 11.6 dB HL; range 2–26 dB HL) were randomly assigned to a Caption group (n = 11) or No Caption group (n = 11) in a between-groups design.
  • Participants completed a sustained ~50-min audiovisual sentence recognition dual task (AV CST) in quiet at individually adjusted low speech levels targeting ~70% correct without captions; the secondary task was speeded button-press RT to topic words. Captions had a 500-ms delay and only 60% of scoring key words accurately captioned (86%–88% of all spoken words accurately captioned).
  • Subjective fatigue was measured with the Right Now Fatigue Scale before, twice during, and after the dual task; behavioral measures included AV CST scores, topic-word RTs, and a Psychomotor Vigilance Task before and after the dual task, plus a posttask content quiz. Linear mixed-effects models included group, block, and their interaction as fixed factors and participant as a random intercept; RTs 3 SD above or below the mean were removed.

Limitations

  • Stated by the authors: the quiet, low-sensation-level audiovisual test condition may not generalize to noisy classroom or conference settings; captions contained deletion errors and a 500-ms delay but no substitution errors; the between-groups design reduced statistical power and could allow between-group differences in motivation, cognitive ability, or motor RTs to confound behavioral effort/fatigue estimates; topic-word RTs embedded in sentences are novel and may not optimally assess fatigue; participants had normal/near-normal hearing, so effects for listeners with hearing loss are unknown.
  • Evident from the text: the sample was small (n = 22; 11 per group), caption quality was fixed at one imperfect level, and there was no accurate-caption comparison, so the study cannot separate the effects of delay versus deletion errors or determine whether better captions would reduce fatigue.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1044/2025_aja-25-00018

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

8 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Figures

Figure 1. Schematic depicting the testing procedure in chronological order. PVT = Psychomotor Vigilance Task; RNFS = Right Now Fatigue Scale.
Figure 1. Schematic depicting the testing procedure in chronological order. PVT = Psychomotor Vigilance Task; RNFS = Right Now Fatigue Scale.CC BY-NC · PubMed Central
Figure 2. Mean sentence recognition performance as a function of test block for the No Caption group (red line) and the Caption group (blue line). Error bars represent ± 1 standard error. Analysis revealed the Caption group outperformed the No Caption group.
Figure 2. Mean sentence recognition performance as a function of test block for the No Caption group (red line) and the Caption group (blue line). Error bars represent ± 1 standard error. Analysis revealed the Caption group outperformed the No Caption group.CC BY-NC · PubMed Central
Figure 3. Mean subjective fatigue rating as a function of test block for the No Caption group (red line) and the Caption group (blue line). Block 0 occurred prior to the start of the dual task. Error bars represent ± 1 standard error. Ratings were not significantly different before the start of the dual task. Analysis of the group by block interaction revealed a significantly greater increase in ratings from Block 0 to Block 3 in the Caption group.
Figure 3. Mean subjective fatigue rating as a function of test block for the No Caption group (red line) and the Caption group (blue line). Block 0 occurred prior to the start of the dual task. Error bars represent ± 1 standard error. Ratings were not significantly different before the start of the dual task. Analysis of the group by block interaction revealed a significantly greater increase in ratings from Block 0 to Block 3 in the Caption group.CC BY-NC · PubMed Central
Figure 4. Mean response times (RTs) in milliseconds as a function of test block for the No Caption group (red line) and Caption group (blue line). Error bars represent ± 1 standard error. RTs were not significantly different between the groups overall. However, there was a significant increase in RTs from the second to the third block for the Caption group, suggestive of an increase in fatigue over time.
Figure 4. Mean response times (RTs) in milliseconds as a function of test block for the No Caption group (red line) and Caption group (blue line). Error bars represent ± 1 standard error. RTs were not significantly different between the groups overall. However, there was a significant increase in RTs from the second to the third block for the Caption group, suggestive of an increase in fatigue over time.CC BY-NC · PubMed Central
Figure 5. Mean response times (RTs) in milliseconds as a function of test block for the No Caption group (red line) and Caption group (blue line). Error bars represent ± 1 standard error. RTs increased significantly from the pretest to posttest block in both groups, consistent with an increase in fatigue. PVT = Psychomotor Vigilance Task.
Figure 5. Mean response times (RTs) in milliseconds as a function of test block for the No Caption group (red line) and Caption group (blue line). Error bars represent ± 1 standard error. RTs increased significantly from the pretest to posttest block in both groups, consistent with an increase in fatigue. PVT = Psychomotor Vigilance Task.CC BY-NC · PubMed Central
Figure 6. Mean quiz performance as a function of test block for the No Caption group (red line) and the Caption group (blue line). Error bars represent ± 1 standard error. Quiz performance was significantly different between groups only for quiz questions sourced from passages that were presented during the first block of the dual task.
Figure 6. Mean quiz performance as a function of test block for the No Caption group (red line) and the Caption group (blue line). Error bars represent ± 1 standard error. Quiz performance was significantly different between groups only for quiz questions sourced from passages that were presented during the first block of the dual task.CC BY-NC · PubMed Central

Verification

Claims the paper makes

0 Valid · 1 Not valid · 39 Not checked · 63 Own results and interpretations

30 more claims

Create a free account to see all 103 claims and to verify the 37 references still open.

See all claims — free account
  • Probably not a substantial claim. 46
  • It may state two separate claims as one. 29
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upF(3, 60) = 2.89reported p < .01 · recomputed p = .043

    “F(3, 60) = 2.89, p < .01”

  • ConsistentF(1, 20) = 5.3reported p = .032 · recomputed p = .032

    “F(1, 20) = 5.30, p = .032”

  • ConsistentF(2, 481) = 3.63reported p = .027 · recomputed p = .027

    “F(2, 481) = 3.63, p = .027”

  • ConsistentF(3, 60) = 33.92reported p < .0001 · recomputed p < .001

    “F(3, 60) = 33.92, p < .0001”

  • ConsistentF(1, 4423.8) = 28.94reported p < .001 · recomputed p < .001

    “F(1, 4423.8) = 28.94, p < .001”

5 of 8 · the rest aren’t loaded

3 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 46 references

Research works printed but not linked: 1

  1. Jiline, M., Kirk, D., Quirk, G., Sandler, M., & Monette, M. (2020). A review of state-of-the-art automatic speech recognition services for CART and CC applications. In Proceedings of the 2020 Broadcast Engineering and IT Conference.

Metadata

Authors
Esser, K., Picou, E. M., & Hornsby, B. W. Y.
Affiliations
  • Towson University (US)
  • Vanderbilt University Medical Center (US)
Corresponding author
Hornsby, Benjamin W. Y.
Venue
American Journal of Audiology
Published
2025-09-02
Volume
34
Issue
3
Pages
661-675
ISSN
  • 1059-0889 (ISSN-L)
  • 1558-9137
OpenAlex
W4413116510
Cited by
1
References
46
Citations by year
2026: 1
FWCI
1.22
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • American Journal of Audiology (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc)
  • Figshare (submittedVersion, cc-by)
  • figshare ASHA Publications (cc-by)
Funders
  • Vanderbilt University
  • Vanderbilt University Medical Center
  • National Institutes of Health
  • National Institute on Deafness and Other Communication Disorders (T35DC008763)
Topics
  • Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
  • Health Sciences › Health Professions › Speech and Hearing › Hearing Loss and Rehabilitation
  • Social Sciences › Psychology › Developmental and Educational Psychology › Hearing Impairment and Communication
Keywords
captions, audio-visual speech recognition, speech recognition (audiology), dual-task performance, speech intelligibility, auditory fatigue, sustained attention
MeSH terms
Adolescent, Adult, Attention, Fatigue, Female, Humans, Male, Middle Aged, Photic Stimulation, Reaction Time, Speech Perception, Speech Recognition Software, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-19, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-19 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Imperfect Captions Paired to Audiovisual Stimuli Improve Speech Recognition but Do Not Reduce Listening-Related Fatigue. https://app.cochat.ai/papers/10.1044/2025_aja-25-00018
The paper
Esser, K., Picou, E. M., & Hornsby, B. W. Y. (2025). Imperfect Captions Paired to Audiovisual Stimuli Improve Speech Recognition but Do Not Reduce Listening-Related Fatigue. American Journal of Audiology. https://doi.org/10.1044/2025_aja-25-00018

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.