CoChat

NSX-Net: A Neurolinguistic and Acoustic Multimodal Deep Learning Framework for Speech Disorder Classification

Open access

The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index; 2 of the 5 sources its claims cite were checked for support.

Abstract

Background/Objectives: Speech disorders caused by neurological, articulatory, cognitive, and behavioral impairments require accurate multimodal analysis frameworks capable of jointly understanding acoustic abnormalities and neurolinguistic inconsistencies for reliable computer-assisted speech disorder assessment. Recent multimodal deep learning approaches have utilized speech signals, linguistic transcripts, attention mechanisms, and contextual fusion strategies to improve disordered speech analysis and intelligent disorder prediction. However, existing methods frequently suffer from insufficient temporal synchronization, ineffective local–global feature learning, multimodal redundancy, poor interpretability, and reduced robustness under heterogeneous speech disorders. To address these challenges, Methods: The proposes NSX-Net (NeuroSpeech Explainable Network), an intelligent multimodal deep learning framework for speech disorder classification using Acoustic Speech Signal Data and Neurolinguistic Text/Linguistic Data. Initially, the Adaptive Speech Refinement Module (ASRM) performs noise removal, silence elimination, signal normalization, spectrogram generation, MFCC extraction, and transcript preprocessing to improve multimodal speech consistency. Subsequently, the Hierarchical Multimodal Feature Learning Unit (HMFLU) extracts discriminative feature representations through the Cross-Domain Representation Encoder (CDRE), Fine-Grained Speech Pattern Analyzer (FGSPA), and Global Sequential Dependency Learner (GSDL) for capturing both local articulation abnormalities and long-range semantic dependencies. Furthermore, the Dual-Path Attention Enhancement Block (DPAEB) emphasizes clinically important disordered speech regions using adaptive local–global attention mechanisms, while the Temporal Resolution Synchronization Module (TRSM) aligns rhythm-level and beat-level multimodal contextual structures to improve temporal consistency and suppress noise disturbances. Results: Experimental evaluation across six publicly available multimodal speech disorder datasets demonstrated that NSX-Net achieved superior performance, with 99.27% Accuracy, 99.11% Precision, 98.97% Recall, 99.02% F1-Score, and 99.41% AUC, significantly outperforming existing state-of-the-art frameworks. Conclusions: Finally, optimized multimodal contextual representations are forwarded into a Softmax classification layer for intelligent multiclass speech disorder prediction with improved interpretability and potential clinical applicability.

Overview

CoChat Summary

## TL;DR

NSX-Net is a multimodal deep learning framework for speech disorder classification that combines acoustic speech signals and neurolinguistic text/language features with attention and temporal synchronization modules. The paper reports strong results across six public datasets, including 99.27% Accuracy and 99.41% AUC, while noting that broader real-world and multilingual validation is still needed.

## Key findings

  • Experimental evaluation across six publicly available multimodal speech disorder datasets showed NSX-Net achieved 99.27% Accuracy, 99.11% Precision, 98.97% Recall, 99.02% F1-Score, and 99.41% AUC.
  • In one comparison, NSX-Net outperformed baselines including NAA (91.42%), MML (93.65%), BioNeuroFusionNet (96.21%), TMTLN (95.38%), and CogniAlign (96.87%) with 99.12% classification accuracy.
  • The paper reports additional values of Recall (98.84%), Precision (98.91%), F1-Score (98.96%), MCC (98.41%), Specificity (99.02%), and AUC (99.31%) in another experimental summary.
  • The authors state that visualization and quantitative analyses showed improved context learning, temporal synchronization, attention optimization, noise robustness, cross-dataset generalization, and severe speech disorder detection.

## Methods

  • The study proposes NSX-Net (NeuroSpeech Explainable Network), a multimodal deep learning framework that combines acoustic speech signal data and neurolinguistic text/linguistic data.
  • The framework uses ASRM for preprocessing, HMFLU with CDRE, FGSPA, and GSDL for feature extraction, DPAEB for attention enhancement, TRSM for temporal synchronization, and a Softmax classification layer for multiclass prediction.
  • Experiments were run in Python 3.11 with PyTorch; transcript analysis used HuggingFace Transformers API and NLTK, and acoustic preprocessing used LibROSA and Praat. Evaluation used five-fold cross-validation, early stopping, categorical cross-entropy loss, and Adam with learning rate 0.0001 and batch size 32.

## Limitations

  • The paper states that NSX-Net was evaluated only on publicly available speech disorder datasets, and that robustness in real-world clinical environments, multilingual populations, and unseen recording conditions needs further validation.
  • It also states that the current framework does not perform disorder severity estimation, progression monitoring, or longitudinal patient assessment.
  • The paper notes that NSX-Net currently uses only acoustic speech signals and neurolinguistic textual information, and future work will explore additional modalities such as articulatory movement, facial expressions, EEG, or neuroimaging.
  • Evident from the text: the evaluation is described as offline rather than real-time, so the paper does not demonstrate live clinical deployment.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/diagnostics16152377

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 HMFLU module hierarchically integrates cross-domain multimodal encoding, fine-grained articulation analysis, and global sequential feature learning to generate highly discriminative multimodal speech representations for accurate speech disorder classification.
Figure 1 HMFLU module hierarchically integrates cross-domain multimodal encoding, fine-grained articulation analysis, and global sequential feature learning to generate highly discriminative multimodal speech representations for accurate speech disorder classification.CC BY · PubMed Central
Figure 2 CDRE module jointly learns global semantic dependencies and local fine-grained disordered speech representations through adaptive channel–spatial attention mechanisms to generate enriched multimodal contextual features for robust speech disorder analysis.
Figure 2 CDRE module jointly learns global semantic dependencies and local fine-grained disordered speech representations through adaptive channel–spatial attention mechanisms to generate enriched multimodal contextual features for robust speech disorder analysis.CC BY · PubMed Central
Figure 3 DPAEB module enhances multimodal disordered speech representations by jointly integrating spatial, channel, and pixel attention mechanisms to emphasize clinically significant articulation regions and contextual semantic abnormalities for improved interpretability and classification performance.
Figure 3 DPAEB module enhances multimodal disordered speech representations by jointly integrating spatial, channel, and pixel attention mechanisms to emphasize clinically significant articulation regions and contextual semantic abnormalities for improved interpretability and classification performance.CC BY · PubMed Central
Figure 4 TRSM module adaptively synchronizes rhythm-level and beat-level multimodal feature representations through temporal attention learning and multimodal alignment mechanisms to enhance temporal consistency, contextual reliability, and disordered speech understanding.
Figure 4 TRSM module adaptively synchronizes rhythm-level and beat-level multimodal feature representations through temporal attention learning and multimodal alignment mechanisms to enhance temporal consistency, contextual reliability, and disordered speech understanding.CC BY · PubMed Central
Figure 5 Demonstration that abnormal speech signals exhibit unstable pitch variations, increased jitter and shimmer values, reduced harmonic-to-noise ratio (HNR), and distorted spectrotemporal patterns compared with stable harmonic structures observed in normal speech.
Figure 5 Demonstration that abnormal speech signals exhibit unstable pitch variations, increased jitter and shimmer values, reduced harmonic-to-noise ratio (HNR), and distorted spectrotemporal patterns compared with stable harmonic structures observed in normal speech.CC BY · PubMed Central
Figure 6 Comparative performance analysis demonstrates that NSX-Net consistently achieves superior convergence stability and higher classification performance than existing multimodal frameworks across accuracy, precision, recall, F1-score, MCC, and AUC metrics during training.
Figure 6 Comparative performance analysis demonstrates that NSX-Net consistently achieves superior convergence stability and higher classification performance than existing multimodal frameworks across accuracy, precision, recall, F1-score, MCC, and AUC metrics during training.CC BY · PubMed Central
Figure 7 Contextual and robustness evaluation demonstrates that NSX-Net consistently outperforms existing multimodal frameworks in context learning efficiency, temporal synchronization, attention optimization, noise robustness, cross-dataset generalization, and severe disordered speech disorder detection across training epochs.
Figure 7 Contextual and robustness evaluation demonstrates that NSX-Net consistently outperforms existing multimodal frameworks in context learning efficiency, temporal synchronization, attention optimization, noise robustness, cross-dataset generalization, and severe disordered speech disorder detection across training epochs.CC BY · PubMed Central
Figure 8 Demonstration that speech disorders exhibit reduced pitch stability, increased pitch variance, decreased rhythmic consistency, and higher voice break abnormalities compared with normal speech, thereby highlighting significant acoustic biomarkers associated with neurological speech disorders.
Figure 8 Demonstration that speech disorders exhibit reduced pitch stability, increased pitch variance, decreased rhythmic consistency, and higher voice break abnormalities compared with normal speech, thereby highlighting significant acoustic biomarkers associated with neurological speech disorders.CC BY · PubMed Central
Figure 9 The attention and feature learning analysis illustrates that NSX-Net achieves superior attention stability, semantic alignment, contextual dependency learning, temporal preservation, attention drift reduction, and feature interpretability compared with existing multimodal speech disorder classification frameworks across different training epochs.
Figure 9 The attention and feature learning analysis illustrates that NSX-Net achieves superior attention stability, semantic alignment, contextual dependency learning, temporal preservation, attention drift reduction, and feature interpretability compared with existing multimodal speech disorder classification frameworks across different training epochs.CC BY · PubMed Central
Figure 10 The fusion and robustness analysis demonstrates that NSX-Net consistently achieves superior multimodal fusion efficiency, synchronization accuracy, noise suppression, cross-dataset robustness, disordered speech localization, and redundant feature elimination compared with existing multimodal speech analysis frameworks across training epochs.
Figure 10 The fusion and robustness analysis demonstrates that NSX-Net consistently achieves superior multimodal fusion efficiency, synchronization accuracy, noise suppression, cross-dataset robustness, disordered speech localization, and redundant feature elimination compared with existing multimodal speech analysis frameworks across training epochs.CC BY · PubMed Central
Figure 11 Demonstration that NSX-Net effectively captures clinically significant disordered speech regions through ASRM preprocessing, FGSPA local articulation attention, DPAEB contextual attention enhancement, and TRSM temporal synchronization for robust multimodal speech disorder analysis.
Figure 11 Demonstration that NSX-Net effectively captures clinically significant disordered speech regions through ASRM preprocessing, FGSPA local articulation attention, DPAEB contextual attention enhancement, and TRSM temporal synchronization for robust multimodal speech disorder analysis.CC BY · PubMed Central
Figure 12 Demonstration that NSX-Net effectively transforms irregular token-acoustic temporal misalignment into synchronized multimodal feature representations using TRSM, while DPAEB further enhances clinically significant articulation attention regions. (A) Temporal Misalignment in Pathological Speech (Before Synchronization), (B) Temporal Alignment after TRSM Synchronization, (C) Attention Focus Shift Enabled by DPAEB after Synchronization. Red dashed boxes indicate misaligned disordered speech regions before synchronization, and green dashed boxes represent synchronized and attention-refined speech regions after contextual alignment.
Figure 12 Demonstration that NSX-Net effectively transforms irregular token-acoustic temporal misalignment into synchronized multimodal feature representations using TRSM, while DPAEB further enhances clinically significant articulation attention regions. (A) Temporal Misalignment in Pathological Speech (Before Synchronization), (B) Temporal Alignment after TRSM Synchronization, (C) Attention Focus Shift Enabled by DPAEB after Synchronization. Red dashed boxes indicate misaligned disordered speech regions before synchronization, and green dashed boxes represent synchronized and attention-refined speech regions after contextual alignment.CC BY · PubMed Central
Figure 13 The clinical robustness evaluation demonstrates that NSX-Net consistently achieves superior clinical reliability, diagnostic consistency, disordered speech understanding, interpretability, severity prediction accuracy, and real-time applicability compared with existing multimodal speech disorder analysis frameworks across training epochs.
Figure 13 The clinical robustness evaluation demonstrates that NSX-Net consistently achieves superior clinical reliability, diagnostic consistency, disordered speech understanding, interpretability, severity prediction accuracy, and real-time applicability compared with existing multimodal speech disorder analysis frameworks across training epochs.CC BY · PubMed Central
Figure 14 The clinical adaptability analysis demonstrates that NSX-Net consistently achieves superior rehabilitation monitoring, emotional speech adaptability, cross-clinical deployment scalability, and clinical decision support capability compared with existing multimodal speech pathology assessment frameworks across training epochs.
Figure 14 The clinical adaptability analysis demonstrates that NSX-Net consistently achieves superior rehabilitation monitoring, emotional speech adaptability, cross-clinical deployment scalability, and clinical decision support capability compared with existing multimodal speech pathology assessment frameworks across training epochs.CC BY · PubMed Central

Verification

Claims the paper makes

1 Valid · 6 Not valid · 2 Not checked · 51 Own results and interpretations

12 more claims

Create a free account to see all 60 claims and to verify the 3 references still open.

See all claims — free account
  • It may state two separate claims as one. 26
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 20

Create a free account to see these claims.

Citations

Cites

Show all 39 references

Research works printed but not linked: 5

  1. Wang, J., Jiang, H., Liu, Y., Ma, C., Zhang, X., Pan, Y., Liu, M., Gu, P., Xia, S., and Li, W. (2024). A comprehensive review of multimodal large language models: Performance and challenges across different tasks. arXiv.
  2. Hu, G., Xin, Y., Lyu, W., Huang, H., Sun, C., Zhu, Z., Gui, L., Cai, R., Cambria, E., and Seifi, H. (2024). Recent trends of multimodal affective computing: A survey from NLP perspective. arXiv.
  3. Farabi, S., Ranasinghe, T., Kanojia, D., Kong, Y., and Zampieri, M. (2024). A survey of multimodal sarcasm detection. arXiv.
  4. Anuprabha, M., Gurugubelli, K., Kesavaraj, V., and Vuppala, A.K. (2025). A multi-modal approach to dysarthria detection and severity assessment using speech and text information. Proceedings of the ICASSP 2025—2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Hyderabad, India, 6–11 April 2025, IEEE.
  5. Chhabra, A., and Vishwakarma, D.K. (2024). Mhs-stma: Multimodal hate speech detection via scalable transformer-based multilevel attention framework. arXiv.

Cited by

Still being checked.

Metadata

Authors
Nadeem, A., Khan, M. Z., Sirshar, M., & Thirunavukkarasu, U.
Affiliations
  • King Salman Center for Disability Research (SA)
  • Saveetha University (IN)
Corresponding author
Khan, Mohammad Zubair
Venue
Diagnostics
Publisher
MDPI AG
Published
2026-07-28
Volume
16
Issue
15
Pages
2377
ISSN
2075-4418 (ISSN-L)
OpenAlex
W7171521658
Cited by
0
References
38
FWCI
0.00
Type
journal article
Language
en
Open access
Locations
  • Diagnostics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
King Salman Center for Disability Research (KSRG-2026-341)
Topics
  • Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords
multimodal deep learning, acoustic features, neurogenic speech disorders, global-local attention, temporal synchronization, explainable neural networks, speech preprocessing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: NSX-Net: A Neurolinguistic and Acoustic Multimodal Deep Learning Framework for Speech Disorder Classification. https://app.cochat.ai/papers/10.3390/diagnostics16152377
The paper
Nadeem, A., Khan, M. Z., Sirshar, M., & Thirunavukkarasu, U. (2026). NSX-Net: A Neurolinguistic and Acoustic Multimodal Deep Learning Framework for Speech Disorder Classification. Diagnostics. https://doi.org/10.3390/diagnostics16152377

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.