Do speech features for detecting cognitive load depend on specific languages?
The DOI matches its Crossref record; 31 of the 32 references it cites were found in a registry or an index.
Abstract
Speech-based cognitive load modeling recently proposed in English have enabled objective, quantitative and unobtrusive evaluation of cognitive load without extra equipment. However, no evidence indicates that these techniques could be applied to speech data in other languages without modification. In this study, a modified Stroop Test and a Reading Span Task were conducted to collect speech data in English and Chinese respectively, from which twenty non-linguistic features were extracted to investigate whether they were language dependent. Some discriminating speech features were observed language dependent, which serves as an evidence that there is a necessity to adapt speech-based cognitive load detection techniques to diverse language contexts for a higher performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Here, speech features such as the flow of words (e.g., tempo [42] or pauses [117,118]), but also more nuanced features such as the lexical density [115], the spectral domain [259], or the pitch contour [39] while talking were shown to be indicators of changes in participants' cognitive workload.”
From A Survey on Measuring Cognitive Workload in Human-Computer Interaction (2023) · cites this paper as [42]
Citations
Cites
Cognitive Load Theory and Instructional Design: Recent Developments
Paas, F., Renkl, A., & Sweller, J. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_1 · Cited by 3,041
Cognitive Load Theory and the Format of Instruction
Chandler, P., & Sweller, J. (1991) · Cognition and Instruction · doi:10.1207/s1532690xci0804_2 · Cited by 2,894 · cited 1×
“It is attributable to the limited capacity of the person's working memory and his or her ability to process novel information [4].”
BackgroundCognitive Load Measurement as a Means to Advance Cognitive Load Theory
Paas, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_8 · Cited by 2,527 · cited 1×
“Evidence has been found in [16] that highest performance could be achieved by maintaining the load experienced by users within an appropriate range, neither too low nor too high.”
BackgroundAn automated version of the operation span task
Unsworth, N., Heitz, R. P., Schrock, J. C., & Engle, R. W. (2005) · Behavior Research Methods · doi:10.3758/bf03192720 · Cited by 1,861 · cited 1×
“The reading span task is based on CLSE-Span [24] and RSPAN [22].”
MethodProceedings of Interspeech 2013
Lu, L., Ghoshal, A. K., & Renals, S. J. (2013) · Proceedings - International Symposium on Computer Architecture/Proceedings · Cited by 1,003
Modeling Player Experience for Content Creation
Pedersen, C., Togelius, J., & Yannakakis, G. N. (2010) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2010.2043950 · Cited by 231 · cited 1×
“These formulas are similar to those proposed in [17] to capture the spatial diversity of the placement of gaps.”
MethodAutomated stress detection using keystroke and linguistic features: An exploratory study
Vizer, L. M., Zhou, L., & Sears, A. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.07.005 · Cited by 220 · cited 2×
“[23] applied five machine learning techniques to automatically differentiate cognitive and physical stress from baseline based on keystroke and linguistic features extracted from spontaneously generated text.”
MethodA spectral/temporal method for robust fundamental frequency tracking
Zahorian, S. A., & Hu, H. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2916590 · Cited by 180 · cited 2×
“A Pitch-based Voice Activity Detector (VAD), YAAPT [31], was adopted to extract pitch of an utterance so as to distinguish voiced and unvoiced region, where unvoiced region was masked to 0 by pitch.”
MethodAssessing Cognitive Load with Physiological Sensors
Ikehara, C. S., & Crosby, M. E. (2005) · Proceedings of the 38th Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2005.103 · Cited by 150 · cited 3×
“Ikehara and Crosby [7] embedded two features of mouse pressure into their physiological sensor system, P1 and P2.”
BackgroundInterpreting Symptoms of Cognitive Load in Speech Input
Berthold, A., & Jameson, A. (1999) · Courses and lectures · doi:10.1007/978-3-7091-2490-1_23 · Cited by 102 · cited 2×
“Cognitive load evaluation based on speech features has been broadly exploited [1, 2, 6, 15].”
BackgroundInvestigation of spectral centroid features for cognitive load classification
Le, P. N., Ambikairajah, E., Epps, J., Sethu, V., & Choi, E. H. C. (2011) · Speech Communication · doi:10.1016/j.specom.2011.01.005 · Cited by 87 · cited 1×
“Investigated frame level speech features include Pitch, Intensity and MFCC (Mel-frequency cepstral coefficients) [30]; SCA (Spectral centroid amplitude) and SCF (Spectral centroid frequency) [13], and Formant [25].”
BackgroundSpeech-based cognitive load monitoring system
Yin, B., Chen, F., Ruiz, N., & Ambikairajah, E. (2008) · 2008 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2008.4518041 · Cited by 87 · cited 2×
“Features extracted from pen-input trajectories [14, 19], keystrokes [23], mouse pressure [7], eye movements [7] or speech [5, 21, 28] were examined.”
MethodRecognizing Time Pressure and Cognitive Load on the Basis of Speech: An Experimental Study
Müller, C. A., Großmann-Hutter, B., Jameson, A., Rummer, R., & Wittig⋆, F. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44566-8_3 · Cited by 86 · cited 1×
“Cognitive load evaluation based on speech features has been broadly exploited [1, 2, 6, 15].”
BackgroundAnalysis and detection of cognitive load and frustration in drivers' speech
Bořil, H., Sadjadi, S. O., Kleinschmidt, T., & Hansen, J. H. L. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-208 · Cited by 83
Rothkrantz, L. J. M., Wiggers, P., van Wees, J.-W. A., & van Vark, R. J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30120-2_57 · Cited by 79 · cited 4×
“[18] quantitatively analyzed some statistical features from speech signals in a Stroop Test conducted on 108 native speakers of Dutch to examine the effect of workload on speech production.”
BackgroundMethodResultInfluentialKhawaja, M. A., Chen, F., & Marcus, N. (2014) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2013.860579 · Cited by 67 · cited 1×
“They furthered this study in the same laboratory setup in 2014 [9].”
Automatic cognitive load detection from speech features
Yin, B., Ruiz, N., Chen, F., & Khawaja, M. A. (2007) · Proceedings of the 19th Australasian conference on Computer-Human Interaction: Entertaining User Interfaces · doi:10.1145/1324892.1324946 · Cited by 55 · cited 1×
“Investigated frame level speech features include Pitch, Intensity and MFCC (Mel-frequency cepstral coefficients) [30]; SCA (Spectral centroid amplitude) and SCF (Spectral centroid frequency) [13], and Formant [25].”
BackgroundTowards Automatic Cognitive Load Measurement from Speech Analysis
Yin, B., & Chen, F. (2007) · Springer eBooks · doi:10.1007/978-3-540-73105-4_111 · Cited by 51 · cited 19×
“[28] Yin, B., Chen, F., Ruiz, N. and Ambikairajah, E. 2008.”
BackgroundMethodResultInfluentialFormant Frequencies under Cognitive Load: Effects and Classification
Yap, T. F., Epps, J., Ambikairajah, E., & Choi, E. H. C. (2010) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/2011/219253 · Cited by 36 · cited 3×
“Some studies were based on frame features with GMM [25, 27, 31].”
BackgroundMethodAssessment of a User’s Time Pressure and Cognitive Load on the Basis of Features of Speech
Jameson, A., Kiefer, J., Müller, C. A., Großmann-Hutter, B., Wittig⋆, F., & Rummer, R. (2010) · Cognitive technologies · doi:10.1007/978-3-540-89408-7_9 · Cited by 32 · cited 5×
“Response latency (lr) [8, 12] increased significantly when load increased from low to medium.”
BackgroundResultInfluentialUsing language complexity to measure cognitive load for adaptive interaction design
Khawaja, M. A., Chen, F., & Marcus, N. (2010) · Proceedings of the 15th international conference on Intelligent user interfaces · doi:10.1145/1719970.1720024 · Cited by 32 · cited 3×
“Khawaja, Chen, and Marcus [10] proposed a novel speech content analysis approach to evaluate the current cognitive load of users based on their language and dialogue complexity.”
BackgroundEffect of cognitive load on articulation rate and formant frequencies during simulator flights
Huttunen, K. H., Keränen, H. I., Pääkkönen, R. J., Eskelinen-Rönkä, R. P., & Leino, T. K. (2011) · The Journal of the Acoustical Society of America · doi:10.1121/1.3543948 · Cited by 28 · cited 1×
“Cognitive load evaluation based on speech features has been broadly exploited [1, 2, 6, 15].”
BackgroundPotential speech features for cognitive load measurement
Khawaja, M. A., Ruiz, N., & Chen, F. (2007) · Proceedings of the 19th Australasian conference on Computer-Human Interaction: Entertaining User Interfaces · doi:10.1145/1324892.1324902 · Cited by 26 · cited 17×
“[28] Yin, B., Chen, F., Ruiz, N. and Ambikairajah, E. 2008.”
BackgroundInfluentialSlyh, R. E., Nelson, W. T., & Hansen, E. G. (1999) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1999.758345 · Cited by 26 · cited 3×
“Features extracted from pen-input trajectories [14, 19], keystrokes [23], mouse pressure [7], eye movements [7] or speech [5, 21, 28] were examined.”
BackgroundCognitive Load Measurement from User’s Linguistic Speech Features for Adaptive Interaction Design
Khawaja, M. A., Chen, F., Owen, C., & Hickey, G. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-03655-2_54 · Cited by 21 · cited 1×
“showed significant differences between low and high cognitive load tasks [11].”
BackgroundGlottal features for speech-based cognitive load classification
Yap, T. F., Epps, J., Choi, E. H. C., & Eliathamby Ambikairajah, A. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5494987 · Cited by 18 · cited 1×
“Glottal features [24, 26] have also been coupled to improve the classification accuracy.”
MethodA New Technique for the Estimation of Jitter and Shimmer of Voiced Speech Signal
Shahnaz, C., Zhu, W., & Ahmad, M. O. (2006) · Conference proceedings - Canadian Conference on Electrical and Computer Engineering · doi:10.1109/ccece.2006.277799 · Cited by 14 · cited 1×
“Those differences were expected as jitter and shimmer are laryngeal behaviors during speech production and are dependent of physical - psychological state and the linguistic structure of language [20].”
BackgroundInvestigating speech features and automatic measurement of cognitive load
Yin, B., Ruiz, N., Chen, F., & Eliathamby Ambikairajah, A. (2008) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2008.4665218 · Cited by 13 · cited 2×
“As Stroop Test is a designed experiment, it cannot represent the speech under spontaneous circumstance, such as in traffic management scenarios [27] or on the Reading-Comprehension dataset [29]; Secondly, all participants in Stroop Test have Chinese as their first language and English the second, and no participants h…”
MethodResultAutomatic Speech Feature Extraction for Cognitive Load Classification
Gorovoy, K., Tung, J. Y., & Poupart, P. (2010) · CMBES Proceedings · Cited by 13
Automatic cognitive load evaluation using writing features: An exploratory study
Lin, T., Xie, T., Chen, Y., & Tang, N. (2013) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2013.02.002 · Cited by 11 · cited 2×
“[14] studied 21 potential writing features extracted from sentence-making training to classify three levels of cognitive load, where participants were required to compose sentences based on given words.”
BackgroundSelf-citationCognitive skills learning: pen input patterns in computer-based athlete training
Ruiz, N., Feng, Q. Q., Taib, R., Handke, T., & Chen, F. (2010) · International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction · doi:10.1145/1891903.1891955 · Cited by 10 · cited 2×
“Features extracted from pen-input trajectories [14, 19], keystrokes [23], mouse pressure [7], eye movements [7] or speech [5, 21, 28] were examined.”
Background
Research works printed but not linked: 1
- Campione , E. and Véronis , J . 2002. A large-scale multilingual study of silent pause duration . In Proceedings of the Speech Prosody 2002 Conference (Aix-en-Provence , 2002 ), 199–202. Campione, E. and Véronis, J. 2002. A large-scale multilingual study of silent pause duration. In Proceedings of the Speech Prosody 2002 Conference (Aix-en-Provence, 2002), 199–202.
Cited by
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023) · ACM Computing Surveys · doi:10.1145/3582272 · Cited by 304
“The fixation rate of an area of interest describes the number of times an object was looked at – this can be interpreted as a repeated interest in an area and relevant for the current cognitive activity [208].”
Estimating cognitive load from speech gathered in a complex real-life training exercise
Vukovic, M., Sethu, V., Parker, J., Cavedon, L., Lech, M., & Thangarajah, J. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.12.003 · Cited by 21
Human-centered Behavioral and Physiological Security
Alt, F., Hassib, M. E., & Distler, V. (2023) · New Security Paradigms Workshop · doi:10.1145/3633500.3633504 · Cited by 12
“A change in the users’ cognitive load can be measured through many of the features of speech, such as tempo [25], pauses [51], or lexical density [51].”
BackgroundBroadening the mind: how emerging neurotechnology is reshaping HCI and interactive system design
Schneegass, C., Wilson, M. L., Shaban, J., Niess, J., Chiossi, F., Mitrevska, T., & Woźniak, P. W. (2024) · i-com · doi:10.1515/icom-2024-0007 · Cited by 11
“Speech analysis can help estimate, e.g. mental workload through sentence length or complexity, tempo, 26 or pauses.”
Influence of Cognitive Load on Voice Production: A Scoping Review
Pyfrom, M., Lister, J. J., & Anand, S. (2023) · Journal of Voice · doi:10.1016/j.jvoice.2023.08.024 · Cited by 9
Metadata
- Authors
- Chen, R., Xie, T., Xie, Y., Lin, T., & Tang, N.
- Affiliations
- Sichuan University (CN)
- University of Maryland, Baltimore County (US)
- Venue
- Proceedings of the 18th ACM International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2016-10-31
- Pages
- 76-83
- OpenAlex
- W2546712879
- Cited by
- 8
- References
- 32
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 3 · 2022: 1 · 2020: 1 · 2018: 1
- FWCI
- 0.39
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Funders
- Science and Technology Supporting Program, Sichuan Province (2013GZX0138,2014GZ0154)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- cognitive load detection, speech features, language dependence, Chinese speech, English speech, Stroop task, reading span task, cross-lingual adaptation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.