Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention

“Sood et al. (2020) propose a text saliency model which predicts fixation durations that are then used to compute the attention scores in a transformer network.”

“We find that the heuristic model correlates well with human reading, as has been reported in Sood et al. (2020b).”

“Seminal work by [51] who used synthetic data generated by E-Z Reader, a cognitive model of eye-movement-control in reading [46], demonstrated that synthetic eye movement data bears the potential to not only improve neural language models on standard NLP tasks such paraphrase generation, sentence compression or visual…”

“Prompt learning is originated from the field of natural language processing[25][26].”

“76% in MAF1, respectively, which provides justification for our model design choices.”

“The inductive bias of language processing models can be improved using the eye-tracking signal (Barrett et al., 2018; Klerke and Plank, 2019) and the modification leads to more “human-like” output in generative tasks (Takmaz et al., 2020; Sood et al., 2020b).”

“…create synthetic scanpaths but also, especially within NLP, to leverage them for different use cases: the interpretability of language models (LMs) (Sood et al., 2020a; Hol-lenstein et al., 2021, 2022; Merkx and Frank, 2021), enhancing the performance of LMs on downstream tasks (Barrett et al.,…”

“Eye-tracking data have also been used to enhance or constrain neural attention in (Barrett et al., 2018; Sood et al., 2020b,a; Takmaz et al., 2020).”

“It has proven to be a useful signal for inferring human cognitive processing (Sood et al., 2020; Hollen-stein and Zhang, 2019; Mathias et al., 2020).”

“…and Sood, Tannert, Frassinelli, Bulling, and Vu (2020) collected eye-tracking data in reading comprehension, and showed improvements in the accuracy of automated question answering by training models to predict human reading measures (Malmaud et al., 2020; Sood, Tannert, Mueller, & Bulling, 2020).”

“…object usage [3], detecting privacy-sensitive situations [38], finding attended objects [27], assisting large language models in classification tasks [16], enhancing visual tasks [44, 46], improving egocentric human motion prediction [45], and aiding natural language processing tasks [36].”

“First, it fa-081 cilitates seamless integration with existing LM-082 sentence representation, which is then fed into a newly initialized (series of) dense layer(s) with out-115 put neurons corresponding to the number of labels in the task.”

“In this context, the work by Sood et al. [2020] stood out for adopting human gaze information into neural attention mechanisms for natural language processing tasks.”

“Several studies have shown a strong correlation between human eye movements and attention patterns in transformer-based models [Wang et al., 2024a, Bensemann et al., 2022, Sood et al., 2020].”

“…et al., 2023; Yang & Hollenstein, 2023); named entity recognition (NER) (Hollenstein & Zhang, 2019); sentence compression and paraphrase generation (Sood et al., 2020); the generation of image captions (Takmaz et al., 2020); reading task classification (Hollenstein et al., 2021b); co-reference…”

“…and Sood, Tannert, Frassinelli, Bulling, and Vu (2020) collected eye-tracking data in reading comprehension, and showed improvements in the accuracy of automated question answering by training models to predict human reading measures (Malmaud et al., 2020; Sood, Tannert, Mueller, & Bulling, 2020).”

“A range of studies have already demonstrated their utility for a variety of purposes, including paraphrase generation and reinforcement learning with human feedback [16, 17, 43, 50, 68, 87].”

“…processes of language understanding (Reichle et al., 2003; Engbert et al., 2005), but the performance of computational language models can also be improved if their inductive bias is adjusted using human cognitive signals (Hollen-stein and Zhang, 2019; Sood et al., 2020; Deng et al., 2023a).”

“Some studies also use human attention as supervision to assist specific tasks (Sood et al., 2020; Bansal et al., 2023), or use sequencial models to predict eye-tracking data (Chersoni et al., 2022).”

“The seminal work of Sood et al. (2020), which integrates eye movement data generated by a computational cognitive model of eye-movement-control-during-reading for tasks such as sentence compression and paraphrase generation, demonstrated the potential of synthetic eye-gaze data.”

“We develop and release the OASST-ETC (OASST-Eye-tracking Corpus) 1 , a first-of-its-kind ET dataset that captures the eye movements of 24 participants as they read and evaluate responses generated with the OASST1 [53] dataset.”

“…robustness’ has been spotted (Goodkind and Bicknell, 2018); on the other hand, models combined with ET data to simulate human reading behaviour have been proposed (Malmaud et al., 2020; Sood et al., 2020; Reichle et al., 2003) to interpret LMs (Hollenstein et al., 2021; Eberle et al., 2022).”

“…measures, such as first-pass reading times and regression counts, cannot be re-used to analyze the readers’ scanpaths [von der Malsburg and Vasishth 2011] or for developing neural sequence architectures for eye-tracking-while-reading data [Bolliger et al. 2023; Deng et al. 2023; Sood et al. 2020].”

“Yet, there is a lack of research focused on designing an effective hierarchical feature extraction module specifically for fine-grained fashion retrieval.”

“The work most similar to us is Sood et al. (2020b), which investigates the integration of the gaze-based text saliency model with vanilla trans-formers (Vaswani et al., 2017) for directly incorporating gaze predictions into the attention mechanisms for paraphrase detection and sentence compression…”

“For example, there have been works to use behavioral signals to infer cognitive signals, which help in improving AI models’ content understanding abilities [54–57], content generation abilities [58–60], and alignment with human values [61–64].”

“Currently, the attention mechanism has made breakthroughs and is one of the hottest research directions in computer vision [33], image segmentation, natural language processing [34-37], etc.”

“Deep neural networks have been successfully used to solve many complicated learning tasks, for example, computer vision [1, 2, 3], natural language processing [4, 5, 6] and other engineering applications [7, 8].”

“As a consequence, we propose to design a multi-view architecture with inter-view attention (shown with orange color in Figure 1) that transfer attention scores from one view to another, following the cross-attention paradigm (Sood et al., 2020).”

“The paper (Sood et al., 2020b) evaluated correlations on the reading comprehension task for fine-tuned XLNet.”

“Application T I Data Integration [34] Text Comprehension ✓ [25] Attention [35] Reading Comprehension ✓ Real Input, Cross Att.”

“…signals causes performance improvement in tasks like visual and natural language question answering (Patro & Namboodiri, 2018; Khurana et al., 2023; Sood et al., 2020), text and image sentiment analysis (Khurana et al., 2023; Barrett et al., 2018; Fan et al., 2018), natural language inference…”

“Besides using human gaze as a ground truth to evaluate models’ attention, some works explored the integration of gaze as a supervisory signal to guide neural attention mechanisms, e.g. for NLP [97] or object grasping [98].”

“Obviously, the annotation is very labor-intensive and expensive in real-world scenarios, so an alternative is to use implicit signals such as eye gaze [9], [16], [17], [18].”

“The use of human gaze can be found in computer vision [30, 59, 70, 72, 73] and natural lanauge processing (NLP) [34, 57, 61, 76] tasks, such as visual question answering [51], object referral [63], paraphrasing and sentence compression [61], sequence labeling [34], classifying pronouns [71], key phrase extraction [76]…”

“Furthermore, there have been attempts to guide attention to improve interpretability or performance in downstream tasks (Deshpande and Narasimhan, 2020; Sood et al., 2020).”

“Moreover, E-Z Reader-based simulations over the CNN and DM corpora have demonstrated utility in downstream tasks, achieving top performance in text saliency prediction when modeled during pre-training (Sood et al., 2020).”

“Recently, however, eye-tracking data — especially from grammatical sentences — has been demonstrated to bear significant potential for enhancing language-related technological applications [Deng et al. 2023, 2024; Sood et al. 2020; Yang and Hollenstein 2023].”

“…combine eye-tracking data with word embeddings in neural models to improve NLP tasks, including sentiment analysis (Mishra et al., 2017) and named entity recognition (NER) (Hol-lenstein and Zhang, 2019) or revising neural attention (Barrett et al., 2018; Sood et al., 2020; Tak-maz et al., 2020).”

“There have been attempts in crowdsourcing image categories to record human gaze during task execution [107, 63, 90, 89, 44, 72, 87, 45].”

“…and interpretation, including the assessment of a model’s and cognitive plausibility (Bolliger et al., 2024; Beinborn and Hol-lenstein, 2024; Haller et al., 2024; Goodkind and Bicknell, 2018; Bensemann et al., 2022; Eberle et al., 2022; Sood et al., 2020a; Hollenstein and Beinborn, 2021).”

“Furthermore, there have been attempts to guide attention to improve interpretability or performance in downstream tasks (Deshpande and Narasimhan, 2020; Sood et al., 2020).”

“Besides, some researches also improve the prediction of gaze behavior such as scanpath and calibration process based on the connection between gaze and text [10, 27, 42].”

“Integrating human attention into neural network models has been applied to a large number of natural language processing tasks, such as prediction of multiword expressions (Rohanian et al., 2017), paraphrase generation (Sood et al., 2020), machine reading (Li et al., 2018).”

“Several studies found differences in eye movements be-tween ordinary reading and tasks that define a reading manner: skimming, speed reading and proof-reading (Just et al., 1982; Kaakinen and Hyönä, 2010; Schotter et al., 2014; Strukelj and Niehorster, 2018; Chen et al., 2023).”

“Following Sood et al. (2020), our work leverages the output of E-Z Reader, a computational cognitive model for automatic generation of reading scanpath trajectories (Reichle et al., 1998, 2003, 2009; Veldre et al., 2023).”

“Sood et al. (2020a) focused on the attention layer in deep learning models and eye movements in question-answering (QA) tasks.”

“On the other hand, eye-tracking data is also used to investigate the relationship between human behavior and neural attention (Hahn and Keller, 2018; Sood et al., 2020a; Brandl and Hollenstein, 2022).”

“…object usage [3], detecting privacy-sensitive situations [38], finding attended objects [27], assisting large language models in classification tasks [16], enhancing visual tasks [44, 46], improving egocentric human motion prediction [45], and aiding natural language processing tasks [36].”

“Several models utilize relations between the words to understand the given sentences, such as entities, part-of-speech (POS), and role semantics (Wu et al., 2018; Sood et al., 2020; Zhang et al., 2022).”

“Only a few studies embed a set of eye-movements (e.g. velocity, acceleration, pupil size) into a rich vector space (Sood et al., 2020; Takmaz et al., 2020; Park et al., 2019; Karessli et al., 2017).”

“Examples include action recognition, object classification, image captioning, and medical imaging analysis [1], [2], [3], [4].”

“, 2019), text (Sood et al., 2020), and regularly-spaced time series (Ismail et al.”

“…et al., 2016; Yang et al., 2016; Sadhu et al., 2021) or linguistic-visual attention (Antol et al., 2015; Shih et al., 2016; Zhu et al., 2018; Sood et al., 2020) along with datasets that exist for this purpose (e.g., Kontogiorgos et al., 2018; Chen et al., 2019; Shridhar et al., 2020), and…”

“Prompt learning [10], [41] has achieved remarkable success in the natural language processing (NLP) [42]–[44], providing a lightweight mechanism to condition model behavior via task-relevant cues.”

“Most existing approaches remain heuristic, rule-based, or hybrid, as cognitive modeling prioritizes interpretability.”

“Furthermore, researchers have successfully integrated these human gaze signals as a supervisory signal in neural attention layers to improve performance across various downstream natural language processing tasks (Sood et al., 2020b).”

“…gaze duration and fixation counts have been used to improve model performance in tasks like question answering, paraphrasing, and text simplification (Zhang and Hollenstein, 2024; Hollenstein, 2021; Sood et al., 2020); or to evaluate cognitive plausibility of language models (Eberle et al., 2022).”

“…additional signal when training models for various NLP tasks, including named entity recognition (Hollenstein et al., 2019; Tokunaga et al., 2017), paraphrasing (Sood et al., 2020b), part-of-speech tagging (Barrett et al., 2018), and sentiment analysis (see also Mathias et al. (2020) for a review).”

“…interpretation models [5] paves the way to adapt cognitively-inspired saliency mechanisms applied to text comprehension and summarisation systems [19], [20] to systems devoted to the interpretation and summarisation of structured data, e.g., in the field of STI [9], [8], and entity summarisation.”

“Prior work reports that gaze regularization improves attention alignment (Pani & Yang, 2025; Yu et al., 2017; Sood et al., 2020); our results agree on attention, but the gain does not carry over to choices and is matched by a constant average gaze map, echoing VQA gains from human-attention…”

“Modelling the relationship between human eye gaze and tasks that people perform while looking at images is a long-standing research challenge in computer vision and human-centred computing [Borji et al. 2012; Sood et al. 2023, 2021, 2020; Sugano and Bulling 2016; Yarbus 1967].”

“Sood et al.’s (2020b) joint modeling approach— training the NLP model to solve the task and predict human gaze simultaneously— outperformed state-of-the-art baselines.”

“At the same time, SSMs have made significant progress in the field of NLP. [2].”

“…, ∗ These authors contributed equally. which aims to estimate the sequence of word fixations during reading, while also supporting down-stream applications that leverage gaze to improve NLP models (Mishra et al., 2017; Hollenstein and Zhang, 2019; Sood et al., 2020; Khurana et al., 2023, i.a.).”

“Studies suggest that cognitive models like E-Z Reader (Reichle et al., 2003), which simulate gaze patterns during reading, can improve language models in standard NLP tasks (Sood et al., 2020).”