Neural Machine Translation by Jointly Learning to Align and Translate

Previous versions of attention in machine learning have relied on recurrent processing, and in this way could be considered more similar to biological attention.

“Previous versions of attention in machine learning have relied on recurrent processing, and in this way could be considered more similar to biological attention (Mnih et al. 2014, Bahdanau et al. 2014).”

Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences.

“Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences [2, 19].”

Additive attention is computationally more expensive and less parallelizable than dot-product attention.

“The Additive Attention introduced in Bahdanau et al. (2014) is computationally more expensive (matrix operation + nonlinearity) and less parallelizable.”

Recurrent neural networks, including long short-term memory and gated recurrent networks, are state-of-the-art approaches in sequence modeling and transduction problems such as language modeling and machine translation.

“Recurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation [35, 2, 5].”

Pure model- and data-scaling based on recurrent models will not be enough to match the performance of attention-based architecture.

“This finding is non-surprising given the impact of the attention mechanism on the field of machine translation (Bahdanau et al., 2014), indicating that pure model- and data-scaling based on recurrent models will not be enough to match the performance of attention-based architecture (Rodchenko et al., 2025).”

The second wave was driven by attention mechanisms and sequence-to-sequence models, culminating in foundation models and large language models.

“The second wave was driven by the emergence of attention mechanisms and sequence-to-sequence models [6], culminating in foundation models and large language models [7].”

Bahdanau et al. (2014) introduced the attention mechanism into neural networks to address the challenge of aligning sentences across languages and improve translation quality.

“To address this challenge and improve translation quality, researchers in Bahdanau et al. (2014) introduced the attention mechanism into neural networks.”

The attention mechanism enables the network to selectively focus on specific sections of the source text during translation.

“This mechanism enables the network to selectively focus on specific sections of the source text during translation.”

Attention can be useful for learning context features related to the target and achieve joint alignment and translation, showing better performance for long sentences.

“Attention can be useful for learning context features related to the target and achieve joint alignment and translation, showing better performance for long sentences.”

Attention mechanism is used to attend to all source words (global attention) or only part of them (local attention) when decoding at each step of RNN [173], [174], [175].

“Later, attention mechanism is used to attend to all source words (i.e., global attention) or only part of them (i.e., local attention) when decoding at each step of RNN [173], [174], [175].”