Forgetting Transformer: Softmax Attention with a Forget Gate

Lin, Zhixuan, Nikishin, Evgenii, He, Xu Owen · arXiv · 2025