On the token distance modeling ability of higher RoPE attention dimension

Hong, Xiangyu, Jiang, Che, Qi, Biqing · arXiv · 2024