Relative positional embeddings with RoPE

Word order matters to a language model. In this post I'll explain RoPE: a technique that makes attention between two tokens depend on the distance between them instead of their absolute sentence position.

Self-attention is a core part of modern transformer-based language models. Self-attention adjusts the embedding of each token (or word) using the embeddings of its neighbors, so the token absorbs the context of its sentence. For example, the word "station" means different things in the sentences "I am listening to a radio station" and "I just left the train station".

Self-attention by itself is order-agnostic and would compute the same results for the sentence "left train I station the just" or "I just left the train station". That is a problem, because word order matters in language. How do we ensure that self-attention will consider token positions?

Position as a sinusoidal signal

The trick presented in the original "Attention is all you need" paper is to encode the position of a token into the embedding before computing attention values (Vaswani et al. 2017). These position encodings, or positional embeddings, come in several forms; a popular one is the sinusoidal positional embedding. The d-dimensional positional encoding for a position n is defined as follows:

rni={sin⁡(nLi/d)if i is evencos⁡(nL(i−1)/d)if i is oddr_{ni} = \begin{cases} \sin(\frac{n}{L^{i/d}}) & \text{if } i \text{ is even}\\ \cos(\frac{n}{L^{(i-1)/d}}) & \text{if } i \text{ is odd}\end{cases}

Let's plot these values for varying n and i to see what this encoding looks like.

The encoding lies in the range [-1, 1], and the lower embedding dimensions oscillate faster than the higher ones. This is similar to binary numbers, where the least significant bits flip far more often than the most significant bits (Bishop, 2023). This positional embedding vector is added to a token embedding before computing attention:

xn′=xn+rn\textbf{x}_n' = \textbf{x}_n + \textbf{r}_n

This approach encodes the absolute position into the token embedding. In many sequences, the relative ordering of words or tokens is more important. For example, in the sentences "I just left the train station" and "Because I was late, I left the train station," the link between "train" and "station" is the same, though the words sit at different positions in each.

This is where "Rotary Positional Encodings" (RoPE) come in (Su et al. 2023). RoPE similarly uses a sinusoidal signal to encode the position. However, instead of adding this signal to the original input vector, RoPE multiplies the sinusoidal signal with the input after projection into query or key space. For example, to compute a position-adjusted query vector from the original token at position mm, with the embedding dimension d=2d=2:

qm′=RWqxm=[cos⁡mθ−sin⁡mθsin⁡mθcos⁡mθ][Wq11Wq12Wq21Wq22][xm1xm2]\textbf{q}' _m = \textbf{R} \textbf{W}_q \textbf{x}_m = \begin{bmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta\end{bmatrix} \begin{bmatrix} W _q ^{11} & W _q ^{12} \\ W _q ^{21} & W _q ^{22}\end{bmatrix} \begin{bmatrix} x _m ^1 \\ x _m ^2\end{bmatrix}

Those familiar with linear algebra might recognize the rotation matrix. The input thus gets rotated, by an angle that depends on the token position mm. We generalize to embedding dimensions greater than two by expanding matrix R\textbf{R} as follows:

Rd=[cos⁡mθ1−sin⁡mθ100…00sin⁡mθ1cos⁡mθ100…0000cos⁡mθ2−sin⁡mθ2…0000sin⁡mθ2cos⁡mθ2…00⋮⋮⋮⋮⋱⋮⋮0000…cos⁡mθd/2−sin⁡mθd/20000…sin⁡mθd/2cos⁡mθd/2]\textbf{R}^d = \begin{bmatrix} \cos m\theta_1 & -\sin m\theta _1 & 0 & 0 & \ldots & 0 & 0\\ \sin m\theta _1 & \cos m\theta _1 & 0 & 0 & \ldots & 0 & 0\\ 0 & 0 & \cos m\theta _2 & -\sin m\theta _2 & \ldots & 0 & 0\\ 0 & 0 & \sin m\theta _2 & \cos m\theta _2 & \ldots & 0 & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots\\ 0 & 0 & 0 & 0 & \ldots & \cos m\theta _{d/2} & -\sin m\theta _{d/2}\\ 0 & 0 & 0 & 0 & \ldots & \sin m\theta _{d/2} & \cos m\theta _{d/2}\end{bmatrix}

Pairs of features get rotated with varying base frequencies θi\theta _i. θi\theta_i is defined similarly as in the absolute positional embeddings, with lower embedding dimensions rotating more quickly than higher embedding dimensions:

θi=1L2i/d\theta _i = \frac{1}{L^{2i/d}}

Rotated vectors have a benefit: the angle between two position-adjusted vectors at mm and nn will only depend on their relative position, not on the absolute positions mm and nn.

Animation showing how the rotation angle between two vectors only depends on their relative position.

Interpreting input features as complex numbers to derive the RoPE equation

How did RoPE's authors arrive at its rotation equation? To explain its derivation, let's recall that the attention weights are computed by taking the softmax of the inner product between query and key vectors. In the case of self-attention, the query and key vectors are derived from different tokens in the sequence:

anm=exp⁡⟨qnT,km⟩Σm′=1Nexp⁡⟨qnT,km′⟩a_{nm} = \frac{\exp{\langle \textbf{q}_n^{T}, \textbf{k}_m\rangle}}{\Sigma_{m'=1}^N \exp{\langle \textbf{q}_n^{T}, \textbf{k}_{m'}\rangle}}

When using absolute positional encodings, the query and key vectors will be derived from the position-adjusted vector xn′\textbf{x}_n'. For RoPE, we aim to find an alternative function ff to encode positional information, such that the inner product between two position-adjusted vectors only depends on the relative position between two tokens. Mathematically, the following should hold:

⟨f(q,m),f(k,n)⟩=g(q,k,n−m)(1)\langle f(\textbf{q}, m), f(\textbf{k}, n)\rangle = g(\textbf{q}, \textbf{k}, n - m) \tag{1}

To derive such a function, we turn to the complex domain. A d-dimensional vector vn∈Rd\textbf{v}_n \in \mathbb{R}^d can be interpreted as a complex (d/2)-dimensional vector vn′∈Cd/2\textbf{v}_n' \in \mathbb{C}^{d/2}. For example:

vn=[v1v2v3v4]→vn′=[v1+v2iv3+v4i]\textbf{v}_n = \begin{bmatrix} v_1\\ v_2\\ v_3\\ v_4\end{bmatrix} \rightarrow \textbf{v}_n' = \begin{bmatrix}v_1 + v_2i\\ v_3 + v_4i\end{bmatrix}

A complex number a+bia + bi can alternatively be written in polar form ReiθR e^{i\theta}, where RR and θ\theta are called the radial and angular part of a complex number, respectively, and which are defined as R=∣a+bi∣=a2+b2R = |a + bi| = \sqrt{a^2 + b^2} and θ=arctan⁡(b/a)\theta = \arctan(b/a).

The standard inner product on Cd\mathbb{C}^d is defined as ⟨u,v⟩=u‾Tv\langle \textbf{u}, \textbf{v}\rangle = \overline{\textbf{u}}^T\textbf{v}, that is, taking the complex conjugate and transpose of u\textbf{u} before multiplication with v\textbf{v}. Using this definition, and writing the complex numbers returned by ff and gg in polar form, we can get the following:

Rf(q,m)eiΘf(q,m)‾⋅Rf(k,n)eiΘf(k,n)=Rg(q,k,n−m)eiΘg(q,k,n−m)(2)\overline{R_f(\textbf{q}, m) e^{i\Theta_f(\textbf{q}, m)}} \cdot R_f(\textbf{k}, n) e^ {i\Theta_f(\textbf{k}, n)} \\= R_g(\textbf{q}, \textbf{k}, n-m) e^ {i\Theta_g(\textbf{q}, \textbf{k}, n-m)} \tag{2}

Here we have explicitly written the inner product between two complex numbers on the left-hand side, equating it to a complex number on the right-hand side, representing g(q,k,n−m)g(\textbf{q}, \textbf{k}, n-m). We have also defined functions Rf(q,m)R_f(\textbf{q}, m), Θf(q,m)\Theta_f(\textbf{q}, m), Rg(q,k,n−m)R_g(\textbf{q}, \textbf{k}, n-m), and Θg(q,k,n−m)\Theta_g(\textbf{q}, \textbf{k}, n-m), representing the radial and angular part of the complex numbers returned by ff and gg, respectively.

The conjugate of a complex number in polar form ReiθRe^{i\theta} is Re−iθRe ^{-i\theta}. With this definition, we can further simplify the left-hand side of Equation 2 by combining the product of two complex numbers:

Rf(q,m)Rf(k,n)ei(Θf(k,n)−Θf(q,m))=Rg(q,k,n−m)eiΘg(k,q,n−m)(3)R_f(\textbf{q}, m) R_f(\textbf{k}, n) e^{i(\Theta_f(\textbf{k}, n) - \Theta_f(\textbf{q}, m))} = R_g(\textbf{q}, \textbf{k}, n-m) e^{i\Theta_g(\textbf{k}, \textbf{q}, n-m)} \tag{3}

Equating the radial and angular part in Equation 3, we get:

Rf(q,m)Rf(k,n)=Rg(q,k,n−m)Θf(k,n)−Θf(q,m)=Θg(q,k,n−m)R_f(\textbf{q}, m) R_f(\textbf{k}, n) = R_g(\textbf{q}, \textbf{k}, n-m) \\ \Theta_f(\textbf{k}, n) - \Theta _f(\textbf{q}, m) = \Theta_g(\textbf{q}, \textbf{k}, n-m)

To make functions RfR_f and RgR_g more concrete, we start with a few simple initial conditions. Let's require that a token at the start of a sequence is unmodified by our positional embedding: f(v,0)=vf(\textbf{v}, 0) = \textbf{v}. Here v\textbf{v} can refer to q\textbf{q} or k\textbf{k}. We also define v=∣∣v∣∣eiθv=Rf(v,0)eiΘf(v,0)\textbf{v} = ||\textbf{v}|| e^{i\theta_v} = R_f(\textbf{v}, 0) e^{i\Theta_f(\textbf{v}, 0)}, that is, Rf(v,0)=∣∣v∣∣R_f(\textbf{v}, 0) = ||\textbf{v}||, and Θf(v,0)=θv\Theta_f(\textbf{v}, 0) = \theta_v, where θv\theta_v is the angular part of v\textbf{v}, that is the angular part of the reinterpreted values of q\textbf{q} or k\textbf{k} as a complex number.

Now, let's assess the case when n=mn=m:

Rf(q,m)Rf(q,m)=Rg(q,k,0)=Rf(q,0)Rf(k,0)=∣∣q∣∣∣∣k∣∣(4)R_f(\textbf{q}, m) R_f(\textbf{q}, m) = R_g(\textbf{q}, \textbf{k}, 0) = R_f(\textbf{q}, 0) R_f(\textbf{k}, 0) = ||\textbf{q}|| ||\textbf{k}|| \tag{4}

Here, we have applied Equation 3 to get the result Rg(q,k,0)R_g(\textbf{q}, \textbf{k}, 0), and applied it another time to get Rg(q,k,0)=Rf(q,0)Rf(k,0)R_g(\textbf{q}, \textbf{k}, 0) = R _f(\textbf{q}, 0) R _f(\textbf{k}, 0). We use our initial condition definitions to get the final result.

Equation 4 must hold for all token positions mm; the final result, however, doesn't depend on mm. Thus Rf(x,m)=∣∣x∣∣R_f(\textbf{x}, m) = ||\textbf{x}||.

What can we infer for the angular part when n=mn=m? Applying Equation 3 as we did for the radial part, we get:

Θf(k,m)−Θf(q,m)=Θg(q,k,0)=Θf(k,0)−Θf(q,0)=θk−θq(5)\Theta_f(\textbf{k}, m) - \Theta _f(\textbf{q}, m) = \Theta _g(\textbf{q}, \textbf{k}, 0) = \Theta _f(\textbf{k}, 0) - \Theta _f(\textbf{q}, 0) = \theta_k - \theta_q \tag{5}

Rearranging terms of the first and last part of Equation 5, we get:

Θf(k,m)−θk=Θf(q,m)−θq\Theta_f(\textbf{k}, m) - \theta_k = \Theta_f(\textbf{q}, m) - \theta_q

This holds for all q,k\textbf{q}, \textbf{k} and mm, which implies that the change in the angular part of the position-adjusted complex number is independent of the q\textbf{q} or k\textbf{k}.

This insight gives us some freedom to define Θf(v,m)\Theta_f(\textbf{v}, m). For example, we can define it as follows:

Θf(v,m)=θv+ϕ(m)\Theta_f(\textbf{v}, m) = \theta_v + \phi(m)

In other words, we add a position-dependent value ϕ(m)\phi(m) to the original angular part θv\theta_v. But what would ϕ(m)\phi(m) be? We can gain some insight by analyzing the difference between two subsequent positions, that is the case when n=m+1n = m +1.

Θf(k,m+1)−Θf(q,m)=θk+ϕ(m+1)−θq−ϕ(m)Θf(k,m+1)−Θf(q,m)+θq−θk=ϕ(m+1)−ϕ(m)Θg(q,k,1)+θq−θk=ϕ(m+1)−ϕ(m)\Theta_f(\textbf{k}, m+1) - \Theta _f(\textbf{q}, m) = \theta _k + \phi(m+1) - \theta_q - \phi(m) \\ \Theta _f(\textbf{k}, m+1) - \Theta _f(\textbf{q}, m) + \theta_q - \theta_k = \phi(m+1) - \phi(m)\\ \Theta _g(\textbf{q}, \textbf{k}, 1) + \theta_q - \theta_k = \phi(m+1) - \phi(m)

Here we moved θq\theta_q and θk\theta _k to the left-hand side to get an expression for ϕ(m+1)−ϕ(m)\phi(m+1) - \phi(m). By applying Equation 3, we transformed the left-hand side to an expression that does not depend on mm. This implies that the difference ϕ(m+1)−ϕ(m)\phi(m+1) - \phi(m) does not depend on mm either, and can be constant. We can define ϕ(m)\phi(m) as a simple arithmetic progression:

ϕ(m)=mθ+γ\phi(m) = m\theta + \gamma

In this definition, θ\theta and γ\gamma are free to choose. For example, we can set θ\theta to an embedding-dimension-specific value, and set γ=0\gamma = 0.

Combining all pieces, we arrive at the following:

Rf(v,m)=∣∣v∣∣Θf(v,m)=θv+mθf(v,m)=∣∣v∣∣eiθv+mθ=veimθR_f(\textbf{v}, m) = ||\textbf{v}||\\\Theta _f(\textbf{v}, m) = \theta _v + m\theta\\ f(\textbf{v}, m) = ||\textbf{v}||e^{i\theta _v + m\theta} = \textbf{v}e^{im\theta}

Through our constraint on relative position and a few simple initial conditions, we were able to define a simple expression that encodes the sequence position into a query or key vector. Finally, using Euler's formula, we get the rotation matrix we discussed at the beginning of this post:

eimθ=cos⁡mθ+isin⁡mθ=cos⁡mθ[1001]+sin⁡mθ[0−110]=[cos⁡mθ−sin⁡mθsin⁡mθcos⁡mθ]\begin{aligned}e^{im\theta} &= \cos m\theta + i\sin m\theta = \cos m\theta \begin{bmatrix}1 & 0\\0 & 1\end{bmatrix} + \sin m\theta \begin{bmatrix} 0 & -1 \\1 & 0\end{bmatrix} \\&= \begin{bmatrix} \cos m\theta & -\sin m\theta\\ \sin m\theta & \cos m\theta\end{bmatrix}\end{aligned}

Conclusion

Token order matters in language and in biological sequences. RoPE requires the inner product of position-adjusted query and key vectors to depend only on relative position, and that leads to a simple transformation: rotating input vectors in the complex domain. So when "train" and "station" sit side by side, attention sees them as neighbors, wherever in the sentence they fall.

References

Bishop, C. M., Bishop, H. & Cham, S. (ed.) (2023). Deep Learning - Foundations and Concepts. ISBN: 978-3-031-45468-4

Su, Jianlin, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. “RoFormer: Enhanced Transformer with Rotary Position Embedding.” arXiv:2104.09864. Preprint, arXiv, November 8, 2023. https://doi.org/10.48550/arXiv.2104.09864.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, et al. “Attention Is All You Need.” arXiv:1706.03762. Preprint, arXiv, August 2, 2023. https://doi.org/10.48550/arXiv.1706.03762.