Deep Learning

Relative positional embeddings with RoPE

In language models, the order of tokens is of critical importance. This post explains RoPE: a technique that ensures that the pairwise token attention weights only depend on their relative position.