
Relative positional embeddings with RoPE
Word order matters to a language model. In this post I'll explain RoPE: a technique that makes attention between two tokens depend on the distance between them instead of their absolute sentence position.

1 post