Foundations

The math and mental model behind attention.

Reading level

Before you write a single line of transformer code, you need three things in your head: why attention exists, what the residual stream really is, and what positional encoding actually encodes. This chapter walks you through those, anchored to the original 2017 paper.

After this chapter

Explain attention without waving your hands.

§1 · 6 min read

What attention buys you

RNNs sequential, convolutions local — attention is both global and parallel.

Before transformers, sequence models were either recurrent (LSTM, GRU) or convolutional. RNNs handled long-range dependencies in principle but, because each step depended on the last, they couldn't be parallelised across time. Convolutions parallelised but only saw a fixed local window. Both bled accuracy at long sequence lengths because gradients had to propagate through many steps.

Attention is the answer: every token in the sequence directly looks at every other token, in one matrix multiplication. The result is a model that is global in receptive field and parallel in compute, so it scales gracefully on modern accelerators. The cost is quadratic memory — but that's a tractable engineering problem, not a learning problem.

Prerequisite map

What you need before transformers

100%

ArcXiv concept graph

The concept stack underneath a transformer block.

Builds on

§2 · 8 min read

Scaled dot-product attention

Three projections, one softmax, one matmul. That's it.

Each token starts as a vector in the residual stream. Attention turns each token into three projections: a query asking 'what am I looking for?', a key answering 'this is what I am', and a value carrying the information to mix back in. Queries dot with keys to score relevance, the scores are normalised by a softmax, and the values are mixed in proportion to those scores.

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V

Attention is softmax-normalised dot products of queries with keys, used to weight values.

The $d_{k}$ divisor is not cosmetic. Without it, as the head dimension grows, the dot products $Q K^{⊤}$ scale with $d_{k}$ in standard deviation. That pushes the softmax inputs into a regime where one entry dominates, the distribution collapses to almost one-hot, and gradients through the softmax vanish. The scale factor keeps softmax temperature roughly invariant to $d_{k}$ .

AsideWhy softmax, specifically?

Any monotone normaliser would give you a distribution. Softmax wins because (a) it's differentiable everywhere, (b) the gradient has a clean form $diag (p) - p p^{⊤}$ , and (c) the exponential makes the model commit — a small score advantage compounds into a clear winner. Linear attention drops the softmax and pays for it with much worse pattern selectivity.

Self-check

Why divide attention scores by √d_k?

§3 · 5 min read

Positional encoding

Attention is order-blind. Position has to be encoded by hand.

Self-attention is permutation-equivariant: shuffle the input tokens and the output shuffles the same way. Language is not permutation-equivariant — 'dog bites man' and 'man bites dog' mean different things. Something has to carry order, and that something is the positional encoding.

The original paper added a fixed sinusoidal vector to the token embedding before the first attention layer. Modern systems prefer rotary positional embeddings (RoPE), which rotate Q and K in 2-D subspaces by an angle proportional to position. RoPE encodes *relative* position naturally and extrapolates better to longer contexts.

Tools