Transformers: a research guide

Before transformers, sequence models were either recurrent (LSTM, GRU) or convolutional. RNNs handled long-range dependencies in principle but, because each step depended on the last, they couldn't be parallelised across time. Convolutions parallelised but only saw a fixed local window. Both bled accuracy at long sequence lengths because gradients had to propagate through many steps.

Attention is the answer: every token in the sequence directly looks at every other token, in one matrix multiplication. The result is a model that is global in receptive field and parallel in compute, so it scales gracefully on modern accelerators. The cost is quadratic memory — but that's a tractable engineering problem, not a learning problem.

Each token starts as a vector in the residual stream. Attention turns each token into three projections: a query asking 'what am I looking for?', a key answering 'this is what I am', and a value carrying the information to mix back in. Queries dot with keys to score relevance, the scores are normalised by a softmax, and the values are mixed in proportion to those scores.

The $\sqrt{d_k}$ divisor is not cosmetic. Without it, as the head dimension grows, the dot products $Q{K}^{\top }$ scale with $\sqrt{d_k}$ in standard deviation. That pushes the softmax inputs into a regime where one entry dominates, the distribution collapses to almost one-hot, and gradients through the softmax vanish. The scale factor keeps softmax temperature roughly invariant to $d_k$.

Self-attention is permutation-equivariant: shuffle the input tokens and the output shuffles the same way. Language is not permutation-equivariant — 'dog bites man' and 'man bites dog' mean different things. Something has to carry order, and that something is the positional encoding.

The original paper added a fixed sinusoidal vector to the token embedding before the first attention layer. Modern systems prefer rotary positional embeddings (RoPE), which rotate Q and K in 2-D subspaces by an angle proportional to position. RoPE encodes *relative* position naturally and extrapolates better to longer contexts.

Tokens are first embedded into the residual stream and combined with a positional signal. From there, every block reads from the residual stream, computes something, and writes the result *back* to the same stream via a residual addition. That's the central engineering trick: the residual stream is a shared bus, and each block is a side-channel that adds its contribution without erasing the rest.

Inside one block, the two side-channels are attention (mixes information across positions) and the MLP (mixes information within a position). LayerNorm sits before each, in the now-dominant pre-norm configuration, to keep activations well-scaled at every depth. Without it, deep stacks become unstable around block 30.

A single attention head learns one similarity metric. That's fragile: a head can be the right tool for 'match the subject to its verb' but the wrong tool for 'attend to the previous punctuation'. Multi-head attention runs $h$ independent attentions in parallel, each with its own learned $Q,K,V$ projections, then concatenates the outputs and projects back to model dimension.

In practice each head specialises. Mechanistic-interpretability work has named specific head types: induction heads, name-mover heads, copy-suppression heads. The model doesn't 'know' it's doing this — it falls out of training.

At training time, the model sees the whole sequence at once. At inference time it generates one token at a time, and every step would otherwise recompute attention over the entire past. The trick is that for past tokens the $K$ and $V$ projections never change — only the current token's $Q$ matters. Cache the past $K$s and $V$s, and per-step inference drops from $O(n^2)$ to $O(n)$.

Vanilla attention is $O(n^2)$ in both compute and memory. Past ~32k tokens that's a serious wall, especially during training where you don't have a KV cache to amortise. The frontier here is a spectrum: sparse attention (only attend to a learned subset), linear attention (drop the softmax for an associative kernel), flash attention (re-tile the computation so the $O(n^2)$ matrix never materialises), and state-space models (Mamba and friends, which replace attention with a learned linear recurrence).

Flash attention is the boring-but-essential one: it doesn't change the math, just the I/O pattern. Everyone uses it. The exotic ones — Mamba, RWKV, RetNet — keep appearing but haven't dethroned the transformer for general language modelling yet.

Kaplan et al. (2020) and then Hoffmann et al. (2022) showed that, for a fixed compute budget, there is a specific ratio of parameters to training tokens that minimises loss. Earlier work had over-parameterised and under-trained; Chinchilla's rule of thumb of roughly 20 tokens per parameter shifted the field.

The practical consequence in 2024-2026 is the data wall: the web's high-quality text supply is finite. Modern training pipelines spend serious effort on filtering, deduplication, and synthetic data because more parameters can no longer compensate for thin data.