Transformer Architecture

The Transformer is the foundational neural network architecture for all modern LLMs, introduced by Vaswani et al. in the 2017 paper "Attention Is All You Need."

Core Innovation

The Transformer replaces recurrence (RNNs) and convolution (CNNs) with a purely attention-based mechanism. This allows:

Parallel computation — All tokens processed simultaneously during training
Long-range dependencies — Direct connections between any two positions
Scalability — Architecture scales to trillions of parameters

Architecture Components

Encoder-Decoder vs Decoder-Only

The original Transformer used an encoder-decoder structure, but most modern LLMs (GPT-4o, Llama 4, Claude 4) use decoder-only architecture with causal masking.

Key Components

Multi-Head Self-Attention — Each token attends to all others
Feed-Forward Networks (FFN) — Position-wise MLP layers
positional-encoding — Adds position information (often RoPE in modern models)
Layer Normalization — Pre-norm or post-norm placement
Residual Connections — Skip connections around each sub-layer

Modern Innovations

Grouped Query Attention (GQA) — Reduces KV cache size
Mixture of Experts (MoE) — Sparse activation for larger models
Flash Attention — IO-aware attention algorithm for efficiency
Rotary Position Embeddings (RoPE) — Relative position encoding
SwiGLU / GELU — Modern activation functions

Scaling Laws

The Transformer exhibits predictable scaling-laws: performance improves with increased parameters, data, and compute according to power-law relationships.

Training

Transformers are trained via pretraining on large text corpora, followed by SFT and RLHF or DPO.

Related Concepts

Attention Mechanism — The attention mechanism in depth
multi-head-attention — Parallel attention heads
KV Cache — Efficient inference caching
Context Window — Maximum sequence length