llm.dyyln.dev

LLMs from scratch

A guide to building a language model from nothing: the ideas, and an implementation in C you write yourself. Switch to Concepts in the header to fold the implementation away and read only the ideas.

Primer

Neural networks from nothing

Optional. For readers new to neural networks: every idea the rest of the guide assumes, with no maths beyond school algebra.

  1. P.1 A model is a function
  2. P.2 Learning is walking downhill
  3. P.3 Neurons and layers
  4. P.4 Vectors and matrices
  5. P.5 Text is already numbers
  6. P.6 Characters, words or pieces
  7. P.7 Predicting the next token
  8. P.8 A language model, end to end

Part 0

Foundations

In C. Ends with: a character-level transformer trained in C, and your experiment harness.

  1. 0.1 Matrix multiply in C
  2. 0.2 Attention from first principles
  3. 0.3 Backprop by hand
  4. 0.4 A tiny character model in C
  5. 0.5 Onto the GPU: the roofline
  6. 0.6 The ablation harness

Part 1

A solid baseline

Ends with: a GPT-2-style model you fully understand. Every later chapter is measured against it.

  1. 1.1 Byte-pair encoding
  2. 1.2 The data pipeline
  3. 1.3 The GPT-2 block
  4. 1.4 Training loop and learning rates
  5. 1.5 Evaluation

Part 2

The modern architecture

Ends with: a small model with a current open-model design.

  1. 2.1 RMSNorm and pre-norm
  2. 2.2 SwiGLU
  3. 2.3 Rotary position embeddings
  4. 2.4 GQA and QK-norm
  5. 2.5 Multi-head latent attention
  6. 2.6 Hybrid and linear attention
  7. 2.7 Mixture of experts

Part 3

Training like a lab

Ends with: the capstone pretraining run.

  1. 3.1 Scaling laws new
  2. 3.2 Data quality and dedup
  3. 3.3 Data mixing
  4. 3.4 Warmup-stable-decay
  5. 3.5 The Muon optimizer
  6. 3.6 µP
  7. 3.7 Mixed precision
  8. 3.8 Keeping training stable new
  9. 3.9 Distributed training new
  10. 3.10 Multi-token prediction
  11. 3.11 Mid-training and long context
  12. 3.12 The capstone run

Part 4

Post-training

Ends with: a chat model that visibly learns to reason on arithmetic.

  1. 4.1 Chat templates and SFT
  2. 4.2 DPO
  3. 4.3 RL with verifiable rewards
  4. 4.4 Distillation new
  5. 4.5 Evaluating reasoning

Part 5

Using it

Ends with: your model running in your own C engine, behind a chat interface.

  1. 5.1 The KV cache
  2. 5.2 Sampling
  3. 5.3 Speculative decoding
  4. 5.4 Quantization
  5. 5.5 An inference engine in C
  6. 5.6 A chat interface