llm.dyyln.dev

LLMs from scratch

This site is a project to achieve a few goals:

  • explain key architecture and concepts of LLMs
  • guide the reader through implementing an LLM themself I will walk through the fundamentals and building blocks needed to understand how LLMs work, the limitations and key advancements. I will also outline code to implement an LLM from scratch in a low-level systems programming language (for those who still like to code themselves). I hope the book will be accessible to both audiences who simply want to understand what’s going on under-the-hood of LLMs, and those who want to deep-dive into making one themselves and the implementations details.

I’m doing this to:

  • learn more in-depth myself by trying to explain and implement the concepts
  • practise writing and publishing technical content

The guide has a Concepts switch in the header which hides implementation details away if you are only interested in reading about the concepts themselves, and not interested in implementing your own LLM. Since I expect the implementation to be non-trivial, it’s likely to cut out a lot of reading if you want to do so.

The implementation will be written in JAI, just because it’s a language I’ve been using in hobby projects and such, I wanted to use a low-level language, but you can bring your own!

I will start with some background on machine learning and neural networks, then jump from those to LLMs. I will try not to assume any background knowledge, but there may be some context on math/programming that I skip over too quickly.

Introduction

As mentioned the book is two-fold :

  • Build an LLM
  • Explain the concepts behind an LLM We will build such that you can run most things on a local machine or laptop with a GPU to keep everything smaller scale. Whilst we will go through training and implement it ourselves, our end model will support open weights, so you don’t have to spend obsurd amounts of time/compute on training your own model to get good results.

There are lots of resources on LLMs in general which tend to stop at GPT-2 level LLMs. I would like to go further than that and expand on concepts used by frontier models (where the information is publically available) but that is a long way off for now.

You can read this book straight through, or with concepts only. Even if you choose not to implement the code yourself, you may still be interested in the code and implementation details to skim over.

The expected path is :

  • Read and understand the concepts in a chapter
  • Implement the code describe/guided in the chapter
  • Later chapters will likely build on code from earlier chapters

I will add references for each chapter for sources/further reading, and try to draw my own diagrams to explain topics where I see fit

Tokenization

Tokenization is the process of turning a string into a series of tokens.

A string is simply a series of characters, if we wanted we could use this as the input to an LLM. But since computers can’t read as we do, it means the model also needs to learn to spell words before it can do anything else. This means additional compute as the early layers spend their capacity reassembling words, and attention cost grows as the sequence length you pass it grows.

Instead we give the model tokens, which represent one or more characters. The representation is just an integer which we can lookup in the vocabulary which is simply a fixed list of every token the model knows.

Tokenization

1 Representing Text

Computer at the base level represent everything at bits (0s or 1s). The common unit of a byte is 8 bits. Since each bit can be either 0 or 1, 8 bits can represent 2^8 (256) possible values.

We typically represent text on computers in one of a few formats :

  • ASCII
  • Unicode
  • UTF8

If we used these as our vocabulary for our model, that would mean we need no transformation of the text.

Tokenization

2 Token Granularity

Small vocabulary leads to long sequences, huge vocabulary leads to unknown words. Characters vs Words vs Somewhere in the middle