Tokenization
Tokenization is the process of turning a string into a series of tokens.
A string is simply a series of characters, if we wanted we could use this as the input to an LLM. But since computers can’t read as we do, it means the model also needs to learn to spell words before it can do anything else. This means additional compute as the early layers spend their capacity reassembling words, and attention cost grows as the sequence length you pass it grows.
Instead we give the model tokens, which represent one or more characters. The representation is just an integer which we can lookup in the vocabulary which is simply a fixed list of every token the model knows.