llm.dyyln.dev

Tokenization

2 Token Granularity

Small vocabulary leads to long sequences, huge vocabulary leads to unknown words. Characters vs Words vs Somewhere in the middle