Character Tokenization
Character tokenization splits text into individual characters, giving a tiny vocabulary (~100 for ASCII) but very long sequences. It handles any word but loses semantic unit granularity.
What is Character Tokenization?
Character tokenization splits text into individual characters, giving a tiny vocabulary (~100 for ASCII) but very long sequences. It handles any word but loses semantic unit granularity.
Character tokenization splits text into individual characters, giving a tiny vocabulary (~100 for ASCII) but very long sequences. It handles any word but loses semantic unit granularity.
Where is it used?
Character-level models appear in research and some early language models (char-RNN); they are rarely used in production LLMs because sequence lengths become prohibitive.
How to build it
Implement with `list('hello')` → `['h','e','l','l','o']`, build a char-to-id dictionary, and train a small char-RNN to generate text one character at a time.