BPE
Byte-Pair Encoding (BPE) is a subword tokenization algorithm that starts from characters and iteratively merges the most frequent adjacent pair into a new token until the vocabulary reaches a target size.
What is BPE?
Byte-Pair Encoding (BPE) is a subword tokenization algorithm that starts from characters and iteratively merges the most frequent adjacent pair into a new token until the vocabulary reaches a target size.
Byte-Pair Encoding (BPE) is a subword tokenization algorithm that starts from characters and iteratively merges the most frequent adjacent pair into a new token until the vocabulary reaches a target size.
Where is it used?
BPE is used by GPT-2, GPT-3, GPT-4 (via tiktoken), and Llama 2; it is the most widely deployed tokenizer in LLMs today.
How to build it
Use `tiktoken.get_encoding('gpt2').encode('tokenization')` to see the BPE split, or train one with `tokenizers.Tokenizer(models.BPE()).train(files)` and inspect merges.
Code
A practical example:
Further reading
Want to go deeper? These resources cover BPE in more detail: