WordPiece
WordPiece is a subword tokenization algorithm similar to BPE but selects merges that maximize the likelihood of the training corpus, not just frequency. It uses '##' prefixes for word-internal subwords.
What is WordPiece?
WordPiece is a subword tokenization algorithm similar to BPE but selects merges that maximize the likelihood of the training corpus, not just frequency. It uses '##' prefixes for word-internal subwords.
WordPiece is a subword tokenization algorithm similar to BPE but selects merges that maximize the likelihood of the training corpus, not just frequency. It uses '##' prefixes for word-internal subwords.
Where is it used?
WordPiece is used by BERT, DistilBERT, and Electra; it is optimized for the masked-language-modeling objective rather than generation.
How to build it
Load `BertTokenizer.from_pretrained('bert-base-uncased')`, call `.tokenize('unaffable')` to see `['un', '##aff', '##able']`, and inspect `vocab.txt` for the merge set.