One-Hot Encoding
One-hot encoding represents each token as a vector of length V (vocabulary size) with a 1 at the token's index and 0 elsewhere. It is the simplest text representation but has no semantics and is high-dimensional.
What is One-Hot Encoding?
One-hot encoding represents each token as a vector of length V (vocabulary size) with a 1 at the token's index and 0 elsewhere. It is the simplest text representation but has no semantics and is high-dimensional.
One-hot encoding represents each token as a vector of length V (vocabulary size) with a 1 at the token's index and 0 elsewhere. It is the simplest text representation but has no semantics and is high-dimensional.
Where is it used?
One-hot vectors are conceptually the input to embedding layers — multiplying the one-hot by the embedding matrix selects the corresponding row, which is what `nn.Embedding` does efficiently.
How to build it
Use `F.one_hot(torch.tensor([2,5,1]), num_classes=10)` to create one-hot vectors, then multiply by an embedding matrix to see how `nn.Embedding` is an efficient lookup of the same operation.