Attention Mask
An attention mask is a tensor that selectively blocks certain attention positions — used for causal masking, padding tokens, or document boundaries in packed sequences.
What is Attention Mask?
An attention mask is a tensor that selectively blocks certain attention positions — used for causal masking, padding tokens, or document boundaries in packed sequences.
An attention mask is a tensor that selectively blocks certain attention positions — used for causal masking, padding tokens, or document boundaries in packed sequences.
Where is it used?
LLM training packs multiple documents into one sequence separated by attention masks so tokens don't attend across documents; padding masks ignore pad tokens in batched inference.
How to build it
Build a 2D mask tensor of 0s and 1s, pass it to `F.scaled_dot_product_attention(Q,K,V,attn_mask=mask)`, and verify that masked positions contribute zero to the output.