Floating-Point Numbers
Floating-point numbers represent real numbers in hardware using a sign bit, exponent bits, and mantissa bits. The format (FP32, FP16, BF16) determines the range and precision of values representable, directly affecting model memory and accuracy.
What is Floating-Point Numbers?
Floating-point numbers represent real numbers in hardware using a sign bit, exponent bits, and mantissa bits. The format (FP32, FP16, BF16) determines the range and precision of values representable, directly affecting model memory and accuracy.
Floating-point numbers represent real numbers in hardware using a sign bit, exponent bits, and mantissa bits. The format (FP32, FP16, BF16) determines the range and precision of values representable, directly affecting model memory and accuracy.
Where is it used?
LLMs store all weights and activations as floats. FP32 is the training default; FP16 and BF16 halve memory for inference. The IEEE 754 standard defines these formats. Quantization to INT8/INT4 further reduces precision for edge deployment.
How to build it
Check a tensor's dtype: `tensor.dtype` (e.g., `torch.float32`, `torch.float16`, `torch.bfloat16`). Convert: `tensor.to(torch.bfloat16)`. Inspect bits: `import struct; struct.pack('f', 3.14)` shows the 32-bit representation. Understand that FP16 has range ±65504 while BF16 has the same range as FP32.