Quantization Explained: Q4 vs Q8 vs FP16 in Plain Python
Four bits in blocks of 32 lost about 11 percent; four bits with one scale lost almost 98.
Video #72 ยท Local and open models
Steps in the video
Files
- quant.py (35 lines)
- run.py (39 lines)
- weights.py (15 lines)
Tested with
Python 3.12.15
What this does not show
Made-up weights, not a model: this measures weight error only, not answer quality, speed or memory, and the block scheme is a simplification of the legacy Q8_0 and Q4_0 shape.
Sources
Learning IT by Example.