learning-it by example

Back to the video

Quantization Explained: Q4 vs Q8 vs FP16 in Plain Python

Four bits in blocks of 32 lost about 11 percent; four bits with one scale lost almost 98.

Video #72 ยท Local and open models

Steps in the video

  1. 0:00 Four bits in blocks: 11 percent error. One scale for all: 98.
  2. 0:36 The idea
  3. 1:07 Setup
  4. 1:45 The made-up weights
  5. 2:17 Sixteen bits
  6. 2:35 Eight and four bits
  7. 3:02 Counting the cost
  8. 3:18 The five formats
  9. 4:26 What this does not show
  10. 5:05 Mistakes to avoid
  11. 5:30 My take and next steps
  12. 5:55 Recap

Files

  • quant.py (35 lines)
  • run.py (39 lines)
  • weights.py (15 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

Made-up weights, not a model: this measures weight error only, not answer quality, speed or memory, and the block scheme is a simplification of the legacy Q8_0 and Q4_0 shape.

Sources

Learning IT by Example.