#72 Local and open models 6.4 min
Quantization Explained: Q4 vs Q8 vs FP16 in Plain Python
Four bits in blocks of 32 lost about 11 percent; four bits with one scale lost almost 98.
Work out the size of a block scheme
Copy the code folder first. Read bits_per_weight in quant.py. By hand, work out the bits per weight for 4-bit integers with a scale for every block of 64 weights out of 4096. Then check your answer by running: from quant import bits_per_weight; print(bits_per_weight(4, 64, 4096)). You should get 4.25. Say how it compares with the 4.5 the video shows for blocks of 32, and what you give up by using bigger blocks.
A hint
Each block adds one 16-bit scale. Divide the total scale bits by the number of weights.
Try blocks of 16
In run.py find the q4, blocks of 32 line in FORMATS and change both 32 values on that line to 16 (the quantize call and the bits_per_weight call). Predict whether the bits per weight and the error go up or down, then run python run.py table. You should see 5.0 bits per weight and about 8.85 percent error, against 4.5 and 10.93 before. This is the next step the video suggests. Say what you pay for the lower error. The row label will still print as 'q4, blocks of 32' although it now uses blocks of 16; leave the label alone, because python run.py sample looks the row up by that name.
A hint
Smaller blocks mean more scales to store.
Remove the two outliers
Restore run.py. In weights.py comment out the two lines that set weights[100] and weights[3000]. Predict which rows of python run.py table change most before you run it. You should see q8 with one scale fall from 10.63 percent to 0.85 percent, q4 with one scale fall from 97.66 to 15.31 percent, and the zeroed weights drop from 4091 to 796. What does this tell you about what two outliers did to the earlier result?
A hint
Outliers set the scale, and the scale sets the step between the levels the other weights can take.
Does a different seed change the story?
Restore weights.py, then change seed=7 to seed=8 in make_weights and run python run.py table. Before you run it, predict whether the order of the five formats by error will change. It should not: you should see about 11 percent for q4 in blocks of 32 and about 97.67 percent for q4 with one scale, with 4091 weights zeroed. The output numbers change a little. Write down why one seed is not enough to be sure.
A hint
Compare each row to the table in the video. The weights are random, but the two outliers are still there.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up weights, not a model: this measures weight error only, not answer quality, speed or memory, and the block scheme is a simplification of the legacy Q8_0 and Q4_0 shape.
Up next in Local and open models
More in Local and open models
Related videos
- #67 Distillation: Teach a Small Model From a Big One (Python) LLM engineering and evals
- #01 Local AI Agent in Python: No API Key (Ollama) AI agents and MCP
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local) RAG and vector search
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results RAG and vector search
