learning-it by example

#72 Local and open models 6.4 min

Quantization Explained: Q4 vs Q8 vs FP16 in Plain Python

Four bits in blocks of 32 lost about 11 percent; four bits with one scale lost almost 98.

1. What does quantization do to a model's weights?
2. In a plain Python test on 4,096 made-up weights, how much weight error did 4-bit rounding give with a scale for every block of 32 weights, and with one scale for the whole list?
3. In the symmetric round-to-nearest block quantizer built in the video 'Quantization explained: Q4 vs Q8 vs FP16', how is the scale for each block chosen?
4. Why do the two outlier weights, 0.9 and -0.7, matter so much when 4-bit rounding uses one scale for all 4,096 made-up weights?
5. A plain Python test on 4,096 made-up weights showed about 11 percent weight error for 4-bit blocks of 32. What does that tell you about the answers of a real model?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Work out the size of a block scheme

    Copy the code folder first. Read bits_per_weight in quant.py. By hand, work out the bits per weight for 4-bit integers with a scale for every block of 64 weights out of 4096. Then check your answer by running: from quant import bits_per_weight; print(bits_per_weight(4, 64, 4096)). You should get 4.25. Say how it compares with the 4.5 the video shows for blocks of 32, and what you give up by using bigger blocks.

    A hint

    Each block adds one 16-bit scale. Divide the total scale bits by the number of weights.

  2. Try blocks of 16

    In run.py find the q4, blocks of 32 line in FORMATS and change both 32 values on that line to 16 (the quantize call and the bits_per_weight call). Predict whether the bits per weight and the error go up or down, then run python run.py table. You should see 5.0 bits per weight and about 8.85 percent error, against 4.5 and 10.93 before. This is the next step the video suggests. Say what you pay for the lower error. The row label will still print as 'q4, blocks of 32' although it now uses blocks of 16; leave the label alone, because python run.py sample looks the row up by that name.

    A hint

    Smaller blocks mean more scales to store.

  3. Remove the two outliers

    Restore run.py. In weights.py comment out the two lines that set weights[100] and weights[3000]. Predict which rows of python run.py table change most before you run it. You should see q8 with one scale fall from 10.63 percent to 0.85 percent, q4 with one scale fall from 97.66 to 15.31 percent, and the zeroed weights drop from 4091 to 796. What does this tell you about what two outliers did to the earlier result?

    A hint

    Outliers set the scale, and the scale sets the step between the levels the other weights can take.

  4. Does a different seed change the story?

    Restore weights.py, then change seed=7 to seed=8 in make_weights and run python run.py table. Before you run it, predict whether the order of the five formats by error will change. It should not: you should see about 11 percent for q4 in blocks of 32 and about 97.67 percent for q4 with one scale, with 4091 weights zeroed. The output numbers change a little. Write down why one seed is not enough to be sure.

    A hint

    Compare each row to the table in the video. The weights are random, but the two outliers are still there.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up weights, not a model: this measures weight error only, not answer quality, speed or memory, and the block scheme is a simplification of the legacy Q8_0 and Q4_0 shape.

Up next in Local and open models

More in Local and open models

All Local and open models videos

Related videos