learning-it by example

#164 MLOps and deployment 6.4 min

Batch vs Real-Time Inference: A Latency Simulation

In a made-up simulation, batches of 256 processed 481 items a second against 45 one at a time.

1. What does the batch-versus-real-time video suggest asking first when you choose between offline batch and online serving?
2. In a Python batch-versus-real-time simulation of an offline job of 10,000 items (each call costs 20 ms plus 2 ms per item), how long did one at a time take compared with batches of 256?
3. In the Python batching simulation at heavy load (40.2 requests a second), one at a time had a 95th percentile latency of 169 ms with the worker 88% busy. What did batches of up to 8 with a 10 ms wait give?
4. What does the video say its batching numbers do not show?
5. In the Python batching simulation at light load (20.1 requests a second), what happened with batches of up to 32 and a 50 ms wait?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Try other batch sizes

    Copy the code folder first. In run.py change the offline batch sizes (1, 32, 256) to (1, 8, 64, 1024). Predict where the gains flatten, then run python run.py offline. On my machine batch 8 took 45.0 s, batch 64 took 23.1 s and batch 1024 took 20.2 s.

    A hint

    Once the overhead is shared, the 2 ms per item set the pace.

  2. Make the overhead cheaper

    In model.py change OVERHEAD_MS from 20 to 5 and run python run.py offline. Predict whether batching helps by more or less than before. On my machine one at a time took 70.0 s and batches of 256 took 20.2 s.

    A hint

    Work out the time for 10,000 single-item calls with the new overhead.

  3. Add a policy with no wait

    Put OVERHEAD_MS back to 20. In run.py add a fourth policy to POLICIES: ("up to 8, wait 0", 8, 0). Predict how its median and 95th percentile compare with up to 8, wait 10 at each load, then run python run.py online. On my machine it gave 22 and 43 at light load, and 30 and 45 at heavy load.

    A hint

    With a wait of 0 the worker only takes requests that have already arrived.

  4. Does batching ever lose?

    Set OVERHEAD_MS to 5 again and run python run.py online. Predict whether any of the three batching policies beats one at a time on the 95th percentile, at either load. On my machine none of them did.

    A hint

    Compare the p95 column for each policy against one at a time at the same load.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A simulation with a made-up cost model (20 ms per call plus 2 ms per item), one worker and seeded made-up traffic; no model runs and no real time is measured.

Up next in MLOps and deployment

More in MLOps and deployment

All MLOps and deployment videos

Related videos