#164 MLOps and deployment 6.4 min
Batch vs Real-Time Inference: A Latency Simulation
In a made-up simulation, batches of 256 processed 481 items a second against 45 one at a time.
Try other batch sizes
Copy the code folder first. In run.py change the offline batch sizes (1, 32, 256) to (1, 8, 64, 1024). Predict where the gains flatten, then run python run.py offline. On my machine batch 8 took 45.0 s, batch 64 took 23.1 s and batch 1024 took 20.2 s.
A hint
Once the overhead is shared, the 2 ms per item set the pace.
Make the overhead cheaper
In model.py change OVERHEAD_MS from 20 to 5 and run python run.py offline. Predict whether batching helps by more or less than before. On my machine one at a time took 70.0 s and batches of 256 took 20.2 s.
A hint
Work out the time for 10,000 single-item calls with the new overhead.
Add a policy with no wait
Put OVERHEAD_MS back to 20. In run.py add a fourth policy to POLICIES: ("up to 8, wait 0", 8, 0). Predict how its median and 95th percentile compare with up to 8, wait 10 at each load, then run python run.py online. On my machine it gave 22 and 43 at light load, and 30 and 45 at heavy load.
A hint
With a wait of 0 the worker only takes requests that have already arrived.
Does batching ever lose?
Set OVERHEAD_MS to 5 again and run python run.py online. Predict whether any of the three batching policies beats one at a time on the 95th percentile, at either load. On my machine none of them did.
A hint
Compare the p95 column for each policy against one at a time at the same load.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
A simulation with a made-up cost model (20 ms per call plus 2 ms per item), one worker and seeded made-up traffic; no model runs and no real time is measured.
Up next in MLOps and deployment
More in MLOps and deployment
Related videos
- #318 LLM Latency and Cost Budget: Which Lever Matters? Forward deployed engineer interviews
- #157 Model Registry by Hand: Versions, Promote, Roll Back MLOps and deployment
- #158 Detect Data Drift: PSI and K-S by Hand in Python MLOps and deployment
- #159 Feature Stores: When a Table and a Join Are Enough MLOps and deployment
