Batch vs Real-Time Inference: A Latency Simulation
In a made-up simulation, batches of 256 processed 481 items a second against 45 one at a time.
Video #164 ยท MLOps and deployment
Steps in the video
Files
- model.py (8 lines)
- offline.py (11 lines)
- online.py (31 lines)
- run.py (31 lines)
Tested with
Python 3.12.15
What this does not show
A simulation with a made-up cost model (20 ms per call plus 2 ms per item), one worker and seeded made-up traffic; no model runs and no real time is measured.
Sources
Learning IT by Example.