learning-it by example

Back to the video

Batch vs Real-Time Inference: A Latency Simulation

In a made-up simulation, batches of 256 processed 481 items a second against 45 one at a time.

Video #164 ยท MLOps and deployment

Steps in the video

  1. 0:00 One at a time: 45 items a second. Batches of 256: 481. A simulation.
  2. 0:41 Two meanings of batch
  3. 1:18 Setup
  4. 1:39 The cost model
  5. 1:54 The offline job
  6. 2:43 Live traffic
  7. 2:53 One worker
  8. 3:19 Percentiles and policies
  9. 4:28 What this does not show
  10. 5:01 Mistakes to avoid
  11. 5:25 My take and next steps
  12. 5:56 Recap

Files

  • model.py (8 lines)
  • offline.py (11 lines)
  • online.py (31 lines)
  • run.py (31 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

A simulation with a made-up cost model (20 ms per call plus 2 ms per item), one worker and seeded made-up traffic; no model runs and no real time is measured.

Sources

Learning IT by Example.