learning-it by example

#157 MLOps and deployment 6.6 min

Model Registry by Hand: Versions, Promote, Roll Back

Version 3 scored 0.867 against 0.897 in production: the registry refused it, a forced push went through, and one call rolled it back.

1. Which questions does the video say a model registry answers?
2. In a hand-written SQLite model registry for a made-up fraud model, version 3 scored 0.867 on held-out data and version 2 in production scored 0.897. What happened?
3. How far back does the video's rollback go, and how does it find the version?
4. What does the video's registry do before it loads a model file, and why?
5. Which of these does the video give as a mistake to avoid?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Give Cy more rows

    Copy the code folder first. In run.py change "cy": NEW[:40] to "cy": NEW[:200]. Predict what version 3 will score and whether the gate will still refuse it. Run python run.py register, then python run.py promote, and look for a refused line. On my machine version 3 got threshold 250 and test accuracy 0.897, and no refused line appeared.

    A hint

    Compare version 3's accuracy with version 2's.

  2. Refuse ties as well

    Keep the change from the last task. In registry.py, in promote, the gate refuses when the new accuracy is below the old one. Change that test so that it also refuses when the two are equal (use less-than-or-equal). Predict what python run.py promote prints first, then run it. On my machine a refused line appeared even though the two accuracies were equal.

    A hint

    The gate compares the new version's test accuracy with the one in production.

  3. Rewrite the file with the same bytes

    In run.py the tamper mode overwrites version 2's file with b'{"threshold": 0}'. Replace those bytes with b'{"threshold": 250}' and run python run.py tamper, then try b'{"threshold":250}' with no space. Predict for each whether the load is refused. On my machine the first was not refused (no refused line means the load was accepted) and the second was.

    A hint

    The check compares hashes of bytes, not the meaning of the content.

  4. Roll back twice

    In run.py in the promote function, add a second call reg.rollback("fraud", "production", "ana") followed by another show line. Predict which version production is on after the second rollback and what the audit will list, then run python run.py promote. On my machine production stayed on version 2 and a second rollback row appeared.

    A hint

    Read what rollback looks up in the audit table: the last promote or force row.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A made-up one-number fraud model on made-up payments; a hand-written SQLite registry with one metric as the gate, a counter instead of a clock, one process and no logins; this is not MLflow.

Up next in MLOps and deployment

More in MLOps and deployment

All MLOps and deployment videos

Related videos