learning-it by example

#159 MLOps and deployment 6.5 min

Feature Stores: When a Table and a Join Are Enough

On made-up data, a naive join scored a churn rule at 0.96; point in time, the same rule scored 0.69.

1. What goes wrong when training rows for a churn model are made by joining each label to a table of the latest feature values?
2. In a SQLite demo with 200 made-up users (63 churned), a one-line rule picked on the naive join scored 0.96. What did the same rule score on point-in-time rows?
3. Why does the video say the gap between 0.96 and 0.69 is not a general result?
4. How does the video's point-in-time join choose which feature snapshot to use for a label?
5. What did EXPLAIN QUERY PLAN show for the video's online lookup of one user in the latest table?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Look up another user

    Copy the code folder first. In run.py change the user id 7 in the online part (in the query arguments and in the message text) to 42, then run python run.py online. Check that the query plan line is the same as in the video. On my machine user 42 had 3 tickets as of day 90.

    A hint

    The lookup is by the primary key of the latest table.

  2. Look strictly before the label

    In joins.py, in the POINT_IN_TIME query, change the less-than-or-equal comparison of h.day with l.day to a plain less-than. Predict whether the point-in-time join will still show 0 rows from later, then run python run.py leak. On my machine the future count stayed 0 and the cut picked on point-in-time rows had an accuracy of 0.73.

    A hint

    Label days are 30, 45 and 60, and snapshots are every 15 days.

  3. Switch the built-in leak off

    In data.py, in the line that adds 2 tickets after a churned user leaves, change the 0.5 to 0.0 so no extra complaint tickets are added after a user leaves. Predict whether the gap between the naive and point-in-time scores will close, then run python run.py leak. On my machine the rule picked on naive rows was 6 tickets or more with accuracy 0.84, and the same rule on point-in-time rows scored 0.72.

    A hint

    The naive join still takes the latest value, even without the extra tickets.

  4. Fail the build on a leak

    The video's tip is to put a future column in every training query and fail the build if it is ever above zero. At the end of the leak branch in run.py add a check that raises an error if the future column of the point-in-time rows sums to more than zero. Test that the same check would raise an error on the naive rows.

    A hint

    Each row is a tuple and the future flag is its fourth item, so sum(row[3] for row in rows) counts them.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up users with a leak I built in on purpose, one seed, a one-line rule scored on the rows it was picked on; no feature store is run, nothing is timed, and vendor claims come from their own docs.

Up next in MLOps and deployment

More in MLOps and deployment

All MLOps and deployment videos

Related videos