#159 MLOps and deployment 6.5 min
Feature Stores: When a Table and a Join Are Enough
On made-up data, a naive join scored a churn rule at 0.96; point in time, the same rule scored 0.69.
Look up another user
Copy the code folder first. In run.py change the user id 7 in the online part (in the query arguments and in the message text) to 42, then run python run.py online. Check that the query plan line is the same as in the video. On my machine user 42 had 3 tickets as of day 90.
A hint
The lookup is by the primary key of the latest table.
Look strictly before the label
In joins.py, in the POINT_IN_TIME query, change the less-than-or-equal comparison of h.day with l.day to a plain less-than. Predict whether the point-in-time join will still show 0 rows from later, then run python run.py leak. On my machine the future count stayed 0 and the cut picked on point-in-time rows had an accuracy of 0.73.
A hint
Label days are 30, 45 and 60, and snapshots are every 15 days.
Switch the built-in leak off
In data.py, in the line that adds 2 tickets after a churned user leaves, change the 0.5 to 0.0 so no extra complaint tickets are added after a user leaves. Predict whether the gap between the naive and point-in-time scores will close, then run python run.py leak. On my machine the rule picked on naive rows was 6 tickets or more with accuracy 0.84, and the same rule on point-in-time rows scored 0.72.
A hint
The naive join still takes the latest value, even without the extra tickets.
Fail the build on a leak
The video's tip is to put a future column in every training query and fail the build if it is ever above zero. At the end of the leak branch in run.py add a check that raises an error if the future column of the point-in-time rows sums to more than zero. Test that the same check would raise an error on the naive rows.
A hint
Each row is a tuple and the future flag is its fourth item, so sum(row[3] for row in rows) counts them.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up users with a leak I built in on purpose, one seed, a one-line rule scored on the rows it was picked on; no feature store is run, nothing is timed, and vendor claims come from their own docs.
Up next in MLOps and deployment
More in MLOps and deployment
Related videos
- #185 Cross-Validation Done Right: The Leakage Mistake Machine learning and data science
- #187 Feature Engineering Beat 500 Trees, on Made-Up Data Machine learning and data science
- #237 Where Should AI Agent State and Memory Live? A Crash Test Enterprise architecture and AI agents
- #316 Build an Eval Set From Customer Tickets Without Leakage Forward deployed engineer interviews