#194 Machine learning and data science 6.4 min
Anomaly Detection Basics: z-Score, IQR, Isolation Forest
A z-score found 1 of 3 planted bad readings; a median-based check found all 3.
Predict a smaller glitch
Copy the code folder first. In data.py change temp[17] = 250.0 to temp[17] = 100.0. Before you run, predict how the mean and standard deviation on the first line will change and whether the plain z-score will now find more than row 17. Then run python run.py one and compare.
A hint
A smaller wild value drags the mean and the standard deviation less.
Stricter box plot fences
In run.py, in the one function, change iqr_flags(x) to iqr_flags(x, 3.0), which uses fences 3 interquartile ranges out instead of 1.5. Predict whether row 174, the false alarm, will still be flagged. Run python run.py one to check.
A hint
The video says NIST calls points beyond 1.5 ranges mild outliers and beyond 3 ranges extreme ones.
Ask the forest for fewer rows
In run.py, in the forest function, change contamination=0.02 to contamination=0.01. Predict how many rows the isolation forest will flag out of 300, and whether all 4 planted rows are still among them. Run python run.py forest and compare.
A hint
The video says contamination asks it to flag about that share of the rows, and that you choose it.
Plant your own bad row
In data.py, in make_readings, add the line temp[100] = 70.0 after the other planted rows. Predict which of the three checks (z-score, modified z, IQR fences) will flag row 100. Run python run.py one and read the rows listed on each line.
A hint
Remember the z-score cut is 3 standard deviations, and the glitch at row 17 still inflates the standard deviation.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up data with planted anomalies, one seed for the isolation forest, and a contamination value chosen by hand.
Up next in Machine learning and data science
More in Machine learning and data science
- #182 Linear Regression From Scratch, Then scikit-learn
- #183 Precision, Recall, F1: Which Metric Matters?
- #184 Overfitting You Can See: Train vs Validation Error
- #185 Cross-Validation Done Right: The Leakage Mistake
- #186 Data Leakage: The Bug That Fakes Accuracy, Reproduced
- #187 Feature Engineering Beat 500 Trees, on Made-Up Data
Related videos
- #182 Linear Regression From Scratch, Then scikit-learn Machine learning and data science
- #296 Attention in 40 Lines of NumPy: The Core of Transformers LLM engineering and evals
- #183 Precision, Recall, F1: Which Metric Matters? Machine learning and data science
- #184 Overfitting You Can See: Train vs Validation Error Machine learning and data science
