Distillation: Teach a Small Model From a Big One (Python)
A small model trained on 12 hand labels got 149 of 200 right; trained on a teacher's labels, 188.
Python 3.12.1511 videos tagged "machine learning basics".
A small model trained on 12 hand labels got 149 of 200 right; trained on a teacher's labels, 188.
Python 3.12.15
#182 · Machine learning and data science · 6.2 minBy hand and with scikit-learn, the slope came out at 5.77 both times. The true slope was 5.
Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2
#183 · Machine learning and data science · 6.5 minA model that never flags fraud scored 98% accuracy and caught 0 of 20 frauds.
Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2
#184 · Machine learning and data science · 6 minDegree 14 had a training error of 0.00 and a validation error of 3.27.
Python 3.12.15 | numpy==2.5.4
#185 · Machine learning and data science · 6.3 minOn pure noise with random labels, cross-validation reported 0.89 accuracy, until one step moved.
Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2
#186 · Machine learning and data science · 6.2 minOne column filled in after delivery lifted test accuracy from 0.69 to 0.95.
Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2
#187 · Machine learning and data science · 6.2 minOn made-up rents, a straight line with one extra column missed by 31.2 a month; a 500-tree forest on the raw columns missed by 39.4.
Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2
#189 · Machine learning and data science · 6.3 minOne deep tree scored 0.728 on test; a hundred random trees together scored 0.842.
Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2
#193 · Machine learning and data science · 6.2 mink-means matched 299 of 300 points on round clouds, but turned a small cloud of 40 into a cluster of 102.
Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2
#194 · Machine learning and data science · 6.4 minA z-score found 1 of 3 planted bad readings; a median-based check found all 3.
Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2
#296 · LLM engineering and evals · 6.4 minWith hand-set weights, the word tired gave 0.79 of its attention to cat, in about 40 lines of NumPy.
Python 3.12.15 | numpy==2.5.4