learning-it by example

Back to the video

Clean Messy Data with Pandas 3: End to End, 3 Mistakes

Clean a messy CSV with pandas 3 from start to finish in about twenty lines, and check your work with counts. Includes three real mistakes.

Video #09 ยท Python for data

Steps in the video

  1. 0:00 Clean messy data
  2. 0:22 The messy file
  3. 0:54 Setup
  4. 1:10 Clean the text
  5. 1:40 Numbers and dates
  6. 2:05 Duplicates and checks
  7. 2:50 Mistake one
  8. 3:12 Mistake two
  9. 3:26 Mistake three
  10. 4:06 What this does not show
  11. 4:38 Mistakes to avoid
  12. 5:05 My take and next steps
  13. 5:34 Recap

Files

  • clean.py (26 lines)
  • gotchas.py (23 lines)
  • messy.csv (9 lines)

Download the code (zip)

Tested with

Python 3.12.15 | pandas==3.1.0

What this does not show

One small made-up orders file, cleaned with rules chosen for it; the chained assignment trap is shown for pandas 3.1.0 on Python 3.12 only, and day-first or mixed date formats are not covered.

Sources

Learning IT by Example.