learning-it by example

#216 Data governance 6.5 min

Audit Evidence From Pipeline Logs in Python (SHA-256)

Six made-up pipeline runs, three gaps: an evidence report built straight from the logs.

1. How does the evidence report approach audit evidence from a data pipeline?
2. In the Python evidence report built from six made-up pipeline runs, what gaps did the three controls find?
3. In the Python pipeline evidence report, what does the SHA-256 hash of the run log show, and what does it not show?
4. In the Python pipeline evidence report, the row-count control compares which two numbers?
5. In the Python pipeline evidence report demo, the hash is stored right next to the log. Why does the video list that as a mistake to avoid?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Fill in the missing approver

    Copy the code folder first. In runs.jsonl, change the word null after approver on the run r2 publish line to "data-lead" (with the quotes). Run python run.py. Predict what happens to the first control. When I ran it, r2 appeared as evidence approved by data-lead, and the first control had no gap left. Then run python run.py check and read the last two lines. Why do both of those last two lines now say True?

    A hint

    The check command hashes runs.jsonl as it is now, then compares it with a second copy that it edits. Note that the program writes evidence.md, evidence.json and runs_edited.jsonl into your copy of the folder.

  2. Add a clean re-run for run 5

    At the end of runs.jsonl, add two lines for a run r7: a load step with in and out of 895 and the field "rerun_of": "r5", then a check step with in and out of 895 and "failed": 0. Copy the field layout of the r4 lines, and give each line an at time. Predict the second control, then run python run.py. When I ran it, r5 turned into evidence with clean re-run r7, and r7 was added to the runs covered.

    A hint

    The second control maps each run to the run that re-ran it using rerun_of.

  3. A step that disagrees with itself

    In runs.jsonl, change the r6 check line so its in value is 930 instead of 925 and its out value stays 925. Predict whether the row count gap for r6 goes away, then run python run.py. When I ran it, the gap disappeared and r6 was listed as carrying its counts. Why does the control not notice 5 rows missing inside that step?

    A hint

    Read the row_counts function in controls.py and see which pairs of numbers it compares.

  4. Edit the log and see the hash change

    Run python run.py and note the first 16 characters of the sha256 on the Input line. Now change any one number in runs.jsonl, such as the time on the first line, and run it again. Predict first whether the hash will change. Then write one sentence on why this proves the file changed but not whether the old or new version was true.

    A hint

    The hash is taken from the bytes of the file, after Windows line endings are turned into plain newlines.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up runs and invented controls. This is evidence, not compliance: the hash shows the log did not change after hashing, not that it was true.

Up next in Data governance

More in Data governance

All Data governance videos

Related videos