learning-it by example

Back to the video

Schema Design for RAG: Chunks, Versions, Sources in SQLite

Asked one question, a one-table design I built without tenant, version or access columns returned 3 chunks in its top 4 that the user should not get: an old version, another tenant's text, and a note she may not read.

Video #269 ยท Data modelling

Steps in the video

  1. 0:00 One table: 3 of the top 4 chunks should not reach the user
  2. 0:37 The idea
  3. 1:04 Setup
  4. 1:23 Documents, versions, chunks
  5. 1:47 Embeddings, people, members
  6. 2:13 The made-up data
  7. 2:39 Filter first, then rank
  8. 2:54 The naive design
  9. 3:47 Re-index after a change
  10. 4:34 Delete a person's data
  11. 5:16 What this does not show
  12. 5:49 Mistakes to avoid
  13. 6:18 My take and next steps
  14. 6:36 Recap

Files

  • corpus.py (55 lines)
  • naive.py (24 lines)
  • queries.py (39 lines)
  • run.py (66 lines)
  • schema.py (28 lines)
  • toy_embed.py (16 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

Made-up documents, companies and people; the one-table design was written without tenant, version or access columns on purpose; toy five-word vectors instead of an embedding model and no language model; the delete covers this SQLite database only, not backups or copies, and is not legal advice.

Sources

Learning IT by Example.