Schema Design for RAG: Chunks, Versions, Sources in SQLite
Asked one question, a one-table design I built without tenant, version or access columns returned 3 chunks in its top 4 that the user should not get: an old version, another tenant's text, and a note she may not read.
Video #269 ยท Data modelling
Steps in the video
- 0:00 One table: 3 of the top 4 chunks should not reach the user
- 0:37 The idea
- 1:04 Setup
- 1:23 Documents, versions, chunks
- 1:47 Embeddings, people, members
- 2:13 The made-up data
- 2:39 Filter first, then rank
- 2:54 The naive design
- 3:47 Re-index after a change
- 4:34 Delete a person's data
- 5:16 What this does not show
- 5:49 Mistakes to avoid
- 6:18 My take and next steps
- 6:36 Recap
Files
- corpus.py (55 lines)
- naive.py (24 lines)
- queries.py (39 lines)
- run.py (66 lines)
- schema.py (28 lines)
- toy_embed.py (16 lines)
Tested with
Python 3.12.15
What this does not show
Made-up documents, companies and people; the one-table design was written without tenant, version or access columns on purpose; toy five-word vectors instead of an embedding model and no language model; the delete covers this SQLite database only, not backups or copies, and is not legal advice.
Sources
Learning IT by Example.