Project · v0 · predicted, not yet measured
Retrieval-augmented generation is not one technique any more. It is seven designs that solve seven different problems, and most teams pick one by accident. This page runs all seven on the same small, real knowledge base and the same six questions, so the only thing that changes between columns is the architecture.
The knowledge base is my own bookshelf: a Goodreads export, my Kindle highlights, a list of who recommended what, and a few photographed pages. I chose it because I can tell a right answer from a plausible one in a second, which is the whole job when you evaluate retrieval, and because "it invented a book I never read" is the most legible failure a RAG system can have. Every trace below is a prediction of how each pipeline handles the question. Nothing is measured yet. The method page says what gets measured and when.
Teams upgrade the embedding model, switch to a stronger LLM, add more chunks to the prompt, and still get shallow or wrong answers. The reason is usually upstream of the model: the retrieval design cannot reach the information the question needs. Similarity search has no idea what "2024" or "under 300 pages" means. A text index cannot look at a photograph. A single retriever cannot see that two books argue with each other, because that fact lives in neither book. And no retrieval at all can compose four constraints into a reading list.
The real skill isn't building RAG. It's choosing the right pattern for the problem, and being able to show why.
Comparing patterns on different demos measures the demos. To compare the patterns you need to hold everything else still: same corpus, same questions, same generation model, and answers you can grade. That is what this page does.
Three files and a handful of photos. The Goodreads export is the ledger: every book, when I read it, what I rated it, how long it is, what everyone else rated it, and the review I wrote. The Kindle clippings are the quotes: one short highlight per block, hundreds of them. A hand-written list says who recommended what. Nothing is fabricated, private notes are never ingested, and the export is curated before it goes anywhere near a public page.
| source | what it holds | what it contributes |
|---|---|---|
| Goodreads export | one row per book: title, author, ISBN, pages, date read, my rating, the average rating, shelves, my review | the ledger. Queried, never embedded. |
| Kindle highlights | one short quotation per block, with book, location and date | the semantic index, and the misattribution trap: a highlight in one book quoting another author |
| Recommendations | title, who (initials), where, when | the people edges in the graph |
| Photos | covers, a few photographed pages, one shelf | answers with no text form |
| Extracted graph | who wrote what, who recommended what, which books mention or argue with which, themes | built once by a model with a fixed schema, published as JSON so a missed edge is visible |
One per failure mode, plus one that everything should get right. Each has a reference answer written before any pipeline runs. The exact titles get fixed once the export is inspected; the shape of each question does not change. The graded predictions are in the explorer below.
Each trace shows what gets retrieved, in what order, what the model is given, and the answer it is likely to produce, graded against the reference. The "why" line under each answer is the point of the page.
| step | what the pipeline is expected to do |
|---|
v0: every trace on this page is a prediction written from how the pipeline is designed and what the corpus contains. No pipeline has run. Each cell is replaced by a real trace, with the date, as the version that measures it ships; the schedule is on the method page.
Three things to read off the grid. Every pattern gets the easy question right, so the easy question tells you nothing about which pattern to use. The Q2 column splits the table cleanly into patterns with a tool and patterns without one, and no amount of embedding quality moves a pattern across that line. And reranking fixes the most common failure for the price of one extra call, which is why it should be the first upgrade, not the graph.
The prediction I most expect to be wrong is naive on Q1. Kindle highlights are short, so the nearest chunk is sometimes a different book quoting the same author. If that happens, Q1 stops being a control and reranking earns its call one column earlier.
Don't start complex. Start naive, add reranking, give the model a tool for the numbers, and evolve only when a fixed question set shows you the failure the next pattern fixes. A design space is only useful if you can say which point in it you are standing on, and why.