How it's built · v0 · predicted, not yet measured
Seven retrieval patterns, one corpus that happens to be my own bookshelf, six fixed questions with reference answers written before any pipeline runs, the same generation model everywhere. This page is the lab notebook: what I decided, what I turned down, and what replaces prediction with measurement as each version ships.
I build the way I learn: make the thing, then find out where it breaks, and publish the finding-out. Version 0 is the design and the predicted traces. The measurements start at v1. Every claim on the comparison page that is currently a prediction is labelled as one, and gets a date here when it becomes a measurement.
The seven-pattern RAG picture is everywhere in 2026, and it is right that these are different designs. What it does not give you is a way to choose. Every pattern is shown on its own demo, so any comparison is really a comparison of demos: different data, different questions, different models, different definitions of "worked".
The question I wanted to answer is the one a team asks before they build: which of these do we actually need, and how would we know? Answering that means holding everything still except the pattern.
The corpus is my reading life: what I read and when, what I rated it, the lines I highlighted, who told me to read it. Not because the world needs another chat-with-your-highlights app, but because evaluation is the hard part of RAG and I can only grade answers in a domain I know cold. When a pipeline says I read a book in 2023 that I read in 2021, or quotes a line back from the wrong book, or recommends something already on my read shelf, I know in one second. In an enterprise-docs demo I would have had to guess, or pay a model to guess for me.
The second reason is that the data has, naturally, the properties that separate the seven patterns. I did not have to invent them; I had to make sure the export kept them.
| property | where it comes from | which pattern it separates |
|---|---|---|
| Numbers in a ledger, not in text | dates read, ratings, page counts in the Goodreads export | patterns with a tool from patterns without |
| Sources that disagree | my review, my own highlights from the same book, and the crowd's average rating | naive from reranked |
| Knowledge in relationships | a highlight in one book naming the author of another; who recommended what | vector from graph |
| Answers only in images | photographed pages and covers | text from multimodal |
| Questions needing several sources | a reading list built from shelf, page counts, recommendations and abandoned books | single-path from multi-agent |
The earlier draft of this site used a handstand-coaching notebook I would have had to write from scratch. Real, exported data won on every count: it took an afternoon instead of a week, it cannot be accused of being built to flatter a pattern, and its failures are legible to anyone who has ever kept a reading list.
| decision | chose | instead of | why |
|---|---|---|---|
| Corpus | My own exports, curated | A fabricated corpus, or scraped reviews | I know every fact in it, and it is real. Cost: it is small, and small corpora flatter naive RAG; the question set compensates. |
| Full text | Highlights only | Whole books | A short quotation with attribution can be published; a book cannot. Highlights are also the more personal signal. |
| Recommenders | Initials, from a hand-written CSV | Full names, or nothing | The people edges make the graph interesting. The people themselves stay off a public page. |
| The ledger | Kept as rows in SQLite, queried by a tool | Embedding every row as text | Embedding a ledger is how a system invents a book. Keeping it as rows is what makes the Q2 column an honest test of "do you have a tool or not". |
| Question set | 6 fixed questions, one per failure mode | 50 random questions | Each question is a diagnostic: it should turn one pattern from wrong to right. Random questions average that out. More questions come at v5, once the pipelines run. |
| v0 as predictions | Predicted traces, labelled as such | Waiting for live pipelines | The design thinking is the product; the numbers are the proof. Shipping the thinking first, honestly labelled, beats shipping nothing. Cost: a reader who skips the label will think these are results. |
| Graph construction | LLM extraction with a fixed schema (Book, Author, Person, Theme; WROTE, RECOMMENDED, ABOUT, MENTIONS, ARGUES_WITH, READ_AFTER) | A hand-built graph | A hand-built graph hides the real failure mode of Graph RAG, which is the extractor missing an edge. I want that failure to be possible, visible, and published as JSON. |
| Reranker | A local cross-encoder | An LLM reranking call | Cheaper per query, and a second LLM in the loop makes it harder to attribute a fix to retrieval rather than to more model. |
| Generation model | One model, one prompt template, every pattern | The "natural" model for each pattern | Change one thing at a time. If the model varies, the comparison is of models. |
| Grading | Hand labels against a written reference, plus an automatic invented-book count | An LLM judge | 6 questions × 7 patterns × 5 runs is 210 answers. I can read 210 answers. A judge needs its own validation set, which at this size costs more than reading. |
| Hosting | Static pages now; a small Python service for the pipelines | A full app from day one | Static pages deploy in seconds and cannot go down. The service only needs to exist for the eval runs and, at v5, the demo box. |
| source | what is used | what is never used |
|---|---|---|
| Goodreads export | title, author, ISBN, pages, publication year, date read, date added, my rating, average rating, shelves, my review, read count | private notes, and any title I remove before ingest |
| Kindle highlights | the highlight text, its book, location, and date; my notes on it | bookmarks; clippings under 20 characters |
| Recommendations | title, who (initials), where, when. Written by hand. | the mapping from initials to people, which stays on my machine |
| Photos | covers from Open Library, a few pages I photograph, one shelf | |
| Extracted graph | edges from the files (who wrote what, who recommended what, what I read after what) and edges a model extracts from highlights and reviews (themes; which books mention or argue with which) | nothing; the whole graph is committed as JSON |
The highlights are the semantic index and the ledger is a table. That split is the single most important design decision on this site: it is what makes the "numbers" question an honest test rather than a trick.
Exact counts (books, highlights, match rate between the two exports) are published here at v1, after ingest. The curated exports themselves stay out of the repo; the photos, the graph JSON, the question set and every result CSV go in.
| question | tests | expected to separate | |
|---|---|---|---|
| Q1 | Where did I read the line about …? | a quotation in the index | nothing. A control, unless a same-author highlight from another book outranks it |
| Q2 | What did I read in 2024, rated five stars, under 300 pages? | a filter over the ledger | patterns with a tool (right) from patterns without (invented titles) |
| Q3 | Why did I give [book] two stars when everyone else loved it? | my review vs my highlights vs the crowd | naive (wrong) from reranked (right) |
| Q4 | Which of my books argue with each other? | a relationship between two books | vector (partial) from graph and hybrid (right) |
| Q5 | [photo] Which book is this, and what did I highlight near it? | an answer that starts from pixels | text-only (partial) from multimodal (right) |
| Q6 | A two-week holiday list from my to-read shelf, under 400 pages, one from a friend, nothing like the last three I abandoned | composing four sources into a deliverable | every single-path pattern (partial) from multi-agent (right) |
Reference answers are written before I trace a single pipeline and are shown on the comparison page next to each question. The bracketed titles get fixed once the export is inspected. A second reader grades a sample before v5; the agreement rate gets published as the ceiling, as I did for fixmybanana.
| measure | how | why it matters |
|---|---|---|
| Verdict | correct / partial / wrong against the reference, by hand | the headline cell in the grid |
| Invented books | any title in the answer that is not in the library, counted by code | the failure a reader can feel; the check that catches what I miss |
| Retrieval hit | did the chunk or row holding the reference answer reach the model? | separates "retrieval failed" from "the model ignored it" |
| Consistency | same verdict on all 5 runs? | a pattern that is right 3 times in 5 is not right |
| Model calls | counted by the code | structural cost, independent of vendor pricing |
| Latency and $ | wall clock and token usage per run | the columns that decide what ships |
The eval calls the same function the site's demo box will call, so what is measured is what a visitor gets. The question set does not change between versions, so the only thing that moves the grid is the change I made.
Predicted traces. Corpus chosen and its rules written, six question shapes fixed, seven pipelines traced by prediction on all six, the scorecard and decision guide published. No code has run. Every score on the comparison page is a design argument, not a measurement.
Ingest, naive, reranked, and the ledger tool, live. Parse both exports, build the tables, embed the highlights, add the cross-encoder and the SQL tool. Fix the real titles in the questions. Run 5× each. The first two rows of the grid become measurements, and Q2 gets its first real answer through the tool so the number exists early.
Graph and hybrid. LLM extraction over highlights and reviews with the fixed schema, then traversal. Publish the extracted graph so a missed edge is visible. The Q4 column becomes real.
Multimodal. Multimodal embeddings for the photos, vision-language generation. The Q5 column becomes real.
Router and multi-agent. A small routing model over the five retrievers; then a planner, sub-agents and a synthesizer. The Q6 column, and the cost column, become real.
The numbers. Full grid with latency and cost per cell, consistency across runs, invented-book counts, a second reader's agreement rate, ten more questions drawn from how people actually ask, and a live "ask the librarian" box with the trace panel. The "predicted" label comes off the comparison page.
| part | choice | note |
|---|---|---|
| These pages | Plain HTML, inline CSS, no build step | Cloudflare Pages, deploys on push. Same as my other sites. |
| Pipelines (v1+) | Python service, five retrievers, seven patterns behind one interface: answer(question, pattern, image) → {answer, context, route, calls, ms, usage} | the eval imports the same function the demo box calls |
| Ledger | SQLite, queried by the SQL tool | a few hundred rows; never embedded |
| Embeddings | Voyage text embeddings; the multimodal model for v3; vectors stored in the same SQLite file | one file, no vector database to run |
| Reranker | A local cross-encoder | no API call, so its cost shows up as milliseconds, not dollars |
| Graph | NetworkX in memory, extracted by Claude with a fixed schema, committed as JSON | a graph database would be a fourth thing to run for a graph with under a thousand nodes |
| Generation | Claude Opus 5 with adaptive thinking, one prompt template for every pattern | the router uses Claude Haiku 4.5: it is a classifier and should cost like one |
| Hosting the service | Railway, Cloudflare in front | the setup I already run for fixmybanana |