Retrieval is a memory problem

First, how RAG works today

If you already know what "chunks" and "embeddings" are, this just sets the words straight. Skip to §2.

A language model only knows what it learned during training, and it can only read a limited amount of text at once — that limited space is called its context window. RAG (retrieval-augmented generation) is the standard trick for giving it your documents without retraining it.

It has four steps. Chop every document into small pieces (chunks). Index each piece by turning it into a string of numbers that captures roughly what it's about (an embedding), and store those. When a question comes in, find the handful of pieces whose numbers look most like the question. Then answer: paste those pieces in front of the question and let the model reply.

It works, and for a while it was the whole game. But look at the shape: every important decision is made before the model is even involved, and the lookup happens once, based only on the user's opening words. That shape is behind nearly every problem in the next section.

The one idea this paper is built on

Nearly every RAG problem comes from the same root. Today's systems chase one thing: find the documents that look most like the question, once. That made sense when the model could only read a tiny amount and looking things up was the only way in. It's the wrong goal now.

Flip it around. Don't ask "what looks like the question?" Ask: what does this model need in front of it to give an answer it can stand behind — including the answer that the documents simply don't cover it?

This isn't a proof, it's a way of seeing. But it earns its place, because it lines up the whole history as one story. Old search engines returned documents. RAG returned pieces of documents. Both just fetch. The step this paper names is from fetching to managing: deciding what belongs in the model's small reading space at each moment, and keeping everything else in cheaper storage until it's needed.

If the model's reading space is the scarce thing, there's a familiar picture that fits: the way a computer's memory is arranged. A processor has tiny, lightning-fast registers, then cache, then main memory, then a big slow disk. The whole craft is moving the right bits up to the fast level at the right moment and leaving the rest below. Line that up with retrieval and the design falls out.

The rest of the paper is the two halves of that idea. §3 is what goes wrong when you chase "looks similar." §4 onward is what falls out when you chase "earns a place."

Six ways today's RAG breaks

There are dozens of ways RAG goes wrong. Almost all of them come back to six root causes — each one a symptom of chasing "looks similar" instead of "earns a place." Grouping them like this matters: every one is answered by a part of the design in §7.

Retrieval as a bouncer at the door

Accept the core idea and the lookup's job changes completely. It stops being a thing that hunts for answers and becomes a bouncer at the door of the model's reading space: it decides what gets in, arranges it neatly, and — importantly — is allowed to say "nothing here qualifies" instead of padding the room with filler. "What matches the question?" becomes "what does this model actually need to give an answer it can defend — including saying the documents don't cover it?"

A door is only worth having if it keeps its word. So the design makes a few plain promises that must hold every time, or it's built wrong:

The design, part by part

Do the heavy thinking up front, keep an organised store, let the model look things up as it reasons, tidy what it finds before it reads it, make it cite sources, and let it learn. A fast lane keeps simple questions cheap. Read it from the bottom up.

A fast lane and a careful lane

In front of everything sits a cheap sorter that reads the question and picks a lane. Simple, low-stakes questions take the fast lane: a quick search, cite the source, done. Hard, high-stakes, or multi-step questions take the careful lane: the full design below. Without this, you build something lovely that's far too slow and expensive on the everyday questions that don't need it. And the sorter's mistakes aren't equal: sending an easy question down the careful lane just costs a little time, but sending a hard rules question down the fast lane gives a confident wrong answer with no date check and no source. So the sorter is built to err toward caution — when it's unsure, it sends the question the careful way.

The same design, wired up as a system

The figures so far explain the idea. Engineers will want the shape they'd actually deploy — so here it is as a reference architecture, with roles rather than product names, so it doesn't date. Read it for one thing above all: the seam the whole design turns on. The expensive work lives on two offline planes — loading documents (0) and, later, learning (5). The shared stores sit in the middle. The live query plane only ever reads them. Nothing a user waits on writes to the store — which is exactly what lets the live path stay quick and the records stay trustworthy.

A concrete stack you could build on — August 2026

The section above stays deliberately vendor-neutral, because the design outlives any tool. But a real team wants a starting point, so here's one credible stack as of August 2026 — with an eye on what a regulated shop can self-host. Treat it as a snapshot: this is the one part of the paper guaranteed to date fastest, so check every choice against its own current docs and your own tests before committing.

Two cross-cutting pieces don't need their own row. Warm caching is native to the major model APIs now — switch on the provider's prompt/context cache and keep the changeable content last. And permissions belong in the store, not a bolt-on filter: Postgres row-level security (via Supabase or your own roles) or per-tenant indexes are the usual way to keep "check access before admission" true by construction. Once more, though: this list is an August-2026 snapshot — the design is the durable part; the names will churn.

Roughly what it costs to run

A design should show it has thought about cost. These are rough shapes, not exact figures — but they show where the effort goes.

One honest correction: the prep work pays off because the documents change slowly. If they change constantly — news, filings, chat — the prep work isn't a one-off but a never-ending job, and keeping it up to date can cost as much as answering questions. That's not a flaw; it's the edge of where this design fits.

A worked example

Designs get nodded at and not built. So here's one real question, start to finish. The details are made up for illustration.

The setup. An assistant inside a bank is asked: "Does the monthly fee apply to this savings account?" Two things make it tricky. First, the fee rule changed at the start of 2026, so the answer depends on which year the question is about. Second, the document store still holds last year's rate sheet next to this year's — and they disagree.

That's the whole idea in one question: the right rule for the right year, a conflict shown instead of hidden, and "I don't know" instead of a confident guess.

Which part fixes which problem

The six problems from §3, matched to the parts that answer them. Notice the caveats: in a few cases the design moves a hard problem somewhere you can work on it, rather than making it vanish. Saying so is what separates a design from a sales pitch.

What it costs to build

Benefits without a bill are marketing. Here's how hard each part is, where the effort goes, and how you'd know it's actually working. Read the difficulty column as the order to build things in.

The sorter is cheap — build it early. Otherwise: the sources rule first (cheap, and "shows its sources" is a feature you can ship day one), then search (mostly off-the-shelf), then the prep work and the store together (one investment), then the tidy-up step. Learning goes last on purpose, not by default: the system's trustworthiness doesn't depend on it — that's already carried by the sources rule and the permissions gate — and its real yield is narrower than it first sounds (see part 5), so it's the one part worth building only once everything with a surer payoff is already standing. Most of parts 1–4 are known techniques the average system just skips. The learning part, and truly knowing what the model doesn't know, are the parts still genuinely open. One caveat outranks the whole order, though: the difficulty table above rates the six parts, but the single hardest and least-automatable piece for a regulated buyer isn't in it — it's building the graded, date-aware test set the whole thing is measured against (see §5). That's expert-labour, not engineering, and it's the real first line of any honest budget.

Making it fast enough

A weakness the grand idea glosses over: the careful lane is slow. Owning that is part of the design.

On a hard question, the careful lane does a lot — sort, search several times, tidy up, write, check sources — and that can take several seconds, sometimes more than ten. For slow-moving, high-stakes work, a few seconds for a careful, up-to-date, sourced answer is usually the right trade — the expensive mistake in tax or law is a fast wrong number, not a slow right one. But "acceptable" isn't "ignore it." Four levers, biggest first.

1 · The sorter does the most. Most questions are easy and never enter the loop. Make the fast lane quick and you've handled the bulk of traffic; the full design runs only for the minority that earns it.

2 · Keep the warm part warm. The fixed parts — the standing instructions and the steady core of the store for an active session — barely change. Modern tools let you keep that already-read part "warm" so a follow-up question only pays for the bit that changed. It cuts both cost and the wait before the answer starts. A rule of thumb falls out of it: put the things most likely to change last in that warm part.

3 · Drop stale bits from the warm part cleanly. Keeping things warm risks serving something out of date. The two clocks fix it: when a fact is removed or replaced, drop the warm bits that leaned on it. One honest limit — the warm part is a running start, so dropping something early forces re-reading everything after it. That's exactly why you put the changeable things last.

4 · Draft-and-check — with a warning that matters here. There's a real trick where a small fast model drafts while a bigger one checks, roughly halving the wait. Note the order in the real version: it checks before it shows anything. The tempting shortcut — show the draft instantly and fix it if the check fails — breaks this design's core promise in exactly the places it's meant for. A wrong tax or legal figure shown for two seconds and then corrected isn't a speed win; it's a wrong answer someone may already have acted on. So here, checking comes before showing. Show-then-fix is fine for low-stakes chat — but that's the everyday case the fast lane already covers.

There's also a genuine middle ground the show-then-fix idea misses: show the work, not the claims. While the careful lane runs, the user can watch what's happening — "found the 2024 and 2026 versions; working out which applies; checking the figure against the current sheet" — which kills the dead wait without ever showing an unchecked fact. Showing what the system is doing is honest; showing what it hasn't checked is not.

How this compares, and the obvious objections

Why "just use a knowledge graph" isn't enough

A knowledge graph — facts stored as things and the links between them — is the comparison people reach for. It's a great search tool (part 2): it's how you follow links for a multi-step question. But that's one part. On its own a graph doesn't tell you which version applies on a given date, doesn't tidy anything up before the model reads it, doesn't decide what earns a place (it just returns neighbours), doesn't make the answer cite its sources, and doesn't learn what actually helped. The door and the tidy-up are separate from link-following — you put them on top of a graph. The graph is a tool inside part 2, not the whole design.

Where this actually sits — and what's genuinely new

Honesty means naming what already exists, because most of the pieces do. Zep's Graphiti already stores facts with two clocks (when-true and when-recorded), pulls facts out of documents as they arrive, and marks facts as removed rather than deleting them — that's the prep work, the store, and clean deletion, in production today. Microsoft's GraphRAG already pulls out facts and relationships when it loads documents. Keyword-plus-meaning-plus-links search with a re-sort is a standard toolkit. So parts 0–2 are largely known techniques the average system just hasn't adopted.

And the framing isn't new either — this is the honest part to own. MemGPT (Packer et al., 2023, now Letta) already cast the context window as RAM and external memory as disk, with the model paging in what it needs — that is the exact metaphor behind Figure 3. "Context engineering" (the term Anthropic and others made standard through 2025) is by now the accepted name for "curate what earns a place in the window" — my §2 objective, near-verbatim. And putting the model in the retrieval loop is agentic RAG, with its own surveys. So neither the components nor the memory frame are the contribution, and a reviewer who says "this is context engineering plus MemGPT's metaphor plus agentic RAG" is not wrong about those parts.

So what is the contribution? Something narrower, and I think more durable for being honest about its size. Not the memory frame and not the objective — but four things the frame's usual literature doesn't do. The tidy-up step as a distinct, guarded, mostly-deterministic stage, with the "information, not instructions" boundary as a real job rather than a hope (plenty of systems compile context; few treat the compile as a governed stage with an injection boundary of its own). The show-your-sources rule with a real support check, not merely a citation. The present-but-restricted handling of §11 — the third ending, and keeping that decision out of the model. And, most of all, carrying the whole frame into changing, regulated document retrieval, where the two clocks, the source trail, and the governance actually bite. Almost all context-engineering work is about agents; very little is about governed retrieval over a corpus that changes and has to answer to a regulator. That gap is the real contribution — the coherence and the application, not the inventory.

There's a second family worth naming, because it answers the same complaint from the opposite direction: the compiled-wiki pattern (Karpathy's "LLM Wiki" and the many tools built on it). It shares this design's prep work — read a source, pull out the facts, note where they clash — but then makes a different bet. Where this design assembles the working set fresh from raw, citable facts on every question, the compiled wiki treats the synthesis itself as the durable thing: a good answer gets written back as a new page, so the next related question reads an already-worked-out page instead of re-deriving it. That compounds beautifully — insight accumulates instead of being rebuilt each time — and for personal research or a slow-moving knowledge base it may be the better shape.

But it carries a risk this design was built to avoid, and it's worth being plain about because it's the crux of the trade. A compiled wiki answers more confidently than raw retrieval — that's the point of it — but confidence drifts loose from whether the answer is actually grounded, and in a compounding store that failure feeds itself: a fabricated synthesis gets filed back as a page, and from then on it is a source. This design refuses that by construction — the model's own synthesis never becomes a citable fact; every admitted span traces to a real document, with its source and its dates. So the two aren't rivals so much as two answers to "stop re-deriving from scratch," tuned for different worlds: accumulate synthesis for speed and reach, or re-derive from sources every time to keep provenance clean. A personal wiki can live with a wrong page it corrects next week; a bank answering "does the fee apply" cannot let a fabrication quietly become the truth. That's the whole reason this design lands where it does.

Google's Open Knowledge Format (OKF, mid-2026) is worth a word here, because it's this same wiki pattern given a portable file convention — a directory of markdown files, each a "concept," with a little YAML on top (type, title, tags, a timestamp) and plain links between them. It's explicitly a format, not a service: no store, no retrieval, no runtime. Which places it cleanly in this design — it's a tidy, version-controlled, vendor-neutral way to serialise the document-shaped parts of the structured store (part 1) and to ship them between systems, and their reference "enrichment agent" that drafts a concept per table is a nice instance of the prep work (part 0). Two of this design's promises even get easier: knowledge that lives as plain files in git is knowledge whose every change is an auditable diff, and files move across tools without lock-in. But note what OKF carries per concept — a single timestamp, and no notion of source-versus-synthesis. It has one clock where this design needs two, and it's provenance-agnostic where this design insists a fact trace to a real document. So it slots in underneath — a good serialization for part 1's contents — but the two clocks, the source trail, and the "synthesis is never a citable fact" rule are exactly what you'd add around it. It standardises the wiki pattern; it doesn't make it safe for a regulated answer. That remains the work.

One of those existing systems also carries a warning. Graphiti deliberately keeps the model out of the search loop — plain search only, very fast — for speed and reliability. This design puts the model back in the loop so it can adapt mid-question. That's a real trade: you buy flexibility and pay in speed and in the reliability risk named first in §11. A production system chose the other side of that trade for good reasons; the sorter and the "stop when you have enough" rule are how this design tries to afford its choice.

How you'd check the claim

Plainly: this is a design paper, and it ships no benchmarks yet. What it would be judged on is the test set from §5 — does the answer match its sources, does it get dates right, does it handle multi-step questions, is it efficient, is it fast enough, does it say "I don't know" when it should, does it resist hidden instructions. The core claim is small and can be proven wrong: a small, tidied, "earns-a-place" set beats an equally-sized pile of "most similar" pieces on accuracy and on getting dates right, at the same or better speed. If it doesn't, the tidy-up step is just for show and the paper is wrong.

Objections worth answering up front

What this design still can't do

The most useful part of any design write-up — where no part fully closes the gap, and the interesting work is left.

What it deliberately isn't

To keep the scope honest, this design is:

• not a replacement for ordinary reporting over neat, structured databases;
• not a model trained end-to-end to do its own searching;
• not a way to bake a changing set of documents into a model's weights;
• not a full permissions-and-access product.

Search gave you documents. RAG gave you pieces.
What comes next manages what the model reads.

A note on how this was made: the framing is mine — design retrieval around the way an LLM actually works — and I carried it through the whole process. I drafted the approach, then put it through several AI models in turn, instructing each to attack and extend the last version, doing my own research alongside and judging which criticisms to keep. The models were sharp critics; the direction and the decisions were mine.