A Pile of Documents Is Not Knowledge

My research agent had read about 20,400 documents. It had concluded nothing.

Not metaphorically. The knowledge store held roughly twenty thousand rows — papers, articles, PDFs, nightly log entries — and there was nowhere in the system that said what any of it meant. Research answers were stored as aegis://research/<hash> rows filed among ten thousand PDFs. The nightly journal wrote one dated entry into the same pile, in a format nobody could open.

This is the third of three posts about rebuilding lanes of AEGIS, my self-hosted agent platform, between 5 and 13 September. The first two were alerting and money. All three ended at the same rule, and this lane is where it's easiest to see why.

Measure use, not ingestion

Every RAG dashboard I have ever seen measures the wrong end of the pipe. Documents ingested. Chunks embedded. Index size. All of it is input, and input is the part that is trivially easy to grow.

The measurement that mattered took one join: a log of which documents were injected into which prompt, joined back to where each document came from. Over the 30 days to 12 September:

• arXiv: 1,889 papers, 89,669 chunks — 90% of all RSS chunks. Prompts used 14 papers.
• Across the whole corpus, arXiv PDFs were 93% of all chunks, and 78 of 10,284 had ever reached a prompt.
• The intelligence scans had stored 371 items. Exactly one was ever used in a prompt.
• Those same scans created 277 task-manager items in 30 days, every one auto-classified as reference material on arrival. My task list was being used as a log file.

A retention preview said the same thing in bytes. One rule — keep only the first chunk of any PDF no prompt has used in 30 days — would touch 8,297 PDFs and drop 362,153 of 500,055 chunks, freeing about 530 MB of text and 1 GB of vectors from a 4.8 GB table. The PDFs themselves stay, and nothing has been deleted: the preview is a dry run, and deleting is a separate, explicit decision.

This is not a story about arXiv being low quality. It is excellent. It is a story about a system that was optimising the metric it could see.

The filter I measured and did not ship

The obvious fix is a topic gate: only store the full text when the item matches something you care about. I built it, measured it, and made it opt-in.

Here is why. On arXiv the gate would pass 41% of papers — about 2.3× fewer chunks, for real added complexity. And on every other feed, the gate would have kept the full text of only 2 of the 10 documents a prompt actually used.

That second number killed it as a default. A filter that removes 80% of the value you can prove, to save storage you have plenty of, is not an optimisation. The measurement changed the design, which is the only reason to take a measurement.

The actual win was somewhere else entirely.

Store less, fetch on demand

arXiv goes to abstract mode: one row per paper from the title and summary the feed already carries, nothing fetched. About 47× fewer chunks — and the full paper is still one paper_read call away whenever a question genuinely needs it.

The same move, again, for books. A Calibre library connects over OPDS, and the index holds one metadata row per book — title, authors, tags, description — and never the text. A book is read on demand, bounded, cited, and not stored.

That's the pattern:

If the full text is still reachable at a stable address, indexing it is a cache — and it is a cache you are paying for in embedding cost, storage, and retrieval noise, on the 99% of documents nobody will ever ask about.

Index what you need to find a thing. Fetch what you need to read it. Full-text indexing earns its place only for sources you cannot re-fetch.

Feeds now also get graded on this. A table records every entry a run stored, with the knowledge row it produced, so "which feeds have documents a prompt actually used?" is a query rather than an opinion. Once a month, the briefing names active feeds with 90 days of history and no use in that time. Three failed fetches in a row, or no entry for a configured staleness window, becomes a finding on the problem hub.

That matters more than it sounds, because the RSS reader this replaced had been dead for five and a half months and nobody noticed. Every one of its feeds had been erroring since 28 March. Nothing caught it, because AEGIS read only its feed list, once, at startup, and the health check only asked whether the API answered. Both of those are true of most integrations you have: you are monitoring that the service is up, not that it is doing anything.

Research is a workflow, not a chat turn

Research used to happen inside a chat reply. That is wrong for a boring reason: chat turns time out, and users retry them.

Now a question goes to a durable workflow whose id is research-<sha256 of the question>. Three consequences fall straight out of that one line:

• A retried turn re-attaches to the run already in flight, instead of paying for the same research twice.
• Past 45 seconds the tool says "still researching" and the flow posts the answer to the agent's channel when it lands. The answer is never lost to a closed connection.
• Only a real answer is saved. A run whose synthesis failed says so and stores nothing — no empty row for a future search to rank.

The same seam is used for ledger writes in the money lane and note writes here: a tool validates a write; a workflow performs it. Put the write back in the tool and you have a write the chat loop cannot cancel and cannot re-attach to.

Letting it fetch

Letting an agent research means letting it fetch, and that is the other half of the risk. Every fetch of a URL a model chose, a page named or a feed published goes through one guard. The important detail is that it checks the first request and every redirect — a public page redirecting inward was the actual hole, and checking only the initial URL is the version of this guard that everyone writes first. Bodies stream with a hard cap of 10 MB. What it does not close, stated plainly in the docs so nobody assumes otherwise: DNS rebinding, where the host resolves differently between the check and the connect.

A tracked topic is a problem, not a feed of chores

Those 277 auto-closed tasks came from treating every interesting article as an event worth interrupting a human about. The replacement reuses the alerting primitive: a tracked topic is one problem, and each matching article is an occurrence of it, keyed on the URL so a story arriving by two routes counts once.

A round earns a task only when it holds enough items — two for a high-priority topic, three for medium, five for low. Below that it lives in the hub and the daily briefing, visible and silent.

And one detail I would not have predicted: completing the task resolves and closes the round at once. The hub's normal behaviour is to reopen a problem when it recurs, which here would mean the task you just dismissed comes back tomorrow because one more article appeared. Closing the round means the next article opens a fresh one. "Seen" has to be a real state, or dismissal is meaningless.

The seeding decision is my favourite piece of restraint in this lane. I could have seeded tracked topics from the existing scan configuration. I didn't, because those topics are single broad words — ai, world, tech, macro — and as whole-word terms they would cross the threshold on every single scan and raise a task every round. That is precisely the noise the change removed. A fresh install therefore tracks nothing until you ask for something.

The vault: the file is the record

The last piece is the one that ties the series together.

I already keep an Obsidian vault: 1,023 markdown notes, 582 of them journal entries from 2014 to 2023 — and idle since 25 October 2023. Meanwhile AEGIS was writing a nightly log entry into a database table nobody would ever open.

So the vault became the record, and the knowledge store became its index. The nightly, weekly and monthly entries are now notes in the vault, filed the way the vault files its own: journal/<year>/<month>/<day>.md, the week's note in its Monday's month folder, the month's entry in the month folder's own note. If today's note is already open where the periodic-notes plugin creates it, the write goes there instead.

Three rules make this survivable, and all three are enforced in code rather than requested in a prompt:

Insert-only. A write creates a note, or inserts exactly one contiguous block into an existing one. The check is mechanical: the new content must be the old content with precisely one block inserted, or the write is refused. Nothing I wrote is ever rewritten, reordered or deleted — because the same files are being edited from my phone while this runs. Each block carries a hidden marker, so a re-run is a no-op instead of a duplicate.

A write counts only once pushed. An auto-commit plugin commits from my phone and my laptop. A rejected push, or a rebase that conflicts, drops the agent's own unpushed commit, pulls fresh, re-applies once, then reports. It never force-pushes, and it never leaves a divergent commit sitting in the checkout.

Encrypted blocks never leave the file. The vault holds ciphertext inline between markers, from an encryption plugin. That is stripped before anything is indexed, shown by a read tool, or sent to a model. The encryption is not there to defend against my own agent, but the agent has no business being the exception.

What all three lanes agreed on

Alerting, money, knowledge. Three rebuilds, three different problem domains, nine days, and the same sentence at the end of each:

The record is a thing a human can open — a problem row, a journal file, a markdown note. Everything else is an index, and an index is allowed to be wrong, because you can rebuild it from the record.

For alerting that meant a problem exists before any ticket about it does, so deleting the ticket changes nothing. For money it meant a plain-text double-entry journal that hledger checks and git remembers, with Postgres demoted to an index. For knowledge it means the file I have been keeping since 2014, which now has a diligent contributor and no editor but me.

The nice consequence is that this is also the answer to "what happens when the agent is wrong?" — a question I get from every client who is nervous about putting a model near real work. If the model's output lands in a file you own, in a format you can read, appended and never rewritten, then being wrong is a line you delete. That is a very different risk profile from an agent with UPDATE on the only copy of the truth.

AEGIS is open source and MIT-licensed: github.com/hikmahtech/aegis — the design documents with every measurement above are in the repo. The wider tour is at /aegis.

The rest of the series: your alerts have no identity and a ledger is not a database table.