My AI Remembered Everything. That Was the Problem.
I asked the knowledge base who Fredrik works with and got back 85 documents.
An email from 2014 said Softhouse. A meeting note from 2024 said CodePath. A Slack message mentioned his consulting company Metallapan. A ChatGPT conversation from the initial bootstrap somehow attributed my daughter's Python programming project to him. All of these were in the index. All of them came back as results.
The knowledge base could find Fredrik. It just couldn't always tell me a coherent story about him.
The search evolution was about retrieval mechanics. How to find things. This story is about what happens after retrieval works, when the problem shifts from "I can't find anything" to "I found too much, and some of it is wrong."
The "just store everything" trap
The default approach for any RAG system is simple: index everything, retrieve what matches, let the LLM sort it out. And it works at small scale because the noise-to-signal ratio is low enough that the LLM can sort through it.
But the noise grows with the corpus. More documents means more empty files, more stale fragments, more encoding errors in generated summaries. At some point the junk starts showing up in your results. For me that point came somewhere around 20,000 documents, but the number isn't the point. The point is that it's proportional, and it only goes one direction.
Empty files from old Google Drive exports were being indexed as real documents. Doc card summaries had encoding corruption (the LLM occasionally mangled characters during generation, producing text that couldn't be stored or searched properly). Empty shells that look like results but contain nothing useful.
"Just store everything" is the RAG equivalent of "just log everything." You end up with so much data you can't find the signal. And it's insidious, because these aren't hard errors. Nothing crashes. The search still returns results. They're just slightly worse, and you don't notice until you do.
The Data Constitution
So I built a governance layer. Every document passes through a 7-gate pipeline before it enters the index. Not a filter (that implies losing data). A quarantine system.
The core principle is no silent coercion. If an LLM is generating a doc card summary and its response is missing a required field, the system doesn't fill in a blank or silently skip it. It quarantines the item with the full payload preserved and logs exactly why. If the GPU runs out of memory during embedding, the batch gets quarantined, not dropped. When resources recover, the system replays the quarantined items automatically.
761 items have been quarantined since the Constitution went live. 300 of those have been successfully replayed after the underlying issue was fixed. The system is self-healing, not just self-protecting.
The numbers by category tell the story. 77 empty files caught at the parsing gate. 270 encoding failures caught during doc card generation. 30 embedding batches preserved for retry after CUDA out-of-memory errors. Every one of those would have silently degraded the index without the Constitution.
This is boring infrastructure by design. It's plumbing. But try building a memory system on top of an index you can't trust. I already learned that lesson with ChromaDB.
Data quality before data intelligence. Always.
What is an entity?
Here's the real problem. Fredrik appears in 85 documents. Some people in the system appear in over 200. Each document has a fragment of context about them, but no single document has the full picture. And some of those fragments contradict each other across time.
If I ask "tell me about Fredrik," the system has to search across all 85 documents, pull the relevant chunks, and hope the LLM synthesizes them correctly right there at query time. That's expensive. It's unreliable. And it gets worse as more documents accumulate, because the ratio of stale fragments to current truth keeps growing.
So I introduced entity docs. A single, curated document per entity (person, project, organisation, system) that represents what is true right now. Fredrik's entity doc has his current client, his consulting company, his relationship to me, his work history, his Slack ID, even his preferred language. The 85 mention documents are the evidence. The entity doc is the conclusion.
This is the core philosophical shift. Instead of re-synthesizing "who is Fredrik?" from 85 scattered fragments every time someone asks, you do it once during curation, maintain the result, and serve it directly. Move the intelligence from query time to build time.
And the doc cards from the search post fit in naturally. Each entity doc (which can get quite long) gets its own compressed summary. So the retrieval chain becomes three levels of compression: 85 source documents become one entity doc, which becomes one doc card. A search for "who works on oneapp?" can scan doc card summaries (tiny, fast), find Fredrik's entity (mentions oneapp, CodePath, backend team), and load the full entity doc or individual docs only if it needs the detail.
Three levels. Each one serves a different retrieval need. And each one is small enough that the system doesn't choke on context.
Here's what that looks like in practice. This is the doc card for one of the entities in my system:
Summary, key facts, topics for search matching, and pointers to the 7 source documents it was built from. The full entity doc has more detail, but this card is enough for the system to decide whether Latte is relevant to a query without loading everything.
When categories help and when they hurt
Once you have entities, you need types. A person is not a project is not a conference.
The system currently has 10 entity types and 264 entities. Person (164), project (39), organisation (27), and system (15) carry 93% of everything. Then there's a long tail: conference (7), process (5), course (3), pet (2), vehicle (1), team (1).
These types reflect my life and what's in my knowledge base. I'm an independent consultant who goes to testing conferences and builds software systems, so that's what the taxonomy looks like. Someone else would end up with different types. A researcher might need "paper" and "dataset." A product manager might need "feature" and "customer segment." The types aren't universal. They're shaped by what you actually need to keep track of.
Yes, pet. I have two British Shorthairs in the knowledge base. They didn't fit anywhere else.
The tension is real though. Too few types and you lose the ability to filter and browse. "Show me all the people I work with" requires type metadata.
Too many types and you spend curation time debating categories instead of capturing knowledge. Is "backend-verification-recipes" a process or a system? Is "team-culture-notes" a process or a project? Every new type creates classification ambiguity at the boundary.
So I started small and split when browsing broke down. The four core types (person, organisation, project, system) were obvious from day one. Conference got added when I had enough conference-related entities that they cluttered the organisation list. Process emerged when I needed a home for recurring workflows. The cats forced pet into existence.
Don't design the taxonomy upfront. Let it emerge from real data and refine when friction shows up. The types that exist earned their place by being needed.
Facts change. Documents don't.
An email from 2014 says Fredrik works at Softhouse. A meeting note from 2024 says he's at CodePath. Both are true. At different points in time.
This gets worse with less obvious changes. A project status doc from November says "admissions flow is in staging." A doc from January says "admissions flow launched to production." Neither contradicts the other on its face. But one is stale. Multiply this across 24,000 documents and temporal ambiguity becomes a real problem.
The design decision: source documents are immutable. An email from 2014 should still say what it said in 2014. Rewriting source documents to reflect current state would destroy provenance and make the knowledge base untrustworthy as an archive.
The entity doc is the mutable layer. It represents "what is true now," curated from the immutable evidence below it. The architecture becomes: immutable source documents (evidence), mutable entity docs (current truth), compressed doc cards (retrieval summaries). Each layer has a different relationship to time. Source docs are frozen at their creation date. Entity docs reflect the present. Doc cards summarise the entity doc's current state.
I should be honest about what isn't solved. There's no automated mechanism yet for detecting when a new document contradicts an existing entity doc. No way to flag that the last curation was three months ago but twenty new mentions have arrived. Staleness is caught opportunistically during curation sessions, or when a search result feels wrong and I dig in.
That's the weakest link in the entity pipeline right now. The Constitution ensures quality at ingestion. But it doesn't track truth drift over time. I'd rather be honest about an unsolved problem than pretend the system handles it.
Manual before automated
The entity pipeline started with no backend code, no schema changes, just Claude Code skills running manually. A curation session looked like this: run /kb-curate, get back 17 unincorporated captures across 6 files. Group them by entity. Analyse for contradictions. Create 3 new entity docs. Update 1 existing one. Mark 15 source notes as incorporated.
Everything worked. It just required a human to run the command and approve every decision.
That was deliberate. I wanted to prove that entity docs were worth having before automating their creation. Same philosophy as the PA suggestions system, where I validated the suggestion concept manually before building the automation. Discovery and decision should be validated together before you split them apart.
The growth curve tells the story. Manual curation produced about 0.4 entity docs per day. Tooling-assisted curation (Phase 2) brought it to 2 per day. When the automated aggregation engine went live (Phase 3), it peaked at 9 in a single day and settled at about 1.2 per day sustained.
But here's the thing. When I ran the aggregation engine against the full corpus, it surfaced 1,512 entity candidates. I processed them in a batch session. 79% were rejected as noise. Days of the week. Programming language names. API terms. "Monday" is not an entity. "Python" is not an entity (in this context). "GET" is definitely not an entity.
The rejection rate is part of the design. The aggregation engine casts a wide net with confidence scoring (source quality, frequency, diversity, recency). The more something is mentioned in different types of sources, the more confident the system gets. Candidates surface in a review interface. The human decides. We automated discovery while keeping human approval.
Each phase removed friction without removing judgment. Human-in-the-loop, always, it's been a core philosophy of my local AI systems from the start for a good reason.
What's still missing
So now the knowledge base has documents you can trust (Constitution), organised around entities that represent current truth (entity docs), compressed for efficient retrieval (doc cards). Search finds things. The index is clean. Entities capture who and what matters.
But the entities exist in isolation. Fredrik's entity doc knows everything about Fredrik. It doesn't know that his contact card and the oneapp migration doc are always opened together during Monday planning sessions. It doesn't know that the admissions dossier and Daphne's contact card travel as a pair. Content tells you what things say. It doesn't tell you how they're used together.
That's a different kind of signal. And getting it required putting a filesystem in the middle of everything.
That's the next chapter.
This is part of a series about building a personal knowledge base. Previously: I Gave My AI a Memory, I Removed the Friction. That Was the Problem., I Called Them Suggestions. There Was a Reason., and Teaching a Knowledge Base to Search.
