Models Come and Go, the Memory Stays

For about a year, everything my AI assistants knew about me lived inside other people's products. A memory feature in one, a "projects" area in another, a config file that only a single tool ever read. It worked well enough. It also meant the most valuable thing I had built was not the automation and not the code - it was the accumulated context. Who I am. How I want an agent to behave. Every host and service in my homelab, the safety rules, the shape of my family's week. And all of it was scattered across features I do not own and cannot move.

That is a strange spot to end up in. You spend months teaching a system your life, and the part that remembers it is rented back to you.

So a few weeks ago I moved all of it into a folder.

Not a database, not a platform. A folder of Markdown files - _AGENTS/, one numbered subfolder inside my everyday Obsidian vault, sitting right next to my daily notes, my dev projects, my blog, even the autobiography I am slowly writing. 114 files today. Headings, a few tables, wiki-links between notes. Nothing you couldn't open in any text editor from the last thirty years. It syncs across all my machines through Nextcloud and it is versioned in git on my own Forgejo, so history and diffs just work - because it is plain text, and plain text is the one format nobody gets to take away from me.

That placement is not an accident. _AGENTS/ is the agents' corner - they keep it and organize it themselves, within the rules, I do not hand-file it. But it is not walled off. Because it lives in my real vault and not in some sealed store, the same agents that remember me can step out of their corner and into my actual notes - read them, draft things for me, work on a project next to me, in the place I already think. One workspace, an area for me and an area for them. The wall most AI setups have - my knowledge here, the assistant's context there, me ferrying between the two - is simply gone. This post is being written into that same vault, in its blog folder, by an agent working next to those notes.

It did not start clean, though. Before the folder, that same context lived in each tool's own config file - every AI tool I use had its own instruction file and its own little memory store - and those copies had quietly drifted apart. The trigger to finally fix it was concrete and a bit embarrassing: the per-machine memory on one laptop had silently run weeks ahead of the store that was supposed to be canonical, because nothing ever forced the two to agree, so they didn't. That is the failure mode of memory-as-a-feature in one sentence - you cannot even see the drift.

So I migrated it by hand, not by dumping everything into one big file. Back it all up first - I do not delete a file I am not sure about. Then walk the old config fact by fact and ask one thing of each: is this true everywhere, or only on this machine, or only for this one tool? Four buckets. Duplicate, already captured - drop it. Device-specific - it stays a local scratchpad fact. Tool-specific - same, one layer over. Genuinely new and true everywhere - promote it into the shared folder. Anything I could not cleanly sort, I asked about instead of guessing. When it was done, the fat per-tool config files had collapsed into a slim pointer that basically says "read the folder." The knowledge had moved out of the config and into the thing I own.

That is the whole claim, and I want to keep it at its real size. The interesting thing is not the folder. It is what stops mattering once your AI's memory is a folder you own.

Two things follow.

The first: the model underneath becomes swappable. When the memory is a feature inside a product, switching products means starting over - re-teaching the same context to the next assistant. When the memory is a folder, the harness is just the thing that reads it this week. I know this holds because I have actually done it, not because it looks good on a diagram. Three different agent setups - Claude Code, Opencode, and Hermes - have read and written to this same folder, across five machines: a MacBook, two Mac minis, a Mac Studio, a Linux box. The folder's own work-logs are grouped by which agent on which device wrote them, and there are seven such identities in there now. Some of those agents run against local models on my own hardware, so this is not "trade one cloud vendor for another." The memory layer does not need a cloud provider at all.

The second follows from the first, and it is the part I did not expect to like as much as I do. I stopped configuring agents and started onboarding them.

There is an ONBOARDING.md that a brand-new agent reads before it is allowed to write a single line - read the whole thing first, the same instruction you would give a human on day one. There is a MIGRATION.md for an agent that already has its own memory and needs to converge on the shared one. A new tool does not get "set up." It reads the handbook - the same rules, the same voice, the same safety kernel every other agent already follows - and gets to work. You point it at the folder the way you point a new-hire at the wiki.

Underneath, it is boring on purpose, and the boring is doing the work. Two layers. A local scratchpad on each machine for things that are only true there - paths, installed versions - and the shared, synced folder for everything durable that applies everywhere. The rule of thumb is the same one from the migration: if a fact would still be true on another machine or under another tool, it belongs in the shared one.

Inside that shared folder the top-level directories are numbered, so their order is obvious at a glance - who I am and how I want an agent to behave, safety rules and my writing voice included, then the homelab, then the personal and family context, then one note per machine, one per AI tool, and the append-only logs. Every note is the same dumb shape - a little frontmatter with an updated: date, some headings, a few tables, and wiki-links to the related notes, so facts that belong together are connected. Two indexes do two jobs: a human-facing README on how to use the folder, and a flat, machine-readable index an agent can use to jump straight to the right note - with one rule keeping it honest, which is that if you move a file you fix the index in the same commit. None of that is decoration. It is the whole reason an agent that has never seen the folder can land in it on its first run and find the fact it needs before anyone has told it anything.

And because the folder reaches every device, there is one rule no agent gets to skip: propose the exact change, wait for my okay, then write. Ownership is not a free-for-all - it comes with a write protocol, which is also what makes it safe to let them out of their own corner and into the rest of my notes. My wife runs her agents on the same substrate, so it has to be safe for both of us.

There is an obvious objection to all of this, and it is the right one: plain text rots. Dates pass and nobody updates the note. Links break. Two notes start quietly disagreeing. A fact that should live everywhere gets stranded in one machine's scratchpad and never makes it up to the shared folder. A vendor's memory feature at least hides that mess from you - a folder puts it right in front of you and leaves the tidying to you.

So the upkeep became a job with a name. I have a gardener - a routine whose only task is to walk the whole thing and find where trust is decaying. Contradictions, where the same fact is stated two different ways. Stale facts, where a date passed and nobody noticed. Dead links and holes in the index. Facts sitting in the wrong layer that should be promoted up. It reports exact proposed fixes and it does not get to write on its own - I approve every change, I am the merge gate. Three limits on it, and all three are the point. It fixes facts, not style - a note written in my phrasing keeps my phrasing. A difference between two notes is not automatically a contradiction - it might be scoped, or historical - so when it is unsure it asks instead of "fixing." And it maintains, it does not redesign. The folder is only worth something for as long as the agents can trust it blindly. Maintenance is not overhead sitting next to that - it is the thing that keeps the trust alive.

One honest note, because this is exactly the kind of thing that gets dressed up as an architecture. It is not one. It is plain files and a couple of conventions - nothing to install, nothing to buy. But I am not going to pretend the cost is zero. The discipline of keeping facts where they belong is real, and I had to stand up a recurring maintenance routine just to pay it. It works because it is dumb technology, not despite it - but dumb technology you still have to tend. Markdown in a folder, and a bit of gardening. If I sold it to you as a framework I would be doing the exact thing I am trying to get out from under.

That is the trade I would make again. The tools will keep changing - faster models, new harnesses, whatever ships next quarter. The folder stays. I would rather own the memory and rent the intelligence than the other way around.