The name is about writing.
Once a model finishes training it stops changing. Whatever it works out while you are talking to it sits in a temporary buffer that gets thrown away when the conversation ends.
The work here is on models that can write to themselves while they are running: they carry a memory of their own, put things into it as they go, and still have them later.
First principles of stateful machine learning
Most of what gets called memory today is one of three things: a bigger temporary buffer, a search index bolted to the side, or another round of training. None of them changes the model itself. Four things follow from making the model's own memory writeable, and they are why it is worth the trouble.
The memory is a fixed size and stays that size whether the model has been running a minute or a month. Nothing has to re-read the whole history every time you say something.
What it works out from you goes into that memory and is still there after the conversation ends.
A process that stays up and carries its memory forward, rather than starting cold every time it is called.
A fixed-size memory cannot keep everything word for word. What to condense, what to drop, and what to do when a new fact contradicts an old one are each a separate operation the design has to actually perform.
The alternatives
Searching a document store, squeezing a long conversation down to fit, and running extra training on the side are all real techniques, and none of them is this. Each leaves the model itself unchanged and keeps the memory somewhere outside it. The work here is on a fixed-size memory that sits inside the computation and stays writeable while the model runs. Exactly how that relates to older ideas along the same lines is part of the technical work, and the papers will say so precisely.
| Property | Bigger buffer | Retrieval | Continual fine-tuning | Writable internal state |
|---|---|---|---|---|
| Where the information lives | A temporary buffer | External store | Updated weights | Inside model state |
| How it gets updated | Append tokens | Write to the store | Gradient steps | The model writes it |
| Bounded without replaying the history | No | Partly | Yes | The target |
| Survives the session | Not inherently | Yes | Yes | Yes |
| Takes part in the computation | Yes, as prompt tokens | Only once retrieved | Implicitly, in weights | Yes, as state |
Whether a model whose weights never change can do open-ended work purely by reading and rewriting a fixed-size memory it continuously reads and rewrites is still an open question. Some of it works with no training at all. Some works only after training. Some does not work yet. Sorting out which is which, and why, is what the first papers are about.
How you would know
A model's training loss falls when it learns what text usually looks like. It also falls when it genuinely holds something and gives it back later. One number, two abilities, and nothing in it that separates them, so loss cannot answer the question this work is about.
What can answer it is asking a memory to perform one operation at a time, on streams where knowing what text usually looks like buys you nothing. Hold something across a long gap. Pick the right one out of many. Overwrite a fact rather than stack a second one beside it. Recover a whole from a fragment. Say whether something has been seen before at all. Erase a binding. Look something up backwards, from its value to its key. Those are not a wish list, they are what four separate fields independently converged on as the operations a memory has, and each one is a different question you can put to a store.
Then the score has to be taken away again. Every result is re-run with the memory emptied. If it survives that, the memory was never carrying it and whatever passed was reachable from the surrounding text all along. That one control is the difference between a benchmark number and a claim about state, and applying it to published architectures is a large part of what this line of work has produced so far.
Why ownership matters past privacy
A stateless model you run locally is the same model on your last day with it as on your first. One that accumulates on your hardware, from your data, becomes specific to your problems without any of that history leaving the machine. A memory that never grows is also what makes the cost of running one predictable, and predictable cost is what makes ordinary hardware a realistic place to put it.
What the second line is for
Building something that behaves well is not the same as knowing why it behaves that way, and the second line is how you check. It runs on models anyone can download: forty-six of them from fourteen different groups, from 160 million internal parameters up to 14 billion. A lot of published work in this area tests one model against an easy comparison and stops there.
Several of those models are not built the usual way. RWKV, Mamba, Falcon-H1 and recurrentgemma carry a running internal memory instead of re-reading everything each time, which is the same idea the first line is about, and that is why they are in there. The two lines are one question asked from both ends. The limit on how many steps a model can chain internally is not my finding, the existing literature reports the same range. What I measured is that it barely moves across fourteen models from 28 to 48 layers deep, and holds on an architecture with no attention in it at all, which puts it below attention, in the computation itself. Commit-then-transport holds on a model with no attention layers at all.
What has come out of it so far is a calibration of how far to trust the existing literature, and it runs in both directions. The measured results are on their own page, along with the ones that cut against the thesis.
The divider
The mark between axio and gram is a vertical spine crossed by a horizontal, with cells scattered either side, some filled and some empty. The spine is the memory that persists; the cells are what has been written into it and what has not. It is the architecture, at the size of a piece of punctuation.
Contact
Research and collaboration: [email protected]
General inquiries: [email protected]