6 August 2026
A trained neural network is frozen. You talk to it, it answers, and not one of its weights moves. Everything it appears to "remember" about your conversation is really just text you keep handing back to it.
Here is a different idea, and it is small enough to build in a blog post.
Leave the big frozen network alone, and bolt on a few dozen extra numbers that
do change while it runs. Call them m. Every observation
nudges them. Between observations they fade. And here is the part worth
testing: give every one of those numbers its own half-life.
Some forget in three steps. Some forget in two hundred. Most sit somewhere in
between, spread out on a log scale.
The claim I wanted to test: a spread of half-lives should beat any single one, because a memory needs to be good at remembering both what just happened and what keeps happening, and one forgetting rate can only be good at one of those. It sounds obviously true. It is mostly false, and the way in which it is false turns out to be the interesting part.
Everything below runs in your browser. The training is real: about 200 lines of JavaScript, no libraries, gradients computed by hand-rolled autodiff. You can watch it learn.
Weights normally look like a big matrix W. We are not going to
touch it. Instead we bolt a small correction onto it:
W_now = W_frozen + U · diag(m) · VT
U and V are thin: if the network is 1000 wide,
these might be 1000×48. They are learned once and then frozen too.
m is just 48 numbers, and m is the only thing that
moves while the model is running. That is the memory. Forty-eight numbers.
Written as code, storing and recalling a fact are four lines each:
// remember: "this key means this value" a = key · V // where does this key live in the basis? c = value · UT // where does this value live? d = a * c // elementwise: the two, agreeing m = lambda * m + eta * d // recall: "what did this key mean?" a = query · V logits = (m * a) · U // elementwise again, then project out
That is the entire mechanism. d is computed from the thing in
front of you and nothing else: no history, no backpropagation through the
conversation. And lambda is a vector, not a number. That vector is
the whole point of this post.
If you have seen Hebbian learning, d = a * c is "fire together,
wire together" squashed onto 48 directions. If you have not: it is the outer
product of key and value, compressed.
Each channel j fades by a factor lambda_j every
step, so a fact written t steps ago survives in that channel with
strength lambda_jt. Pick lambda close to
zero and the channel is a goldfish. Pick it close to one and the channel never
lets go, which sounds good until you realise it also never lets go of the
thousand irrelevant things it saw in between.
Drag the slider. Nothing is being trained here; this is just arithmetic on how much of a memory survives.
The green curve is what a spread of half-lives buys you. It starts high like the fast channel and it has a tail like the slow one, because at any delay some channel is in its sweet spot. Plot it on log-log axes and it is close to a straight line: a power law, not an exponential. That is the shape human forgetting curves have, which is not a coincidence and which I will come back to.
Arithmetic is not evidence. The question is whether a model can learn to use this thing, so let us train one here, on this page.
The task: a stream of events. A write says "key K means value V".
A query says "what does K mean?" Keys are freshly invented random
vectors every single episode, so the network cannot memorise any of them in
U and V. Whatever it knows at query time it learned
seconds ago and put in those 48 numbers.
Crucially, the stream is crowded: facts arrive faster than 48 numbers can hold, so the model has to let go of something. (My first version was not crowded, and I will show you later why that made the whole experiment worthless.)
m moves.It should climb past 45% within a couple of hundred steps, against a chance rate of 12.5%. That is the "it works" bar: 48 numbers, updated by one multiply-add per step, doing real associative recall on keys they have never seen.
Once it is trained, sort the queries by how many times that fact had been written before it was asked about. Nothing in the model is told which facts matter. There is no importance flag, no priority, no attention over history.
Repeated facts survive dramatically longer than one-off facts. And it falls
straight out of the arithmetic: repeating a fact writes the same
d again, so the writes add up — but only in the channels
slow enough not to have faded in between. Repetition automatically routes a
memory into the long-lived channels. Nobody designed that; it is what
m = lambda*m + eta*d does when lambda is a spread.
In the full run, one-off facts sat at 45% and facts seen four times at 78%. The effect is even sharper for the slow variants: a model with a single slow rate goes from 37% on one-off facts to 94% on repeated ones. Repetition buys survival everywhere, and nothing in any of these models was told which facts were important.
A demo that agrees with me is worth very little, so here is the comparison run properly: nine variants, identical in every respect except the vector of half-lives, trained from scratch in Node.
The one number to look at is the shape across delays, not the average.
And here the idea I started with mostly falls over.
Read the two highlighted rows. A single decay rate, tuned, beats the log-spaced spectrum — 49.3% against 47.0% overall, and better at almost every delay in between. The spectrum wins at delay 0 and nowhere else that matters. The gap is small but it is five seeds wide: ±0.4 and ±0.5 points respectively, so it is not noise.
Worse for the idea, look at the shuffled row. Scrambling which channel gets which half-life changes essentially nothing: 46.7% against 47.0%, and delay curves that agree to within a point almost everywhere. I had listed that as a thing that would falsify the idea, and it fired. The ordering of the spectrum carries no information; the model is just averaging over a bag of rates.
When I ran this with one seed I got the opposite conclusion and wrote an excited paragraph here about the spectrum being the only row good at both ends. Five seeds later, it is not. I have left the shape of that mistake visible rather than quietly swapping the numbers, because a single seed producing a clean story is the most common way to fool yourself in this kind of work, and I did exactly that.
Two things are solid, and neither is the thing I set out to show.
Forgetting matters enormously. Never forgetting is the worst variant in the table — 38.2%, dead last, below every single decay rate. It is excellent at facts it has seen many times (94% at four repeats) and hopeless at facts it has seen once (35%), because everything it ever saw is still in there, drowning the recent stuff. A memory that cannot forget is not a memory with infinite capacity; it is a memory full of noise.
Learning the half-lives beats prescribing them. The best row in the table, by a clear margin, is the one where the timescales themselves are trained: 50.8% against 49.3% for the best fixed rate, and 92% versus 76% at delay 0. Same channel count, same parameters otherwise, same everything. Give the model a spread of rates and let it move them, and it does better than any spread I chose by hand and better than any single rate.
So the useful version of the claim is not "spread your timescales out on a log scale". It is "let the model choose its own timescales, and give it more than one to choose from". That is a weaker and much less romantic statement than the one I started with, and it is what the data actually supports.
One question is worth more than all the accuracy numbers. When this thing remembers that key K meant value V, where is that fact? Spread across all 48 channels, or sitting in one of them?
It matters because the boring answer would deflate the whole idea. If each fact owns a channel, this is a lookup table with 48 slots wearing a costume, and calling it "weights" is marketing. The interesting answer is that every fact is smeared across every channel, and no single channel is load-bearing.
There is a clean way to find out, and it needs no training at all: take the trained model and hold channels at zero. If it is a table, knocking out one of 48 channels should cost you about a 48th of your facts, roughly 2.1 points. If it is distributed, it should cost almost nothing.
| channels removed (of 48) | accuracy | % retained | a lookup table would give |
|---|---|---|---|
| none | 50.8% | 100% | — |
| 1 | 50.6% | 100% | 50.0% |
| 8 | 48.6% | 96% | 44.4% |
| 24 — half of them | 41.9% | 83% | 31.6% |
| 40 | 28.0% | 55% | 18.9% |
| 47 — one channel left | 14.8% | 29% | 13.3% |
Throw away half the memory and it keeps 83% of its accuracy. Throw away 40 of 48 channels and it is still at 28%, more than double chance. No single channel is worth more than three quarters of a point. This is a distributed memory, decisively, and it is the one prediction of the original idea that came through cleanly.
Now the finding that reframes everything above.
The best variant in the table was the one that learns its own half-lives. I initialised those half-lives log-spaced from 2 to 200 steps and let gradient descent move them. Here is where they ended up:
initial spread: tau = 2 ... 200 (log-spaced, 48 channels) after training: tau = 8.9 ... 11.2 (all 48 channels)
Given the freedom to place its timescales anywhere across two orders of magnitude, the model collapsed all of them onto approximately one value. It did not distribute them. It did not keep a few fast and a few slow. It threw the spectrum away and picked a single rate of about ten steps — and by doing so it beat every hand-designed spectrum I gave it.
So "learned timescales win" and "spectra do not help" are the same finding, not two. The learned model wins because it converges on one rate while keeping all 48 channels working at it. A prescribed spectrum spends half its channels on timescales this task has no use for.
The lesion data says exactly that. In the fixed-spectrum model, removing the 24 fastest channels drops short-delay accuracy from 74.6% to 46.6%, while removing the 24 slowest barely touches it (70.5%). The fast channels are doing the work. The slow ones sit at 15-17% on long delays, which is barely above the 12.5% floor: they are close to dead weight. In the learned model the same test shows nothing at all — removing the "slowest" and the "fastest" are identical, because after training there is no difference between them.
Channels genuinely do specialise by timescale when you force a spectrum on them. It is just that in this task, specialising is a waste of capacity.
All of which suggests the obvious experiment, and I think it is the one that would actually settle the idea. This task has a single dominant timescale, so of course one rate wins. Build a task with genuinely two: facts that matter for ten steps mixed with facts that matter for five hundred, both queried often. A spectrum should win there, and if it still does not, the idea is dead rather than merely unsupported.
That is a real prediction and I am writing it down before running it, which is the only way any of this counts for anything.
My first version of this experiment had four facts and forty-eight channels. The spectrum lost to a plain single slow rate, and "no forgetting at all" beat everything.
It took me an embarrassing minute to see why. With four facts and forty-eight numbers, the memory fits everything comfortably. Forgetting is pure loss. I had built a task where the correct policy is to never forget, and then expressed surprise that forgetting did not help.
Decay only pays for itself when memory is scarce. Once facts arrive faster than the state can hold them, forgetting stops being damage and becomes triage. Every result on this page comes from the crowded version, and I have left the uncrowded numbers in the repository because a negative result you cannot reproduce is just an anecdote.
There was a second bug, and it is the kind worth naming. I had a variant where the half-lives themselves were learned rather than prescribed. It produced results identical to the fixed version to four decimal places, which I briefly took as an interesting finding — "learning does not help here!" — before noticing that I had built the decay vector as a constant, so no gradient ever reached it. It was not learning anything. When two conditions agree that exactly, you have found a bug, not a result.
The honest answer is: the mechanism, no; the framing, a little.
Spreading timescales across channels is what diagonal state-space models (S4D, Mamba) already do, and what RetNet does explicitly by giving each attention head a different decay rate. Updating weights online from a local surprise signal, with forgetting, is what Google's Titans does at scale. Storing memories in fast-changing weights goes back to Schmidhuber's fast weights in the early nineties.
And the specific claim that a spread of exponentials gives you power-law forgetting is older than all of it: Fusi, Drew and Abbott showed it for cascades of synaptic states in 2005, and Benna and Fusi built the continuous version in 2016. They were modelling actual synapses. The reason your memory of last Tuesday fades faster than your memory of your childhood may be exactly this mechanism, running in wetware.
What I have not found anyone doing cleanly is the cheap version: freeze the basis, keep only the diagonal, and treat those few dozen numbers as a persistent per-user memory that rides on top of a model nobody is allowed to fine-tune. That is an engineering story rather than a scientific one, and this post is the smallest experiment I could build that would falsify it.
Stated up front, so I cannot move the goalposts afterwards:
The first two fired. A tuned single rate does match and slightly beat the prescribed spectrum, and shuffling the assignment changes nothing. By the standard I set before running it, the specific claim — that a continuous log-spaced spectrum is the thing that helps — is not supported. What replaced it is the finding that learned timescales beat both.
The fourth did not fire, and that is the good news: the lesion results above show a thoroughly distributed memory, not a lookup table. The third I still have not measured properly — the state does not obviously blow up over 480-step streams, but I have not tracked its norm.
git clone <repo> && cd code node smoke.js # smallest thing that could fail: does it learn at all? node study.js # the nine-variant comparison, 5 seeds node study.js --quick
No Python, no GPU, no dependencies. The whole study is about ten minutes on a laptop. The three files it loads are the same three this page is running.