Chapter 05 · Updating stored associations

DeltaNet: learning to overwrite

The fixed-size memory in chapter 04 adds every new association to the same matrix. DeltaNet first checks what the matrix already returns for a key, then writes the difference between that result and the new value. This delta rule lets the model revise an association. Making those sequential updates run efficiently on GPUs took additional work.

5.1Read before write

When a new key-value pair (k_t, v_t) arrives, DeltaNet reads the current value for that key from S before updating it:

β_t ∈ [0, 1] is a learned, per-token write strength.

old_value  = key @ state                    # read the current value for this key
correction = beta * (new_value - old_value) # scale the prediction error
state      = state + outer(key, correction) # update the matching association
output     = query @ state

Pseudocode adapted from the source article by @waterloo_intern.

The update has three useful properties:

• If the memory already returns the right value, e_t ≈ 0, stable facts are left alone by this write (later, overlapping writes can still disturb them).
• If the key holds stale information, the write subtracts the current value and adds the new one, scaled by the write strength.
• β_tcontext The model computes this per-token write strength from a learned projection of the current token, then restricts it to [0,1]. The strength can vary with context instead of being a fixed setting. controls the size of the correction: β = 0 ignores the token; β = 1 fully corrects what this key direction retrieves (for unit-norm keys; neighboring keys can still be disturbed, ch. 04's lesson).

5.2Compare additive and delta updates

5.3Where the delta rule came from

The update above is the delta rule, introduced in Widrow & Hoff's 1960 workcontextThe same rule trained ADALINE, one of the first adaptive machines, and it lives on as the LMS algorithm of signal processing. Echo cancellers and adaptive filters still use its descendants.. A 2021 result connected it to linear attention: "Linear Transformers Are Secretly Fast Weight Programmers" (Schlag, Irie & Schmidhuber), which showed linearized attention is formally equivalent to the fast weight programmerscontextSchmidhuber's 1991 construction: one network's outputs set the weights of another during the forward pass, "fast" weights change per input, while "slow" weights change during training. of the early 1990s. The authors proposed a "delta rule-like programming instruction" to address capacity problems. In this view, the slow network (ordinary weights, learned over months of training) emits keys, values, and write strengths that program a fast, temporary memory (S) during the forward pass. The prompt updates this temporary memorycontext Examples in a prompt can install key-value associations in S for later queries. The trained weights stay fixed. This description applies directly to this linear-attention family; applying it to full softmax attention is less straightforward..

5.4Each write is a learning step

The update reads the current prediction for a key, measures its error, and changes the state to reduce that error. This is one step of gradient descent on the squared error between the returned and desired values, with β acting as the learning rate. For the loss ½‖v − S^⊤k‖², the gradient in S is −k e^⊤. One step downhill is S ← S + βk e^⊤, exactly the delta write.

Chapter 01 described the slow training process that produces the model's long-lived weights. Those weights stay fixed during inference. The delta rule adds a faster learning loop inside the forward pass: each token supplies another training example for the temporary state S. Given a key, this small linear model predicts a value, measures the error against the token's target value, and takes one gradient step. It builds a compact sketch of the current context and is reset for the next one.

The trained network produces the keys, target values, and write strengths that guide this fast student. In that sense, it has learned how to make a smaller model learn at test time, without changing its own long-lived weights. Sun et al.'s 2024 paper, Learning to (Learn at Test Time): RNNs with Expressive Hidden States, develops the related idea of making the hidden state an explicitly trained mini-model. DeltaNet's delta-rule state is a simpler one-layer linear example of fast adaptation, not the same architecture.

5.5Making sequential updates run on GPUs

Rearranged, the delta update is a state transition:

Each state depends on the previous one through a different matrix every step, unlike ch. 04, where updates were plain sums that could be computed in any order. A naive implementation processes tokens one by one: thousands of tiny sequential operations, a poor fit for GPUscontextGPUs are throughput machines: tens of thousands of threads in flight, hiding memory latency behind sheer parallel work. A length-T chain of dependent updates strands that army: one tiny op finishes, the next can't start elsewhere. It's the same reason Mamba shipped with a custom scan kernel rather than plain PyTorch ops.. This blocked delta-rule models at scale until Yang, Wang, Zhang, Shen & Kim (NeurIPS 2024) derived a hardware-efficient algorithm, exploiting the structure of those (I − βkk^⊤) factors (generalized Householder matricescontextA Householder matrix (I − 2kk⊤) is a mirror: it reflects space across the plane perpendicular to k. The delta transition is the same object with the reflection dialed by β: at β=1 it erases the k direction entirely; at β=0 it's the identity. Numerical linear algebra has fast, stable tricks for products of these (they've powered QR factorization since 1958), and that sixty-year-old toolbox is what made DeltaNet trainable at scale.) to compute chunks of the sequence with dense matrix multiplication, carrying state only between chunks. They trained a 1.3B-parameter DeltaNet on 100B tokens that outperformed their Mamba and GLA baselines, showing that the update could work at that scale.

Contents · Glossary · Derivations follow the papers and blog credited inline.