๐Ÿ† ICML 2026 ยท Seoul, South Korea

Why Limit the Residual Stream to Layers and Not Tokens?

Persistent Memory for Continuous Latent Reasoning. We find that latent-reasoning LLMs forget what they just figured out โ€” and fix it with three tiny learned gates and a memory stream that costs just 1.41% extra parameters.

Mujtaba FarhanAlgoverse AI Research
Maheep ChaudharyIndependent
TL;DR

The paper in three bites

๐Ÿ•ณ๏ธ

The Concept Bottleneck

In latent reasoning (CoCoNuT), each reasoning pass overwrites the previous hidden state. Facts computed early are lost by the time the answer is generated โ€” 87% of pass-1 information is gone by pass 6.

๐Ÿšช

Three Learned Gates

AGCLR adds a persistent concept stream โ€” a residual memory across passes โ€” controlled by read, forget, and write gates. Like an LSTM, but gating memory across reasoning passes, not timesteps.

๐Ÿ“ˆ

Compounding Gains

+2.6% on GSM8K, +3.6% EM on HotpotQA, +4.0% on ProsQA over vanilla CoCoNuT โ€” and the gap grows with reasoning depth, exactly where the bottleneck bites hardest.

The Problem

Watch a model forget ๐Ÿง ๐Ÿ’จ

Latent reasoning feeds the model's hidden state back as the next input โ€” powerful, but each pass overwrites the last. Drag the slider to simulate reasoning passes and watch what happens to the facts. Values are real cosine similarities from the paper (Fig. 3).

Pass 1 / 6

Vanilla CoCoNuT no memory

Similarity to pass-1 state: 1.000
๐Ÿ“ Manor Township โ†’ Pennsylvania ๐Ÿ‘ค Pennsylvania โ†’ William Penn ๐Ÿ—“๏ธ founded in 1655 โ“ question asks "who founded"
Information retained

AGCLR gated memory

Similarity to pass-1 state: 1.000
๐Ÿ“ Manor Township โ†’ Pennsylvania ๐Ÿ‘ค Pennsylvania โ†’ William Penn ๐Ÿ—“๏ธ founded in 1655 โ“ question asks "who founded"
Information retained
๐Ÿ”’ Concept stream holds facts written in early passes โ€” read back when needed
Both models start with everything they needโ€ฆ
The Method

AGCLR: a memory stream with three doors ๐Ÿšช๐Ÿšช๐Ÿšช

At every latent token position, a persistent vector ct (the concept stream) rides alongside the hidden state ht. Three sigmoid gates decide what gets remembered, retrieved, and pruned. Click a gate to learn what it does.

Concept stream cโ‚โ‚œโ‚‹โ‚โ‚Ž โ†’ cโ‚œ ๐Ÿ”’ Hidden state hโ‚œ โ†’ hโ€ฒโ‚œ (fed back as next input) ๐Ÿง  READ rโ‚œ FORGET fโ‚œ WRITE wโ‚œ

๐Ÿ‘† Click a gate

Each gate is a learned sigmoid over the layer-normed hidden state โ€” e.g. rt = ฯƒ(Wr ฤฅt). Together they cost only 1.41% extra parameters on GPT-2.

The full update, four lines ๐Ÿงฎ

1ฤฅโ‚œ = LayerNorm(hโ‚œ)normalize the current thought
2rโ‚œ, fโ‚œ, wโ‚œ = ฯƒ(Wยทฤฅโ‚œ)three gates, each โˆˆ [0,1]แตˆ
3hโ€ฒโ‚œ = (1โˆ’fโ‚œ)โŠ™hโ‚œ + rโ‚œโŠ™cโ‚œโ‚‹โ‚prune noise, pull in memory
4cโ‚œ = LayerNorm(cโ‚œโ‚‹โ‚ + wโ‚œโŠ™hโ€ฒโ‚œ)commit to the stream
Playground

Drive the gates yourself ๐ŸŽ›๏ธ

Slide the three gates and watch how the next hidden state hโ€ฒt gets mixed, and how fast the concept stream fills up. The presets are the paper's actual learned initializations (Appendix A.3).

Next hidden state hโ€ฒโ‚œ is made of:
current thought (1โˆ’f)ยทhretrieved memory rยทc
Concept stream fill over 6 passes:

Results

Numbers that compound ๐Ÿ“Š

All methods use the same GPT-2 (117M) base. Pick a dataset โ€” bars animate to the paper's Table 1. Hover a bar for details.

๐Ÿง  Memory retention across passes

Cosine similarity of each pass's hidden state to pass 1 (100 validation samples, epoch 15). Vanilla decays monotonically to 0.126 (โ‰ˆ87% information loss); AGCLR stabilizes at 0.216 โ€” retaining 71% more information at final generation.

1.0 0.5 0.0 P1P2P3P4P5P6 0.126 0.216 Vanilla CoCoNuT AGCLR (ours)

๐Ÿ“‰ Depth helps us, hurts them

On ProsQA, vanilla CoCoNuT peaks at 95% at curriculum stage 5, then degrades to 92% at stage 6 (fully latent). AGCLR keeps climbing to 96% โ€” the concept bottleneck, resolved.

Vanilla @ stage 5
95%
Vanilla @ stage 6
92% ๐Ÿ˜ฐ
AGCLR @ stage 6
96% ๐ŸŽ‰

The pattern is consistent: AGCLR and vanilla perform similarly at early stages, then the gap compounds as more reasoning goes latent โ€” exactly where a persistent memory should matter.

Interpretability

What actually gets written? ๐Ÿ”

Probing the concept stream against the vocabulary (logit-lens style) shows it stores answer-relevant entities at 0.6โ€“0.8 cosine similarity. Three cases where AGCLR wins and vanilla hallucinates โ€” click a card to flip it.

โ“ Who founded Manor Township, Pennsylvania?
Penn ยท806William ยท681Pennsylvania ยท614
tap to see the answers โ†ป
โœ… AGCLR: William Penn
โŒ Vanilla: Henry David Thoreau

The stream keeps the Pennsylvaniaโ†’William Penn binding; vanilla loses it across passes and hallucinates a plausible historical figure.

โ“ When did Oakland Assembly close?
War ยท760World ยท684Assembly ยท713
tap to see the answers โ†ป
โœ… AGCLR: World War I
โŒ Vanilla: World War II

Preserved temporal context stops the drift toward the statistically more common "World War II".

โ“ What country is WCDL radio station in?
Federal ยท507country ยท495Pennsylvania ยท479
tap to see the answers โ†ป
โœ… AGCLR: United States
โŒ Vanilla: Pennsylvania

Both the base entity and abstraction markers survive, enabling the second hop from state โ†’ country. Vanilla stops at hop one.

Ablations

Memory, not parameters ๐Ÿงช

Is it just extra weights? No. Freezing the write gate after pass 2 costs only โˆ’0.8% EM โ€” the stream is persistent storage, not a scratchpad: write early, read often. And every gate earns its keep (HotpotQA EM).

Write-gate ablation

Vanilla CoCoNuT
10.4
AGCLR w/o write gate
8.8
Write frozen after pass 2
13.2
Full AGCLR
14.0

Remove one gate at a time

Full AGCLR
14.0
w/o read gate
9.4
w/o write gate
8.8
w/o forget gate
8.4

Biggest drop from removing forget (โˆ’5.6): without pruning, the stream gets polluted with noise. All three gates are indispensable.

โš–๏ธ Honest limitations

Single-seed runs on GPT-2 124M across three benchmarks; scaling to 1B+ models, multi-seed variance estimates, and broader benchmarks (MuSiQue, 2WikiMultihopQA, StrategyQA) are future work. The architecture itself is parameter-agnostic. Total training: ~8 hours on a single A10 GPU.

Citation

Cite this work ๐Ÿ“š

@inproceedings{farhan2026agclr,
  title     = {Why Limit the Residual Stream to Layers and Not Tokens?
               Persistent Memory for Continuous Latent Reasoning},
  author    = {Farhan, Mujtaba and Chaudhary, Maheep},
  booktitle = {Proceedings of the 43rd International Conference
               on Machine Learning (ICML)},
  year      = {2026},
  publisher = {PMLR}
}