learning machines

02 / attention and memory

How much does a model need to keep?

Some layers keep a growing record of earlier tokens. Others fold each new token into a compact state.

In lesson one, the last piece listened to the card of every earlier piece. That only works if those cards are still there.

The·cat·sat·on·the·mat.
Seven pieces so far. To produce the next one, the last piece listens to the six before it, so six cards have to be kept. One more piece, one more card.

01 / a growing context

More to read. More to keep?

During cached decoding, full-attention layers retain keys and values for earlier tokens. Recurrent layers update a bounded state.

In lesson one’s terms, a key is how well a card matches a listener, and a value is what the card offers. Keeping both for every piece is keeping every card on the table. A recurrent layer folds each piece into one running summary and puts the card away.

1,024 tokens262,144 tokens
Linear-scale growth chartIllustrative context-state growthFull attention grows fastest; hybrid grows with a smaller slope; recurrent state remains flat. Abstract units, not measured memory.1K262Killustrative units
Full attention
16× baseline
Qwen hybrid
4.75× baseline
All recurrent
1× baseline

Growth illustration, not GPU memory. Each layer starts at one abstract unit at 1,024 tokens. Attention units grow with context; recurrent units stay fixed. Actual layer sizes differ.

Fixed recurrent state does not mean constant total memory, constant processing cost or perfect recall. Full attention does not guarantee exact retrieval either.

02 / inside the stack

64 layers. Two arrangements.

Qwen3.8-27B repeats three GatedDeltaNet (GDN) layers, then one full-attention layer. A community experiment replaces those attention layers with GDN.

Architecture to inspect

48 original GDN layers (G), 16 full-attention layers (A). Pattern: G G G A, repeated 16 times.

A = full attention · G = original GDN · C = converted to GDN · D = original GDN, later adapted. Layers numbered 1–64.

This switch compares architectures. The conversion required distillation and further training; it is not a free, quality-preserving toggle. The 48 original GDN layers were also adapted in the later LoRA stage.

03 / keeping details

Every card, or one summary?

Read a short story one piece at a time, with the pieces from lesson one. The record keeps every card. The summary has two slots, who and where, and each new piece writes over the slot it fills.

0 / 6 updates

Keep every card

  1. No updates yet

Fold into one summary

who
—
where
—

ask both

Where did the cat sit first?

The record says
—
The summary says
—

Start with an empty table and an empty summary.

A summary has limited room, like the six numbers that held eight notes in lesson one. Fold enough pieces into it and something gets written over.

Deterministic teaching illustration. This summary is deliberately lossy. It is not GDN, actual model output or a benchmark.

Sources, assumptions & limits

Credit to the Qwen team and summerAIO. Architecture comes from the official Qwen config and community config, rather than the community README's hybrid label. The community config makes all 64 layers linear, converting zero-based layers 3, 7, …, 63.

The author's training report records per-layer distillation, then rank-32 LoRA with native_lora: true. Configured context is 262,144 tokens. Needle results and needle lengths are empty; the training/cache plan goes up to 16,384. This is not evidence of demonstrated 262K retrieval. We ran no models and report no quality scores.

The growth comparison assigns equal abstract units to unlike layer states to show shape only: full = 64n/1024; hybrid = 16n/1024 + 48; recurrent = 64. It excludes weights, activations, buffers, vision and runtime overhead. The recurrent state exists even before any tokens arrive.

Verified 1 November 2026. Official revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0; community revision 46319430c16af66860b20f14986ad1032e991796.

Source: attention and memory experiment · exact mechanics commit · integrity manifest · mechanics license.

The source is public. The browser uses the pinned copy served with this page; the experiment also runs independently of this site. Original experiment code is MIT licensed; the source repository preserves the upstream snapshots under Apache-2.0 with attribution.