01 / attention and memory
How much does a model need to keep?
Some layers keep a growing record of earlier tokens. Others fold each new token into a compact state.
01 / a growing context
More to read. More to keep?
During cached decoding, full-attention layers retain keys and values for earlier tokens. Recurrent layers update a bounded state.
Height uses a logarithmic scale; compare exact values below. Abstract state units, not measured GPU memory.
Linear-scale growth chart
- Full attention
- 16× baseline
- Qwen hybrid
- 4.75× baseline
- All recurrent
- 1× baseline
Growth illustration, not GPU memory. Each layer starts at one abstract unit at 1,024 tokens. Attention units grow with context; recurrent units stay fixed. Actual layer sizes differ.
Fixed recurrent state does not mean constant total memory, constant processing cost or perfect recall. Full attention does not guarantee exact retrieval either.
02 / inside the stack
64 layers. Two arrangements.
Qwen3.8-27B repeats three GatedDeltaNet (GDN) layers, then one full-attention layer. A community experiment replaces those attention layers with GDN.
48 original GDN layers (G), 16 full-attention layers (A). Pattern: G G G A, repeated 16 times.
A = full attention · G = original GDN · C = converted to GDN · D = original GDN, later adapted. Layers numbered 1–64.
This switch compares architectures. The conversion required distillation and further training; it is not a free, quality-preserving toggle. The 48 original GDN layers were also adapted in the later LoRA stage.
03 / keeping details
A record, or the latest state?
Step through six updates. A log keeps each change. A two-slot dictionary replaces the previous value for each object.
Keep the record
- No updates yet
Update the state
- door
- —
- cup
- —
Start with an empty log and two empty state slots.
Deterministic teaching illustration. This dictionary is deliberately lossy. It is not GDN, actual model output or a benchmark.
Sources, assumptions & limits
Credit to the Qwen team and summerAIO. Architecture comes from the official Qwen config and community config, rather than the community README's hybrid label. The community config makes all 64 layers linear, converting zero-based layers 3, 7, …, 63.
The author's training report records per-layer distillation, then rank-32 LoRA with native_lora: true. Configured context is 262,144 tokens. Needle results and needle lengths are empty; the training/cache plan goes up to 16,384. This is not evidence of demonstrated 262K retrieval. We ran no models and report no quality scores.
The growth comparison assigns equal abstract units to unlike layer states to show shape only: full = 64n/1024; hybrid = 16n/1024 + 48; recurrent = 64. It excludes weights, activations, buffers, vision and runtime overhead. The recurrent state exists even before any tokens arrive.
Verified 1 November 2026. Official revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0; community revision 46319430c16af66860b20f14986ad1032e991796.
Source: attention and memory experiment · exact mechanics commit · integrity manifest · mechanics license.
The source is public. The browser uses the pinned copy served with this page; the experiment also runs independently of this site. Original experiment code is MIT licensed; the source repository preserves the upstream snapshots under Apache-2.0 with attribution.