Chapter 10 · Kimi K3's memory systems

Kimi K3, assembled

Kimi K3 combines the mechanisms from the previous chapters. The components address different limitscontext KDA uses fixed-size but lossy state; periodic MLA layers preserve token-level lookup. MoE stores far more parameters than it activates for one token. AttnRes lets deep layers select earlier outputs. Together they balance retrieval accuracy, arithmetic work, memory bandwidth, capacity, and communication across devices. This chapter puts the published specifications in one place and separates reported measurements from architectural facts.

10.1The spec sheet, with provenance

Each row names its source. The model card and technical blog are Moonshot publications and describe the model's design. Their large-scale efficiency and quality claims have not all been independently reproducedcontext Checking a model of this size can mean running released weights, reproducing smaller experiments, or measuring serving performance. Retraining all 2.8T parameters independently is much more costly..

All values re-checked against the linked sources on 2026-08-15.

10.2One macro-cycle, all six systems

The six systems serve different purposes (adapted from the source article):

1. KDA: cheap, lossy, managed recurrent sequence memory;
2. Gated MLA: expensive, compressed, token-addressable retrieval, rationed 1-in-4;
3. Stable LatentMoE: conditional parameter capacity, 16-of-896;
4. Block AttnRes: selective depth retrieval;
5. Quantization-aware training: a low-precision serving path built in, not bolted on;
6. Expert-parallel infrastructure: the routing, balancing, and communication engineering that makes 1–5 executable.

10.3Precision changes memory use

K3 trains quantization-aware from the SFT stagecontextSupervised fine-tuning, the post-pretraining stage where the model learns from curated instruction/response examples. Starting QAT here is a pragmatic compromise: pretraining still runs in higher precision (stability). The later assistant-training stage exposes the model to rounding used by the serving format.: the model learns while experiencing the rounding of its serving format, MXFP4context"MX" is microscaling. Small blocks of numbers share a scale factor, giving their 4-bit floats a wider useful range than each could represent alone. for weights and MXFP8 for activations. Bytes per number also determine memory use. A 2.8T-parameter model at 4 bits per weight needs roughly a quarter of the raw weight storage of fp16 (before block-scale metadata and any tensors kept at higher precision). Details are vendor-reported in the blog and model card.

10.4Questions to ask about performance claims

The source article closes with a useful checklist. For claims such as "6× faster" or "beats full attention," check:

• At what context length, and prefill or decodecontextThe two phases stress different resources (ch. 03), so a genuine 6× decode win can coexist with unchanged or slower prefill. The phase matters when comparing results.?
• What batch size, hardware, and precision?
• Against what baseline implementation: tuned or strawman?
• Is the number measured end-to-end or extrapolated from one subsystem? (Kimi Linear's 6×-at-1M is a controlled result for a 48B hybrid; it is not automatically K3's serving speed.)
• Is it independently reproduced, or vendor-reported? (At 2.8T, assume the latter and read accordingly.)

Checklist items consolidated from the caveats sections of the source article.

10.5Compare the systems with a memory hierarchy

The six systems resemble parts of a computer's memory hierarchy. The analogy has an important limitcontextThe deepest mismatch: a CPU hierarchy caches the same bytes at different distances, while the LLM systems hold different representations: a compressed sequence state, token-addressed details, and learned expert parameters. They are not copies of one underlying set of bytes. The table compares their roles:

Classical hierarchies often use fixed management policies. Several K3 choices, including KDA's decay and MoE's routing, depend on the current input and were learned during training. The table is an analogy for those choices, not a claim that K3 literally pages expert weights like a CPU pages memory.

Comparing LLM memory with a computer's hierarchy is not new: PagedAttention (ch. 03) manages the KV cache the way an operating system pages virtual memory, and MemGPT applies an OS memory hierarchy to context management. This table maps that on to K3's subsystems and notes the limits of that comparison.

10.6The whole road, one table

Table structure from the source article's "tutorial in miniature," extended with chapter mapping.

10.7Why this design may change

Each system reduces a different cost. KDA and MLA limit how much token-addressable context the model keeps. MoE limits active parameters per token. MXFP4 reduces weight storage, Block AttnRes limits access across depth, and expert parallelism has to manage communication between devices. These are distinct resource constraints.

K3 reflects the costs of current hardware: memory bandwidth, arithmetic relative to data movement, and transfers between accelerators. If exact attention becomes cheaper, the case for KDA and MLA could change. More abundant memory and interconnect bandwidth could change the value of aggressive quantization and expert routing. Parameter count and depth-related training issues may remain. When you read a new model card, check which costs its design reduces and which costs it accepts.

Contents · Glossary · Spec values verified against the model card and blog on 2026-08-15; vendor-reported items labeled throughout.