# From GPT-2 to Kimi K3 > Kimi K3 is 22,580 GPT-2s by parameter count, but its real story is how modern models learned to compress, update and selectively retrieve memory. _Source: ali (@waterloo_intern) architecture worklog, cross-checked against Moonshot AI's public Kimi K3 materials · 2026-07-29 · 9 min read · Verified against primary sources_ Canonical: https://iyu.app/e/gpt2-to-kimi-k3-architecture ## The 60-second version Kimi K3 is 22,580 times GPT-2's parameter count, but the architectural leap is a new division of labor between compressed recurrent memory, selective full attention, sparse experts and depth retrieval. **Key points** - GPT-2-style softmax attention uses a KV cache that grows with context. - Linear attention compresses history into a fixed matrix, trading bounded memory for interference. - DeltaNet corrects old associations before writing new ones; gating adds learned forgetting. - KDA makes decay fine-grained, while periodic MLA preserves direct token retrieval. - MoE and AttnRes add selective capacity across experts and network depth. **Verdict.** The 22,580× number is a scale comparison, not an intelligence score. K3's real advance is deciding what to remember, erase and retrieve at each level of the model. ## Full explainer > **i** This explainer distills ali's observed architecture worklog and cross-checks Kimi K3's released structure against Moonshot's public materials. Interpretations of the lineage are the author's; K3 performance figures remain vendor-reported unless independently reproduced. ### The headline — 22,580 times larger is not 22,580 times smarter GPT-2's 124 million parameters fit into Kimi K3's 2.8 trillion total parameter count about **22,580 times**. That arithmetic is correct. It is not a capability multiplier: K3 activates only part of its sparse expert network for each token, and seven years of progress changed the memory system, routing and training machinery as much as the raw scale. - **124M** — GPT-2 baseline parameters - **2.8T** — Kimi K3 total parameters - **22,580×** — Parameter-count ratio - **104B** — K3 parameters active per token, vendor specification ### Stage one — Softmax attention remembers every token A GPT-2-style decoder turns tokens into queries, keys and values. During generation, a KV cache saves earlier keys and values so the model does not recompute the entire prefix. The trade-off is physical: the cache grows with sequence length, and reading it can become a memory-bandwidth bottleneck. - **Softmax attention:** Keeps token-level keys and values; expressive retrieval, but the KV cache grows with context. - **Linear attention:** Folds past key-value associations into a fixed-size matrix state; bounded memory, but associations can interfere. - **Delta rule:** Reads what a key currently retrieves, then writes only the correction needed to replace that association. - **Gating:** Learns when and how strongly old state should decay, preventing memory from accumulating forever. ### Stage two — Linear attention turns a list into a memory board Linear attention changes the order of multiplication. By applying a feature map to queries and keys before they interact, it can summarize the growing history into a fixed matrix. Think of replacing a filing cabinet that gains one folder per token with a whiteboard of fixed dimensions. The whiteboard is cheaper to carry forward, but new notes eventually overwrite or blur old ones. > The efficiency gain and the memory problem are the same operation: many token-level associations are compressed into one fixed-size state. ### Stage three — DeltaNet learns to erase before it writes DeltaNet treats the state as fast weights. For each key, it first asks what the current memory would return. It subtracts that old value from the value it wants to store, scales the correction, and writes the difference back. Gated DeltaNet adds a learned decay control, combining targeted replacement with broader forgetting. The difficult engineering step is training this recurrence efficiently. Chunk-wise formulations keep normal attention-like work inside a small block while carrying a recurrent state between blocks. Smaller chunks reduce arithmetic, but hardware often runs larger matrix multiplications faster, so minimum FLOPs and minimum wall-clock time are not the same target. ### Stage four — Kimi mixes bounded memory with selective retrieval Kimi Delta Attention adds fine-grained, per-channel decay instead of one scalar forget gate. Kimi K3 then combines three KDA layers with one gated Multi-head Latent Attention layer in each four-layer macrocycle. KDA carries constant-size recurrent memory; periodic MLA can still retrieve directly from the token context when compression would lose detail. - **KDA:** Constant-state recurrent memory with fine-grained decay. - **Periodic MLA:** Full softmax-style retrieval at selected layers to recover token-level detail. - **Latent MoE:** Routes each token through a small subset of experts, adding capacity without activating the whole 2.8T model. - **AttnRes:** Lets later blocks selectively retrieve earlier depth representations instead of relying only on an ever-growing residual sum. > **⚑ Caveat:** Moonshot reports advantages such as up to 6× decode throughput for Kimi Linear and publishes K3 architecture and benchmark figures. These are first-party measurements; hardware, kernels, sequence length and comparison settings determine whether users see the same result. ### Bottom line — The lineage is a sequence of memory policies The useful lesson is not that transformers simply became enormous. Each step answers a specific memory question: what must remain token-addressable, what can be compressed, what should be overwritten, and where selective retrieval is worth the cost. Read K3 as a hybrid memory system with sparse compute, not as 22,580 GPT-2 models stacked together. ## Primary sources - [ali (@waterloo_intern) — 22580: From GPT2 to Kimi3, Explained](https://x.com/waterloo_intern/status/2081762065392541951) - [Moonshot AI — Kimi K3 official repository](https://github.com/MoonshotAI/Kimi-K3) - [Moonshot AI — Kimi K3 model card](https://huggingface.co/moonshotai/Kimi-K3) --- _Published by iyu (https://iyu.app) — the day's AI news, checked against primary sources and rewritten in plain language. Free to quote with attribution and a link to the canonical URL._