From GPT-2 to Kimi K3
Kimi K3 is 22,580 GPT-2s by parameter count, but its real story is how modern models learned to compress, update and selectively retrieve memory.
The 60-second version
Kimi K3 is 22,580 times GPT-2's parameter count, but the architectural leap is a new division of labor between compressed recurrent memory, selective full attention, sparse experts and depth retrieval.
Key points
- GPT-2-style softmax attention uses a KV cache that grows with context.
- Linear attention compresses history into a fixed matrix, trading bounded memory for interference.
- DeltaNet corrects old associations before writing new ones; gating adds learned forgetting.
- KDA makes decay fine-grained, while periodic MLA preserves direct token retrieval.
- MoE and AttnRes add selective capacity across experts and network depth.
Verdict. The 22,580× number is a scale comparison, not an intelligence score. K3's real advance is deciding what to remember, erase and retrieve at each level of the model.
The headline22,580 times larger is not 22,580 times smarter
GPT-2's 124 million parameters fit into Kimi K3's 2.8 trillion total parameter count about 22,580 times. That arithmetic is correct. It is not a capability multiplier: K3 activates only part of its sparse expert network for each token, and seven years of progress changed the memory system, routing and training machinery as much as the raw scale.
Stage oneSoftmax attention remembers every token
A GPT-2-style decoder turns tokens into queries, keys and values. During generation, a KV cache saves earlier keys and values so the model does not recompute the entire prefix. The trade-off is physical: the cache grows with sequence length, and reading it can become a memory-bandwidth bottleneck.
| Softmax attention | Keeps token-level keys and values; expressive retrieval, but the KV cache grows with context. |
|---|---|
| Linear attention | Folds past key-value associations into a fixed-size matrix state; bounded memory, but associations can interfere. |
| Delta rule | Reads what a key currently retrieves, then writes only the correction needed to replace that association. |
| Gating | Learns when and how strongly old state should decay, preventing memory from accumulating forever. |
Stage twoLinear attention turns a list into a memory board
Linear attention changes the order of multiplication. By applying a feature map to queries and keys before they interact, it can summarize the growing history into a fixed matrix. Think of replacing a filing cabinet that gains one folder per token with a whiteboard of fixed dimensions. The whiteboard is cheaper to carry forward, but new notes eventually overwrite or blur old ones.
The efficiency gain and the memory problem are the same operation: many token-level associations are compressed into one fixed-size state.
Stage threeDeltaNet learns to erase before it writes
DeltaNet treats the state as fast weights. For each key, it first asks what the current memory would return. It subtracts that old value from the value it wants to store, scales the correction, and writes the difference back. Gated DeltaNet adds a learned decay control, combining targeted replacement with broader forgetting.
The difficult engineering step is training this recurrence efficiently. Chunk-wise formulations keep normal attention-like work inside a small block while carrying a recurrent state between blocks. Smaller chunks reduce arithmetic, but hardware often runs larger matrix multiplications faster, so minimum FLOPs and minimum wall-clock time are not the same target.
Stage fourKimi mixes bounded memory with selective retrieval
Kimi Delta Attention adds fine-grained, per-channel decay instead of one scalar forget gate. Kimi K3 then combines three KDA layers with one gated Multi-head Latent Attention layer in each four-layer macrocycle. KDA carries constant-size recurrent memory; periodic MLA can still retrieve directly from the token context when compression would lose detail.
| KDA | Constant-state recurrent memory with fine-grained decay. |
|---|---|
| Periodic MLA | Full softmax-style retrieval at selected layers to recover token-level detail. |
| Latent MoE | Routes each token through a small subset of experts, adding capacity without activating the whole 2.8T model. |
| AttnRes | Lets later blocks selectively retrieve earlier depth representations instead of relying only on an ever-growing residual sum. |
Bottom lineThe lineage is a sequence of memory policies
The useful lesson is not that transformers simply became enormous. Each step answers a specific memory question: what must remain token-addressable, what can be compressed, what should be overwritten, and where selective retrieval is worth the cost. Read K3 as a hybrid memory system with sparse compute, not as 22,580 GPT-2 models stacked together.