From GPT2 to Kimi3, Explained

✍ ali🕐 2026-07-28📦 27.8 KB 🟢 已读 𝕏 文章列表

文章回顾了从 GPT-2 (2019) 到 KimiK3 (2026) 的大语言模型架构演进，重点对比了参数规模从 1.24 亿到 2.8 万亿的巨大跨越。深入解析了 GPT-2 的 Transformer 解码器结构、KV 缓存机制，并探讨了线性注意力机制在降低计算复杂度方面的原理与权衡。

# 22580: From GPT2 to Kimi3, Explained **作者**: ali **日期**: 2026-07-27T15:22:08.000Z **来源**: [https://x.com/waterloo_intern/status/2081762065392541951](https://x.com/waterloo_intern/status/2081762065392541951) --- ![Image](/doc/3490/image/img_a58c224ca437.jpg) ![Image](/doc/3490/image/img_a74034570f39.jpg) Twenty-two thousand five hundred and eighty. That’s how many GPT-2 (2019) models fit inside KimiK3 (2026). We scaled up by a factor of 22,580 in seven years. But is it just... scale? In this worklog, I’ll walk through how we got here and how much, or how little, has actually changed since then. We’ll trace the major architectural developments leading to KimiK3. ![Image](/doc/3490/image/img_72a9257021d5.jpg) # GPT-2 GPT-2 is a decoder-only architecture: ``` tok_emb = self.transformer.wte(idx) # token embeddings of shape (b, t, n_embd) pos_emb = self.transformer.wpe(pos) # position embeddings of shape (t, n_embd) x = self.transformer.drop(tok_emb + pos_emb) for block in self.transformer.h: x = block(x) x = self.transformer.ln_f(x) logits = self.lm_head(x) return logits ``` The input receives token and positional embeddings: ![Image](/doc/3490/image/img_3e9f496b82d1.jpg) Each transformer block, zoomed in, looks like this: ``` class Block(nn.Module): def __init__(self, config): super().__init__() self.ln_1 = LayerNorm(config.n_embd, bias=config.bias) self.attn = CausalSelfAttention(config) self.ln_2 = LayerNorm(config.n_embd, bias=config.bias) self.mlp = MLP(config) def forward(self, x): x = x + self.attn(self.ln_1(x)) x = x + self.mlp(self.ln_2(x)) return x ``` ![Image](/doc/3490/image/img_c7fe41cb54aa.jpg) The attention process: ``` B, T, C = x.size() # batch size, sequence length, embedding dimensionality (n_embd) # calculate query, key, values for all heads in batch and move head forward to be the batch dim q, k, v = self.c_attn(x).split(self.n_embd, dim=2) k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) # manual implementation of attention att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1))) att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf')) att = F.softmax(att, dim=-1) att = self.attn_dropout(att) y = att @ v # (B, nh, T, T) x (B, nh, T, hs) -> (B, nh, T, hs) y = y.transpose(1, 2).contiguous().view(B, T, C) # re-assemble all head outputs side by side # output projection y = self.resid_dropout(self.c_proj(y)) return y ``` Once the final hidden-state matrix is produced, the language-model head maps it into vocabulary logits. During autoregressive decoding, only the logits at the final position are needed to select the next token. > This is an inefficiency of decoder-only generation: the model computes representations for every input position, but each decode step consumes only the final position’s logits. Without caching, much of that work would be repeated for the next token. ![Image](/doc/3490/image/img_f914047e8f8b.jpg) The KV cache comes from a straightforward observation: after appending the generated token to the input, the model would otherwise recompute projections for all previous tokens. Storing their key and value vectors avoids that redundant work. That storage is the KV cache. It retains vectors for the previous N-1 tokens and can become large enough to create a memory-bandwidth bottleneck. Overall, with about 50k possible tokens, 12 blocks, 12 heads, and an embedding dimension of 768, our baseline model is about 124M parameters. ``` vocab_size: int = 50304 # GPT-2 vocab_size of 50257, padded up to nearest multiple of 64 for efficiency n_layer: int = 12 n_head: int = 12 n_embd: int = 768 ``` At 2.8 trillion parameters, one KimiK3 model contains roughly as many parameters as 22,580 GPT-2 models. # Linear Attention Softmax attention applies its nonlinearity after the q·k product, coupling every query to every key. Linear attention instead applies a feature map, such as ELU+1, to q and k separately. This makes the product re-associable, so the growing set of K and V vectors can be folded into a fixed D×D state. The paper’s O(N²) framing threw me off. It's not true that "the cost per time-step for transformers scales with the square of the current sequence length". That's what Flash Attention fixes... then I saw that it was released in 2020. At the time, training commonly materialized the full N×N attention matrix, FlashAttention did not exist, and reference autoregressive implementations often recomputed the token history without a KV cache. ``` def forward(self, x, mask=None, past_kv=None): # x is b,t,d b,t,d=x.shape d_head=d//self.num_heads h=self.num_heads qkv=self.qkv_proj(x) q=qkv[:, :, :d].view(b,t,h,d_head).transpose(1,2) k=qkv[:, :, d:2*d].view(b,t,h,d_head).transpose(1,2) v=qkv[:, :, 2*d:].view(b,t,h,d_head).transpose(1,2) # at prefill, q,k,v have shapes b,h,t,d # at decode, shape is b, h, 1, d # so i cat at the t dimension, dim(2) if past_kv is not None: k_past=past_kv[0] v_past=past_kv[1] k=torch.cat((k_past, k), dim=2) v=torch.cat((v_past, v), dim=2) scores=(q@k.transpose(-1,-2))/math.sqrt(d_head) if past_kv is None: #we're in prefill and need to mask causal_mask=torch.ones(t,t,dtype=bool, device=q.device) causal_mask=torch.triu(causal_mask, diagonal=1) scores=scores.masked_fill(causal_mask, float('-inf')) if mask is not None: scores=scores.masked_fill(~mask, float('-inf')) #get attn (bhtt x bhtd) attn=scores.softmax(-1)#bhtt o=attn@v #bhtd o=o.transpose(1,2).contiguous().view(b,t,d) #b,t,d # use x to get qkv o_proj=self.o_proj(o) past_kv=(k, v) return o_proj, past_kv ``` The same process is easier to see visually. Each decode step performs two ND reads and two 1D writes to HBM, while the KV cache grows linearly, in O(N), with the sequence length. ![Image](/doc/3490/image/img_071552a485c7.jpg) Notice the excessive reads and writes, which this paper replaces with: ``` def forward(self, x, mask=None, cache=None): # x is b,t,d b,t,d=x.shape d_head=d//self.num_heads h=self.num_heads qkv=self.qkv_proj(x) q=qkv[:, :, :d].view(b,t,h,d_head).transpose(1,2) k=qkv[:, :, d:2*d].view(b,t,h,d_head).transpose(1,2) v=qkv[:, :, 2*d:].view(b,t,h,d_head).transpose(1,2) k=F.elu(k)+1 k=k.transpose(-1,-2) q=F.elu(q)+1 S,z=cache if cache is not None else (0.0, 0.0) S=S+k@v z=z+k o=q@S #bhtd denom=q@z o_scaled=o/denom o_scaled=o_scaled.transpose(1,2).contiguous().view(b,t,d) o_proj=self.o_proj(o_scaled) cache=(S,z) return o_proj, cache ``` There is a trade-off. Here, we replace the exponential used by softmax with ELU+1 applied separately to q and k before they interact. Both approaches normalize the resulting scores, but the feature map used by linear attention is a less expressive approximation of the softmax kernel. That approximation can reduce fidelity, although the practical accuracy loss depends on the architecture and workload. Notice that we still divide by the sum of qk, which is omitted from the diagram for simplicity. At a high level, attention consists of three steps: 1. Make the qk scores non-negative. Linear attention uses ELU+1, while softmax uses exponentiation. 2. Divide by the sum. 3. Compute the weighted average of the values. This preserves the basic attention contract, but uses a less expressive feature map to make the QK scores non-negative. # DeltaNet (Fast Weight Programmers) A finite cache must overwrite or combine with information already stored. The state from token i-1 does not receive its own slot; it is added to the same D by D matrix. New queries can therefore no longer retrieve a perfectly isolated representation of each earlier token. That addition is also the source of the efficiency gain. Updating the cache additively rather than by concatenation prevents it from growing in O(N), but the same operation causes information to interfere. DeltaNet addresses this loss of recoverability. ![Image](/doc/3490/image/img_1a158d3689ae.jpg) Eloquently put by Schlag’s paper (Fast Weight Programmers): “when the sequence length exceeds storage capacity, the model may end up in an overcapacity regime. To properly operate under such a regime, the model should learn to dynamically interact with the memory contents and selectively decide which key-value associations to keep and which ones to delete. The purely additive instruction may be inappropriate for this purpose…. endlessly adding new associations to a memory of finite size, as in Eq. 17, inevitably will reach a limit.“ The regime that makes linear attention attractive, where N is much larger than D, also exposes its main limitation. Once the state exceeds its effective capacity, associations begin to interfere because the update is additive and nothing leaves the cache. ``` def forward(self, x, mask=None, cache=None): # x is b,t,d b,t,d=x.shape d_head=d//self.num_heads h=self.num_heads qkv=self.qkv_proj(x) q=qkv[:, :, :d].view(b,t,h,d_head).transpose(1,2) k=qkv[:, :, d:2*d].view(b,t,h,d_head).transpose(1,2) v=qkv[:, :, 2*d:].view(b,t,h,d_head).transpose(1,2) q = F.normalize(F.silu(q), dim=-1) k = F.normalize(F.silu(k), dim=-1) beta = torch.sigmoid(self.w_beta(x)).view(b, 1, t, 1) # new: per-token write strength S = cache if cache is not None else 0.0 v_old = k @ S # read the board at this key u = beta * (v - v_old) # the delta: only what's actually new S = S + k.transpose(-1, -2) @ u # same outer-product write as before o = q @ S # read, no denominator o = o.transpose(1, 2).contiguous().view(b, t, d) return self.o_proj(o), S ``` A visual example makes this easier to follow. ![Image](/doc/3490/image/img_e30acd7f4e2f.jpg) Take a single association written as S = k.T @ v. If read back with the same key and you get k @ (k.T @ v), which is (k @ k.T) v, which is the squared norm of k times v. So read returns scaled by key's squared norm, and if normalize k to unit length, or just divide result by norm, get v back exactly. Q is also a learned pointer. Wq and Wk read the same residual stream, and the query for a fact points at the key direction that fact was written into. The update first asks what information the current key retrieves from the cache. It subtracts that existing information from the value we want to store, multiplies the key by the difference, and adds the result back. Old information is removed and new information is written in its place. # DeltaNet (Parallelizing Linear Transformers with Delta Rule) This is the most difficult section of the post. It took me about seven hours to develop a working understanding of it, so I will build the explanation from the implementation. In short, DeltaNet implements a first-order linear recurrence with generalized Householder transition matrices, enabling chunk-wise parallel forward passes for hardware-efficient linear-time training. It splits the inputs and outputs into several chunks of size C, and computes outputs for each chunk based on the final state of the previous chunk and the query key value blocks of the current chunk. The practical problem is prefill. A direct implementation of the Delta rule over a sequence of T tokens would look like this: ``` S = torch.zeros(b, h, dh, dh) if cache is None else cache outs = [] for i in range(t): k_i = k[:, :, i:i+1] v_i = v[:, :, i:i+1] b_i = beta[:, :, i:i+1] v_old = k_i @ S u_i = b_i * (v_i - v_old) S = S + k_i.transpose(-1, -2) @ u_i # write outs.append(q[:, :, i:i+1] @ S) o = torch.cat(outs, dim=2) ``` Unlike standard attention, this formulation requires a correction at every key vector, so the path to a parallel matrix multiplication is not immediately obvious. Even without the Delta rule, a direct linear-attention prefill remains sequential: ``` S = torch.zeros(b, h, dh, dh) if cache is None else cache outs = [] for i in range(t): q = q[:, :, i:i+1] k = k[:, :, i:i+1] v = v[:, :, i:i+1] S=S_old+k@v o=q@S #bhtd o=self.norm(o) o=o.transpose(1, 2).contiguous().view(b, t, d) out=self.o_proj(o) cache=S outs.append(out) o = torch.cat(outs, dim=2) ``` A chunked formulation provides a more efficient approach. The mechanics are easier to understand through an example: ![Image](/doc/3490/image/img_e3828d1b0d1b.jpg) Setting C=N recovers standard O(N^2) attention, while C=1 gives regular linear attention. Intermediate values we interpolate between trade additional within-chunk work for better hardware utilization. In practice, C is often 64 or 128 because tensor-core instructions operate efficiently at that granularity; UMMA is one example. The intermediate tiles are folded into S as part of the state update: ![Image](/doc/3490/image/img_e134a81e2f6f.jpg) ``` S = torch.zeros(b, h, dh, dh) if cache is None else cache outs = [] for i in range(t//C): q_c = q[:, :, i*C:(i+1)*C] k_c = k[:, :, i*C:(i+1)*C] v_c = v[:, :, i*C:(i+1)*C] o_prev=q_c@S #this is everything up to this block attn=(q_c@k_c.transpose(-1,-2)).tril() #masked attention o_curr=attn@v_c o=o_prev+o_curr S_new=k_c.transpose(-1,-2)@v_c #recurrent attention S=S+S_new outs.append(o) o = torch.cat(outs, dim=2) ``` Within a block, we do q(kᵀv). This is score first, the normal attention order with masking. Across blocks, we follow (kᵀv)q, so we’re doing recurrent order, state first. Attention grows in O(N²) and this does not. Inside a block I do real attention (the masked QKᵀ times V), and across blocks I fold everything into the state and read it back with one matmul. So the cost splits in two. There's a fixed piece, 2Ld², which is the state work and doesn't care about C at all. And there's a growing piece, 2LCd, which is the score matrices sitting on the diagonal. Full attention is just the case where C equals L, and then that second term becomes 2L²d, quadratic. So the smaller I make C, the fewer FLOPs I do. C=1 is the cheapest option in pure FLOP terms, but not necessarily in wall-clock time. A GPU can complete more arithmetic faster when the work maps efficiently onto its matrix-multiply hardware. The next step is to extend the same approach to DeltaNet. ![Image](/doc/3490/image/img_6dbd8db35019.jpg) The underlying issue is simple: the chunking method used for purely additive attention does not directly apply to the delta updates: ``` v_old = k_i @ S u_i = b_i * (v_i - v_old) ``` We need every single state in order to compute the information that needs to be subtracted out. We can't parallelize it the same way without some mathematical re-parameterization. The authors therefore rewrite the delta updates from: ``` u=v_new-v_old S_t= S_(t-1)+K.T@u o=q@S_T ``` Here, a sequential loop computes one delta per iteration. The reparameterized form is: ``` S_t = S_{t-1}(I − β_t k_t k_tᵀ) + β_t v_t k_tᵀ o_t = S_t q_t ``` This formulation allows the chunked code to compute all C deltas at once: ``` def chunk_delta_rule_forward(Q, K, V, beta, C): # L: sequence length, d: head dimension L, d = Q.shape # chunking Q, K, V = map(lambda x: x.reshape(-1,C,d), [Q, K, V]) beta = beta.reshape(-1, C) K_beta = K * beta.unsqueeze(-1) V_beta = V * beta.unsqueeze(-1) # compute eq. 10 with vectorized forward substitution for fast inverse T = -(K_beta @ K.t()).tril(-1) for i in range(1, C): T[i, :i] = T[i, :i] + (T[i, :, None] * T[:, :i]).sum(-2) T += torch.eye(C) W = T @ K_beta U = T @ V_beta # chunkwise parallel. Eq. 8-9 S = torch.zeros(d, d) O = torch.empty_like(V) for i in range(L//C): q_i, k_i, w_i = Q[i], K[i], W[i] u_i = U[i] - w_i @ S # the corrections, all of one chunk o_inter = q_i @ S A_i = (q_i @ k_i.t()).tril() #qk.t o_intra = A_i @ u_i # attention @ v (with corrections, so u) S += k_i.t() @ u_i # update state with addition O[i] = o_intra + o_inter #update output with flash + recurrent return O.reshape(L, d) ``` This gets us to our first comparison point: MHA vs DeltaNet Transformers: ![Image](/doc/3490/image/img_78fccf48389d.jpg) # Gated Delta Net We now have a method for making precise changes to the cache. With each new fact (each new key vector), we can look at exactly the old information stored at that point and replace it with the new information we want to attend to. However, this mechanism can forget only an association for which it has a specific replacement. It cannot efficiently clear multiple associations during a context switch or decay memory generally to free capacity. If we were doing purely additive linear attention: Adding the ability to forget would be simple. We'd just need a parameter controlling the forgetful state: ``` S_old=cache S_new=k@v # cache=S_old+S_new cache=alpha * S_old + S_new ``` ![Image](/doc/3490/image/img_83f03e15a8af.jpg) This is the Mamba-2 contribution. We decay the previous cache, then add the new cache at full strength, preventing the state from growing without bound. Uniformly decaying all key-value associations at each time step by a dynamic ratio is a working approach, and it's what Mamba does. But it doesn't account for the varying importance of different key-value associations. That is, if the model needs to forget one specific association, all associations are forgotten equally. The Delta rule, in contrast, can update a single fact but has no way to make the rest of the facts decay. So the Gated Delta rule combines Mamba's gated update rule with the Delta rule. It adds a parameter, alpha, that switches to the pure Delta rule when set to one and clears the memory when set to zero. The challenge is implementing this with the same parallel-chunks method. The implementation uses the same DeltaNet reparameterization described in the previous section. The mathematics is nearly identical, with one addition: a data-dependent scalar between zero and one that controls the decay of the previous state. This combines effective key-value association learning with adaptive memory management. The corresponding code changes are shown below: ![Image](/doc/3490/image/img_5dc2e5718f17.jpg) The γʳ/γⁱ term accounts for cumulative decay. A token written at time step x and read at x+t has been multiplied by αₓαₓ₊₁αₓ₊₂…αₓ₊ₜ. This is the multiplicative analogue of a prefix-sum calculation. The resulting architecture looks like this: ![Image](/doc/3490/image/img_85579c5011b9.jpg) # KDA/Kimi Linear At this point, researchers began experimenting with hybrid models that combine multiple forms of attention within one architecture, like Gated DeltaNet withM Mamba. Kimi Linear drew attention for one central claim: under controlled comparisons, it outperformed full attention. The authors presented it as a drop-in architectural replacement with better quality and up to 6x higher decode throughput. Kimi Linear improves on Gated DeltaNet by introducing fine-grained gating. Instead of a single scalar decay, it learns a separate decay value for each channel. ![Image](/doc/3490/image/img_95b5987a3172.jpg) The KDA update rule remains similar, but the code now looks more like this: ![Image](/doc/3490/image/img_24fddfd80aa7.jpg) Here, alpha.reshape(nb, C, d) captures the paper’s most significant contribution: fine-grained control over memory decay. Placed beside the DeltaNet Transformer, the Kimi Linear architecture introduces three major changes: 1. It uses a hybrid system that interleaves Multi-head Latent Attention (MLA) layers. 2. It replaces the MLP with a Mixture-of-Experts (MoE) layer. 3. It adds capacity to DeltaNet through the alpha projection. ![Image](/doc/3490/image/img_3e1acc480ddf.jpg) The later sections cover MLA and MoE in more detail. For now, the important point is that this is not blind scaling. The additional capacity has a specific mathematical purpose: the per-channel scale gives the model finer control over memory decay. Scaling laws remain relevant, but capacity must be added in the right place and in a form the system can use. Each architecture in this progression adds capacity to address a concrete limitation in the preceding system. # Kimi K3 Ultimately, the KimiK3 language backbone looks similar to the Kimi Linear model above. It contains 23 four-layer macrocycles. In each macrocycle, three layers use Kimi Delta Attention and the fourth uses Multi-head Latent Attention. The first layer uses a dense feed-forward network; every remaining layer uses a latent Mixture-of-Experts. At first glance, the changes from Kimi Linear appear modest: - A substantial increase in scale - Blockwise AttnRes every 12 layers - MLA query LoRA and output gating - Latent-space MoE - SiTU activations - Gated MLA KDA supplies constant-state recurrent memory, while periodic MLA layers retain full softmax retrieval over the context. The following simplified visualization provides a useful reference for the changes discussed below. ![Image](/doc/3490/image/img_10630b6c18c7.jpg) We will begin with the more direct changes: Gated MLA, latent-space MoE, and SiTU activations. Gated MLA determines how much of each retrieved feature passes from MLA into the residual stream. It does this through element-wise multiplication with a gate projected from the input. In a conventional MoE, a learned router uses dot-product similarity to send each token to a subset of expert networks. KimiK3 has 898 experts in total. Two are shared and process every token; of the remaining 896, the router selects 16 for each token. KimiK3 also changes the expert activation. Instead of applying SiLU to the up projection, multiplying it element-wise by the gate, and then applying the down projection, it uses SiTU: ``` d = x.shape[-1] // 2 gate = x[..., :d].to(torch.float32) up = x[..., d:].to(torch.float32) situ_a = self.beta * torch.tanh(gate / self.beta) * torch.sigmoid(gate) if self.linear_beta is not None: up = self.linear_beta * torch.tanh(up / self.linear_beta) return (situ_a * up).to(x.dtype) ``` The model also down-projects inputs to the shared experts and up-projects their final sum: ![Image](/doc/3490/image/img_d301afeb8f94.jpg) This illustrates a recurring challenge in model inference. Without a fused kernel, the new activation is almost 3x slower than the original path. One offsetting optimization is that the experts operate in a compressed latent space, which makes their forward pass much faster and nearly halves the FLOPs. The remaining changes are MLA query LoRA, output gating, and blockwise Attention Residuals every 12 layers. AttnRes adds roughly 2% inference latency, but provides two important benefits: - Selective retrieval of earlier representations, which mitigates residual dilution and hidden-state growth - A 1.25x compute advantage AttnRes and MLA address the same underlying limitation from different directions. KDA layers operate with constant-size state and must inevitably discard information. MLA retrieves from the token context, while AttnRes retrieves from earlier depth-wise representations. # AttnRes Thanks to @chloey3k for help with this section. In each forward pass, the input passes through a stack of layers. Here, each layer consists of an attention block (KDA or MLA) and an MLP or MoE block. Normally, the input to each layer is the sum of the original embedding and every preceding layer's output, all weighted equally. Here, h_i is the input to layer i, h_1 is the embedding of the current token (the last token in the sequence so far), and f_i(h_i) is the output of layer i (an attention or MLP block). The problem is the lack of selective access. Different layer types receive the same aggregated state, even though they may benefit from different weightings. Because the recurrence is purely additive, later layers must also learn increasingly large outputs to influence the accumulated residual, which can destabilize training. Instead of treating all the layers equally, AttnRes multiplies each term of that sum by a specialized weight, which lets the model give more importance to whichever layers are most useful in context. Each weight alpha_i is computed from a query-key dot product. The query is learned for each layer, while the keys and values come from earlier residual-stream states. The scores are normalized to sum to one, then used to form a weighted combination of those states. ![Image](/doc/3490/image/img_0b34b3fb6b34.jpg) The model therefore does not have to condition only on its immediate predecessor. AttnRes gives each layer selective access to earlier layer outputs, allowing its learned query to retrieve the representations most useful for the current computation. The pseudocode below applies the same idea at block granularity. A block is the element-wise sum of the attention and MLP outputs accumulated across 12 decoder layers, stored as a single depth representation for later AttnRes mixing. Applying residual attention at every layer would add too much training and inference cost. Applying it only at fixed block boundaries captures most of the benefit at a lower cost. In KimiK3, each boundary occurs after 12 decoder layers. Across 23 four-layer macrocycles, this produces eight AttnRes blocks, which increases our inference speed. This is possibly the most important part of the block_attn_res function ``` V = torch.stack(blocks + [partial_block]) # [N+1, B, T, D] K = norm(V) logits = torch.einsum('d, n b t d -> n b t', proj.weight.squeeze(), K) h = torch.einsum('n b t, n b t d -> b t d', logits.softmax(0), V) return h ``` This completes the progression from GPT-2 to KimiK3. The central change is not scale alone. Each architectural step changes what the model stores, how it updates that state, or how it retrieves information that a fixed-size state cannot preserve. KimiK3 combines constant-state recurrent memory, periodic softmax retrieval, sparse expert capacity, and selective depth-wise residual access. The result is a system that spends additional capacity where it has a specific functional role. In essence, a fixed-capacity associative memory (fixed dimensions) needs an eviction policy, since a purely additive linear operation eventually adds interference once at capacity. To that end, learned selection, like gating, routing, or decay, is necessary, and attention is the most effective selective-read mechanism. ## 相关链接 - [ali](https://x.com/waterloo_intern) - [@waterloo_intern](https://x.com/waterloo_intern) - [1.1M](https://x.com/waterloo_intern/status/2081762065392541951/analytics) - [@chloey3k](https://x.com/@chloey3k) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [11:22 PM · Jul 27, 2026](https://x.com/waterloo_intern/status/2081762065392541951) - [1.1M Views](https://x.com/waterloo_intern/status/2081762065392541951/analytics) - [View quotes](https://x.com/waterloo_intern/status/2081762065392541951/quotes) --- *导出时间: 2026/7/28 12:35:59* --- ## 中文翻译 # 22580: 从 GPT2 到 Kimi3，详解 **作者**: ali **日期**: 2026-07-27T15:22:08.000Z **来源**: [https://x.com/waterloo_intern/status/2081762065392541951](https://x.com/waterloo_intern/status/2081762065392541951) --- ![Image](/doc/3490/image/img_a58c224ca437.jpg) ![Image](/doc/3490/image/img_a74034570f39.jpg) 两万两千五百八十。这是在一个 KimiK3 (2026) 模型中能塞进多少个 GPT-2 (2019) 模型。我们在七年内将规模扩大了 22,580 倍。但这仅仅是……规模吗？在这篇工作日志中，我将梳理我们是如何走到这一步的，以及自那时以来实际上发生了多少（或多小）变化。我们将追溯导致 KimiK3 的主要架构发展。 ![Image](/doc/3490/image/img_72a9257021d5.jpg) # GPT-2 GPT-2 是一个仅解码器的架构： ``` tok_emb = self.transformer.wte(idx) # 形状为 (b, t, n_embd) 的 token 嵌入 pos_emb = self.transformer.wpe(pos) # 形状为 (t, n_embd) 的位置嵌入 x = self.transformer.drop(tok_emb + pos_emb) for block in self.transformer.h: x = block(x) x = self.transformer.ln_f(x) logits = self.lm_head(x) return logits ``` 输入接收 token 和位置嵌入： ![Image](/doc/3490/image/img_3e9f496b82d1.jpg) 每个 Transformer block 放大后看起来像这样： ``` class Block(nn.Module): def __init__(self, config): super().__init__() self.ln_1 = LayerNorm(config.n_embd, bias=config.bias) self.attn = CausalSelfAttention(config) self.ln_2 = LayerNorm(config.n_embd, bias=config.bias) self.mlp = MLP(config) def forward(self, x): x = x + self.attn(self.ln_1(x)) x = x + self.mlp(self.ln_2(x)) return x ``` ![Image](/doc/3490/image/img_c7fe41cb54aa.jpg) 注意力过程： ``` B, T, C = x.size() # batch size, 序列长度, 嵌入维度 (n_embd) # 批量计算所有头的 query, key, values 并将 head 维度前移至 batch 维度 q, k, v = self.c_attn(x).split(self.n_embd, dim=2) k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) # 手动实现注意力 att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1))) att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf')) att = F.softmax(att, dim=-1) att = self.attn_dropout(att) y = att @ v # (B, nh, T, T) x (B, nh, T, hs) -> (B, nh, T, hs) y = y.transpose(1, 2).contiguous().view(B, T, C) # 重新组合所有 head 的输出 # 输出投影 y = self.resid_dropout(self.c_proj(y)) return y ``` 一旦生成了最终的隐藏状态矩阵，语言模型头就会将其映射到词汇表的 logits。在自回归解码期间，只需要最后位置的 logits 来选择下一个 token。 > 这是仅解码器生成的一个低效之处：模型计算每个输入位置的表示，但每个解码步骤只消耗最后位置的 logits。如果没有缓存，许多工作将在下一个 token 时被重复计算。 ![Image](/doc/3490/image/img_f914047e8f8b.jpg) KV 缓存源于一个直接的观察：在将生成的 token 追加到输入后，模型否则会重新计算所有先前 token 的投影。存储它们的键和值向量可以避免这种冗余工作。这种存储就是 KV 缓存。它保留先前 N-1 个 token 的向量，并且可能变得足够大以至于产生内存带宽瓶颈。总的来说，大约有 50k 个可能的 token，12 个 block，12 个头，以及 768 的嵌入维度，我们的基线模型大约有 124M 参数。 ``` vocab_size: int = 50304 # GPT-2 的 vocab_size 是 50257，为了效率填充到最近的 64 的倍数 n_layer: int = 12 n_head: int = 12 n_embd: int = 768 ``` 在 2.8 万亿参数的情况下，一个 KimiK3 模型包含的参数大约相当于 22,580 个 GPT-2 模型。 # 线性注意力 Softmax 注意力在 q·k 乘积之后应用其非线性，将每个查询与每个键耦合。线性注意力则分别对 q 和 k 应用特征映射，例如 ELU+1。这使得乘积可以重新结合，因此不断增长的 K 和 V 向量集合可以被折叠成一个固定的 D×D 状态。论文中的 O(N²) 表述让我困惑。说“transformer 每个时间步的成本随当前序列长度的平方缩放”是不对的。那是 Flash Attention 解决的问题……然后我看到它是 2020 年发布的。当时，训练通常会实现完整的 N×N 注意力矩阵，FlashAttention 尚不存在，而且参考的自回归实现通常在没有 KV 缓存的情况下重新计算 token 历史。 ``` def forward(self, x, mask=None, past_kv=None): # x 是 b,t,d b,t,d=x.shape d_head=d//self.num_heads h=self.num_heads qkv=self.qkv_proj(x) q=qkv[:, :, :d].view(b,t,h,d_head).transpose(1,2) k=qkv[:, :, d:2*d].view(b,t,h,d_head).transpose(1,2) v=qkv[:, :, 2*d:].view(b,t,h,d_head).transpose(1,2) # 在 prefill 时，q,k,v 的形状是 b,h,t,d # 在 decode 时，形状是 b, h, 1, d # 所以我在 t 维度上连接，即 dim(2) if past_kv is not None: k_past=past_kv[0] v_past=past_kv[1] k=torch.cat((k_past, k), dim=2) v=torch.cat((v_past, v), dim=2) scores=(q@k.transpose(-1,-2))/math.sqrt(d_head) if past_kv is None: # 我们在 prefill 阶段，需要 mask causal_mask=torch.ones(t,t,dtype=bool, device=q.device) causal_mask=torch.triu(causal_mask, diagonal=1) scores=scores.masked_fill(causal_mask, float('-inf')) if mask is not None: scores=scores.masked_fill(~mask, float('-inf')) # 获取 attn (bhtt x bhtd) attn=scores.softmax(-1)#bhtt o=attn@v #bhtd o=o.transpose(1,2).contiguous().view(b,t,d) #b,t,d # 使用 x 来获取 qkv o_proj=self.o_proj(o) past_kv=(k, v) return o_proj, past_kv ``` 从视觉上更容易看到相同的过程。每个解码步骤对 HBM 执行两次 ND 读取和两次 1D 写入，而 KV 缓存随序列长度呈线性增长，即 O(N)。 ![Image](/doc/3490/image/img_071552a485c7.jpg) 注意过度的读写，这篇论文用以下代码替代了它们： ``` def forward(self, x, mask=None, cache=None): # x 是 b,t,d b,t,d=x.shape d_head=d//self.num_heads h=self.num_heads qkv=self.qkv_proj(x) q=qkv[:, :, :d].view(b,t,h,d_head).transpose(1,2) k=qkv[:, :, d:2*d].view(b,t,h,d_head).transpose(1,2) v=qkv[:, :, 2*d:].view(b,t,h,d_head).transpose(1,2) k=F.elu(k)+1 k=k.transpose(-1,-2) q=F.elu(q)+1 S,z=cache if cache is not None else (0.0, 0.0) S=S+k@v z=z+k o=q@S #bhtd denom=q@z o_scaled=o/denom o_scaled=o_scaled.transpose(1,2).contiguous().view(b,t,d) o_proj=self.o_proj(o_scaled) cache=(S,z) return o_proj, cache ``` 这是一种权衡。在这里，我们用分别在 q 和 k 交互之前应用的 ELU+1 替换了 softmax 使用的指数运算。两种方法都对结果分数进行归一化，但线性注意力使用的特征映射是对 softmax 核心的表达能力较弱的近似。这种近似可能会降低保真度，尽管实际精度损失取决于架构和工作负载。请注意，我们仍然除以 qk 的总和，为了简化起见，图中省略了这一点。在高层次上，注意力由三个步骤组成： 1. 使 qk 分数非负。线性注意力使用 ELU+1，而 softmax 使用指数运算。 2. 除以总和。 3. 计算值的加权平均值。这保留了基本的注意力契约，但使用表达能力较弱的特征映射来使 QK 分数非负。 # DeltaNet (快速权重程序员) 有限的缓存必须覆盖或组合已存储的信息。来自 token i-1 的状态不会接收自己的槽位；它被添加到同一个 D x D 矩阵中。因此，新的查询无法再检索每个早期 token 的完全隔离的表示。这种加法也是效率提升的来源。通过加法而不是拼接来更新缓存可以防止其以 O(N) 增长，但同样的操作会导致信息干扰。DeltaNet 解决了这种可恢复性的损失。 ![Image](/doc/3490/image/img_1a158d3689ae.jpg) 正如 Schlag 的论文（快速权重程序员）所言：“当序列长度超过存储容量时，模型可能会进入过容量状态。为了在此类状态下正常运行，模型应该学习动态地与内存内容交互，并有选择地决定保留哪些键值关联以及删除哪些。纯粹的加法指令可能不适合此目的……不断地将新关联添加到有限大小的内存中，如式 17 所示，不可避免地会达到极限。” 使线性注意力具有吸引力的状态，即 N 远大于 D，也暴露了其主要局限性。一旦状态超过其有效容量，关联就会开始干扰，因为更新是加法的，并且没有任何东西离开缓存。 ``` def forward(self, x, mask=None, cache=None): # x 是 b,t,d b,t,d=x.shape d_head=d//self.num_heads h=self.num_heads qkv=self.qkv_proj(x) q=qkv[:, :, :d].view(b,t,h,d_head).transpose(1,2) k=qkv[:, :, d:2*d].view(b,t,h,d_head).transpose(1,2) v=qkv[:, :, 2*d:].view(b,t,h,d_head).transpose(1,2) q = F.normalize(F.silu(q), dim=-1) k = F.normalize(F.silu(k), dim=-1) beta = torch.sigmoid(self.w_beta(x)).view(b, 1, t, 1) # 新增：每个 token 的写入强度 S = cache if cache is not None else 0.0 v_old = k @ S # 用这个键读取缓存 u = beta * (v - v_old) # 增量：仅包含实际新内容的部分 S = S + k.transpose(-1, -2) @ u # 与之前相同的外积写入 o = q @ S # 读取，无分母 o = o.transpose(1, 2).contiguous().view(b, t, d) return self.o_proj(o), S ``` 一个视觉示例使这更容易理解。 ![Image](/doc/3490/image/img_e30acd7f4e2f.jpg) 取一个写入的关联 S = k.T @ v。如果用相同的键读回，你会得到 k @ (k.T @ v)，即 (k @ k.T) v，也就是 k 的范数平方乘以 v。因此，读回的结果按键的范数平方缩放，如果将 k 归一化为单位长度，或者只是将结果除以范数，就可以精确得到 v。 Q 也是一个学习到的指针。Wq 和 Wk 读取相同的残差流，且一个事实的查询指向该事实写入的键方向。更新首先询问当前键从缓存中检索到了什么信息。它从我们要存储的值中减去该现有信息，将键乘以差值，然后将结果加回。旧信息被移除，新信息被写入其位置。 # DeltaNet (使用 Delta 规则并行化线性 Transformer) 这是这篇文章中最难的部分。我花了大约七个小时才对其形成有效的理解，所以我将根据实现来构建解释。简而言之，DeltaNet 实现了具有广义 Householder 转换矩阵的一阶线性递归，实现了分块并行前向传递以进行硬件高效的线性时间训练。它将输入和输出分成几个大小为 C 的块，并根据前一个块的最终状态和当前块的查询键值块计算每个块的输出。实际的问题是预填充。直接对 T 个 token 的序列实现 Delta 规则看起来像这样： ``` S = torch.zeros(b, h, dh, dh) if cache is None else cache outs = [] for i in range(t): k_i = k[:, :, i:i+1] v_i = v[:, :, i:i+1] b_i = beta[:, :, i:i+1] v_old = k_i @ S u_i = b_i * (v_i - v_old) S = S + k_i.transpose(-1, -2) @ u_i # 写入 outs.append(q[:, :, i:i+1] @ S)) o = torch.cat(outs, dim=2) ``` 与标准注意力不同，这种表述需要每个键向量进行校正，因此并行矩阵乘法的路径并不明显。即使没有 Delta 规则，直接的线性注意力预填充仍然是顺序的： ``` S = torch.zeros(b, h, dh, dh) if cache is None else cache outs = [] for i in range(t): q = q[:, :, i:i+1] k = k[:, :, i:i+1] v = v[:, :, i:i+1] S=S_old+k@v o=q@S #bhtd o=self.norm(o) o=o.transpose(1, 2).contiguous().view(b, t, d) out=self.o_proj(o) cache=S outs.append(out) o = torch.cat(outs, dim=2) ``` 分块表述提供了一种更有效的方法。通过一个例子更容易理解其机制： ![Image](/doc/3490/image/img_e3828d1b0d1b.jpg) 设置 C=N 可恢复标准的 O(N^2) 注意力，而 C=1 给出常规的线性注意力。中间值我们在两者之间进行插值，用额外的块内工作换取更好的硬件利用率。在实践中，C 通常是 64 或 128，因为张量核心指令在该粒度下高效运行；UMMA 就是一个例子。中间图块作为状态更新的一部分被折叠到 S 中： ![Image](/doc/3490/image/img_e134a81e2f6f.jpg) ``` S = torch.zeros(b, h, dh, dh) if cache is None else cache outs = [] for i in range(t//C): q_c = q[:, :, i*C:(i+1)*C] k_c = k[:, :, i*C:(i+1)*C] v_c = v[:, :, i*C:(i+1)*C] o_prev=q_c@S # 这是到此块为止的所有内容 attn=(q_c@k_c.transpose(-1,-2)).tril() # 掩码注意力 o_curr=attn@v_c o=o_prev+o_curr S_new=k_c.transpose(-1,-2)@v_c # 循环注意力 S=S+S_new outs.append(o) o = torch.cat(outs, dim=2) ``` 在一个块内，我们执行 q(kᵀv)。这是先计算分数，即带有掩码的正常注意力顺序。在块之间，我们遵循 (kᵀv)q，所以我们是在做循环顺序，状态优先。注意力呈 O(N²) 增长，而这不会。在一个块内，我执行真正的注意力（掩码 QKᵀ 乘以 V），而在块之间，我将所有内容折叠到状态中，并通过一次矩阵乘法读回。因此，成本分为两部分。有一块固定的部分，2Ld²，这是状态工作，完全不关心 C。还有一块增长的部分，2LCd，这是位于对角线上的分数矩阵。完全注意力只是 C 等于 L 的情况，然后第二项变成 2L²d，即二次方。所以我让 C 越小，我做的 FLOPs 就越少。就纯 FLOP 而言，C=1 是最便宜的选项，但不一定在挂钟时间上最便宜。当工作能高效映射到 GPU 的矩阵乘法硬件时，它可以更快地完成更多算术运算。下一步是将相同的方法扩展到 DeltaNet。 ![Image](/doc/3490/image/img_6dbd8db35019.jpg) 根本问题很简单：用于纯加法注意力的分块方法不能直接应用于 delta 更新： ``` v_old = k_i @ S u_i = b_i * (v_i - v_old) ``` 我们需要每一个状态来计算需要减去的信息。如果没有一些数学上的重新参数化，我们就无法以同样的方式并行化它。因此，作者将 delta 更新从： ``` u=v_new-v_old S_t= S_(t-1)+K.T@u o=q@S_T ``` 在这里，一个顺序循环每次迭代计算一个 delta。重新参数化的形式是： ``` S_t = S_{t-1}(I − β_t k_t k_tᵀ) + β_t v_t k_tᵀ o_t = S_t q_t ``` 这种表述允许分块代码一次计算所有 C 个 delta： ``` def chunk_delta_rule_forward(Q, K, V, beta, C): # L: 序列长度, d: 头维度 L, d = Q.shape # 分块 Q, K, V = map(lambda x: x.reshape(-1,C,d), [Q, K, V]) beta = beta.reshape(-1, C) K_beta = K * beta.unsqueeze(-1) V_beta = V * beta.unsqueeze(-1) # 使用向量化前向替换计算 eq. 10 以进行快速求逆 T = -(K_beta @ K.t()).tril(-1) for i in range(1, C): T[i, :i] = T[i, :i] + (T[i, :, None] * T[:, :i]).sum(-2) T += torch.eye(C) W = T @ K_beta U = T @ V_beta # 分块并行。Eq. 8-9 S = torch.zeros(d, d) O = torch.empty_like(V) for i in range(L//C): q_i, k_i, w_i = Q[i], K[i], W[i] u_i = U[i] - w_i @ S # 校正量，一个块的全部 o_inter = q_i @ S A_i = (q_i @ k_i.t()).tril() #qk.t o_intra = A_i @ u_i # 注意力 @ v (带有校正，所以是 u) S += k_i.t() @ u_i # 通过加法更新状态 O[i] = o_intra + o_inter # 通过 flash + 循环更新输出 return O.reshape(L, d) ``` 这使我们达到了第一个比较点：MHA 与 DeltaNet Transformer： ![Image](/doc/3490/image/img_78fccf48389d.jpg) # 门控 Delta Net 现在我们有了一种对缓存进行精确更改的方法。对于每个新事实（每个新键向量），我们可以查看确切存储在该点的旧信息，并用我们想要关注的新信息替换它。然而，这种机制只能忘记它有特定替换的关联。它无法在上下文切换期间高效清除多个关联，或一般性地衰减记忆以释放容量。如果我们做纯加法线性注意力：添加忘记能力会很简单。我们只需要一个控制遗忘状态的参数： ``` S_old=cache S_new=k@v # cache=S_old+S_new cache=alpha * S_old + S_new ``` ![Image](/doc/3490/image/img_83f03e15a8af.jpg) 这是 Mamba-2 的贡献。我们衰减先前的缓存，然后以全强度添加新缓存，防止状态无限增长。在每个时间步以动态比率均匀衰减所有键值关联是一种可行的方法，这也是 Mamba 所做的。但它没有考虑到不同键值关联的不同重要性。也就是说，如果模型需要忘记一个特定的关联，所有关联都会被同等程度地遗忘。相比之下，Delta 规则可以更新单个事实，但没有办法让其余事实衰减。因此，门控 Delta 规则将 Mamba 的门控更新规则与 Delta 规则结合起来。它添加了一个参数 alpha，当设置为 1 时切换到纯 Delta 规则，当设置为 0 时清除记忆。挑战在于使用相同的并行块方法来实现这一点。该实现使用了上一节描述的相同 DeltaNet 重新参数化。数学几乎相同，但有一个补充：一个介于零和一之间的数据相关标量，它控制先前状态的衰减。这结合了有效的键值关联学习和自适应内存管理。相应的代码更改如下所示： ![Image](/doc/3490/image/img_5dc2e5718f17.jpg) γʳ/γⁱ 项考虑了累积衰减。在时间步 x 写入并在 x+t 读取的 token 已经被乘以 αₓαₓ₊₁αₓ₊₂…αₓ₊ₜ。这是前缀和计算的乘法模拟。结果架构如下所示： ![Image](/doc/3490/image/img_85579c5011b9.jpg) # KDA/Kimi Linear 此时，研究人员开始试验混合模型，即在一个架构中结合多种形式的注意力，例如 Gated DeltaNet 与 Mamba。 Kimi Linear 因一个核心主张而引人注目：在受控比较下，它优于全注意力。作者将其 presented 为一种直接插入的架构替代品，具有更好的质量和高达 6 倍的解码吞吐量。 Kimi Linear 通过引入细粒度门控改进了 Gated DeltaNet。它不是单一的标量衰减，而是学习每个通道的单独衰减值。 ![Image](/doc/3490/image/img_95b5987a3172.jpg) KDA 更新规则保持相似，但代码现在看起来更像这样： ![Image](/doc/3490/image/img_24fddfd80aa7.jpg) 这里，alpha.reshape(nb, C, d) 捕捉了论文最重要的贡献：对内存衰减的细粒度控制。与 DeltaNet Transformer 并列，Kimi Linear 架构引入了三个主要变化： 1. 它使用混合系统，交错多层多头潜在注意力 (MLA) 层。 2. 它用专家混合层替换了 MLP。 3. 它通过 alpha 投影增加了 DeltaNet 的容量。 ![Image](/doc/3490/image/img_3e1acc480ddf.jpg) 后面的部分将更详细地介绍 MLA 和 MoE。目前，重要的一点是这不是盲目扩展。额外的容量具有特定的数学目的：每通道缩放使模型能够更好地控制内存衰减。缩放定律仍然相关，但必须在正确的位置以系统可以使用的形式添加容量。此进展中的每个架构都增加了容量以解决前一个系统的具体限制。 # Kimi K3 最终，KimiK3 语言主干看起来与上面的 Kimi Linear 模型相似。它包含 23 个四层宏循环。在每个宏循环中，三层使用 Kimi Delta Attention，第四层使用多头潜在注意力。第一层使用密集的前馈网络；其余每一层都使用潜在的 MoE。乍一看，与 Kimi Linear 相比的变化似乎是适度的： - 规模大幅增加 - 每 12 层进行一次分块 AttnRes - MLA 查询 LoRA 和输出门控 - 潜在空间 MoE - SiTU 激活 - 门控 MLA KDA 提供恒定状态的循环记忆，而周期性的 MLA 层保留对上下文的完整 softmax 检索。以下简化的可视化提供了下面讨论的更改的有用参考。 ![Image](/doc/3490/image/img_10630b6c18c7.jpg) 我们将从更直接的变化开始：门控 MLA、潜在空间 MoE 和 SiTU 激活。门控 MLA 确定从 MLA 传入残差流的每个检索特征有多少。它通过与从输入投影的门进行逐元素乘法来实现这一点。在传统的 MoE 中，学习到的路由器使用点积相似性将每个 token 发送到专家网络的子集。KimiK3 总共有 898 个专家。两个是共享的，处理每个 token；在剩余的 896 个中，路由器为每个 token 选择 16 个。 KimiK3 还改变了专家激活。它不是对上投影应用 SiLU，逐元素乘以门，然后应用下投影，而是使用 SiTU： ``` d = x.shape[-1] // 2 gate = x[..., :d].to(torch.float32) up = x[..., d:].to(torch.float32) situ_a = self.beta * torch.tanh(gate / self.beta) * torch.sigmoid(gate) if self.linear_beta is not None: up = self.linear_beta * torch.tanh(up / self.linear_beta) return (situ_a * up).to(x.dtype) ``` 该模型还将输入下投影到共享专家，并将其最终和上投影： ![Image](/doc/3490/image/img_d301afeb8f94.jpg) 这说明了模型推理中的一个反复出现的挑战。如果没有融合内核，新的激活几乎比原始路径慢 3 倍。一个抵消的优化是专家在压缩的潜在空间中运行，这使得他们的前向传递快得多，并且几乎将 FLOPs 减半。剩下的变化是 MLA 查询 LoRA、输出门控以及每 12 层的分块注意力残差。AttnRes 增加了大约 2% 的推理延迟，但提供了两个重要的好处： - 选择性检索早期表示，这减轻了残差稀释和隐藏状态增长 - 1.25 倍的计算优势 AttnRes 和 MLA 从不同的方向解决了同样的潜在限制。KDA 层以恒定大小的状态运行，并且必然丢弃信息。MLA 从 token 上下文中检索，而 AttnRes 从早期的深度表示中检索。 # AttnRes 感谢 @chloey3k 对本节的帮助。在每个前向传递中，输入通过一系列层。这里，每一层由一个注意力块 (KDA 或 MLA) 和一个 MLP 或 MoE 块组成。通常，每一层的输入是原始嵌入和每个先前层输出的总和，所有权重相等。这里，h_i 是第 i 层的输入，h_1 是当前 token 的嵌入（到目前为止序列中的最后一个 token），f_i(h_i) 是第 i 层的输出（一个注意力或 MLP 块）。问题是缺乏选择性访问。不同的层类型接收相同的聚合状态，尽管它们可能从不同的权重中受益。因为递归是纯加法的，后来的层也必须学习越来越大的输出才能影响累积的残差，这可能会破坏训练的稳定性。AttnRes 不是平等对待所有层，而是将该总和的每一项乘以一个专门的权重，这使模型能够根据上下文赋予最有用的层更多重要性。每个权重 alpha_i 由查询键点积计算得出。查询是为每一层学习的，而键和值来自早期的残差流状态。分数被归一化为总和为一，然后用于形成这些状态的加权组合。 ![Image](/doc/3490/image/img_0b34b3fb6b34.jpg) 因此，模型不必仅仅依赖于其直接的前驱。AttnRes 为每一层提供对早期层输出的选择性访问，允许其学习的查询检索对当前计算最有用的表示。下面的伪代码在块粒度上应用了相同的想法。一个块是跨 12 个解码层累积的注意力和 MLP 输出的逐元素和，存储为单个深度表示以供以后的 AttnRes 混合。在每一层应用残差注意力会增加太多的训练和推理成本。仅将其应用于固定的块边界可以以较低的成本获得大部分好处。在 KimiK3 中，每个边界出现在 12 个解码层之后。在 23 个四层宏循环中，这产生了 8 个 AttnRes 块，这增加了我们的推理速度。这可能是 block_attn_res 函数最重要的部分 ``` V = torch.stack(blocks + [partial_block]) # [N+1, B, T, D] K = norm(V) logits = torch.einsum('d, n b t d -> n b t', proj.weight.squeeze(), K) h = torch.einsum('n b t, n b t d -> b t d', logits.softmax(0), V) return h ``` 这完成了从 GPT-2 到 KimiK3 的演变。核心变化不仅仅是规模。每个架构步骤都改变了模型存储的内容、它如何更新该状态，或者它如何检索固定大小状态无法保存的信息。 KimiK3 结合了恒定状态循环记忆、周期性 softmax 检索、稀疏专家容量和选择性深度残差访问。结果是一个系统在具有特定功能角色的地方花费额外的容量。本质上，固定容量的关联记忆（固定维度）需要一个驱逐策略，因为一旦达到容量，纯加法线性运算最终会增加干扰。为此，学习选择，如门控、路由或衰减，是必要的，而注意力是最有效的选择性读取机制。 ## 相关链接 - [ali](https://x.com/waterloo_intern) - [@waterloo_intern](https://x.com/waterloo_intern) - [1.1M](https://x.com/waterloo_intern/status/2081762065392541951/analytics) - [@chloey3k](https://x.com/@chloey3k) - [升级到 Premium](https://x.com/i/premium_sign_up) - [11:22 PM · Jul 27, 2026](https://x.com/waterloo_intern/status/2081762065392541951) - [1.1M Views](https://x.com/waterloo_intern/status/2081762065392541951/analytics) - [查看引用](https://x.com/waterloo_intern/status/2081762065392541951/quotes) --- *导出时间: 2026/7/28 12:35:59*

🔗 相关文档