DeepSeek-V4 工程权衡分析：1M 上下文并非无损记忆

GoGoNN 2026-04-28 45.4 KB 16.8K 23 𝕏 文章列表

文章深入剖析了 DeepSeek-V4 如何通过 HCA、CSA 和 SWA 三通道并行架构，将 KV Cache 压缩至基线的 2%，从而实现工程上可行的 100 万 token 上下文。然而，这种极致的压缩是有代价的：MRCR 8-needle 测试准确率在 1M 长度下显著下降至 0.59。文章指出，V4 通过牺牲检索精度换取了推理成本的数量级降低，是一种‘有损的 1M’，并明确论证了该设计原理上无法完全替代 RAG 系统。

DeepSeek 长上下文模型架构 KV Cache CSA HCA 工程权衡 RAG 技术分析

# 1M Context ≠ Lossless Long Memory: What DeepSeek-V4 Reveals About the Engineering Tradeoffs **作者**: GoGoNN **日期**: 2026-04-27T14:37:43.000Z **来源**: [https://x.com/hhjjjFinFin/status/2048773590942970275](https://x.com/hhjjjFinFin/status/2048773590942970275) --- ![Image](/doc/1210/image/img_fe68f582012c.jpg) > 1M context is not lossless long memory.DeepSeek-V4 turns attention into three things running in parallel: a 128:1 global summary (HCA), a 4:1 sparse-selection top-k=0.4% precise read (CSA), and a 128-token local-fidelity window (SWA). Each lane handles a different precision regime.KV cache shrinks to 2% of a BF16 GQA8 baseline. The cost: MRCR 8-needle accuracy drops from 0.94 at 8K to 0.59 at 1M. An order of magnitude cheaper than V3.2, but still 10 points behind Claude Opus 4.6 (0.93).Below is a component-by-component breakdown of how V4 engineers a "seemingly impossible 1M" into a "routinely serviceable 1M" — and why this design fundamentally cannot replace RAG. ## 1. Two narratives behind the "1M" number The open-source community now treats context length as a standalone spec, like RAM size. Llama 4 1M, Qwen3 256K, GLM 4.5 400K, DeepSeek-V4 1M — on paper, it looks like just a number, with the implicit assumption that the model "works on contexts of this length the same way it works on short ones." That's a misread. Closed-source frontier labs like Anthropic and Google haven't published their long-context architectures, but when Claude Opus 4.6 scores 92.9 on MRCR 8-needle at 1M and Gemini 3.1 Pro scores 92.4, you can roughly infer they're using something close to dense — paying engineering cost for a high retrieval ceiling. DeepSeek-V4 takes the opposite approach: accept that attention has to be lossy at 1M, then place the losses where the model is least sensitive. After walking that road, V4-Pro hits 83.5 on MRCR 8-needle at 1M, V4-Flash hits 49.4, and V3.2 simply cannot be deployed at that length. V4's "1M" is not a "lossless 1M" — it's "a 1M that becomes serviceable under acceptable quality cost." That's the anchor for the entire piece. Following V4's architecture diagram layer by layer, this judgment will only get more concrete. ![Image](/doc/1210/image/img_fa75b15f768c.jpg) ## 2. The numbers hook: what 27%/10% and 10%/7% really mean The paper's core pitch is two pairs of percentages: - V4-Pro vs V3.2 @ 1M: single-token inference FLOPs down to 27%, KV cache down to 10% - V4-Flash vs V3.2 @ 1M: single-token inference FLOPs down to 10%, KV cache down to 7% - vs BF16 GQA8 baseline @ 1M: V4 series KV cache ≈ 2% of baseline These numbers are honest — V3.2 already incorporates DSA, so this is incremental on top of an already optimised baseline. But three footnotes get glossed over. Footnote 1 · The comparison baseline is V3.2, not vanilla attention. V3.2 vs Llama-style dense attention already has a 3–5x gap on its own. So "V4 saves 10x over V3.2" is by no means equivalent to "V4 saves 30x over dense attention." The accurate framing is "on top of an already-sparsified baseline, we squeeze another order of magnitude." Footnote 2 · The "1/3 speedup from FP4 × FP8" is a future-hardware option, not realised today. On H100 / H200 / B200, FP4 × FP8 peak FLOPs are identical to FP8 × FP8. The paper's exact wording is "theoretically can be 1/3 more efficient on future hardware" — a forward-looking note for B300 / NVFP4-class hardware. Today's deployments don't get it. Footnote 3 · "2% KV cache vs BF16 GQA8" is the product of four compression layers, not a single trick. 1. Sequence-dimension compression (CSA m=4, HCA m'=128) 2. K and V sharing (MLA-style, half off) 3. BF16 → FP8 (non-RoPE dimensions, another ~50% off) 4. Indexer KV in FP4 (one more notch down) Multiplied together, you get 2%. Each layer pays its own information-loss cost. Treat these numbers as facts, but remember they're achieved under degraded retrieval quality. The MRCR 8-needle curve in Figure 9 — sliding from 0.94 at 8K all the way to 0.59 / 0.49 at 1M — is the real shape of the cost. ## 3. Architecture overview: V4 attention is a three-lane parallel retrieval system ![Image](/doc/1210/image/img_767eba0ea88e.jpg) Before unpacking CSA/HCA individually, establish the global picture. Each attention layer in V4 (excluding the first two) is actually a three-lane parallel structure. At every layer, a query simultaneously taps three information channels: ① HCA (global-summary lane)Compression ratio m'=128. 1M tokens compressed into ~7,812 entries; each entry is a lossy summary of 128 raw tokens. Query attends densely to all of them. Covers the entire 1M (global), but each point is severely degraded in precision. ② CSA (sparse-precise-read lane)Compression ratio m=4. 1M tokens compressed into ~250K 4-token blocks. The Lightning Indexer scores all of them and lets the query attend only to the top-k=1024 most relevant ones. Coverage = the entire 1M, but only ~0.4% is actually read by the query. ③ SWA (local-fidelity lane)No compression. The most recent 128 tokens' K/V were kept in full fidelity. Only the last 128 tokens, but completely preserved. The information budget the query actually "sees": Lane High-fidelity bits Note SWA 128 token-equivalents · 100% fidelity local context CSA 1,024 × 4 = 4,096 token-equivalents · 4:1 lossy indexer-picked precise-read region HCA 7,812 × 128 = full 1M tokens · 128:1 lossy global ambience Total ≈ 4,224 high-fidelity bits + 1M low-fidelity bits = "sparse precise read + blurry global view." This is the anchor for everything that follows. None of the three lanes can stand alone — SWA only sees 128 tokens; whatever CSA misses globally is gone forever; HCA's 128:1 compression is too information-sparse for precise retrieval. That's exactly why V4 has to run all three in parallel rather than picking one. ## 4. CSA: compression + sparse selection, the precise-read lane ![Image](/doc/1210/image/img_852000fc3a8d.jpg) 4.1 The two-step design CSA (Compressed Sparse Attention) thins information in two steps: 1. Step 1 — compression: every m=4 consecutive tokens' hidden states are compressed into 1 KV entry using learned softmax weights. 1M tokens → 250K entries. 2. Step 2 — sparse selection: an independent Lightning Indexer scores all 250K entries; the query attends only to the top-k=1024. That's the two-layer saving from "compression + sparsity." Each query effectively reads 1024 × 4 = 4096 raw-token equivalents — but those 4096 tokens are picked by the indexer based on what it thinks is most relevant, not what the query actually needs. If the indexer picks the wrong one, the blocks the query genuinely needs are simply gone. 4.2 The Lightning Indexer: a cheap top-k decision-maker The Lightning Indexer was introduced in V3.2 and inherited directly by V4. It's essentially a very thin, independent attention sub-network used only for scoring, never participating in the final output: - Independent indexer KV (one per 4 tokens, head dim only 128) - FP4 dot products — even cheaper than the main attention's FP8 - ReLU activation followed by top-k This is a beautiful engineering decoupling: "deciding where to look" and "what to look at" are separated. But it carries a fundamental risk: the indexer's ranking quality determines the upper bound of the main attention. If the indexer ranks a truly relevant entry at position 1100, the top-k=1024 query will never see it. The paper does not provide an indexer-accuracy ablation. On MRCR at 1M, V4's top-k coverage is 1024 / 250000 ≈ 0.41% — meaning the indexer must rank each of the 8 needles' compressed blocks within the top-1024. If even one falls outside, the 8-needle task loses one needle. 4.3 Overlap compression: V4's small innovation on CSA The genuinely new piece in CSA relative to V3.2's DSA is overlap compression: two sets of KV (C^a, C^b) are produced simultaneously, covering staggered 4-token blocks, then cross-block softmax is applied over a wider window. The benefit is mitigating "information fracture at block boundaries" — under naive compression, if a needle straddles two 4-token blocks, its semantics are split. Overlap compression amounts to a 50%-stride convolutional perspective. The cost is concrete: pre-compression KV memory doubles (two sets of K, V instead of one), and the CSA layer's parameter count doubles. The cost gets absorbed by §3.5.3's contextual parallelism, but training-time activation memory pressure is real. ## 5. HCA: heavy compression + full dense, the coarse-grained global summary ![Image](/doc/1210/image/img_927f54f4492f.jpg) 5.1 A counterintuitive design choice HCA (Heavily Compressed Attention) takes the opposite philosophy from CSA: Dimension CSA HCA Compression ratio m = 4 m' = 128 Compression style Overlap (two KV sets) Non-overlap (single KV set) Sparse selection? Yes (top-k=1024) No, full dense Indexer needed? Yes No Tokens condensed per entry 4 128 The interesting questionis: why does HCA dare togo "compress and then dense-attend directly," with no sparse selection? The answer is in the numbers. After m'=128 compression, 1M tokens shrink to 1,000,000 / 128 ≈ 7,812 entries. 7,812 KVs is well within the budget of dense attention — equivalent to dense attention on an 8K context, roughly 4 GFLOPs on an H100. Totally fine. So HCA's core trade is: "trade extreme compression for the elimination of sparse-selection complexity." CSA is "light compression + selection"; HCA is "heavy compression + view all." Both pull attention out of the O(N²) regime into a serviceable range, but through entirely different mechanisms. 5.2 The limit of information density m'=128 means each compressed entry must cram 128 raw tokens' semantics into a c=512-dimensional vector. Given that the original hidden state is also d=4096~7168 dimensions, 128 tokens × 7168 dim = ~917K dimensions of information squeezed down to 512 — a compression ratio of 1800:1. This is the lossiest stage in V4's architecture. HCA entries are destined to capture "global ambience" rather than "specific facts." They can tell the query "what region X of the 1M context is generally about," but cannot precisely say "at token position 543,217, the specific name of variable X is Y." This observation maps directly onto Figure 9's MRCR curve: the drop at ≥256K on precise-retrieval tasks can be read almost literally as "HCA's semantic granularity can't keep up with needle-level precision." ## 6. Sliding Window: the local-safety net for when compression fails 6.1 A patched-on design Looking at CSA + HCA alone, there's a serious problem: causality requires the query to not attend to its own compressed block (otherwise it leaks future-token information). This means: the most recent stretch of tokens including the query itself is invisible to both CSA and HCA — CSA can't see its own 4-token block; HCA can't see its own 128-token block. But in language modelling, the most recent few tokens are precisely the most critical signal for next-token prediction. The contradiction has to be resolved, so V4 adds a third lane: Sliding Window Attention (SWA), in which each query additionally attends to the K/V of the most recent n_win = 128 uncompressed tokens. 6.2 An equality that's not a coincidence Notice a design detail: n_win = m' = 128. These two numbers being equal isn't accidental — SWA's window size is exactly HCA's compression ratio, meaning SWA covers exactly the "current HCA block" range that HCA can't see due to causality. So engineering-wise, the three lanes are in strict complementary division of labour: - SWA covers the most recent 128 tokens (the current HCA block's range) - HCA covers all earlier compressed HCA blocks (start → ~current - 128) - CSA picks top-k=1024 from all 4-token compressed blocks, no positional restriction This division makes one thing clear: pure compressed attention is engineering-infeasible without an uncompressed local window as a fallback. SWA is an "elegant patch," but it's still a patch — its very existence acknowledges that "compression has limits." ## 7. Attention Sink: making "look at nothing" a legitimate option This trick comes from StreamingLLM (Xiao et al. 2024) and was also used in GPT-OSS. V4's implementation: each attention head has a learnable scalar z'_h added to the softmax denominator. Result: attention weights no longer have to sum to 1. They can be 0.7, 0.3, or close to 0. This trick matters especially under sparse attention. Sometimes none of CSA's top-k=1024 selected entries are actually relevant — say the query is "what's the value of this variable?" but the indexer dropped all the relevant blocks. Without a sink, softmax forces attention to spread evenly across those 1024 irrelevant entries, generating massive noise. With a sink, the model can learn "I don't care about any of these 1024 — let attention flow entirely to the sink," and the layer's output is barely affected. > What sink really means: it gives an attention layer the legitimate option to opt out of the current layer. Without it, sparse-attention failure mode is quietly emitting wrong signal; with it, failure mode becomes emitting near-zero signal and letting the next layer recover. Former is silent corruption, latter is graceful degradation. ## 8. Partial RoPE: position encoding done with care V4 uses partial RoPE on CSA / HCA — applying RoPE only to the last 64 dimensions of query and KV vectors, leaving the rest position-agnostic. This dates back to V2's MLA and is inherited by V4. But V4 adds an interesting little detail: inverse RoPE on the output side. The problem: CSA / HCA's KV is K=V shared (MQA-style); attention output is Σ s_j × V_j. If the last 64 dimensions of V_j carry absolute position encoding (pos j), then Σ s_j × V_j mixes RoPEs across all different positions into a "weighted sum of positions" — neither absolute nor relative, basically noise. V4's fix: on the output side, rotate the attention output with RoPE at position = -i. This makes the output's "effective position" become j - i = -dist, meaning each V_j's contribution to the output is re-encoded by the query-key distance, automatically converted into relative position encoding. I haven't seen this trick in any other paper. Mathematically, it's elegant — it resolves the inherent contradiction of "K=V sharing + absolute RoPE." But again, the V4 paper provides no ablation; we just have to trust that "it works fine." ## 9. The cost in quality: the honest curve in Figure 9 ![Image](/doc/1210/image/img_38c772491da7.jpg) Set the architecture aside and look at the data. Figure 9 plots V4's MRCR 8-needle retrieval accuracy across context length: Context 8K 16K 32K 64K 128K 256K 512K 1024K V4-Pro-Max 0.90 0.85 0.94 0.90 0.92 0.82 0.66 0.59 V4-Flash-Max 0.91 0.84 0.87 0.85 0.87 0.76 0.60 0.49 For comparison, frontier models on MRCR 1M 8-needle: - Claude Opus 4.6: 0.929 - Gemini 3.1 Pro: ~0.92 (paper-reported) - V4-Pro-Max: 0.835 (Table 7) A few facts worth surfacing: Fact 1: Within 128K, V4-Pro is close to the frontier (0.92 vs 0.93). 1M is where divergence actually shows up — there V4 trails Claude by 9–10 absolute points. Fact 2: V4-Pro at 1M (0.59) beats V4-Flash (0.49). Bigger total params + a larger top-k (1024 vs 512) directly help with long-context retrieval — Pro has twice as many "high-fidelity bits" as Flash. Fact 3: Figure 9's curve has a "rebound" at 32K, hitting 0.94, then declines monotonically. The 0.59 at 1M is 34 points lower than 0.90 at 8K — that's the concrete number behind "the quality cost of 1M context." The paper's wording is worth quoting: > retrieval performance remains highly stable within a 128K context window. While a performance degradation becomes visible beyond the 128K mark, the model's retrieval capabilities at 1M tokens remain remarkably strong compared to both proprietary and open-source counterparts. "Remarkably strong compared to ... counterparts" is true — among open-source peers, V4 does lead. But what gets glossed over is the drop from 0.92 to 0.59 after 128K. For a rigorous technical reader, that curve is the clearest evidence of V4's engineering trade-off. ## 10. A counterintuitive observation: V4 actually scores higher at 1M than at 128K? Note this set of numbers from Table 7: MRCR 8-needle 128K MRCR 8-needle 1M V4-Pro-Max 78.6 83.5 V4-Pro 78.4 81.0 1M scores higher than 128K? This contradicts the trend in Figure 9 (same task, more lengths) — Figure 9 is monotonically decreasing. The two numbers in Table 7 are "official highlight scores at specific lengths," and they show 1M > 128K. The paper doesn't explain the anomaly. Two guesses: 1. Training distribution: V4's late-stage long-context training concentrates around 1M; the model is "more familiar" with that length. 128K is just a "transit point" in the training schedule and may not have been fully consolidated. 2. HCA's sweet spot: a 1M sequence under m'=128 compression yields 7,812 dense entries — exactly HCA's design optimum. A 128K sequence only produces 1,000 entries, which is too sparse and doesn't let HCA's dense advantage shine. Either way, the anomaly tells us one thing: V4's long-context capability is "trained for 1M," not "generic across lengths." If your real workload is at 200K~500K, you may not be hitting the sweet spot the paper highlights. ## 11. Putting all the pieces back together The previous 10 sections compressed into one sentence: > V4's attention isn't attention — it's a composite retrieval system of "lossy compression + sparse retrieval + local fidelity + explicit abstention." At every layer, a query simultaneously consults three views at different granularities (SWA local, CSA precise, HCA global), relying on the indexer's ranking quality, the attention sink's "give up" mechanism, and partial RoPE's position coupling to engineer the "seemingly impossible" 1M into "routinely serviceable." The real position of this design isn't "innovation" — it's engineering-grade assembly of pre-existing, individually validated components: - Lightning Indexer + top-k: V3.2 DSA - Latent query / low-rank K=V sharing: V2 MLA - Partial RoPE: V2 onward - Sliding Window: Mistral / Gemma mainstream - Attention Sink: StreamingLLM, GPT-OSS - Compress + dense interleaving: Native Sparse Attention lineage V4's real contribution is the engineering composition + end-to-end 1M serviceability: making these components coexist in a stably trained 32T-token, 1.6T-parameter MoE, paired with custom KV cache management, sparse-attention kernels, and contextual-parallelism training infrastructure. It's solid systems work, but it isn't a "lossless 1M breakthrough" — it's an engineering victory of "pulling the marginal cost of 1M context into an acceptable range, at the cost of acceptable precise-retrieval degradation." ## 12. Takeaways for the reader When an open-source model claims 1M context - Ask whether the attention is dense. If not, it's 1M is a lossy 1M. - Ask for MRCR 1M 8-needle scores. 0.85+ is frontier-tier; 0.5–0.6 means "runnable but not for precise retrieval." - Ask for the compression ratio. HCA-style 128:1 compression dooms certain precision tasks — that's not something "more training" can fix. When designing long-context applications - Don't use V4 for strict needle retrieval. Use Claude Opus 4.6 / Gemini 3.1 Pro, or just put RAG in front. - V4's sweet spot is long-document reasoning (e.g., CorpusQA, where V4-Pro-Max scores 33.6 vs Gemini's 35.4) — tasks that need "global ambience + local detail" rather than point-to-point precise matching. - Good fits: large codebase analysis, multi-document summarisation, and agent long-trajectory maintenance. - Bad fits: legal-clause retrieval, extracting specific numbers / named entities from medical literature, citing precise timestamps in long transcripts. When doing architecture research - Lossy long-context is now a consensus path. Native Sparse Attention, MLA, DSA, Linear Attention, State-Space hybrids — they're all on this path, just varying the compression/sparsity/state ratio. - V4's real lesson is "tiering": different layers use different compression strategies (CSA / HCA interleaved); different lanes use different precision (SWA full + CSA dense + HCA sparse). This kind of hybrid clearly outperforms any single strategy. - Three open questions worth pursuing: How does the indexer's top-k ranking accuracy ablate? It's the bottleneck on CSA's ceiling. Is HCA's m'=128 the empirical optimum, or would m'=64 / 256 also work? The paper doesn't sweep this curve. The attention weight allocation across the three lanes is learned — could the sink already be "switching off the entire HCA lane" in some layers? That would imply HCA's actual contribution is overstated. ## 13. Closing "1M context" as a spec line has been refreshed repeatedly during the 2025–2026 open-model race. V4's real contribution is that it wrote down, for the first time, a complete answer to "how is 1M actually built" — not "we widened the attention window," but "we shredded attention with five different compression/selection / windowing mechanisms, each handling the granularity it's best suited for." That kind of transparency deserves encouragement. But readers — especially engineering readers — need to see the architectural cost hiding behind the spec numbers: - 1M's KV cache is only 2% of BF16 GQA8 because information has been squeezed through four compression layers - 1M's FLOPs are only 27% of V3.2's because attention has degenerated from O(N²) into a "tiered approximation, roughly O(N · log N)" - 1M's retrieval accuracy slid from 0.94 at 8K to 0.59 because this approximation has a precision ceiling, and that ceiling won't be cured by training data Next time someone tells you "1M context models are here, RAG is dead," send them the curve from Figure 9 — the one going from 0.94 down to 0.59. 1M context ≠ lossless long memory. It's an engineering-tradeoff victory, and it's an honest disclosure: for some tasks, you still need RAG, you still need dense attention, you still need other external memory systems. V4 didn't solve the long context. V4 just pushed "the serviceable boundary of long context" considerably further. That's already plenty. References: DeepSeek-AI, "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence" (2026); Hugging Face Blog, "DeepSeek-V4: a million-token context that agents can actually use"; NVIDIA Developer Blog, "Build with DeepSeek V4 Using NVIDIA Blackwell"; Andrew Lukyanenko, "DeepSeek-V4 Review: Why Million-Token Context Needs Efficient Attention" (Medium, Apr 2026); BigDataBoutique, "Needle in haystack? Optimizing Retrieval and RAG over Long Context Windows"; Liu et al., "Lost in the Middle: How Language Models Use Long Contexts"; Xiao et al., "Efficient Streaming Language Models with Attention Sinks" (StreamingLLM). ## 相关链接 - [GoGoNN](https://x.com/hhjjjFinFin) - [@hhjjjFinFin](https://x.com/hhjjjFinFin) - [8.2K](https://x.com/hhjjjFinFin/status/2048773590942970275/analytics) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [10:37 PM · Apr 27, 2026](https://x.com/hhjjjFinFin/status/2048773590942970275) - [8,277 Views](https://x.com/hhjjjFinFin/status/2048773590942970275/analytics) --- *导出时间: 2026/4/28 15:26:44* --- ## 中文翻译 # 1M上下文 ≠ 无损长期记忆：DeepSeek-V4 揭示的工程权衡 **作者**: GoGoNN **日期**: 2026-04-27T14:37:43.000Z **来源**: [https://x.com/hhjjjFinFin/status/2048773590942970275](https://x.com/hhjjjFinFin/status/2048773590942970275) --- ![Image](/doc/1210/image/img_fe68f582012c.jpg) > 1M上下文并不等于无损的长期记忆。DeepSeek-V4 将注意力拆分为三条并行运行的部分：128:1 的全局摘要（HCA）、4:1 的稀疏选择 top-k=0.4% 精确读取（CSA），以及一个 128 token 的局部保真窗口（SWA）。每条通道处理不同的精度区间。KV 缓存缩减至 BF16 GQA8 基线的 2%。代价是：MRCR 8-needle 准确率从 8K 时的 0.94 降至 1M 时的 0.59。比 V3.2 便宜一个数量级，但仍落后 Claude Opus 4.6（0.93）10 个百分点。下面将逐个组件剖析 V4 如何把一个“看似不可能的 1M”工程化为“可日常使用的 1M”——以及为什么这种设计从根本上无法取代 RAG。 ## 1. “1M”这个数字背后的两种叙事开源社区如今把上下文长度当作一个独立的技术规格，就像内存大小一样。Llama 4 1M、Qwen3 256K、GLM 4.5 400K、DeepSeek-V4 1M——纸面上看只是一个数字，背后隐含的假设是模型“在此长度的上下文上的表现与其处理短上下文时相同”。这是一种误读。 Anthropic 和 Google 等闭源前沿实验室并未公布其长上下文架构，但当 Claude Opus 4.6 在 1M 长度的 MRCR 8-needle 上拿到 92.9 分、Gemini 3.1 Pro 拿到 92.4 分时，可以大致推断它们使用的是接近稠密注意力（dense）的方案——为高检索上限付出工程成本。 DeepSeek-V4 则采取相反的路线：承认注意力在 1M 长度下必然有损，然后把损失放置在模型最不敏感的地方。走完这条路之后，V4-Pro 在 1M 长度的 MRCR 8-needle 上拿到 83.5，V4-Flash 拿到 49.4，而 V3.2 根本无法部署到该长度。V4 的“1M”不是“无损的 1M”——而是“在可接受的质量成本下变得可用的 1M”。这是整篇文章的锚点。跟随 V4 的架构图逐层剖析后，这个判断只会变得更加具体。 ![Image](/doc/1210/image/img_fa75b15f768c.jpg) ## 2. 数字钩子：27%/10% 和 10%/7% 究竟意味着什么论文的核心卖点是两组百分比： - V4-Pro 对比 V3.2 @ 1M：单 token 推理 FLOPs 降至 27%，KV 缓存降至 10% - V4-Flash 对比 V3.2 @ 1M：单 token 推理 FLOPs 降至 10%，KV 缓存降至 7% - 对比 BF16 GQA8 基线 @ 1M：V4 系列 KV 缓存 ≈ 基线的 2% 这些数字是诚实的——V3.2 已经引入了 DSA，所以这是在已优化基线之上的增量改进。但有三处脚注被一笔带过。脚注 1 · 比较基线是 V3.2，而非原始注意力。V3.2 对比 Llama 式稠密注意力本身已有 3–5 倍的差距。所以“V4 比 V3.2 节省 10 倍”绝不等于“V4 比稠密注意力节省 30 倍”。准确的表述是“在已经稀疏化的基线之上，再压榨出一个数量级”。脚注 2 · “FP4 × FP8 带来的 1/3 加速”是面向未来硬件的选项，今天并未实现。在 H100 / H200 / B200 上，FP4 × FP8 的峰值 FLOPs 与 FP8 × FP8 相同。论文的原文是“理论上在未来硬件上可以再高效 1/3”——这是针对 B300 / NVFP4 级硬件的前瞻性说明。今天的部署享受不到这一点。脚注 3 · “相比 BF16 GQA8 的 2% KV 缓存”是四层压缩叠加的产物，而非单一技巧。 1. 序列维度压缩（CSA m=4，HCA m'=128） 2. K 与 V 共享（MLA 风格，减半） 3. BF16 → FP8（非 RoPE 维度，再省约 50%） 4. 索引器 KV 使用 FP4（再降一档）相乘之后，就是 2%。每一层都要付出各自的信息损失代价。把这些数字当作事实来接受，但请记住它们是以检索质量下降为代价实现的。图 9 的 MRCR 8-needle 曲线——从 8K 时的 0.94 一路滑落到 1M 时的 0.59 / 0.49——才是这一代价的真实形状。 ## 3. 架构总览：V4 注意力是一个三通道并行的检索系统 ![Image](/doc/1210/image/img_767eba0ea88e.jpg) 在分别拆解 CSA/HCA 之前，先建立全局图景。V4 中的每个注意力层（前两层除外）实际上是一个三通道并行结构。在每一层，一个 query 会同时访问三条信息通道： ① HCA（全局摘要通道）压缩比 m'=128。1M token 被压缩为约 7,812 个条目；每个条目是对 128 个原始 token 的有损摘要。Query 对所有条目进行稠密注意力。覆盖整个 1M（全局），但每个点的精度严重退化。 ② CSA（稀疏精确读取通道）压缩比 m=4。1M token 被压缩为约 25 万个 4-token 块。Lightning 索引器对所有块打分，让 query 只关注最相关的 top-k=1024 个块。覆盖率 = 整个 1M，但实际被 query 读取的仅约 0.4%。 ③ SWA（局部保真通道）无压缩。最近 128 个 token 的 K/V 以完整保真度保留。只有最后 128 个 token，但被完整保留。 Query 实际“看到”的信息预算： | 通道 | 高保真比特 | 备注 | |---|---|---| | SWA | 128 token 当量 · 100% 保真度 | 局部上下文 | | CSA | 1,024 × 4 = 4,096 token 当量 · 4:1 有损 | 索引器挑选的精确读取区域 | | HCA | 7,812 × 128 = 完整 1M token · 128:1 有损 | 全局氛围 | 总计 ≈ 4,224 高保真比特 + 1M低保真比特 = “稀疏精确读取 + 模糊全局概览”。这是后续所有内容的锚点。三条通道没有一条能独立成立——SWA 只能看到 128 个 token；CSA 在全局上错过的内容就永远消失了；HCA 的 128:1 压缩信息过于稀疏，无法用于精确检索。这正是 V4 必须让三者并行运行、而不能只选其一的原因。 ## 4. CSA：压缩 + 稀疏选择，精确读取通道 ![Image](/doc/1210/image/img_852000fc3a8d.jpg) 4.1 两步设计 CSA（压缩稀疏注意力，Compressed Sparse Attention）通过两个步骤对信息进行“瘦身”： 1. 第 1 步——压缩：每 m=4 个连续 token 的隐藏状态通过可学习的 softmax 权重被压缩为 1 条 KV 条目。100 万 token → 25 万条条目。 2. 第 2 步——稀疏选择：一个独立的 Lightning Indexer 对全部 25 万条条目进行打分；查询只关注 top-k=1024 的条目。这就是“压缩 + 稀疏”带来的两层节省。每个查询实际上读取的是 1024 × 4 = 4096 个原始 token 的等价信息——但这 4096 个 token 是由索引器根据它认为最相关的内容挑选出来的，而不是查询真正需要的内容。如果索引器选错了，查询真正需要的块就彻底丢失了。 4.2 Lightning Indexer：一个廉价的 top-k 决策器 Lightning Indexer 在 V3.2 中首次引入，并被 V4 直接继承。它本质上是一个非常轻量的独立注意力子网络，仅用于打分，绝不参与最终输出： - 独立的索引器 KV（每 4 个 token 一条，head 维度仅为 128） - FP4 点积——比主注意力的 FP8 还要便宜 - ReLU 激活后接 top-k 这是一种漂亮的工程解耦：“决定往哪看”和“看什么”被分开了。但它带着一个根本性风险：索引器的排序质量决定了主注意力的上限。如果索引器把一条真正相关的条目排在第 1100 位，那么 top-k=1024 的查询就永远看不到它。论文没有提供索引器准确率的消融实验。在 1M 的 MRCR 上，V4 的 top-k 覆盖率为 1024 / 250000 ≈ 0.41%——这意味着索引器必须把 8 根“针”中每一根的压缩块都排进 top-1024。哪怕只有一根落在了外面，8 针任务就会丢掉一根针。 4.3 重叠压缩：V4 在 CSA 上的小创新 CSA 相对于 V3.2 的 DSA，真正新的部分是重叠压缩：同时生成两组 KV（C^a, C^b），覆盖交错的 4-token 块，然后在更宽的窗口上做跨块 softmax。好处是缓解“块边界处的信息断裂”——在朴素的压缩下，如果一根“针”横跨两个 4-token 块，它的语义就会被割裂。重叠压缩相当于步长为 50% 的卷积视角。代价是具体的：压缩前的 KV 显存翻倍（两组 K、V 而不是一组），CSA 层的参数量也翻倍。这部分代价被 §3.5.3 的上下文并行所吸收，但训练时的激活显存压力是真实存在的。 ## 5. HCA：重度压缩 + 全量稠密，粗粒度的全局摘要 ![Image](/doc/1210/image/img_927f54f4492f.jpg) 5.1 一个反直觉的设计选择 HCA（重度压缩注意力，Heavily Compressed Attention）采取了与 CSA 相反的哲学：维度 CSA HCA 压缩比 m = 4 m' = 128 压缩方式重叠（两组 KV）非重叠（单组 KV）是否稀疏选择？是（top-k=1024）否，全量稠密是否需要索引器？是否每条条目凝聚的 token 数 4 128 有趣的问题是：为什么 HCA 敢于“压缩之后直接稠密注意力”，而不做稀疏选择？答案在数字里。经过 m'=128 的压缩后，100 万 token 缩减为 1,000,000 / 128 ≈ 7,812 条条目。7,812 条 KV 完全在稠密注意力的预算之内——相当于在 8K 上下文上做稠密注意力，在 H100 上大约 4 GFLOPs。完全没问题。所以 HCA 的核心权衡是：“用极致的压缩换取稀疏选择复杂性的消除。”CSA 是“轻压缩 + 选择”；HCA 是“重压缩 + 全看”。两者都把注意力从 O(N²) 的量级拉到了可服务的范围，但机制完全不同。 5.2 信息密度的极限 m'=128 意味着每条压缩条目必须把 128 个原始 token 的语义塞进一个 c=512 维的向量中。考虑到原始隐藏状态本身是 d=4096~7168 维，128 个 token × 7168 维 ≈ 91.7 万维的信息被压到 512 维——压缩比高达 1800:1。这是 V4 架构中信息损失最严重的阶段。HCA 条目注定只能捕捉“全局氛围”，而不是“具体事实”。它们可以告诉查询“100 万上下文中 X 区域大致是关于什么的”，但无法精确说出“在 token 位置 543,217，变量 X 的具体名字是 Y”。这个观察直接对应图 9 的 MRCR 曲线：在精确检索任务上 ≥256K 处的下降，几乎可以逐字解读为“HCA 的语义粒度跟不上针级别的精度”。 ## 6. 滑动窗口：压缩失效时的本地安全网 6.1 一个打补丁式的设计单看 CSA + HCA，有一个严重的问题：因果性要求查询不能关注它自己所在的压缩块（否则会泄露未来 token 的信息）。这意味着：包括查询本身在内的最近一段 token，对 CSA 和 HCA 都是不可见的——CSA 看不到自己的 4-token 块；HCA 看不到自己的 128-token 块。但在语言建模中，最近的几个 token 恰恰是下一个 token 预测最关键的信号。这个矛盾必须解决，于是 V4 增加了第三条通道：滑动窗口注意力（SWA），其中每个查询额外关注最近 n_win = 128 个未压缩 token 的 K/V。 6.2 一个并非巧合的等式注意一个设计细节：n_win = m' = 128。这两个数字相等并非偶然——SWA 的窗口大小恰好等于 HCA 的压缩比，也就是说 SWA 覆盖的恰好是 HCA 因因果性而看不到的“当前 HCA 块”的范围。因此在工程上，三条通道形成了严格的互补分工： - SWA 覆盖最近的 128 个 token（当前 HCA 块的范围） - HCA 覆盖所有更早的已压缩 HCA 块（起始 → 约当前 − 128） - CSA 从所有 4-token 压缩块中挑选 top-k=1024，无位置限制这种划分说明了一件事：纯压缩注意力在工程上不可行，除非配一个未压缩的局部窗口作为兜底。SWA 是一个“优雅的补丁”，但它仍然是一个补丁——它的存在本身就承认了“压缩有其极限”。 ## 7. Attention Sink：让“什么都不看”成为一个合法选项这个技巧来自 StreamingLLM（Xiao 等，2024），GPT-OSS 也使用了它。V4 的实现方式：每个注意力头有一个可学习的标量 z'_h，加到 softmax 的分母中。结果是：注意力权重不再必须加总为 1。它们可以是 0.7、0.3，甚至接近 0。这个技巧在稀疏注意力下尤其重要。有时 CSA 选出的 top-k=1024 个条目中没有一个真正相关——比如查询是“这个变量的值是多少？”，但索引器把所有相关块都丢掉了。没有 sink 的话，softmax 会迫使注意力均匀分布在那 1024 个无关条目上，产生大量噪声。有了 sink，模型可以学到“这 1024 个我都不在乎——让注意力全部流向 sink”，这一层的输出几乎不受影响。 > sink 的真正含义：它让注意力层拥有了合法退出当前层计算的选项。没有它，稀疏注意力的失败模式是悄悄输出错误信号；有了它，失败模式变成输出接近零的信号，让下一层去恢复。前者是静默污染，后者是优雅降级。 ## 8. Partial RoPE：精心处理的位置编码 V4 在 CSA / HCA 上使用部分 RoPE——只对 query 和 KV 向量的最后 64 个维度应用 RoPE，其余维度与位置无关。这可以追溯到 V2 的 MLA，并被 V4 继承。但 V4 增加了一个有趣的小细节：输出侧的逆 RoPE。问题在于：CSA / HCA 的 KV 是 K=V 共享的（MQA 风格）；注意力输出是 Σ s_j × V_j。如果 V_j 的最后 64 个维度携带绝对位置编码（位置 j），那么 Σ s_j × V_j 会把不同位置的 RoPE 混成一个“位置的加权和”——既不是绝对也不是相对，基本上就是噪声。 V4 的解决方案：在输出侧，用位置 = -i 的 RoPE 旋转注意力输出。这使得输出的“有效位置”变成 j - i = -dist，也就是说，每个 V_j 对输出的贡献被按 query-key 之间的距离重新编码，自动转换成了相对位置编码。我没有在任何其他论文中见过这个技巧。从数学上看，它很优雅——它解决了“K=V 共享 + 绝对 RoPE”的内在矛盾。但同样，V4 论文没有提供消融实验，我们只能相信“它工作得不错”。 ## 9. 质量上的代价：图 9 中那条诚实的曲线 ![Image](/doc/1210/image/img_38c772491da7.jpg) 抛开架构，看数据。图 9 画出了 V4 在不同上下文长度下的 MRCR 8-needle 检索准确率：上下文 8K 16K 32K 64K 128K 256K 512K 1024K V4-Pro-Max 0.90 0.85 0.94 0.90 0.92 0.82 0.66 0.59 V4-Flash-Max 0.91 0.84 0.87 0.85 0.87 0.76 0.60 0.49 作为对比，前沿模型在 MRCR 1M 8-needle 上的表现： - Claude Opus 4.6: 0.929 - Gemini 3.1 Pro: ~0.92（论文报告值） - V4-Pro-Max: 0.835（表 7）几个值得指出的事实：事实 1：在 128K 以内，V4-Pro 接近前沿水平（0.92 vs 0.93）。1M 才是差距真正显现的地方——在那里 V4 落后 Claude 9 到 10 个绝对百分点。事实 2：V4-Pro 在 1M 下的得分（0.59）优于 V4-Flash（0.49）。更大的总参数量 + 更大的 top-k（1024 vs 512）直接有助于长上下文检索——Pro 拥有的“高保真比特”是 Flash 的两倍。事实 3：图 9 的曲线在 32K 处有一个“反弹”，达到 0.94，然后单调下降。1M 下的 0.59 比 8K 下的 0.90 低了 34 个百分点——这就是“1M 上下文的质量代价”背后的具体数字。论文的措辞值得引用： > 检索性能在 128K 上下文窗口内保持高度稳定。虽然在超过 128K 之后性能下降变得可见，但模型在 1M token 下的检索能力相对于专有和开源同类模型仍然非常强大。 “相对于同类模型非常强大”是真的——在开源同类中，V4 确实领先。但被一笔带过的是 128K 之后从 0.92 跌到 0.59。对严谨的技术读者来说，那条曲线是 V4 工程权衡最清晰的证据。 ## 10. 一个反直觉的观察：V4 在 1M 的得分竟然比 128K 更高？注意表 7 中的这组数字： MRCR 8-needle 128K MRCR 8-needle 1M V4-Pro-Max 78.6 83.5 V4-Pro 78.4 81.0 1M 的得分比 128K 还高？这与图 9 的趋势（同一任务、更多长度）相矛盾——图 9 是单调递减的。表 7 中的这两个数字是“特定长度下的官方亮点分数”，它们显示 1M > 128K。论文没有解释这个异常。两个猜测： 1. 训练分布：V4 后期的长上下文训练集中在 1M 附近；模型对这个长度“更熟悉”。128K 只是训练计划中的一个“中转点”，可能没有得到充分巩固。 2. HCA 的甜点位置：一条 1M 的序列在 m'=128 的压缩下产生 7,812 个稠密条目——恰好是 HCA 的设计最优值。一条 128K 的序列只能产生 1,000 个条目，过于稀疏，无法让 HCA 的稠密优势发挥出来。无论哪种情况，这个异常告诉我们一件事：V4 的长上下文能力是“为 1M 训练的”，而不是“跨长度通用”。如果你的真实工作负载在 200K~500K，你可能并没有落在论文强调的甜点位置上。 ## 11. 把所有拼图重新拼起来前面 10 节可以压缩成一句话： > V4 的注意力并不是注意力——它是一个“有损压缩 + 稀疏检索 + 局部保真 + 显式弃答”的复合检索系统。在每一层，一个查询同时咨询三种不同粒度的视图（SWA 局部、CSA 精确、HCA 全局），依赖索引器的排序质量、注意力汇的“放弃”机制，以及部分 RoPE 的位置耦合，把“看似不可能”的 1M 工程化成“日常可用”。这套设计的真实定位不是“创新”——而是对既有、各自已被验证的组件做工程级组装： - Lightning Indexer + top-k：V3.2 DSA - 潜空间查询 / 低秩 K=V 共享：V2 MLA - 部分 RoPE：V2 起 - 滑动窗口：Mistral / Gemma 主流 - 注意力汇：StreamingLLM、GPT-OSS - 压缩 + 稠密交错：Native Sparse Attention 谱系 V4 的真正贡献是工程组合 + 端到端的 1M 可用性：让这些组件在稳定训练过的 32T token、1.6T 参数的 MoE 中共存，并配以定制的 KV cache 管理、稀疏注意力内核和上下文并行训练基础设施。这是扎实的系统工作，但它不是“无损的 1M 突破”——它是“以可接受的精确检索退化为代价，把 1M 上下文的边际成本拉入可接受范围”的工程胜利。 ## 12. 给读者的要点当开源模型宣称 1M 上下文时 - 问一下注意力是否是稠密的。如果不是，它的 1M 就是有损的 1M。 - 索要 MRCR 1M 8-needle 分数。0.85 以上是前沿水平；0.5–0.6 意味着“能跑，但不能用于精确检索”。 - 索要压缩比。HCA 式的 128:1 压缩注定了某些精确任务无法胜任——这不是“更多训练”能修复的。当设计长上下文应用时 - 不要用 V4 做严格的针式检索。用 Claude Opus 4.6 / Gemini 3.1 Pro，或者干脆在前面加 RAG。 - V4 的甜区是长文档推理（例如 CorpusQA，V4-Pro-Max 得分 33.6，Gemini 为 35.4）——需要“全局氛围 + 局部细节”而非点对点精确匹配的任务。 - 适合的场景：大型代码库分析、多文档摘要、智能体长轨迹维护。 - 不适合的场景：法律条款检索、从医学文献中提取具体数字 / 命名实体、在长转录中引用精确时间戳。当做架构研究时 - 有损长上下文如今已是共识路线。Native Sparse Attention、MLA、DSA、线性注意力、状态空间混合——它们都在这条路上，只是压缩 / 稀疏 / 状态的比例不同。 - V4 的真正启示是“分层”：不同层使用不同压缩策略（CSA / HCA 交错）；不同通道使用不同精度（SWA 全量 + CSA 稠密 + HCA 稀疏）。这种混合明显优于任何单一策略。 - 三个值得追问的开放问题：索引器 top-k 排序准确率如何消融？它是 CSA 上限的瓶颈。 HCA 的 m'=128 是经验最优吗，m'=64 / 256 是否也可行？论文没有扫这条曲线。三条通道间的注意力权重分配是学出来的——是否在某些层里注意力汇已经“关掉了整条 HCA 通道”？那将意味着 HCA 的实际贡献被高估了。 ## 13. 结语 “1M 上下文”这个规格参数在 2025–2026 年的开源模型竞赛中被反复刷新。V4 的真正贡献在于，它第一次完整写下了“1M 到底是如何构建的”这个问题的答案——不是“我们加宽了注意力窗口”，而是“我们用五种不同的压缩 / 选择 / 窗口机制把注意力切成了碎片，每种机制处理它最擅长的粒度”。这种透明值得鼓励。但读者——尤其是工程读者——需要看到规格数字背后隐藏的架构代价： - 1M 的 KV cache 仅为 BF16 GQA8 的 2%，因为信息已经被挤过了四层压缩 - 1M 的 FLOPs 仅为 V3.2 的 27%，因为注意力已经从 O(N²) 退化成“分层近似，大致 O(N · log N)” - 1M 的检索准确率从 8K 时的 0.94 滑落到 0.59，因为这个近似有精度上限，而这个上限不会被训练数据治愈下次有人告诉你“1M 上下文模型来了，RAG 已死”，把图 9 的曲线发给他——那条从 0.94 一路滑到 0.59 的曲线。 1M 上下文 ≠ 无损长记忆。它是一场工程权衡的胜利，也是一次诚实的披露：对某些任务，你仍然需要 RAG，仍然需要稠密注意力，仍然需要其他外部记忆系统。 V4 没有解决长上下文。V4 只是把“长上下文的可用边界”向前推了一大截。这已经足够了。参考文献：DeepSeek-AI, "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence" (2026); Hugging Face Blog, "DeepSeek-V4: a million-token context that agents can actually use"; NVIDIA Developer Blog, "Build with DeepSeek V4 Using NVIDIA Blackwell"; Andrew Lukyanenko, "DeepSeek-V4 Review: Why Million-Token Context Needs Efficient Attention" (Medium, Apr 2026); BigDataBoutique, "Needle in haystack? Optimizing Retrieval and RAG over Long Context Windows"; Liu et al., "Lost in the Middle: How Language Models Use Long Contexts"; Xiao et al., "Efficient Streaming Language Models with Attention Sinks" (StreamingLLM). ## 相关链接 - [GoGoNN](https://x.com/hhjjjFinFin) - [@hhjjjFinFin](https://x.com/hhjjjFinFin) - [8.2K](https://x.com/hhjjjFinFin/status/2048773590942970275/analytics) - [升级到 Premium](https://x.com/i/premium_sign_up) - [2026年4月27日晚上10:37](https://x.com/hhjjjFinFin/status/2048773590942970275) - [8,277 次浏览](https://x.com/hhjjjFinFin/status/2048773590942970275/analytics) --- *导出时间: 2026/4/28 15:26:44*