# Kimi K3's memory savings vs Jevon's paradox - a deep technological & historical analysis...
**作者**: GDP
**日期**: 2026-07-29T08:07:29.000Z
**来源**: [https://x.com/bookwormengr/status/2082377457852920075](https://x.com/bookwormengr/status/2082377457852920075)
---
One of the details that has caught X community's imagination is that Kimi K3 has constant state. In other words it does not have linearly growing KV cache as the context grows, or your conversation lengthens (for the most part - we will see details and caveats soon).
Long context use cases - e.g. agentic coding - has increased memory demand massively, resulting in 10 to 30 times increase in prices of memory stocks. If you don't require as much memory to store KV cache, is this price increase justified? There is a vibrant debate going on this topic on X.
The detractors say Hybrid Linear attention used by Kimi K3 called Kimi Delta Attention (KDA) is around for quite some time. So it is already factored in prices of memory stocks.
However, they miss the point that - unlike previous attempts - Kimi K3 proved that Hybrid Linear could work so well at 1T+ parameter scale and deliver frontier performance. This is NET NEW information and deserves serious attention. Before Kimi K3 Hybrid Linear was believed not to be a technique for frontier but just a neat trick used by likes of
- Qwen (who never quite get to the frontier) or
- Nvidia (who tries everything under the sun, just because they have infinite GPUs).
Let us dive in!
(I want to make it clear - THIS IS NOT INVESTMENT ADVICE).
## Brief history of Hybrid Linear:
Linear models always held great promise on paper just like Communism.
Instead of constantly growing KV cache, as context grew; they have a constant state that gets updated. The state size is constant whether you have thousand tokens or a million tokens.
It sounds to good to be true, because it is! Linear models have issues with recalling content from earlier parts of conversation perfectly - e.g. telephone number from earlier part of the conversation.
Why? This is because such models are constantly updating their internal 'summary' (called state) of the context. However, since this context/summary is fixed in size (e.g. 128*128 in case of Kimi K3, per head), its storage capacity is limited. It compresses information to be efficient (that is what the model learns during its training), but it has to drop some information for sure. In other words, the compression is lossy.
One way to solve the problem is to go for 'hybrid linear' where you combine linear layers (typically 75%) with full attention layers (25%). Hybrid Linear can have efficient version of full attention as well (e.g. DeepSeek style MLA in case of Kimi K3). While this was attempted, the earlier hybrid linear models did not have performance at the frontier level!
What Kimi K3 showed is that - with their variation called Kimi Delta Linear Attention (KDA) - you can achieve near perfect recall, as good as, that of models using full attention in all layer. This is a massive breakthrough, whatever you have been given to believe about it....
## How did Kimi K3 achieve it?
Moonshot are not first to use hybrid linear, but they came up with a variation of it that is highly effective. It allows fine grained control over memory decay.
Off course they did many other things, involving data engineering, kernel development etc., use of attention residuals, stable latent MoE etc. (we will understand them towards the end). But they did something seemingly simple that resulted in exceptional performance gain on the attention part.
The model's KDA state can essentially be thought of as a stack of memories (over simplified version). At every new token position a diagonal matrix is computed and the state is multiplied with it. This results in forgetting some memories more and some memories less (jargon: per channel decay). In earlier versions you would forget all the memories to equal extent. This trick of selectively and judiciously forgetting significantly improved the performance!
This is a beautiful visualisation that shows how state matrix updates happen. I will have another blog coming that explains, how this leads to the holy grail "continuous learning".
## How much memory saving is there with Kimi K3:
K3 has 93 layers and 96 heads. KDA has 128 dv and dk dimensions.
Let us see what it takes to process 128K tokens and 1 million tokens in terms of state and KV storage. Let us see two cases:
1) 93 MLA full attention layers (All-MLA) vs
2) 69 KDA + 24 MLA full attention (actual Kimi K3 model as depicted above)
At the bottom of the article I have posted the calculations.
Key observations:
1. For All-MLA, KV cache storage grows from 13.7 GB to 107 GB from 128K to 1M tokens.
2. As you can see, KDA state for Kimi K3 is constant at 0.22 GB regardless of the context length. Only the MLA part grows from 3.5 GB to 29 GB from 128K to 1M tokens context.
In both cases, there is saving of about 73%. The beauty is KDA does not compromise on frontier performance while offering these saving. MLA is already a massive saving over MHA and GQA (almost 90%), KDA save 73%!
Some will argue savings with DeepSeek V4 architecture at Kimi K3 scale would be even higher. True, DeepSeek V4 is not at the frontier yet. And even if it reaches the frontier, it will improve in exactly same direction: less storage.
This beautiful visualisation produced by Kimi K3 itself shows the savings:
## Is Kimi K3 unique in the industry when it comes to such dramatic memory saving?
DeepSeek V4 (https://arxiv.org/pdf/2606.19348v1) has done exceptional work on cutting KV cache by 98%, but nobody considers them near frontier. When DeepSeek V4 was launched did you observe the same level of activity on X?
Also, for all their brilliance all the layers in DeepSeek V4 are either CSA (Compressed Sparse Attention) of HCA (Highly Compressed Attention) - both types are compressed along the sequence dimension and it is lossy. There is not a single layer that has uncompressed information, that is all the layers are lossy - recall in Kimi K3 25% of layers have full attention.
My hypothesis is that Kimi K3 will have better ability to retrieve specific information from earlier context in long context scenarios. Hence, I foresee Kimi K3 architecture will be widely adopted by other labs as well (e.g. ZAI, MiniMax, Qwen, Tencent, Xiomi etc.), just like DeepSeek V3 (MLA) and later DeepSeek V3.2 (MLA + DSA) became widespread in the industry.
DeepSeek's CSA. Notice the compression along sequence dimension. It results in 1 compressed tokens for every 4 tokens.
DeepSeek's HCA. Notice the compression along sequence dimension. It results in 1 compressed tokens for every 128 tokens.
## What about Funda AI's point that you need to cache KDA state multiple times, as it is not additive like KV cache? Also, will subagents not need KV cache storage?
Recall the previous example, total KDA state across 69 layers is 0.22 GB. So, even when you cache/snap-shot it at every 20K tokens during a long session, it is 50 times across 1M context.
So theses 50 spanshots will take 50*0.22 GB. It all adds up to just 11GB - total l (29.22 GB + 11 GB) stays well below 107 GB required for All MLA at 1M tokens context.
Second, Funda AI's point was that main agents can spawn subagents. People will spawn many subagents. But typically, once a subagent returns the control to main again, its job is done. Its KV + state is deleted. So even if people launch agents that launch hundreds of subagents, KV + state needs are not long term. The main session / main agent's state needs to be preserved much longer; and this where KDA saves a lot!!!
There are patterns where there are long running async agents, in those cases their KV + state would need to be maintained. But I see those patterns implemented rarely. This is not to say it won't change in future.
Synchronous subagents (a very common pattern):
Asynchronous subagents (a relatively rare pattern)
## Will KDA's adoption then will have downward impact on memory demand then?
I am not a fortune teller, but there is a small and non-zero probability of it busting memory bubble.
First why it may not have any impact on memory demand:
- While savings are real, if open source AI gets widely adopted, most deployments are not going to be as efficient as they are at closed frontier labs. Hence, any gains at the model level will be lost at industry level. We will have large number of small deployments that are not very efficient. E.g. for most enterprises & governments use happens across 12 hrs in a day. What their open source model infra and its memory going to do at night? Infra at large labs is used 24*7 due to global customer base.
- Also, we are making a big assumption that leading labs already don't have similar approaches. OpenAI, Anthropic, DeepMind employ some of the smartest people in the world. So there is quite good chances that such innovations are already in use and have been factored into the memory demand.
That said every once in a while there is a SpaceX that comes and shows the industry leaders how their approaches were inefficient. They think from the first principals. Incumbents - due to organization structure issues, or due to not being hungry enough, do not innovate on key dimensions. History of capitalism is filled with thousands of such examples....
So it is an unlikely scenario - but still with some non-zero probability - that if large labs were not as starved for resources, they may not have explored the entire design space to reduce memory demand to such an extent.
In the AI industry itself, it has happened many times before where big labs have shown gross incompetence:
1. XAI's training pipeline had poor MFU (GPU utilisation) - due to habit of throwing compute at the problem, plus other cluster misconfiguration and organization level issues.
2. Despite all the resources at their disposal, Meta did train a sh*tty version of DeepSeek's MoE for Llama 4.
3. Google took 1.5 years since ChatGPT launch to bring Gemini to the forefront and it is still no one's favourite model for the most popular use case: AI coding.
Constraints force innovation, so it is not unlikely that small highly well organised teams outshine large resource rich teams. And now that K3 is out and its architecture is well know, if KDA like technique - that is now successful at the frontier scale - is adopted by closed labs too; it means 75% reduction in memory demand at least for a few labs, and they are quite big individually, so the impact could be in tens of billions.
These memory savings could be eaten up with increased use due to reduction in cost for sure (Jevon's paradox), but it is not guaranteed that consumption growth will always outstrips the savings....
## Diffusion of other innovations: Kimi K3's major innovation is not KDA alone!
Kimi K3 gets rid of positional encoding completely: Positional encodings inform what is sequence number of each token in the context. Its KDA layers, due to their recurrent nature do not need position encodings. They get rid of them even for MLA layers and it does not hurt. Benefits? It does not any special effort to extend model context length compared to earlier models. Elegant.
Kimi K3 implements attention residuals at 2.8T scale: It ensures, effectively, that higher layers are able to pay selective attention to inputs from lower layers. In earlier approaches higher layers could not distinguish between inputs from lower layers at a given token position, as all the values used to be aggregated. This new approach gives the model much higher discernibility. (https://arxiv.org/pdf/2603.15031)
Kimi K3 implements Nvidia's Stable Latent MoE: The main benefit of a stable latent Mixture-of-Experts (LatentMoE) architecture is higher model accuracy and more expert capacity at the same or lower inference cost. It achieves this by projecting token representations into a smaller latent space before running expert routing and computation. They picked if from Nvidia's Jan 2026 paper (https://arxiv.org/pdf/2601.18089v1)
With all these innovations put together, MoonShot team achieve 2.5 training efficiency over their previous architecture Kimi K2. This is to say, they achieve same validation loss at 2.5 times less FLOPs. That is massive saving even for training compute!!!! This will allow many more labs to train such high scale models with limited compute.....
With K3 Moonshot unlocked new scaling laws. Kimi K3 achieves 2.5× gain in scaling efficiency over Kimi K2
MoonShot has also open sourced their FlashKDA GPU kernel for fast computation of KDA state and MoonMoE for dispatch and combine of tokens. They are pretty confident in sharing so much knowledge, because, they are already onto the next set of innovations that will take then even futher (at the end of day, they too are capitalists).
## Scaling laws are not laws on nature, we can always discover better
Furthermore, we may discover newer scaling laws with newer architectures and training paradigms, offering even denser intelligence. Scaling laws are statistical observations, they are not laws of nature that can not be changed. What this means is that with newer models, for smaller size (and lower KV and state) we can have better performance. So demand explosion does not mean running the same large models forever. Models will continue to get denser in terms of intelligence.
We have an ecosystem in form of Chinese labs that is highly constrained, and highly capable - a very potent mix, as discussed. They are discovering corners of the design space that have not been discovered by more well funded labs. They make rapid progress due to open sharing of knowledge. This saves others from wasting their efforts and reinventing the wheel. The evolution of architecture happen much more rapidly than otherwise....
Also, this success by MoonShot team is sure to inspire other equally capable teams across DeepSeek, ZAI, MiniMax, Tencent, Alibaba, ByteDance SEED and even emerging players like Baidu, Ant Ling, Xiomi Mimo, Meituan and many others. to be the best, not just as an open source model, but to be the best overall.
Lot of new innovations will happen across emerging labs in the west, as well, whether it is at xAI or MSL now they have gotten their act together and started delivering great models.
That said, as I keep pointing out, China has most number of young AI researchers based on NEURIPS statistics. Contributions in this field come disproportionately more from younger researchers (the source thread), so many more breakthrough are likely to emerge from that part of the world.
China also has 3 major urban centers with AI talent density rivaling that in the Silicon Valley, they act as idea mixing crucibles (the source thread).
## Banning open source AI will not help:
Whether open source AI is banned or not in the western world, these design spaces will get explored, and papers will get published. West also has lot of new great labs emerging. They will discover lot of new ideas too, now that it has been realised there could be lot of efficiency gains that are still possible.
The best thing is to encourage and adopt these innovation and be responsible with investments and avoid wild speculation.
The future is very very bright due to this technology, but we have to be careful not be crazy speculators who get margin called. We haven't been using as much AI as we could be: whether it is in education or research. E.g. I would love for every student in India to have unlimited access to Opus 4.8 or Kimi K3 level models. They do have access, but to weaker models. Furthermore, we could have much higher value use cases of AI in Science, Engineering and Medicine. Also, we have gotten an aging world that we need to care for. Ai is going to be greatly beneficial there. There exist so many possibilities across entertainment and gaming as well.
## Conclusion:
While the future is uncertain, what is clear in my observations is that, AI bulls and memory bulls, in particular, do not factor in algorithmic innovations and discovering of newer scaling laws. For them everything gets eaten by Jevon's paradox, hence not interesting. I want to introduce new angles into their analysis.
To crash memory stocks, we don't need massive reduction in demand, as Jaya Gupta has pointed out. Even small reduction in demand can start a game theoretical competition wherein existing bag holders start booking profits with intention to buy stocks later, this can crash the whole market. And this seems to have begun.....
It is important to not let such large bubbles form and burst catastrophically for the society. Overall I stay very optimistic about potential of this technology (and even memory demand) and future infra build out. It is the most exciting time to be alive!
(NOT INVESTMENT ADVICE)
# Appendix
## Detailed calculation of memory savings:
To show how good Kimi K3 is, I got K3 itself to do these calculation I have verified they are correct. As mentioned earlier we consider two cases:
1) Kimi K3: 69 KDA + 24 Gated MLA full attention vs
2) Full MLA: 93 MLA full attention
## 相关链接
- [GDP](https://x.com/bookwormengr)
- [@bookwormengr](https://x.com/bookwormengr)
- [123K](https://x.com/bookwormengr/status/2082377457852920075/analytics)
- [https://arxiv.org/pdf/2606.19348v1](https://arxiv.org/pdf/2606.19348v1)
- [https://arxiv.org/pdf/2603.15031](https://arxiv.org/pdf/2603.15031)
- [https://arxiv.org/pdf/2601.18089v1](https://arxiv.org/pdf/2601.18089v1)
- [source thread](https://x.com/bookwormengr/status/2009503245350416441?s=20)
- [source](https://x.com/bookwormengr/status/2009503248756166769?s=20)
- [Jaya Gupta has pointed out.](https://x.com/JayaGup10/status/2082301754872369331)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [4:07 PM · Jul 29, 2026](https://x.com/bookwormengr/status/2082377457852920075)
- [123.4K Views](https://x.com/bookwormengr/status/2082377457852920075/analytics)
- [View quotes](https://x.com/bookwormengr/status/2082377457852920075/quotes)
---
*导出时间: 2026/7/30 10:11:20*
---
## 中文翻译
# Kimi K3 的内存节约与杰文斯悖论——一份深层技术与历史分析……
**作者**: GDP
**日期**: 2026-07-29T08:07:29.000Z
**来源**: [https://x.com/bookwormengr/status/2082377457852920075](https://x.com/bookwormengr/status/2082377457852920075)
---
引起 X 社区想象力的一个细节是，Kimi K3 具有恒定状态。换句话说，随着上下文的增长，或者你的对话变长，它并没有线性增长的 KV 缓存（大部分情况下——我们很快就会看到细节和注意事项）。
长上下文用例——例如智能体编码——极大地增加了内存需求，导致内存股票价格上涨了 10 到 30 倍。如果你不需要那么多的内存来存储 KV 缓存，这种价格上涨是否合理？关于这个话题，X 上正在进行一场激烈的辩论。
批评者说，Kimi K3 使用的混合线性注意力（称为 Kimi Delta Attention，即 KDA）已经存在相当长一段时间了。所以这已经被计入内存股票的价格中了。
然而，他们忽略了这一点——与之前的尝试不同——Kimi K3 证明了混合线性在 1T+ 参数规模下可以如此出色地工作并提供前沿性能。这是全新的信息，值得认真关注。在 Kimi K3 之前，混合线性被认为不是一种用于前沿模型的技术，只是被以下各方使用的一个巧妙技巧：
- Qwen（从未真正达到过前沿水平），或
- Nvidia（尝试了太阳下的一切方法，仅仅因为他们拥有无限的 GPU）。
让我们深入探讨吧！
（我想澄清一下——这不是投资建议）。
## 混合线性简史：
线性模型在纸面上总是像共产主义一样前景光明。
随着上下文的增长，它们不是不断增长的 KV 缓存，而是拥有一个会被更新的恒定状态。无论你有一千个 token 还是一百万个 token，状态大小都是恒定的。
这听起来好得不真实，因为它确实如此！线性模型在完美回忆对话 earlier 部分的内容方面存在问题——例如对话 earlier 部分的电话号码。
为什么？这是因为此类模型不断更新其上下文的内部“摘要”（称为状态）。然而，由于此上下文/摘要的大小是固定的（例如在 Kimi K3 的情况下每个头为 128*128），其存储容量是有限的。它会压缩信息以提高效率（这就是模型在训练期间学习的内容），但它肯定会丢失一些信息。换句话说，这种压缩是有损的。
解决这个问题的一种方法是采用“混合线性”，即将线性层（通常为 75%）与全注意力层（25%）结合在一起。混合线性也可以拥有高效版本的全注意力（例如在 Kimi K3 情况下的 DeepSeek 风格 MLA）。虽然有过尝试，但早期的混合线性模型没有达到前沿水平的性能！
Kimi K3 表明——通过他们称为 Kimi Delta Linear Attention (KDA) 的变体——你可以实现近乎完美的回忆，与在所有层中使用全注意力的模型一样好。这是一个巨大的突破，无论你对此有什么看法……
## Kimi K3 是如何做到的？
Moonshot 不是第一个使用混合线性的公司，但他们提出了一种非常有效的变体。它允许对内存衰减进行细粒度的控制。
当然，他们做了许多其他事情，涉及数据工程、内核开发等，以及使用注意力残差、稳定的潜在 MoE 等（我们将在最后理解它们）。但他们做了一些看似简单的事情，从而在注意力部分产生了显着的性能提升。
模型的 KDA 状态本质上可以被认为是一堆记忆（过度简化的版本）。在每个新的 token 位置，计算出一个对角矩阵，状态与它相乘。这会导致一些记忆被遗忘得更多，一些被遗忘得更少（术语：每通道衰减）。在早期版本中，你会同等地忘记所有记忆。这种有选择性和明智地遗忘的技巧显着提高了性能！
这是一个美丽的可视化图，展示了状态矩阵是如何更新的。我将写另一篇博客来解释，这如何导致“圣杯”——“持续学习”。
## Kimi K3 能节省多少内存：
K3 有 93 层和 96 个头。KDA 有 128 个 dv 和 dk 维度。
让我们看看在状态和 KV 存储方面处理 128K token 和 100 万 token 需要什么。让我们看两种情况：
1) 93 个 MLA 全注意力层 对比
2) 69 个 KDA + 24 个 MLA 全注意力（如上图所示的实际 Kimi K3 模型）
在文章底部我发布了计算结果。
主要观察结果：
1. 对于 All-MLA，KV 缓存存储从 128K token 时的 13.7 GB 增长到 1M token 时的 107 GB。
2. 如你所见，无论上下文长度如何，Kimi K3 的 KDA 状态恒定为 0.22 GB。只有 MLA 部分从 128K 上下文时的 3.5 GB 增长到 1M 上下文时的 29 GB。
在这两种情况下，节省都约为 73%。美妙的是，KDA 在提供这些节省的同时并未牺牲前沿性能。MLA 相比 MHA 和 GQA 已经是巨大的节省（几乎 90%），KDA 节省了 73%！
有些人会认为，在 Kimi K3 规模下使用 DeepSeek V4 架构的节省会更高。的确，DeepSeek V4 尚未达到前沿水平。即使它达到了前沿，它也会朝着完全相同的方向改进：更少的存储。
这张由 Kimi K3 本身生成的精美可视化图展示了节省情况：
## 在如此剧烈的内存节省方面，Kimi K3 在行业中是独一无二的吗？
DeepSeek V4 (https://arxiv.org/pdf/2606.19348v1) 在将 KV 缓存减少 98% 方面做出了出色的工作，但没有人认为他们接近前沿水平。当 DeepSeek V4 发布时，你在 X 上观察到同样水平的活动了吗？
此外，尽管他们才华横溢，DeepSeek V4 中的所有层要么是 CSA（压缩稀疏注意力），要么是 HCA（高度压缩注意力）——这两种类型都沿着序列维度进行压缩，并且是有损的。没有一层具有未压缩的信息，即所有层都是有损的——回想一下 Kimi K3 中 25% 的层具有全注意力。
我的假设是，在长上下文场景中，Kimi K3 从 earlier 上下文中检索特定信息的能力会更好。因此，我预见 Kimi K3 架构也将被其他实验室广泛采用（例如 ZAI、MiniMax、Qwen、Tencent、Xiomi 等），就像 DeepSeek V3 (MLA) 和后来的 DeepSeek V3.2 (MLA + DSA) 在行业中变得广泛一样。
DeepSeek 的 CSA。注意沿序列维度的压缩。结果每 4 个 token 产生 1 个压缩 token。
DeepSeek 的 HCA。注意沿序列维度的压缩。结果每 128 个 token 产生 1 个压缩 token。
## 关于 Funda AI 指出的你需要多次缓存 KDA 状态，因为它不像 KV 缓存那样是可加的观点呢？另外，子智能体不需要 KV 缓存存储吗？
回想一下前面的例子，69 层的总 KDA 状态是 0.22 GB。因此，即使你在长会话期间每 20K token 缓存/快照它一次，在 1M 上下文中也是 50 次。
所以这 50 个快照将占用 50*0.22 GB。总共加起来只有 11GB——总量 (29.22 GB + 11 GB) 远低于 1M token 上下文下 All MLA 所需的 107 GB。
其次，Funda AI 的观点是主智能体可以生成子智能体。人们会生成许多子智能体。但通常，一旦子智能体将控制权交还给主智能体，它的任务就完成了。它的 KV + 状态被删除。因此，即使人们启动的智能体又启动了数百个子智能体，KV + 状态需求也不是长期的。主会话/主智能体的状态需要保留更长时间；而这就是 KDA 节省了大量内存的地方！
有些模式涉及长时间运行的异步智能体，在这些情况下，它们的 KV + 状态需要被维护。但我看到这些模式很少被实现。这并不是说未来不会改变。
同步子智能体（一种非常常见的模式）：
异步子智能体（一种相对罕见的模式）：
## 那么 KDA 的采用会对内存需求产生下行影响吗？
我不是算命先生，但它刺破内存泡沫的概率很小但非零。
首先为什么它可能对内存需求没有任何影响：
- 虽然节省是真实的，但如果开源 AI 被广泛采用，大多数部署将不会像封闭的前沿实验室那样高效。因此，模型层面的任何收益都会在行业层面流失。我们将有大量的小规模部署，效率不是很高。例如，对于大多数企业和政府来说，使用发生在一天中的 12 小时内。他们的开源模型基础设施及其内存在晚上会做什么？大型实验室的基础设施由于全球客户群，是 24*7 使用的。
- 此外，我们做了一个很大的假设，即领先的实验室还没有类似的方法。OpenAI、Anthropic、DeepMind 雇佣了世界上最聪明的一些人。所以很有可能这样的创新已经在使用，并且已经被计入内存需求中。
话虽如此，每隔一段时间就会出现一家 SpaceX，向行业领袖展示他们的方法是多么低效。他们从第一性原理思考。由于组织结构问题，或者由于不够渴望，现有企业不会在关键维度上进行创新。资本主义历史上充满了成千上万个这样的例子……
所以这是一种不太可能的情况——但仍有一些非零概率——即如果大型实验室没有如此渴望资源，他们可能不会探索整个设计空间来将内存需求减少到这种程度。
在 AI 行业本身，这种情况以前发生过很多次，大实验室表现出极度的无能：
1. XAI 的训练流水线 MFU（GPU 利用率）很低——由于习惯向问题投入算力，加上其他集群错误配置和组织层面的问题。
2. 尽管拥有所有资源，Meta 确实为 Llama 4 训练了一个 DeepSeek MoE 的糟糕版本。
3. Google 在 ChatGPT 发布后花了 1.5 年时间才让 Gemini 走到前台，它仍然不是最受欢迎的用例：AI 编码的最爱模型。
限制迫使创新，所以小型高度组织的团队胜过资源丰富的大型团队并不罕见。现在 K3 已经发布，其架构众所周知，如果像 KDA 这样的技术——现在在前沿规模上取得了成功——也被封闭实验室采用；这意味着至少对于一些实验室来说内存需求减少了 75%，而且它们各自都非常大，所以影响可能达到数百亿美元。
由于成本降低，这些内存节省肯定会因使用增加而被抵消（杰文斯悖论），但不能保证消费增长总是会超过节省……
## 其他创新的扩散：Kimi K3 的主要创新不仅仅是 KDA！
Kimi K3 完全摆脱了位置编码：位置编码通知每个 token 在上下文中的序列号。由于其循环性质，其 KDA 层不需要位置编码。他们即使在 MLA 层中也摆脱了位置编码，但这并没有造成伤害。好处？与以前的模型相比，它不需要任何特殊的努力来扩展模型上下文长度。优雅。
Kimi K3 在 2.8T 规模下实现了注意力残差：它有效地确保了更高层能够选择性关注来自低层的输入。在早期的方法中，更高层无法区分给定 token 位置来自低层的输入，因为所有的值曾经被聚合。这种新方法赋予了模型更高的辨别力。(https://arxiv.org/pdf/2603.15031)
Kimi K3 实现了 Nvidia 的 Stable Latent MoE：稳定的潜在混合专家架构的主要好处是，在相同或更低的推理成本下，模型准确率更高，专家容量更大。它通过在运行专家路由和计算之前将 token 表示投影到更小的潜在空间来实现这一点。他们从 Nvidia 2026 年 1 月的论文中选取了这一点 (https://arxiv.org/pdf/2601.18089v1)
将所有这些创新结合在一起，MoonShot 团队实现了比其以前架构 Kimi K2 高 2.5 倍的训练效率。也就是说，他们以 2.5 倍少的 FLOPs 实现了相同的验证损失。即使对于训练算力来说，这也是巨大的节省！！！！这将允许更多实验室用有限的算力训练如此高规模的模型……
凭借 K3，Moonshot 解锁了新的缩放定律。Kimi K3 实现了比 Kimi K2 高 2.5 倍的缩放效率增益
MoonShot 还开源了他们的 FlashKDA GPU 内核，用于快速计算 KDA 状态，以及用于 token 分发和组合的 MoonMoE。他们非常自信地分享了如此多的知识，因为他们已经在进行下一组创新，这将使他们走得更远（归根结底，他们也是资本家）。
## 缩放定律不是自然定律，我们总是能发现更好的
此外，我们可能会随着新的架构和训练范式发现更新的缩放定律，提供甚至更密集的智能。缩放定律是统计观察，它们不是不可改变的自然定律。这意味着通过较新的模型，对于较小的尺寸（以及较低的 KV 和状态），我们可以有更好的性能。因此，需求爆炸并不意味着永远运行相同的大模型。模型将在智能方面继续变得更加密集。
我们有一个由高度受限且高度有能力的中国实验室组成的生态系统，正如所讨论的那样，这是一个非常强大的组合。他们正在发现资金更充裕的实验室尚未发现的设计空间角落。由于知识的开放共享，他们取得了快速进步。这使其他人免于浪费精力并重新发明轮子。架构的进化比 otherwise 要快得多……
此外，MoonShot 团队的这一成功肯定会激励 DeepSeek、ZAI、MiniMax、Tencent、Alibaba、ByteDance SEED 甚至像 Baidu、Ant Ling、Xiomi Mimo、Meituan 等新兴参与者中其他同样有能力的团队成为最好的，不仅是作为开源模型，而是成为整体上最好的。
西方新兴实验室也会发生许多新的创新，无论是在 xAI 还是 MSL，现在他们已经整理好状态并开始提供出色的模型。
话虽如此，正如我一直指出的那样，根据 NEURIPS 的统计数据，中国拥有最多的年轻 AI 研究人员。该领域的贡献不成比例地更多地来自年轻研究人员（源线程），因此许多突破性进展更有可能从世界的那部分出现。
中国还有 3 个主要的 AI 人才密度可与硅谷媲美的大城市中心，它们充当了思想混合的大熔炉（源线程）。
## 禁止开源 AI 并没有帮助：
无论西方世界是否禁止开源 AI，这些设计空间都将被探索，论文也将被发表。西方也有许多很棒的新实验室出现。他们也将会发现许多新想法，现在已经意识到可能仍有很多效率提升是可能的。
最好的事情是鼓励和采用这些创新，对投资负责，避免疯狂的投机。
由于这项技术，未来非常非常光明，但我们要小心不要成为被追加保证金的疯狂投机者。我们没有尽可能地使用 AI：无论是在教育还是研究中。例如，我希望印度的每个学生都能无限访问 Opus 4.8 或 Kimi K3 级别的模型。他们确实有访问权限，但访问的是较弱的模型。此外，我们在科学、工程和医学领域可能会有价值高得多的 AI 用例。此外，我们还有一个需要照顾的老龄化世界。AI 在那里将是非常有益的。在娱乐和游戏方面也存在如此多的可能性。
## 结论：
虽然未来是不确定的，但在我看来，很明显的是，AI 多头，尤其是内存多头，没有考虑到算法创新和更新的缩放定律的发现。对他们来说，一切都被杰文斯悖论吞噬了，因此没有意义。我想向他们的分析引入新的角度。
为了使内存股票崩盘，我们不需要大规模的需求减少，正如 Jaya Gupta 指出的那样。即使是需求的微小减少也可能引发一场博弈论竞争，其中现有的持有者开始获利了结，打算以后再买回股票，这可能会使整个市场崩盘。而这似乎已经开始……
重要的是不要让如此大的泡沫形成并给社会带来灾难性的破裂。总的来说，我对这项技术的潜力（甚至内存需求）和未来的基础设施建设保持非常乐观。这是活着的最激动人心的时刻！
（不是投资建议）
# 附录
## 内存节省的详细计算：
为了展示 Kimi K3 有多好，我让 K3 自己进行了这些计算，我已经验证它们是正确的。如前所述，我们考虑两种情况：
1) Kimi K3: 69 KDA + 24 个门控 MLA 全注意力 对比
2) Full MLA: 93 个 MLA 全注意力
## 相关链接
- [GDP](https://x.com/bookwormengr)
- [@bookwormengr](https://x.com/bookwormengr)
- [123K](https://x.com/bookwormengr/status/2082377457852920075/analytics)
- [https://arxiv.org/pdf/2606.19348v1](https://arxiv.org/pdf/2606.19348v1)
- [https://arxiv.org/pdf/2603.15031](https://arxiv.org/pdf/2603.15031)
- [https://arxiv.org/pdf/2601.18089v1](https://arxiv.org/pdf/2601.18089v1)
- [source thread](https://x.com/bookwormengr/status/2009503245350416441?s=20)
- [source](https://x.com/bookwormengr/status/2009503248756166769?s=20)
- [Jaya Gupta has pointed out.](https://x.com/JayaGup10/status/2082301754872369331)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [4:07 PM · Jul 29, 2026](https://x.com/bookwormengr/status/2082377457852920075)
- [123.4K Views](https://x.com/bookwormengr/status/2082377457852920075/analytics)
- [View quotes](https://x.com/bookwormengr/status/2082377457852920075/quotes)
---
*导出时间: 2026/7/30 10:11:20*