Top Companies Are Secretly Working on This (It Will Replace LLMs)

✍ Siddharth🕐 2026-05-06📦 26.1 KB 🟢 已读 𝕏 文章列表

文章探讨了顶级科技公司正在秘密研发的架构转变——状态空间模型（SSM）。旨在解决 Transformer 架构在长序列处理中的计算成本高和内存消耗大的问题。文章详细解析了 SSM 的工作原理、相比 LLM 的优势（线性时间复杂度、恒定内存），并展示了 NVIDIA、IBM 和 Microsoft 等公司在 Nemotron、Bamba 和 SambaY 等混合架构上的实际应用进展。

SSM Transformer Mamba 架构 Nvidia Deep Learning 推理优化 LLM Inference State Space Models

# Top Companies Are Secretly Working on This (It Will Replace LLMs) **作者**: Siddharth **日期**: 2025-10-28T10:20:36.000Z **来源**: [https://x.com/Pseudo_Sid26/status/2051680656351281565](https://x.com/Pseudo_Sid26/status/2051680656351281565) --- ![Image](/doc/1711/image/img_bfe9cc0950b9_1.jpg) Nvidia, Meta, Open AI, Anthropic, Google, Microsoft, top sf based companies and many more are working on this problem secretly to resolve memory issues, compute issues and find a solution to what LLMs could never. This article is not about replacing LLMs with magic. It is about the architectural shift happening right now inside the labs of NVIDIA, IBM, Microsoft, Meta. All of them are quietly working on this, without press releases. The thing they are all building around: State Space Models. ## The Problem Nobody Says Out Loud Every modern LLM , GPT-4, Claude, Gemini etc runs on the Transformer architecture. Transformers work by doing something called self-attention: Every token looks at every other token in the sequence to figure out what matters. That is genuinely powerful. It is also genuinely expensive. - The math behind attention scales quadratically. - Double the context length, quadruple the compute. - At 128K tokens, your model is doing orders of magnitude more work than it did at 4K. - The KV cache bursts with sequence length. - On a standard 7B Transformer model at 128K context, your KV cache alone can eat 30 to 40 GB of VRAM. The model weights haven't changed. The sequence just got longer. > Transformers face quadratic complexity O(N²) for sequence length, limiting their scalability for long-sequence tasks. This is not a hardware problem waiting for a better GPU. It is a structural one. And that is exactly what State Space Models are designed to solve. ## So What Even Is a State Space Model? Lets start with a very basic intuition: - Think about how you take notes in a long lecture. - You don't write down every single word. - You maintain a running summary in your head - what matters, what to keep, what to discard and update it as new information comes in. - At the end of the lecture, that summary is your state. - You never stored every word. You stored what was relevant. That is roughly what an SSM does. Instead of attending to every past token, it maintains a fixed-size hidden state that gets updated at each step. The entire history of the sequence gets compressed into this state. Memory stays constant regardless of how long the sequence gets. ``` # Minimal SSM: discrete-time state update (the core idea) import torch import torch.nn as nn class MinimalSSM(nn.Module): def __init__(self, d_model, d_state): super().__init__() self.d_model = d_model self.d_state = d_state # A: state transition matrix (what to retain) self.A = nn.Parameter(torch.randn(d_state, d_state) * 0.01) # B: input projection (what to absorb) self.B = nn.Parameter(torch.randn(d_state, d_model) * 0.01) # C: output projection (what to emit) self.C = nn.Parameter(torch.randn(d_model, d_state) * 0.01) def forward(self, x): # x: (batch, seq_len, d_model) batch, seq_len, _ = x.shape h = torch.zeros(batch, self.d_state, device=x.device) outputs = [] for t in range(seq_len): # State update: compress history + absorb new input h = h @ self.A.T + x[:, t, :] @ self.B.T y = h @ self.C.T outputs.append(y) return torch.stack(outputs, dim=1) ``` > State goes in, state gets updated, output comes out. No attention matrix. No quadratic explosion. Linear time, constant memory. ## Why Companies Are Investing in This Because the inference bill is real money. Running a Transformer at 256K context is not just slow but it is expensive in ways that break production systems. ## NVIDIA When NVIDIA published Nemotron-H in 2025, they did not frame it as a research flex. They framed it as an engineering decision. Attention was never enough: Tracing the rise of hybrid LLMs, this bog perfectly states what Nvidia did > NVIDIA replaced 92% of attention layers in their Nemotron-H models with Mamba2 blocks, achieving up to 3x faster inference throughput compared to LLaMA-3.1 and Qwen-2.5 at the same model sizes. ![Image](/doc/1711/image/img_38783004b0f3_1.jpg) Attention was never enough Blog ## IBM IBM did the same with Bamba-9B, a hybrid SSM model that runs comparably to Llama-3.1 8B while cutting memory footprint significantly through quantization. ## Microsoft Microsoft's Phi-4-mini-flash-reasoning introduced their own hybrid architecture called SambaY, mixing Mamba, sliding window attention, and Gated Memory Units to hit 10x throughput gains over its predecessor. This is not academia doing experiments. These are production teams solving inference cost problems. The reason it looks "secret" is that nobody writes a press release saying "we quietly changed 90% of our model's internals." They just ship. ## The SSM Architectures That Matter Right Now The SSM family has been evolving fast. Here is the lineage that actually matters: > **Siddharth@Pseudo_Sid26**: [原文链接](https://x.com/Pseudo_Sid26/status/1983116683796468172) > > Attention mechanism is almost everywhere in ML but is it truly replaceable ? Have there been any attempts to replace attention mechanism ? > There have been several researches that have tried to replace or improve attention - > > State Space Models ( SSMs)' > > Mamba Model built > > ![Image](/doc/1711/image/img_8d0c1144b7ef_1.jpg) S4 (2021): The foundational structured state space model from Albert Gu et al. at Stanford. Proved SSMs could handle long range dependencies better than RNNs, especially on the Long Range Arena benchmark. The architecture was elegant but fixed its parameters didn't depend on the input. Mamba (2023): Albert Gu and Tri Dao's breakthrough. The key innovation: make the SSM parameters (A, B, C) input dependent. Suddenly the model could selectively decide what to remember and what to let fade. Mamba-3B outperformed Transformers of similar size on language benchmarks and ran at 5x inference throughput. ![Image](/doc/1711/image/img_422d840e0ebc_1.jpg) ``` # Selective SSM: the core Mamba idea — parameters depend on input class SelectiveSSM(nn.Module): def __init__(self, d_model, d_state, d_conv=4): super().__init__() self.d_model = d_model self.d_state = d_state # Input-dependent projections (this is what makes it "selective") self.x_proj = nn.Linear(d_model, d_state * 2 + d_model) self.conv = nn.Conv1d(d_model, d_model, kernel_size=d_conv, padding=d_conv - 1, groups=d_model) self.out_proj = nn.Linear(d_model, d_model) # Fixed log-space A init (discretized later) self.A_log = nn.Parameter(torch.log(torch.arange(1, d_state + 1).float()).unsqueeze(0).repeat(d_model, 1)) self.D = nn.Parameter(torch.ones(d_model)) def forward(self, x): B_batch, L, D = x.shape # Local conv for short-range context x_conv = self.conv(x.transpose(1, 2))[:, :, :L].transpose(1, 2) x_conv = torch.sigmoid(x_conv) * x_conv # SiLU activation # Project to get input-dependent B, C, dt x_dbl = self.x_proj(x_conv) dt, B, C = x_dbl.split([D, self.d_state, self.d_state], dim=-1) dt = torch.softplus(dt) # Discretize A using zero-order hold A = -torch.exp(self.A_log) dA = torch.exp(dt.unsqueeze(-1) * A.unsqueeze(0).unsqueeze(0)) dB = dt.unsqueeze(-1) * B.unsqueeze(2) # Selective scan: update hidden state token by token h = torch.zeros(B_batch, D, self.d_state, device=x.device) outputs = [] for t in range(L): h = dA[:, t] * h + dB[:, t] * x_conv[:, t].unsqueeze(-1) y = (h * C[:, t].unsqueeze(2)).sum(-1) outputs.append(y) y = torch.stack(outputs, dim=1) + self.D * x_conv return self.out_proj(y) ``` Mamba-2 (2024): Dao and Gu's ICML paper proved something surprising: Transformers and SSMs are mathematically related through structured semiseparable matrices. Mamba-2's SSD layer ran 2–8x faster than Mamba-1 while staying competitive on language tasks. Even Mamb-3 dropped, i explained it some time back in a post as well !! > **Siddharth@Pseudo_Sid26**: [原文链接](https://x.com/Pseudo_Sid26/status/2034303554559905850) > > Mamba-3 just dropped. You might be ignoring it as just another linear model but this could be the next real phase of State Space Models. > Let's break this down: > -State Space Models (SSMs) > >These models don’t rely on attention > >They treat sequences like a continuous dynamical > > ![Image](/doc/1711/image/img_256f87d8f86b_1.jpg) S5, H3, Hyena: Intermediate architectures that improved training stability and explored SSM attention hybrids at smaller scales. H3 was among the first to mix SSM with attention. Hyena attempted to replace attention entirely with long convolutions. Neither became production dominant, but they informed everything that came after. ![Image](/doc/1711/image/img_d69382e01ab9_1.jpg) ## The Hybrid Architecture: Attention + SSM Pure SSMs have a real weakness. They compress everything into a fixed hidden state, which means recall is hard. Ask a pure Mamba model "what exact phrase did the user mention 8000 tokens ago?" and it may struggle. Transformers with direct attention to all past tokens handle this easily. Think of your brain again. - You have working memory for things you are actively thinking about - You have long-term memory that summarizes and compresses everything else. - You don't replay every conversation word for word. - But when something specific matters, you can pull it with accuracy. Hybrid architectures are exactly this. A small number of attention layers handle precise recall. SSM layers handle the bulk of sequence processing efficiently. The ratio is not 50-50. A Jamba architecture simple code: ``` # Simplified Jamba-style hybrid block class HybridBlock(nn.Module): def __init__(self, d_model, d_state, n_heads, use_attention=False): super().__init__() self.use_attention = use_attention self.norm1 = nn.RMSNorm(d_model) self.norm2 = nn.RMSNorm(d_model) if use_attention: self.mixer = nn.MultiheadAttention(d_model, n_heads, batch_first=True) else: self.mixer = SelectiveSSM(d_model, d_state) self.ffn = nn.Sequential( nn.Linear(d_model, d_model * 4), nn.GELU(), nn.Linear(d_model * 4, d_model) ) def forward(self, x, mask=None): residual = x x = self.norm1(x) if self.use_attention: x, _ = self.mixer(x, x, x, attn_mask=mask) else: x = self.mixer(x) x = x + residual x = x + self.ffn(self.norm2(x)) return x def build_jamba_style(d_model, d_state, n_heads, n_layers, attn_every=8): layers = [] for i in range(n_layers): use_attn = (i % attn_every == 0) layers.append(HybridBlock(d_model, d_state, n_heads, use_attention=use_attn)) return nn.Sequential(*layers) ``` The Jamba paper noted something interesting while building this: in a hybrid setting, Mamba-1 combined with attention actually outperformed Mamba-2 combined with attention. The interaction between the architectures matters more than each component in isolation. This is the kind of finding that only comes from building at scale, not from theory. ![Image](/doc/1711/image/img_f57fab015693_1.jpg) ## The Memory Question: Is SSM the Ultimate Solution? > Transformers have what researchers call episodic memory, they can replay exact tokens from the past because everything is stored in the KV cache. > SSMs have semantic memory, a compressed, lossy representation of history that captures meaning but not necessarily exact text. > The Mamba architecture explicitly notes that the selective scan mechanism lets the model decide what to keep and what to fade, based on content. - SSMs trade explicit KV cache for compressed hidden state. - Transformers: explicit episodic memory. - SSMs: implicit semantic memory, compressed representation of past content. ## 1. The Context Window Is Not the Limit The common framing is "Transformers have longer context." That is not quite right. Transformers have the ability to attend across long context, but the hardware running them sets the real ceiling. The KV cache grows with every token, every layer, every user. At 57K tokens, a Transformer on a consumer GPU is already out of VRAM. SSMs handle up to 220K tokens on the same 24GB budget because the hidden state never grows, it just updates. A 2025 benchmarking study put this directly: while Transformers are up to 1.9x faster at short sequences under 8K tokens, SSMs demonstrate a full performance inversion at long contexts becoming up to 4x faster around 57K tokens with 64% less memory. The context window stopped being a Transformer advantage the moment sequences got long enough to matter. And agents are running very long sequences. Tool calls, observation logs, multi-step reasoning traces, these pile up fast. > SSMs don't just handle long context. They handle it at a fixed, predictable memory cost which is the only thing that matters when you're serving hundreds of concurrent agent sessions. ## 2. Agentic Memory Is Not Retrieval: It Is State Here is a problem every agent system runs into. An agent doing a 50-step task accumulates a massive running context: prior tool outputs, failed attempts, intermediate states, user instructions from 30 steps back. The Transformer approach is to keep all of it in the context window and let attention sort it out. That gets expensive. It also gets incoherent attention spreads across everything equally when you just need the model to remember where it is in the workflow. Take example of a write-manage-read loop: agents need memory that persists, compresses, and selectively recalls across interactions. - The SSM hidden state is structurally that loop. - Every token updates the state. - Irrelevant information fades. - Task-relevant structure accumulates. It is not retrieval from a database ,it is the model carrying the task forward the same way you carry a conversation in your head without replaying every word said. Apple Research took this further When SSMs are given interactive access to external tools, they achieve length generalization on arithmetic, reasoning, and coding tasks solving problems of arbitrary complexity despite their fixed state size. The conclusion of that paper is direct: SSMs are a practical efficient alternative to Transformers specifically in tool-based and agentic settings. > The agent memory problem is not "how do you store more?" It is "how do you carry state efficiently across hundreds of steps?" SSMs were designed for exactly this. ## 3. The Brain Does Not Run Attention and Neither Should Agents While learning about memory transformers (LMS or SSMS), I have this habit that I generally relate everything to human memory or how the human brain works because ultimately neural networks are nothing but a replication of neurons in the brain, right? Whenever I try to relate all of this, I try to relate it to how the brain kind of works and this is how I try to understand this memory or problems in agents as well. Using the Human Brain Analogy. A Harvard paper made a point that most people glossed over: language processing in the human brain appears much more similar to how SSMs process language than how Transformers do. Transformers doing full attention across all prior tokens have no biological analog. The brain maintains a compressed, evolving representation of context, it does not replay every previous word to generate the next one. > This matters for agentic systems because agents running long-horizon tasks are the closest AI analogue to sustained human cognition. A person managing a multi-day project does not hold a transcript of every meeting. They hold a state, a structured understanding of where things stand, what is urgent, what has been resolved. The SSM hidden state is the right inductive bias for this kind of memory. Research across NLP, speech, genomics, and time-series consistently shows SSMs performing at or above Transformer level on tasks requiring long range temporal dependencies which is exactly what sustained agent reasoning is. The problem with Transformers in agentic settings is not intelligence. It is that the architecture accumulates cost with history. Every step taken by an agent grows the KV cache. SSMs do not accumulate cost. They just update state. At scale, across millions of agent calls, that is the difference between a system that works and one that is too expensive to run. ## CONCLUSIVE PROJECT ( Open sourcing soon) So I have been working on this project for a very long time. I've been reading around systems and memory for a long time and I wanted to build this project after getting inspired by the AI Hippocampus paper, where I wanted to replicate the human brain as an agent. Basically this agent will actually be a replication of how the human brain works. Each folder will be named after a certain function or a certain brain part and it will actually do what that brain part does. Not to mention I have previously been a bio student also so this is something that has intrigued me a lot and this is something that has converged my interest from multiple domains. The idea behind this was simple: each brain part that will get involved with an LLM in the loop will be working with O(N²) time. When I will have a file-based retrieval system or a short-term memory system, I will try to allocate it an SSM kind of architecture so that the retrieval is linear. Apart from this, this project has certain phases where the memory gets constrained like how an SSM works and we have basically a compressed memory after a certain session or after a certain time. This is something that I am still trying to pull off. I will mostly be releasing this project by the end of this week because I have been working on this with my work around for almost 1.5 months. I am trying to build this with different ideas, different architecture, and stuff. Yeah so finally this will be out this week. For now just for a reference to make you understand how this project is actually working or what I actually am trying to do, I'm attaching a reference architecture of the work that I am doing. This is just the file directory. Please note that this is the very initial file directory. I shall not be sharing the current one because it may lead to, obviously, since I haven't open sourced it, certain contractions and certain vulnerabilities as well with my project. I'm just releasing a very basic version of my directory structure that I was building initially, attaching it below. ``` cerebra/ │ ├── cerebrum/ # Higher cognition — LLM-powered (quadratic) │ ├── prefrontal_cortex/ # Executive function: planning, reasoning, decisions │ │ ├── planner.py # Decomposes user goals into ordered task graphs │ │ ├── reasoner.py # Runs CoT / ReAct loops for complex reasoning │ │ └── decision_gate.py # Filters candidate actions before execution │ │ │ ├── temporal_lobe/ # Language comprehension and context encoding │ │ ├── language_parser.py # Parses raw input into structured intent objects │ │ └── context_encoder.py # Converts observations into dense context representations │ │ │ └── parietal_lobe/ # Sensorimotor integration — maps intent to action │ └── tool_integrator.py # Tool schema registration, call formatting, output parsing │ ├── hippocampus/ # Memory system — SSM / file-based (linear) │ ├── episodic/ # What happened, in order, tied to this session │ │ ├── session_store.py # Append-only session log of tool calls and decisions │ │ └── consolidator.py # Compresses session logs into semantic summaries (background) │ │ │ ├── semantic/ # Facts and concepts — not tied to when they were learned │ │ ├── knowledge_base.py # Vector-indexed long-term knowledge store (SSM hidden state, persisted) │ │ └── retriever.py # Top-k similarity retrieval for LLM context injection │ │ │ └── working/ # What the agent is actively holding right now │ └── scratch.py # Ephemeral in-memory state for the active task (cleared per task) │ ├── cerebellum/ # Procedural memory and habit execution (linear) │ ├── procedural/ │ │ ├── skill_cache.py # Stored action templates for known task patterns — no LLM needed │ │ └── reflex_handler.py # Deterministic fast-path responses: retries, guards, budget checks │ │ │ └── pattern_cache/ │ └── response_cache.py # Semantic-hash cache of LLM responses with TTL expiry │ ├── amygdala/ # Priority and safety — runs below conscious reasoning (linear) │ ├── priority_tagger.py # Scores tasks by urgency; high-priority interrupts the current plan │ └── safety_filter.py # Hard gate blocking dangerous actions regardless of planner output │ ├── thalamus/ # Central relay — all input passes through here (linear) │ ├── input_router.py # Classifies and routes every incoming signal to the right module │ └── context_gate.py # Manages LLM context window budget; trims and ranks before injection │ ├── hypothalamus/ # System homeostasis — keeps the agent stable (linear) │ ├── resource_governor.py # Monitors token budget, rate limits, memory usage, cost │ └── feedback_loop.py # Feeds task outcomes back into skill scores and memory weights │ ├── brainstem/ # Autonomic run loop — executes without reasoning (linear) │ ├── orchestrator.py # Main agent loop: dispatch tool calls, collect results, route output │ └── lifecycle_manager.py # Session lifecycle: init → run → pause → resume → terminate │ ├── corpus_callosum/ # Bridge between quadratic (LLM) and linear (SSM) subsystems │ ├── bridge.py # Serialises and compresses state crossing the complexity boundary │ └── state_sync.py # Keeps LLM context and SSM memory store in sync bidirectionally │ ├── basal_ganglia/ # Action selection and reward tracking (linear) │ ├── action_selector.py # Picks best action from candidates using learned scores, no LLM │ └── reward_tracker.py # Tracks success/failure per action pattern; feeds cerebellum │ ├── pineal_gland/ # Scheduling and background jobs — the system's sleep cycle (linear) │ └── scheduler.py # Triggers consolidation, cache expiry, feedback aggregation on schedule │ └── __init__.py # Entry point and module index ``` I am still working on this project and I will be releasing it very very soon. Stay tuned till then. ## DISCLAIMER I'm attaching the references and the pre-reads to the paper that I've used. Some part of this article was written by Claude Sonnet 4.6 with certain references. Some of it was rawdogged by me and then improved by Claude. Some of it has been spelled using WisprFlow. You can go with these reads as your pre-reads or eventual understanding of the SSMs in general : - Mamba: Linear-Time Sequence Modeling with Selective State Spaces - Transformers are SSMs - Mamba-3: Improved Sequence Modeling using State Space Principles - Efficiently Modeling Long Sequences with Structured State Spaces - Memory for Autonomous LLM Agents - Attention was never enough: Tracing the rise of hybrid LLMs - Microsoft Research- Routing Mamba: Scaling SSMs with Mixture-of-Experts Projection ## 相关链接 - [Siddharth](https://x.com/Pseudo_Sid26) - [@Pseudo_Sid26](https://x.com/Pseudo_Sid26) - [74K](https://x.com/Pseudo_Sid26/status/2051680656351281565/analytics) - [Attention was never enough: Tracing the rise of hybrid LLMs](https://www.ai21.com/blog/rise-of-hybrid-llms/) - [Oct 28, 2025](https://x.com/Pseudo_Sid26/status/1983116683796468172) - [4.9K](https://x.com/Pseudo_Sid26/status/1983116683796468172/analytics) - [Albert Gu et al. at Stanford](https://arxiv.org/pdf/2111.00396) - [Dao and Gu's ICML paper](https://arxiv.org/abs/2405.21060) - [Mar 19](https://x.com/Pseudo_Sid26/status/2034303554559905850) - [1.5K](https://x.com/Pseudo_Sid26/status/2034303554559905850/analytics) - [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752) - [Transformers are SSMs](https://arxiv.org/abs/2405.21060) - [Mamba-3: Improved Sequence Modeling using State Space Principles](https://arxiv.org/abs/2603.15569) - [Efficiently Modeling Long Sequences with Structured State Spaces](https://arxiv.org/abs/2111.00396) - [Memory for Autonomous LLM Agents](https://arxiv.org/abs/2603.07670) - [Attention was never enough: Tracing the rise of hybrid LLMs](https://www.ai21.com/blog/rise-of-hybrid-llms/) - [Microsoft Research- Routing Mamba: Scaling SSMs with Mixture-of-Experts Projection](https://arxiv.org/html/2506.18145v1) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [11:09 PM · May 5, 2026](https://x.com/Pseudo_Sid26/status/2051680656351281565) - [74.1K Views](https://x.com/Pseudo_Sid26/status/2051680656351281565/analytics) - [View quotes](https://x.com/Pseudo_Sid26/status/2051680656351281565/quotes) --- *导出时间: 2026/5/6 22:31:27* --- ## 中文翻译 # 顶级公司正在秘密研究这项技术（它将取代 LLMs） **作者**: Siddharth **日期**: 2025-10-28T10:20:36.000Z **来源**: [https://x.com/Pseudo_Sid26/status/2051680656351281565](https://x.com/Pseudo_Sid26/status/2051680656351281565) --- ![Image](/doc/1711/image/img_bfe9cc0950b9_1.jpg) 英伟达 (Nvidia)、Meta、OpenAI、Anthropic、谷歌、微软、总部位于旧金山的顶级公司以及许多其他公司，都在秘密地致力于解决这个问题，旨在解决内存问题、计算问题，并找到大语言模型 (LLM) 无法解决的问题的方案。这篇文章不是关于用魔法取代 LLM。它是关于当前正在 NVIDIA、IBM、Microsoft 和 Meta 实验室内部发生的架构转变。所有这些公司都在悄然进行这项工作，没有发布任何新闻稿。它们构建的核心是：状态空间模型。 ## 没有人大声说出来的问题每一个现代的 LLM，无论是 GPT-4、Claude 还是 Gemini 等，都运行在 Transformer 架构之上。Transformer 的工作机制是通过一种叫做“自注意力”的方式：序列中的每一个 token 都会查看序列中的其他每一个 token，以确定什么是重要的。这确实非常强大。但也确实非常昂贵。 - 注意力背后的数学计算量是呈二次方增长的。 - 上下文长度增加一倍，计算量就会增加四倍。 - 当达到 128K tokens 时，你的模型所做的计算量比在 4K 时要多几个数量级。 - KV 缓存会随着序列长度而激增。 - 在标准的 128K 上下文的 7B Transformer 模型上，仅 KV 缓存就可能占用 30 到 40 GB 的显存。模型的权重没有改变。只是序列变长了。 > Transformer 对于序列长度面临二次方复杂度 O(N²)，这限制了它们在长序列任务上的扩展性。这不是一个等待更好的 GPU 出现的硬件问题。这是一个结构性问题。而这正是状态空间模型 (SSM) 设计要解决的问题。 ## 那么，到底什么是状态空间模型？让我们从一个非常基本的直觉开始： - 想象一下你在听一场很长的讲座时是如何记笔记的。 - 你不会写下每一个字。 - 你会在脑海中维护一个不断更新的摘要 - 记住什么是重要的，保留什么，丢弃什么，并在新信息到来时更新它。 - 在讲座结束时，那个摘要就是你的“状态”。 - 你从未存储每一个字。你存储的是相关的内容。这大概就是 SSM 所做的事情。它不关注每一个过去的 token，而是维护一个固定大小的隐藏状态，该状态在每一步都会更新。整个序列的历史被压缩到这个状态中。无论序列变得多长，内存占用都保持不变。 ```python # 最小化 SSM：离散时间状态更新（核心思想） import torch import torch.nn as nn class MinimalSSM(nn.Module): def __init__(self, d_model, d_state): super().__init__() self.d_model = d_model self.d_state = d_state # A: 状态转移矩阵（保留什么） self.A = nn.Parameter(torch.randn(d_state, d_state) * 0.01) # B: 输入投影（吸收什么） self.B = nn.Parameter(torch.randn(d_state, d_model) * 0.01) # C: 输出投影（输出什么） self.C = nn.Parameter(torch.randn(d_model, d_state) * 0.01) def forward(self, x): # x: (batch, seq_len, d_model) batch, seq_len, _ = x.shape h = torch.zeros(batch, self.d_state, device=x.device) outputs = [] for t in range(seq_len): # 状态更新：压缩历史 + 吸收新输入 h = h @ self.A.T + x[:, t, :] @ self.B.T y = h @ self.C.T outputs.append(y) return torch.stack(outputs, dim=1) ``` > 状态进入，状态更新，输出产生。没有注意力矩阵。没有二次方爆炸。线性时间，恒定内存。 ## 为什么公司投资于此因为推理账单是真金白银。在 256K 上下文下运行 Transformer 不仅慢，而且其成本之高足以破坏生产系统。 ## NVIDIA 当 NVIDIA 在 2025 年发布 Nemotron-H 时，他们并没有将其标榜为一种研究展示。他们将其定义为一种工程决策。 Attention was never enough: Tracing the rise of hybrid LLMs，这篇博文完美地阐述了 Nvidia 所做的工作。 > NVIDIA 在其 Nemotron-H 模型中将 92% 的注意力层替换为 Mamba2 块，在与 LLaMA-3.1 和 Qwen-2.5 相同的模型规模下，推理吞吐量提高了高达 3 倍。 ![Image](/doc/1711/image/img_38783004b0f3_1.jpg) Attention was never enough Blog ## IBM IBM 用 Bamba-9B 做了同样的事情，这是一个混合 SSM 模型，通过量化显著减少了内存占用，其性能与 Llama-3.1 8B 相当。 ## Microsoft Microsoft 的 Phi-4-mini-flash-reasoning 引入了他们自己的混合架构 SambaY，混合了 Mamba、滑动窗口注意力和门控记忆单元，实现了比前代产品高 10 倍的吞吐量增益。这不是学术界在搞实验。这些是解决推理成本问题的生产团队。它看起来“秘密”的原因是，没有人会写新闻稿说“我们悄悄更改了模型 90% 的内部结构”。它们只是直接发布产品。 ## 当前重要的 SSM 架构 SSM 家族发展迅速。以下是真正重要的谱系： > **Siddharth@Pseudo_Sid26**: [原文链接](https://x.com/Pseudo_Sid26/status/1983116683796468172) > > 注意力机制在机器学习中几乎无处不在，但它真的不可替代吗？是否有过尝试去替换注意力机制？ > 已经有几项研究试图替换或改进注意力机制 - > > 状态空间模型 > > 构建的 Mamba 模型 > > ![Image](/doc/1711/image/img_8d0c1144b7ef_1.jpg) **S4 (2021)**：由斯坦福大学的 Albert Gu 等人提出的基础性结构化状态空间模型。证明了 SSM 在处理长程依赖方面比 RNN 更好，特别是在 Long Range Arena 基准测试中。架构很优雅，但其固定参数不依赖于输入。 **Mamba (2023)**：Albert Gu 和 Tri Dao 的突破。关键创新：使 SSM 参数 (A, B, C) 依赖于输入。突然间，模型可以有选择地决定记住什么，让什么淡去。Mamba-3B 在语言基准测试中超过了相同规模的 Transformer，并以 5 倍的推理吞吐量运行。 ![Image](/doc/1711/image/img_422d840e0ebc_1.jpg) ```python # 选择性 SSM：核心 Mamba 思想 — 参数依赖于输入 class SelectiveSSM(nn.Module): def __init__(self, d_model, d_state, d_conv=4): super().__init__() self.d_model = d_model self.d_state = d_state # 依赖于输入的投影（这就是使其具有“选择性”的原因） self.x_proj = nn.Linear(d_model, d_state * 2 + d_model) self.conv = nn.Conv1d(d_model, d_model, kernel_size=d_conv, padding=d_conv - 1, groups=d_model) self.out_proj = nn.Linear(d_model, d_model) # 固定的对数空间 A 初始化（稍后离散化） self.A_log = nn.Parameter(torch.log(torch.arange(1, d_state + 1).float()).unsqueeze(0).repeat(d_model, 1)) self.D = nn.Parameter(torch.ones(d_model)) def forward(self, x): B_batch, L, D = x.shape # 用于短程上下文的局部卷积 x_conv = self.conv(x.transpose(1, 2))[:, :, :L].transpose(1, 2) x_conv = torch.sigmoid(x_conv) * x_conv # SiLU 激活 # 投影以获取依赖于输入的 B, C, dt x_dbl = self.x_proj(x_conv) dt, B, C = x_dbl.split([D, self.d_state, self.d_state], dim=-1) dt = torch.softplus(dt) # 使用零阶保持离散化 A A = -torch.exp(self.A_log) dA = torch.exp(dt.unsqueeze(-1) * A.unsqueeze(0).unsqueeze(0)) dB = dt.unsqueeze(-1) * B.unsqueeze(2) # 选择性扫描：逐个 token 更新隐藏状态 h = torch.zeros(B_batch, D, self.d_state, device=x.device) outputs = [] for t in range(L): h = dA[:, t] * h + dB[:, t] * x_conv[:, t].unsqueeze(-1) y = (h * C[:, t].unsqueeze(2)).sum(-1) outputs.append(y) y = torch.stack(outputs, dim=1) + self.D * x_conv return self.out_proj(y) ``` **Mamba-2 (2024)**：Dao 和 Gu 的 ICML 论文证明了一个令人惊讶的结论：Transformer 和 SSM 在数学上通过结构化半可分矩阵联系在一起。Mamba-2 的 SSD 层运行速度比 Mamba-1 快 2–8 倍，同时在语言任务上保持竞争力。甚至 Mamba-3 也发布了，我也在之前的一篇文章中解释过！！！ > **Siddharth@Pseudo_Sid26**: [原文链接](https://x.com/Pseudo_Sid26/status/2034303554559905850) > > Mamba-3 刚刚发布。你可能认为它只是另一个线性模型而忽略了它，但这可能是状态空间模型的下一个真正阶段。 > 让我们拆解一下： > -状态空间模型 > >这些模型不依赖于注意力 > >它们将序列视为连续的动力系统 > > ![Image](/doc/1711/image/img_256f87d8f86b_1.jpg) **S5, H3, Hyena**：中等规模的架构，提高了训练稳定性，并探索了小规模的 SSM-注意力混合体。H3 是最早混合 SSM 和注意力的架构之一。Hyena 试图完全用长卷积来取代注意力。虽然两者都没有在生产中占据主导地位，但它们为后来的一切提供了启发。 ![Image](/doc/1711/image/img_d69382e01ab9_1.jpg) ## 混合架构：注意力 + SSM 纯 SSM 有一个真正的弱点。它们将所有内容压缩到一个固定的隐藏状态中，这意味着回溯很难。如果你问一个纯 Mamba 模型“用户在 8000 个 token 之前提到了哪个确切的短语？”，它可能会很吃力。而 Transformer 通过对所有过去的 token 进行直接注意力，可以轻松处理这个问题。再次思考你的大脑。 - 你有工作记忆，用于存储你正在积极思考的事情。 - 你有长期记忆，它总结和压缩其他所有内容。 - 你不会逐字逐句地重放每一次对话。 - 但当具体的事情变得重要时，你可以准确地提取它。混合架构正是这样做的。少量的注意力层处理精确的回忆。SSM 层高效地处理大部分序列处理。这个比例通常不是 50-50。一个简单的 Jamba 架构代码示例： ```python # 简化的 Jamba 风格混合块 class HybridBlock(nn.Module): def __init__(self, d_model, d_state, n_heads, use_attention=False): super().__init__() self.use_attention = use_attention self.norm1 = nn.RMSNorm(d_model) self.norm2 = nn.RMSNorm(d_model) if use_attention: self.mixer = nn.MultiheadAttention(d_model, n_heads, batch_first=True) else: self.mixer = SelectiveSSM(d_model, d_state) self.ffn = nn.Sequential( nn.Linear(d_model, d_model * 4), nn.GELU(), nn.Linear(d_model * 4, d_model) ) def forward(self, x, mask=None): residual = x x = self.norm1(x) if self.use_attention: x, _ = self.mixer(x, x, x, attn_mask=mask) else: x = self.mixer(x) x = x + residual x = x + self.ffn(self.norm2(x)) return x def build_jamba_style(d_model, d_state, n_heads, n_layers, attn_every=8): layers = [] for i in range(n_layers): use_attn = (i % attn_every == 0) layers.append(HybridBlock(d_model, d_state, n_heads, use_attention=use_attn)) return nn.Sequential(*layers) ``` Jamba 论文在构建时注意到了一件有趣的事情：在混合设置中，结合注意力的 Mamba-1 实际上比结合注意力的 Mamba-2 表现更好。架构之间的相互作用比每个组件单独的作用更重要。这是只有在大规模构建时才能得出的发现，而不是理论。 ![Image](/doc/1711/image/img_f57fab015693_1.jpg) ## 内存问题：SSM 是终极解决方案吗？ > Transformer 拥有研究人员所谓的情景记忆，它们可以重放过去的精确 token，因为所有内容都存储在 KV 缓存中。 > SSM 拥有语义记忆，这是历史的一种压缩的、有损的表示，它捕捉了意义，但不一定是确切的文本。 > Mamba 架构明确指出，选择性扫描机制允许模型根据内容决定保留什么，让什么淡去。 - SSM 用压缩的隐藏状态换取了显式的 KV 缓存。 - Transformer：显式情景记忆。 - SSM：隐式语义记忆，过去内容的压缩表示。 ### 1. 上下文窗口不是限制通常的说法是“Transformer 有更长的上下文”。这不太对。Transformer 有跨越长上下文进行关注的能力，但运行它们的硬件设定了真正的上限。 KV 缓存随着每个 token、每一层、每个用户而增长。在 57K tokens 时，消费级 GPU 上的 Transformer 显存就不够了。 SSM 可以在同样的 24GB 显存预算下处理多达 220K tokens，因为隐藏状态永远不会增长，它只是更新。 2025 年的一项基准测试研究直接指出了这一点：虽然 Transformer 在 8K token 以下的短序列上速度快达 1.9 倍，但 SSM 在长上下文中表现出完全的性能逆转，在 57K tokens 左右速度快达 4 倍，内存减少 64%。当序列变得足够长以至于重要时，上下文窗口就不再是 Transformer 的优势了。而智能体正在运行非常长的序列。工具调用、观察日志、多步推理追踪，这些都会迅速堆积。 > SSM 不仅仅能处理长上下文。它们以固定的、可预测的内存成本处理上下文，当你为数百个并发的智能体会话提供服务时，这是唯一重要的事情。 ### 2. 智能体记忆不是检索：它是状态每个智能体系统都会遇到这个问题。执行 50 步任务的智能体会积累大量的运行上下文：先前的工具输出、失败的尝试、中间状态、30 步之前的用户指令。 Transformer 的方法是将所有这些都保留在上下文窗口中，让注意力来整理。这变得很昂贵。它也会变得不连贯——当你只需要模型记住它在工作流中的位置时，注意力却平均地分散在所有内容上。以写入-管理-读取循环为例：智能体需要能够持久化、压缩并在交互间选择性回溯的记忆。 - SSM 隐藏状态在结构上就是那个循环。 - 每个 token 更新状态。 - 无关信息淡出。 - 任务相关的结构积累。这不是从数据库中检索，而是模型像在脑海中保持对话一样携带任务向前推进，而无需重述说过的每一个字。苹果研究院进一步推进了这项工作。当 SSM 被赋予对外部工具的交互式访问权限时，它们在算术、推理和编程任务上实现了长度泛化，解决了任意复杂度的问题，尽管其状态大小是固定的。那篇论文的结论很直接：SSM 是 Transformer 的实用、高效的替代品，特别是在基于工具和智能体的场景中。 > 智能体记忆问题不是“如何存储更多？”而是“如何在数百步中高效地携带状态？”SSM 正是为此设计的。 ### 3. 大脑不运行注意力，智能体也不应该在学习记忆 Transformer (LMS 或 SSMS) 时，我有这样一个习惯，我通常将所有事情与人类记忆或人类大脑的工作方式联系起来，因为归根结底，神经网络只是大脑中神经元的复制，对吧？每当我试图将所有这些联系起来时，我都会尝试将其与大脑的工作方式联系起来，这也是我尝试理解这种记忆或智能体中的问题的方式。使用人脑类比。哈佛大学的一篇论文指出了一个大多数人忽略的结论：人脑中的语言处理似乎更像 SSM 处理语言的方式，而不是 Transformer。Transformer 对所有先前的 token 进行全注意力，h...

🔗 相关文档