# Top Companies Are Secretly Working on This (It Will Replace LLMs)
**作者**: Siddharth
**日期**: 2025-10-28T10:20:36.000Z
**来源**: [https://x.com/Pseudo_Sid26/status/2051680656351281565](https://x.com/Pseudo_Sid26/status/2051680656351281565)
---
Nvidia, Meta, Open AI, Anthropic, Google, Microsoft, top sf based companies and many more are working on this problem secretly to resolve memory issues, compute issues and find a solution to what LLMs could never.
This article is not about replacing LLMs with magic. It is about the architectural shift happening right now inside the labs of NVIDIA, IBM, Microsoft, Meta. All of them are quietly working on this, without press releases. The thing they are all building around: State Space Models.
## The Problem Nobody Says Out Loud
Every modern LLM , GPT-4, Claude, Gemini etc runs on the Transformer architecture. Transformers work by doing something called self-attention:
Every token looks at every other token in the sequence to figure out what matters. That is genuinely powerful. It is also genuinely expensive.
- The math behind attention scales quadratically.
- Double the context length, quadruple the compute.
- At 128K tokens, your model is doing orders of magnitude more work than it did at 4K.
- The KV cache bursts with sequence length.
- On a standard 7B Transformer model at 128K context, your KV cache alone can eat 30 to 40 GB of VRAM.
The model weights haven't changed. The sequence just got longer.
> Transformers face quadratic complexity O(N²) for sequence length, limiting their scalability for long-sequence tasks.
This is not a hardware problem waiting for a better GPU. It is a structural one. And that is exactly what State Space Models are designed to solve.
## So What Even Is a State Space Model?
Lets start with a very basic intuition:
- Think about how you take notes in a long lecture.
- You don't write down every single word.
- You maintain a running summary in your head
- what matters, what to keep, what to discard and update it as new information comes in.
- At the end of the lecture, that summary is your state.
- You never stored every word. You stored what was relevant.
That is roughly what an SSM does. Instead of attending to every past token, it maintains a fixed-size hidden state that gets updated at each step. The entire history of the sequence gets compressed into this state. Memory stays constant regardless of how long the sequence gets.
```
# Minimal SSM: discrete-time state update (the core idea)
import torch
import torch.nn as nn
class MinimalSSM(nn.Module):
def __init__(self, d_model, d_state):
super().__init__()
self.d_model = d_model
self.d_state = d_state
# A: state transition matrix (what to retain)
self.A = nn.Parameter(torch.randn(d_state, d_state) * 0.01)
# B: input projection (what to absorb)
self.B = nn.Parameter(torch.randn(d_state, d_model) * 0.01)
# C: output projection (what to emit)
self.C = nn.Parameter(torch.randn(d_model, d_state) * 0.01)
def forward(self, x):
# x: (batch, seq_len, d_model)
batch, seq_len, _ = x.shape
h = torch.zeros(batch, self.d_state, device=x.device)
outputs = []
for t in range(seq_len):
# State update: compress history + absorb new input
h = h @ self.A.T + x[:, t, :] @ self.B.T
y = h @ self.C.T
outputs.append(y)
return torch.stack(outputs, dim=1)
```
> State goes in, state gets updated, output comes out. No attention matrix. No quadratic explosion. Linear time, constant memory.
## Why Companies Are Investing in This
Because the inference bill is real money. Running a Transformer at 256K context is not just slow but it is expensive in ways that break production systems.
## NVIDIA
When NVIDIA published Nemotron-H in 2025, they did not frame it as a research flex. They framed it as an engineering decision.
Attention was never enough: Tracing the rise of hybrid LLMs, this bog perfectly states what Nvidia did
> NVIDIA replaced 92% of attention layers in their Nemotron-H models with Mamba2 blocks, achieving up to 3x faster inference throughput compared to LLaMA-3.1 and Qwen-2.5 at the same model sizes.
Attention was never enough Blog
## IBM
IBM did the same with Bamba-9B, a hybrid SSM model that runs comparably to Llama-3.1 8B while cutting memory footprint significantly through quantization.
## Microsoft
Microsoft's Phi-4-mini-flash-reasoning introduced their own hybrid architecture called SambaY, mixing Mamba, sliding window attention, and Gated Memory Units to hit 10x throughput gains over its predecessor.
This is not academia doing experiments. These are production teams solving inference cost problems. The reason it looks "secret" is that nobody writes a press release saying "we quietly changed 90% of our model's internals." They just ship.
## The SSM Architectures That Matter Right Now
The SSM family has been evolving fast. Here is the lineage that actually matters:
> **Siddharth@Pseudo_Sid26**: [原文链接](https://x.com/Pseudo_Sid26/status/1983116683796468172)
>
> Attention mechanism is almost everywhere in ML but is it truly replaceable ? Have there been any attempts to replace attention mechanism ?
> There have been several researches that have tried to replace or improve attention -
> > State Space Models ( SSMs)'
> > Mamba Model built
>
> 
S4 (2021): The foundational structured state space model from Albert Gu et al. at Stanford. Proved SSMs could handle long range dependencies better than RNNs, especially on the Long Range Arena benchmark. The architecture was elegant but fixed its parameters didn't depend on the input.
Mamba (2023): Albert Gu and Tri Dao's breakthrough. The key innovation: make the SSM parameters (A, B, C) input dependent. Suddenly the model could selectively decide what to remember and what to let fade. Mamba-3B outperformed Transformers of similar size on language benchmarks and ran at 5x inference throughput.
```
# Selective SSM: the core Mamba idea — parameters depend on input
class SelectiveSSM(nn.Module):
def __init__(self, d_model, d_state, d_conv=4):
super().__init__()
self.d_model = d_model
self.d_state = d_state
# Input-dependent projections (this is what makes it "selective")
self.x_proj = nn.Linear(d_model, d_state * 2 + d_model)
self.conv = nn.Conv1d(d_model, d_model, kernel_size=d_conv, padding=d_conv - 1, groups=d_model)
self.out_proj = nn.Linear(d_model, d_model)
# Fixed log-space A init (discretized later)
self.A_log = nn.Parameter(torch.log(torch.arange(1, d_state + 1).float()).unsqueeze(0).repeat(d_model, 1))
self.D = nn.Parameter(torch.ones(d_model))
def forward(self, x):
B_batch, L, D = x.shape
# Local conv for short-range context
x_conv = self.conv(x.transpose(1, 2))[:, :, :L].transpose(1, 2)
x_conv = torch.sigmoid(x_conv) * x_conv # SiLU activation
# Project to get input-dependent B, C, dt
x_dbl = self.x_proj(x_conv)
dt, B, C = x_dbl.split([D, self.d_state, self.d_state], dim=-1)
dt = torch.softplus(dt)
# Discretize A using zero-order hold
A = -torch.exp(self.A_log)
dA = torch.exp(dt.unsqueeze(-1) * A.unsqueeze(0).unsqueeze(0))
dB = dt.unsqueeze(-1) * B.unsqueeze(2)
# Selective scan: update hidden state token by token
h = torch.zeros(B_batch, D, self.d_state, device=x.device)
outputs = []
for t in range(L):
h = dA[:, t] * h + dB[:, t] * x_conv[:, t].unsqueeze(-1)
y = (h * C[:, t].unsqueeze(2)).sum(-1)
outputs.append(y)
y = torch.stack(outputs, dim=1) + self.D * x_conv
return self.out_proj(y)
```
Mamba-2 (2024): Dao and Gu's ICML paper proved something surprising: Transformers and SSMs are mathematically related through structured semiseparable matrices. Mamba-2's SSD layer ran 2–8x faster than Mamba-1 while staying competitive on language tasks.
Even Mamb-3 dropped, i explained it some time back in a post as well !!
> **Siddharth@Pseudo_Sid26**: [原文链接](https://x.com/Pseudo_Sid26/status/2034303554559905850)
>
> Mamba-3 just dropped. You might be ignoring it as just another linear model but this could be the next real phase of State Space Models.
> Let's break this down:
> -State Space Models (SSMs)
> >These models don’t rely on attention
> >They treat sequences like a continuous dynamical
>
> 
S5, H3, Hyena: Intermediate architectures that improved training stability and explored SSM attention hybrids at smaller scales. H3 was among the first to mix SSM with attention. Hyena attempted to replace attention entirely with long convolutions. Neither became production dominant, but they informed everything that came after.
## The Hybrid Architecture: Attention + SSM
Pure SSMs have a real weakness. They compress everything into a fixed hidden state, which means recall is hard. Ask a pure Mamba model "what exact phrase did the user mention 8000 tokens ago?" and it may struggle. Transformers with direct attention to all past tokens handle this easily.
Think of your brain again.
- You have working memory for things you are actively thinking about
- You have long-term memory that summarizes and compresses everything else.
- You don't replay every conversation word for word.
- But when something specific matters, you can pull it with accuracy.
Hybrid architectures are exactly this. A small number of attention layers handle precise recall. SSM layers handle the bulk of sequence processing efficiently. The ratio is not 50-50.
A Jamba architecture simple code:
```
# Simplified Jamba-style hybrid block
class HybridBlock(nn.Module):
def __init__(self, d_model, d_state, n_heads, use_attention=False):
super().__init__()
self.use_attention = use_attention
self.norm1 = nn.RMSNorm(d_model)
self.norm2 = nn.RMSNorm(d_model)
if use_attention:
self.mixer = nn.MultiheadAttention(d_model, n_heads, batch_first=True)
else:
self.mixer = SelectiveSSM(d_model, d_state)
self.ffn = nn.Sequential(
nn.Linear(d_model, d_model * 4),
nn.GELU(),
nn.Linear(d_model * 4, d_model)
)
def forward(self, x, mask=None):
residual = x
x = self.norm1(x)
if self.use_attention:
x, _ = self.mixer(x, x, x, attn_mask=mask)
else:
x = self.mixer(x)
x = x + residual
x = x + self.ffn(self.norm2(x))
return x
def build_jamba_style(d_model, d_state, n_heads, n_layers, attn_every=8):
layers = []
for i in range(n_layers):
use_attn = (i % attn_every == 0)
layers.append(HybridBlock(d_model, d_state, n_heads, use_attention=use_attn))
return nn.Sequential(*layers)
```
The Jamba paper noted something interesting while building this: in a hybrid setting, Mamba-1 combined with attention actually outperformed Mamba-2 combined with attention.
The interaction between the architectures matters more than each component in isolation. This is the kind of finding that only comes from building at scale, not from theory.
## The Memory Question: Is SSM the Ultimate Solution?
> Transformers have what researchers call episodic memory, they can replay exact tokens from the past because everything is stored in the KV cache.
> SSMs have semantic memory, a compressed, lossy representation of history that captures meaning but not necessarily exact text.
> The Mamba architecture explicitly notes that the selective scan mechanism lets the model decide what to keep and what to fade, based on content.
- SSMs trade explicit KV cache for compressed hidden state.
- Transformers: explicit episodic memory.
- SSMs: implicit semantic memory, compressed representation of past content.
## 1. The Context Window Is Not the Limit
The common framing is "Transformers have longer context." That is not quite right. Transformers have the ability to attend across long context, but the hardware running them sets the real ceiling.
The KV cache grows with every token, every layer, every user. At 57K tokens, a Transformer on a consumer GPU is already out of VRAM.
SSMs handle up to 220K tokens on the same 24GB budget because the hidden state never grows, it just updates.
A 2025 benchmarking study put this directly: while Transformers are up to 1.9x faster at short sequences under 8K tokens, SSMs demonstrate a full performance inversion at long contexts becoming up to 4x faster around 57K tokens with 64% less memory.
The context window stopped being a Transformer advantage the moment sequences got long enough to matter. And agents are running very long sequences. Tool calls, observation logs, multi-step reasoning traces, these pile up fast.
> SSMs don't just handle long context. They handle it at a fixed, predictable memory cost which is the only thing that matters when you're serving hundreds of concurrent agent sessions.
## 2. Agentic Memory Is Not Retrieval: It Is State
Here is a problem every agent system runs into. An agent doing a 50-step task accumulates a massive running context: prior tool outputs, failed attempts, intermediate states, user instructions from 30 steps back.
The Transformer approach is to keep all of it in the context window and let attention sort it out. That gets expensive. It also gets incoherent attention spreads across everything equally when you just need the model to remember where it is in the workflow.
Take example of a write-manage-read loop: agents need memory that persists, compresses, and selectively recalls across interactions.
- The SSM hidden state is structurally that loop.
- Every token updates the state.
- Irrelevant information fades.
- Task-relevant structure accumulates.
It is not retrieval from a database ,it is the model carrying the task forward the same way you carry a conversation in your head without replaying every word said.
Apple Research took this further
When SSMs are given interactive access to external tools, they achieve length generalization on arithmetic, reasoning, and coding tasks solving problems of arbitrary complexity despite their fixed state size.
The conclusion of that paper is direct: SSMs are a practical efficient alternative to Transformers specifically in tool-based and agentic settings.
> The agent memory problem is not "how do you store more?" It is "how do you carry state efficiently across hundreds of steps?" SSMs were designed for exactly this.
## 3. The Brain Does Not Run Attention and Neither Should Agents
While learning about memory transformers (LMS or SSMS), I have this habit that I generally relate everything to human memory or how the human brain works because ultimately neural networks are nothing but a replication of neurons in the brain, right? Whenever I try to relate all of this, I try to relate it to how the brain kind of works and this is how I try to understand this memory or problems in agents as well. Using the Human Brain Analogy.
A Harvard paper made a point that most people glossed over: language processing in the human brain appears much more similar to how SSMs process language than how Transformers do. Transformers doing full attention across all prior tokens have no biological analog. The brain maintains a compressed, evolving representation of context, it does not replay every previous word to generate the next one.
> This matters for agentic systems because agents running long-horizon tasks are the closest AI analogue to sustained human cognition.
A person managing a multi-day project does not hold a transcript of every meeting. They hold a state, a structured understanding of where things stand, what is urgent, what has been resolved.
The SSM hidden state is the right inductive bias for this kind of memory. Research across NLP, speech, genomics, and time-series consistently shows SSMs performing at or above Transformer level on tasks requiring long range temporal dependencies which is exactly what sustained agent reasoning is.
The problem with Transformers in agentic settings is not intelligence. It is that the architecture accumulates cost with history. Every step taken by an agent grows the KV cache. SSMs do not accumulate cost. They just update state. At scale, across millions of agent calls, that is the difference between a system that works and one that is too expensive to run.
## CONCLUSIVE PROJECT ( Open sourcing soon)
So I have been working on this project for a very long time. I've been reading around systems and memory for a long time and I wanted to build this project after getting inspired by the AI Hippocampus paper, where I wanted to replicate the human brain as an agent. Basically this agent will actually be a replication of how the human brain works. Each folder will be named after a certain function or a certain brain part and it will actually do what that brain part does.
Not to mention I have previously been a bio student also so this is something that has intrigued me a lot and this is something that has converged my interest from multiple domains. The idea behind this was simple: each brain part that will get involved with an LLM in the loop will be working with O(N²) time. When I will have a file-based retrieval system or a short-term memory system, I will try to allocate it an SSM kind of architecture so that the retrieval is linear.
Apart from this, this project has certain phases where the memory gets constrained like how an SSM works and we have basically a compressed memory after a certain session or after a certain time. This is something that I am still trying to pull off. I will mostly be releasing this project by the end of this week because I have been working on this with my work around for almost 1.5 months. I am trying to build this with different ideas, different architecture, and stuff. Yeah so finally this will be out this week. For now just for a reference to make you understand how this project is actually working or what I actually am trying to do, I'm attaching a reference architecture of the work that I am doing. This is just the file directory. Please note that this is the very initial file directory. I shall not be sharing the current one because it may lead to, obviously, since I haven't open sourced it, certain contractions and certain vulnerabilities as well with my project. I'm just releasing a very basic version of my directory structure that I was building initially, attaching it below.
```
cerebra/
│
├── cerebrum/ # Higher cognition — LLM-powered (quadratic)
│ ├── prefrontal_cortex/ # Executive function: planning, reasoning, decisions
│ │ ├── planner.py # Decomposes user goals into ordered task graphs
│ │ ├── reasoner.py # Runs CoT / ReAct loops for complex reasoning
│ │ └── decision_gate.py # Filters candidate actions before execution
│ │
│ ├── temporal_lobe/ # Language comprehension and context encoding
│ │ ├── language_parser.py # Parses raw input into structured intent objects
│ │ └── context_encoder.py # Converts observations into dense context representations
│ │
│ └── parietal_lobe/ # Sensorimotor integration — maps intent to action
│ └── tool_integrator.py # Tool schema registration, call formatting, output parsing
│
├── hippocampus/ # Memory system — SSM / file-based (linear)
│ ├── episodic/ # What happened, in order, tied to this session
│ │ ├── session_store.py # Append-only session log of tool calls and decisions
│ │ └── consolidator.py # Compresses session logs into semantic summaries (background)
│ │
│ ├── semantic/ # Facts and concepts — not tied to when they were learned
│ │ ├── knowledge_base.py # Vector-indexed long-term knowledge store (SSM hidden state, persisted)
│ │ └── retriever.py # Top-k similarity retrieval for LLM context injection
│ │
│ └── working/ # What the agent is actively holding right now
│ └── scratch.py # Ephemeral in-memory state for the active task (cleared per task)
│
├── cerebellum/ # Procedural memory and habit execution (linear)
│ ├── procedural/
│ │ ├── skill_cache.py # Stored action templates for known task patterns — no LLM needed
│ │ └── reflex_handler.py # Deterministic fast-path responses: retries, guards, budget checks
│ │
│ └── pattern_cache/
│ └── response_cache.py # Semantic-hash cache of LLM responses with TTL expiry
│
├── amygdala/ # Priority and safety — runs below conscious reasoning (linear)
│ ├── priority_tagger.py # Scores tasks by urgency; high-priority interrupts the current plan
│ └── safety_filter.py # Hard gate blocking dangerous actions regardless of planner output
│
├── thalamus/ # Central relay — all input passes through here (linear)
│ ├── input_router.py # Classifies and routes every incoming signal to the right module
│ └── context_gate.py # Manages LLM context window budget; trims and ranks before injection
│
├── hypothalamus/ # System homeostasis — keeps the agent stable (linear)
│ ├── resource_governor.py # Monitors token budget, rate limits, memory usage, cost
│ └── feedback_loop.py # Feeds task outcomes back into skill scores and memory weights
│
├── brainstem/ # Autonomic run loop — executes without reasoning (linear)
│ ├── orchestrator.py # Main agent loop: dispatch tool calls, collect results, route output
│ └── lifecycle_manager.py # Session lifecycle: init → run → pause → resume → terminate
│
├── corpus_callosum/ # Bridge between quadratic (LLM) and linear (SSM) subsystems
│ ├── bridge.py # Serialises and compresses state crossing the complexity boundary
│ └── state_sync.py # Keeps LLM context and SSM memory store in sync bidirectionally
│
├── basal_ganglia/ # Action selection and reward tracking (linear)
│ ├── action_selector.py # Picks best action from candidates using learned scores, no LLM
│ └── reward_tracker.py # Tracks success/failure per action pattern; feeds cerebellum
│
├── pineal_gland/ # Scheduling and background jobs — the system's sleep cycle (linear)
│ └── scheduler.py # Triggers consolidation, cache expiry, feedback aggregation on schedule
│
└── __init__.py # Entry point and module index
```
I am still working on this project and I will be releasing it very very soon. Stay tuned till then.
## DISCLAIMER
I'm attaching the references and the pre-reads to the paper that I've used. Some part of this article was written by Claude Sonnet 4.6 with certain references. Some of it was rawdogged by me and then improved by Claude. Some of it has been spelled using WisprFlow. You can go with these reads as your pre-reads or eventual understanding of the SSMs in general :
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Transformers are SSMs
- Mamba-3: Improved Sequence Modeling using State Space Principles
- Efficiently Modeling Long Sequences with Structured State Spaces
- Memory for Autonomous LLM Agents
- Attention was never enough: Tracing the rise of hybrid LLMs
- Microsoft Research- Routing Mamba: Scaling SSMs with Mixture-of-Experts Projection
## 相关链接
- [Siddharth](https://x.com/Pseudo_Sid26)
- [@Pseudo_Sid26](https://x.com/Pseudo_Sid26)
- [54K](https://x.com/Pseudo_Sid26/status/2051680656351281565/analytics)
- [Attention was never enough: Tracing the rise of hybrid LLMs](https://www.ai21.com/blog/rise-of-hybrid-llms/)
- [Oct 28, 2025](https://x.com/Pseudo_Sid26/status/1983116683796468172)
- [4.8K](https://x.com/Pseudo_Sid26/status/1983116683796468172/analytics)
- [Albert Gu et al. at Stanford](https://arxiv.org/pdf/2111.00396)
- [Dao and Gu's ICML paper](https://arxiv.org/abs/2405.21060)
- [Mar 19](https://x.com/Pseudo_Sid26/status/2034303554559905850)
- [1.5K](https://x.com/Pseudo_Sid26/status/2034303554559905850/analytics)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752)
- [Transformers are SSMs](https://arxiv.org/abs/2405.21060)
- [Mamba-3: Improved Sequence Modeling using State Space Principles](https://arxiv.org/abs/2603.15569)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://arxiv.org/abs/2111.00396)
- [Memory for Autonomous LLM Agents](https://arxiv.org/abs/2603.07670)
- [Attention was never enough: Tracing the rise of hybrid LLMs](https://www.ai21.com/blog/rise-of-hybrid-llms/)
- [Microsoft Research- Routing Mamba: Scaling SSMs with Mixture-of-Experts Projection](https://arxiv.org/html/2506.18145v1)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [11:09 PM · May 5, 2026](https://x.com/Pseudo_Sid26/status/2051680656351281565)
- [54.7K Views](https://x.com/Pseudo_Sid26/status/2051680656351281565/analytics)
- [View quotes](https://x.com/Pseudo_Sid26/status/2051680656351281565/quotes)
---
*导出时间: 2026/5/6 16:58:11*
---
## 中文翻译
# 顶级公司正在秘密研发这项技术（它将取代大语言模型）
**作者**: Siddharth
**日期**: 2025-10-28T10:20:36.000Z
**来源**: [https://x.com/Pseudo_Sid26/status/2051680656351281565](https://x.com/Pseudo_Sid26/status/2051680656351281565)
---
英伟达（Nvidia）、Meta、Open AI、Anthropic、谷歌、微软、位于旧金山湾区的顶级公司以及许多其他公司正在秘密致力于解决这一问题，以克服内存瓶颈和算力难题，并寻找大语言模型（LLM）始终无法实现的解决方案。
本文并非探讨用某种魔法取代大语言模型。而是关于当前正在英伟达、IBM、微软和 Meta 实验室内部发生的架构转变。所有这些公司都在悄无声息地推进这项工作，没有发布任何新闻稿。他们构建的核心围绕着一个东西：状态空间模型。
## 大家都避而不谈的问题
每一个现代的大语言模型，无论是 GPT-4、Claude 还是 Gemini 等，都运行在 Transformer 架构之上。Transformer 的工作机制依赖于所谓的“自注意力”：
序列中的每一个 Token 都会注视序列中的其他每一个 Token，以弄清楚什么才是重要的。这确实非常强大。但也确实非常昂贵。
- 注意力背后的数学计算复杂度呈二次方增长。
- 上下文长度翻倍，计算量就会变成原来的四倍。
- 在 128K Token 时，模型所做的计算量比在 4K 时要多出几个数量级。
- KV 缓存随着序列长度的增加而爆发式增长。
- 在一个标准的 128K 上下文的 7B Transformer 模型上，仅 KV 缓存就能占用 30 到 40 GB 的显存。
模型的权重没有改变。只是序列变长了而已。
> Transformer 面临序列长度的二次复杂度 O(N²)，这限制了它们在长序列任务中的扩展性。
这不是一个等待更好的 GPU 来解决的硬件问题。这是一个结构性问题。而这正是状态空间模型设计要解决的。
## 那么到底什么是状态空间模型？
让我们从一个非常直观的例子开始：
- 想象一下你在听一场很长的讲座时是如何记笔记的。
- 你不会写下每一个字。
- 你会在脑海中维持一个不断更新的摘要
- 哪些是重要的，哪些要保留，哪些要舍弃，并随着新信息的到来而更新它。
- 讲座结束时，那个摘要就是你的状态。
- 你从未存储每一个字。你存储的是相关的内容。
这大概就是 SSM 所做的事情。它不关注过去的每一个 Token，而是维护一个固定大小的隐藏状态，该状态在每一步都会更新。整个序列的历史被压缩到这个状态中。无论序列变得多长，内存占用都保持不变。
```python
# 最小化 SSM：离散时间状态更新（核心思想）
import torch
import torch.nn as nn
class MinimalSSM(nn.Module):
def __init__(self, d_model, d_state):
super().__init__()
self.d_model = d_model
self.d_state = d_state
# A：状态转移矩阵（保留什么）
self.A = nn.Parameter(torch.randn(d_state, d_state) * 0.01)
# B：输入投影（吸收什么）
self.B = nn.Parameter(torch.randn(d_state, d_model) * 0.01)
# C：输出投影（输出什么）
self.C = nn.Parameter(torch.randn(d_model, d_state) * 0.01)
def forward(self, x):
# x: (batch, seq_len, d_model)
batch, seq_len, _ = x.shape
h = torch.zeros(batch, self.d_state, device=x.device)
outputs = []
for t in range(seq_len):
# 状态更新：压缩历史 + 吸收新输入
h = h @ self.A.T + x[:, t, :] @ self.B.T
y = h @ self.C.T
outputs.append(y)
return torch.stack(outputs, dim=1)
```
> 状态进入，状态更新，输出产生。没有注意力矩阵。没有二次方爆炸。线性时间，恒定内存。
## 为什么公司要投资于此
因为推理账单是真金白银。运行一个 256K 上下文的 Transformer 不仅慢，而且其昂贵程度足以拖垮生产系统。
## 英伟达 (NVIDIA)
当英伟达在 2025 年发布 Nemotron-H 时，他们并没有将其标榜为研究实力的展示。他们将其界定为一项工程决策。
“Attention was never enough: Tracing the rise of hybrid LLMs”这篇文章完美地阐述了英伟达的做法。
> 英伟达在其 Nemotron-H 模型中将 92% 的注意力层替换为 Mamba2 块，在与 LLaMA-3.1 和 Qwen-2.5 相同的模型规模下，推理吞吐量提高了高达 3 倍。
Attention was never enough Blog
## IBM
IBM 也做了同样的事情，推出了 Bamba-9B，这是一个混合 SSM 模型，通过与 Llama-3.1 8B 相当的性能运行，但通过量化显著减少了内存占用。
## 微软
微软的 Phi-4-mini-flash-reasoning 引入了他们自己的混合架构 SambaY，混合了 Mamba、滑动窗口注意力和门控记忆单元，实现了比前代产品高 10 倍的吞吐量增益。
这不是学术界在做实验。这些是解决推理成本问题的工程团队。之所以看起来“秘密”，是因为没有人会发新闻稿说“我们悄悄改变了模型 90% 的内部结构”。他们只是直接发布产品。
## 当前重要的 SSM 架构
SSM 家族发展迅速。以下是真正重要的演变脉络：
> **Siddharth@Pseudo_Sid26**: [原文链接](https://x.com/Pseudo_Sid26/status/1983116683796468172)
>
> 注意力机制在机器学习中几乎无处不在，但它真的不可替代吗？有没有人尝试过取代注意力机制？
> 已经有几项研究试图替换或改进注意力机制 -
> > 状态空间模型
> > Mamba 模型构建完成
>
> 
**S4 (2021)**：由斯坦福大学的 Albert Gu 等人提出的基础性结构化状态空间模型。证明了 SSM 处理长距离依赖的能力优于 RNN，特别是在长序列竞技场基准测试中。架构很优雅，但参数是固定的，不依赖于输入。
**Mamba (2023)**：Albert Gu 和 Tri Dao 的突破性进展。核心创新在于：让 SSM 参数（A, B, C）依赖于输入。突然之间，模型可以选择性地决定记住什么、让什么淡化。Mamba-3B 在语言基准测试中超越了同等规模的 Transformer，并以 5 倍的推理吞吐量运行。
```python
# 选择性 SSM：核心 Mamba 思想 —— 参数依赖于输入
class SelectiveSSM(nn.Module):
def __init__(self, d_model, d_state, d_conv=4):
super().__init__()
self.d_model = d_model
self.d_state = d_state
# 依赖输入的投影（这正是使其具有“选择性”的原因）
self.x_proj = nn.Linear(d_model, d_state * 2 + d_model)
self.conv = nn.Conv1d(d_model, d_model, kernel_size=d_conv, padding=d_conv - 1, groups=d_model)
self.out_proj = nn.Linear(d_model, d_model)
# 固定的对数空间 A 初始化（稍后离散化）
self.A_log = nn.Parameter(torch.log(torch.arange(1, d_state + 1).float()).unsqueeze(0).repeat(d_model, 1))
self.D = nn.Parameter(torch.ones(d_model))
def forward(self, x):
B_batch, L, D = x.shape
# 用于短距离上下文的局部卷积
x_conv = self.conv(x.transpose(1, 2))[:, :, :L].transpose(1, 2)
x_conv = torch.sigmoid(x_conv) * x_conv # SiLU 激活
# 投影以获取依赖于输入的 B, C, dt
x_dbl = self.x_proj(x_conv)
dt, B, C = x_dbl.split([D, self.d_state, self.d_state], dim=-1)
dt = torch.softplus(dt)
# 使用零阶保持对 A 进行离散化
A = -torch.exp(self.A_log)
dA = torch.exp(dt.unsqueeze(-1) * A.unsqueeze(0).unsqueeze(0))
dB = dt.unsqueeze(-1) * B.unsqueeze(2)
# 选择性扫描：逐个 Token 更新隐藏状态
h = torch.zeros(B_batch, D, self.d_state, device=x.device)
outputs = []
for t in range(L):
h = dA[:, t] * h + dB[:, t] * x_conv[:, t].unsqueeze(-1)
y = (h * C[:, t].unsqueeze(2)).sum(-1)
outputs.append(y)
y = torch.stack(outputs, dim=1) + self.D * x_conv
return self.out_proj(y)
```
**Mamba-2 (2024)**：Dao 和 Gu 在 ICML 的论文证明了一个令人惊讶的结论：Transformer 和 SSM 在数学上通过结构化半可分矩阵联系在一起。Mamba-2 的 SSD 层运行速度比 Mamba-1 快 2–8 倍，同时在语言任务上保持竞争力。
甚至 Mamba-3 也发布了，我之前也在一篇文章中解释过它！
> **Siddharth@Pseudo_Sid26**: [原文链接](https://x.com/Pseudo_Sid26/status/2034303554559905850)
>
> Mamba-3 刚刚发布。你可能因为它只是另一个线性模型而忽略了它，但这可能是状态空间模型的下一个真正阶段。
> 让我们来分解一下：
> -状态空间模型
> >这些模型不依赖注意力
> >它们将序列视为连续的动态系统
>
> 
**S5, H3, Hyena**：中间架构，提高了训练稳定性，并在较小规模下探索了 SSM-注意力的混合体。H3 是最早混合 SSM 和注意力的架构之一。Hyena 试图完全用长卷积来取代注意力。两者都没有在生产中占据主导地位，但它们为后续的所有发展提供了借鉴。
## 混合架构：注意力 + SSM
纯 SSM 有一个真正的弱点。它们将所有内容压缩到一个固定的隐藏状态中，这意味着精确召回很难。如果你问一个纯 Mamba 模型“8000 个 Token 之前用户提到的确切短语是什么？”，它可能会很吃力。而可以对所有过去 Token 进行直接注意的 Transformer 则可以轻松处理这个问题。
再回想一下你的大脑。
- 你有工作记忆，用于处理你正在积极思考的事情。
- 你有长期记忆，用于总结和压缩其他所有内容。
- 你不会逐字逐句地重放每一次对话。
- 但是当某些具体的事情变得重要时，你可以准确地提取出来。
混合架构正是如此。少量的注意力层处理精确召回。SSM 层高效地处理大部分序列处理。这个比例并不是 50:50。
一个简单的 Jamba 架构代码：
```python
# 简化的 Jamba 风格混合块
class HybridBlock(nn.Module):
def __init__(self, d_model, d_state, n_heads, use_attention=False):
super().__init__()
self.use_attention = use_attention
self.norm1 = nn.RMSNorm(d_model)
self.norm2 = nn.RMSNorm(d_model)
if use_attention:
self.mixer = nn.MultiheadAttention(d_model, n_heads, batch_first=True)
else:
self.mixer = SelectiveSSM(d_model, d_state)
self.ffn = nn.Sequential(
nn.Linear(d_model, d_model * 4),
nn.GELU(),
nn.Linear(d_model * 4, d_model)
)
def forward(self, x, mask=None):
residual = x
x = self.norm1(x)
if self.use_attention:
x, _ = self.mixer(x, x, x, attn_mask=mask)
else:
x = self.mixer(x)
x = x + residual
x = x + self.ffn(self.norm2(x))
return x
def build_jamba_style(d_model, d_state, n_heads, n_layers, attn_every=8):
layers = []
for i in range(n_layers):
use_attn = (i % attn_every == 0)
layers.append(HybridBlock(d_model, d_state, n_heads, use_attention=use_attn))
return nn.Sequential(*layers)
```
Jamba 论文在构建过程中注意到了一件有趣的事情：在混合设置中，Mamba-1 结合注意力的表现实际上优于 Mamba-2 结合注意力。
架构之间的相互作用比每个组件单独的性能更重要。这种发现只有在大规模构建中才能得出，而非理论推导。
## 内存问题：SSM 是终极解决方案吗？
> Transformer 拥有研究人员所说的情景记忆，它们可以重放过去的确切 Token，因为所有内容都存储在 KV 缓存中。
> SSM 拥有语义记忆，这是历史的压缩、有损表示，它捕捉了意义但不一定是确切的文本。
> Mamba 架构明确指出，选择性扫描机制让模型能够根据内容决定保留什么、淡化什么。
- SSM 用压缩的隐藏状态交换了显式的 KV 缓存。
- Transformer：显式情景记忆。
- SSM：隐式语义记忆，过去内容的压缩表示。
## 1. 上下文窗口不是限制
通常的说法是“Transformer 有更长的上下文”。这不太对。Transformer 拥有跨越长上下文进行关注的能力，但运行它们的硬件设定了真正的上限。
KV 缓存随着每一个 Token、每一层、每一个用户而增长。在 57K Token 时，消费级 GPU 上的 Transformer 显存已经耗尽。
SSM 可以在同样的 24GB 显存预算下处理高达 220K Token，因为隐藏状态从不增长，它只是更新。
一项 2025 年的基准测试研究直接指出了这一点：虽然在 8K Token 以下的短序列中，Transformer 的速度比 SSM 快高达 1.9 倍，但在长上下文下，SSM 的表现发生了完全的反转，在 57K Token 左右速度快了 4 倍，内存减少了 64%。
当序列长到足以发挥作用时，上下文窗口就不再是 Transformer 的优势了。而智能体正在运行非常长的序列。工具调用、观察日志、多步推理追踪，这些东西堆积得非常快。
> SSM 不仅仅是处理长上下文。它们是以固定的、可预测的内存成本来处理它，这是当你为数百个并发智能体会话提供服务时唯一重要的事情。
## 2. 智能体记忆不是检索：它是状态
这是每个智能体系统都会遇到的问题。一个执行 50 步任务的智能体会积累巨大的运行上下文：先前的工具输出、失败的尝试、中间状态、30 步之前的用户指令。
Transformer 的方法是将所有这些保留在上下文窗口中，让注意力来处理。这变得很昂贵。而且还会变得不连贯 —— 当你只需要模型记住它在工作流中的位置时，注意力却平均地分散在所有内容上。
以“写入-管理-读取”循环为例：智能体需要能够持久化、压缩并在交互之间进行选择性召回的记忆。
- SSM 隐藏状态在结构上就是那个循环。
- 每个 Token 更新状态。
- 不相关的信息淡化。
- 与任务相关的结构积累。
这不是从数据库中检索，而是模型像在脑海中延续对话一样向前推进任务，而无需重放说过的每一个字。
苹果研究院进一步推进了这一点
当 SSM 被赋予对外部工具的交互式访问权限时，它们在算术、推理和编码任务上实现了长度泛化，尽管状态大小固定，却能解决任意复杂度的问题。
那篇论文的结论很直接：SSM 是 Transformer 的一个实用、高效的替代品，特别是在基于工具和智能体的设置中。
> 智能体记忆问题不是“你如何存储更多？”而是“你如何跨数百步高效地携带状态？”SSM 正是为解决此问题而设计的。
## 3. 大脑不运行注意力，智能体也不应如此
在学习记忆 Transformer 或 SSM 时，我有个习惯，我通常将所有事物与人类记忆或人脑的工作方式联系起来，因为归根结底神经网络只是大脑中神经元的复制，对吧？每当我试图将所有这些联系起来时，我会尝试将其与大脑的工作方式联系起来，这也是我尝试理解这种记忆或智能体问题的方式。使用人类大脑类比。
哈佛大学的一篇论文提出了一个大多数人忽略的观点：人脑中的语言处理似乎更接近于 SSM 处理语言的方式，而不是 Transformer。Transformer 对所有先前的 Token 进行全注意力处理，而人脑则……