# Imagine if naked people were stupider. It turns out, naked models are actually are.
**作者**: Garry Tan
**日期**: 2026-04-19T09:36:10.000Z
**来源**: [https://x.com/garrytan/status/2045798603059548364](https://x.com/garrytan/status/2045798603059548364)
---
Sorry, not that kind of naked model. But the disappointment you're feeling right now? That's exactly how Kyle Kingsbury feels about LLMs.
Kyle Kingsbury is one of the best systems engineers alive. His Jepsen project spent a decade methodically proving that distributed databases didn't work as advertised. That CockroachDB, MongoDB, Redis, and dozens of others made consistency guarantees they couldn't keep. He published the results, the vendors fixed the bugs, and the entire industry got more honest. Jepsen is a masterwork of applied skepticism.
Last week he published a 32-page essay called "The Future of Everything is Lies, I Guess: Bullshit About Bullshit Machines." It's beautifully written, deeply researched, genuinely funny, and wrong about the most important thing.
His observations are correct. His conclusion is not.
A note on scope: this essay addresses Kingsbury's technical claims: that LLMs are unreliable bullshit machines incapable of producing trustworthy output. His broader concerns about labor displacement, information ecology, and cultural impact are real, separate questions that deserve their own essays. I'm not dismissing them by not addressing them here. I'm addressing the architecture question: does model unreliability make useful systems impossible, or does it make them an engineering problem? I think it's the latter. Kingsbury's essay assumes the former. That's where we disagree.
## Testing the engine on a bench
Kingsbury's essay is structured as a catalogue of LLM failures. He asked Gemini to apply materials to a 3D bathroom rendering. It forgot the toilet and changed the room's shape. He asked Claude to do image-to-image transformation. It produced thousands of lines of JavaScript creating an incomprehensible garble of nonsense polygons. He asked ChatGPT to put white patches on a blue shirt. It changed the color, moved the patches, deleted them. He watched a colleague's LLM claim to download stock data and produce a graph of randomly generated numbers.
Every single one of these failures is real. I've seen failures like them. Everyone building with LLMs has. Kingsbury is not making anything up.
But here's what's happening in each example: a human sits in front of a raw language model, types a request in natural language, and watches it fail.
No skill file telling the model how to approach the task. No deterministic tool handling the parts that require precision. No resolver routing the request to the right capability. No harness managing context, enforcing safety, or constraining behavior.
He's testing the engine on a bench and concluding that cars are unsafe.
## The architecture question
Some terms for readers who haven't read the first essay in this series. A skill file is a reusable markdown document that teaches the model how to approach a task — a procedure, not a prompt. A resolver is a routing table that tells the model which document to read for which task. Deterministic code is software that produces the same output every time — SQL queries, API calls, math — the parts the model should never touch. A harness is the thin conductor that runs the model in a loop, reads files, and manages context. Together: thin harness, fat skills.
Kingsbury's central claim is that LLMs are "bullshit machines." They confabulate, they're chaotic, they're vulnerable to manipulation, and they can't be trusted. He arrives at this by testing models in isolation, the way you'd test a function: input in, output out, evaluate the output.
The people having Kingsbury's problems are the ones doing exactly this: chatting with a raw model and expecting reliable output. The people who aren't having those problems built harnesses. Not because harnesses make the model trustworthy. Because harnesses make the system trustworthy, even when the model inside it is not.
Kingsbury knows the harness exists, incidentally. He cites the Claude Code source leak: 512,000 lines of engineering that Anthropic built around their own model. Even the makers of the best LLM in the world don't trust the model naked. They wrapped it in live repo context, prompt caching, purpose-built tools, session memory, and parallel sub-agents. That's 512,000 lines of evidence that model reliability is an engineering problem, not a philosophical one.
But I want to be honest about something: 512,000 lines is not simple. The pattern I'll describe: skills, resolvers, deterministic code, testing— is conceptually clean. Robust implementation is real engineering. Harnesses fail. Skills encode wrong procedures. Verification layers are only as good as the invariants they check. The claim is not that harnesses make everything easy. The claim is that they turn model unreliability from a reason-to-stop into a problem-to-solve. That distinction matters.
The stock data example is the clearest illustration. An LLM claimed to download stock prices and produced a graph. The data was random. Kingsbury presents this as evidence that LLMs lie. But what actually happened? Someone asked a language model (a text prediction machine) to fetch data from the internet. It can't fetch data from the internet. It has no tools. It has no HTTP client. It has no API keys. So it did what language models do: it produced text that looked like what a stock data response would look like.
The fix isn't a better model. The fix is a deterministic tool: a function that actually calls a stock API, returns real numbers, and hands them to the model as context. The model never touches the data retrieval. It decides what to look up. The code decides how. Same input, same output, every time.
That's the architecture I described in "Thin Harness, Fat Skills." Push intelligence up into skills. Push execution down into deterministic code. Keep the harness thin. When you do this, the class of failures Kingsbury describes becomes far less likely. Not because the model got smarter, but because the model was never asked to do something it can't do. Not all failures vanish. But the failure mode shifts from "the model hallucinated" to "the skill was wrong" or "the tool had a bug." And those are debuggable, testable, fixable problems. That's the difference between chaos and engineering.
## The bathroom problem
Take Kingsbury's bathroom example. He asked Gemini (a language model) to apply materials to a 3D rendering. Gemini isn't an image editor. It's not a 3D modeling tool. It's a text prediction system that has been given image capabilities as a bolted-on feature. Of course it forgot the toilet. Of course it changed the room's shape. It was playing improv with pixels.
A properly harnessed system would handle this differently. The skill file would say: decompose the task into steps.
- Step 1: identify every surface in the image (use a vision model)
- Step 2: for each surface, select the appropriate material (latent — the model decides)
- Step 3: apply the material using a deterministic image processing tool (code — Pillow, OpenCV, a Blender script)
- Step 4: verify the output geometry matches the input geometry (deterministic comparison)
The model does the judgment. The code does the execution. The resolver routes between them. The harness orchestrates the sequence.
Would this always work? No. The skill might decompose the task wrong. The vision model might misidentify a surface. The deterministic comparison might have the wrong tolerance. These are real failure modes, and I've hit every one of them. But they're debuggable failure modes — you can find the step that went wrong, fix it, and run it again. Kingsbury's failures aren't debuggable because there's no system to debug. There's just a model playing improv.
## Jagged doesn't mean broken
One of Kingsbury's best observations is the "jagged technology frontier" — the idea that LLMs have irregular, unpredictable capability boundaries. They do multivariable calculus and fail simple word problems. They write essays and can't count letters.
This is correct and important. But Kingsbury draws the wrong conclusion. He argues that the jagged frontier makes LLMs unsuitable for tasks requiring reliability. What it actually means is that you need routing.
A resolver is a routing table for context. When task type X appears, load skill Y. When the task requires letter-counting, route it to code. A three-line Python function. When the task requires essay writing, route it to the model. When the task requires image editing, route it to a pipeline that combines both.
The jagged frontier is an argument for harness engineering, not against AI. The irregularity is real. The solution is routing around it! You must put deterministic code where the model is weak and model judgment where the code can't reason. The resolver maps the territory.
In my own system, I designed my OpenClaw's brain resolver to be 80 lines of markdown, in a numbered decision tree that routes every piece of knowledge to the right directory. When we didn't have it, skills made their own filing decisions and 10 out of 13 were wrong. When we added it, misfilings dropped to zero. The model didn't get smarter. The routing got explicit.
## Chaos is an argument for constraints
Kingsbury argues that LLMs are chaotic systems. They're sensitive to small input perturbations, vulnerable to adversarial manipulation, unpredictable in behavior. He's right. Rephrasing a question changes the answer. Rearranging sentences changes the output. Invisible Unicode characters can hijack behavior.
He presents this as a fundamental problem. It is! If you're feeding raw, unstructured input to a naked model.
But it's doesn't have to be if you're constraining the input through skill files.
A skill file is a structured markdown document that defines the procedure. It tells the model what to read, what to consider, what format to output, and what constraints to observe. The input to the model isn't "fix this image" — it's a 200-line document that says: here's the task, here's the process, here's what good output looks like, here are the tools you have, here are the ones you don't, here's what to do if you're uncertain.
Structured input through a skill file is dramatically less chaotic than freeform natural language. The skill constrains the trajectory. It doesn't eliminate chaos (nothing does in a stochastic system) but it channels it, the way banks channel a river. Kingsbury is right that unbanked rivers flood. That's not an argument against rivers. It's an argument for banks.
## The reasoning red herring
Kingsbury makes a clever point about "reasoning" models. He notes that chain-of-thought traces (the stream-of-consciousness text that models emit while "thinking") are "essentially LLMs writing fanfic about themselves." He cites Anthropic's finding that Claude's chain-of-thought traces don't reliably reflect its actual reasoning process.
This is true and irrelevant. The chain-of-thought trace is the scratchpad, not the product. When a human does long division, the intermediate steps on paper aren't a truthful record of their neuronal activity — they're a tool for organizing the computation. Nobody evaluates a mathematician by the beauty of their scratch work.
What matters is the output. Does the system (model plus harness plus tools) produce correct, verifiable results? That's the testable question. Kingsbury doesn't test it, because he doesn't build the system.
He evaluates the scratchpad instead of the answer.
## We don't know why aspirin works either
Near the end of his essay, Kingsbury observes that "we don't really know why transformer models have been so successful, or how to make them better." This is true. It's also true of aspirin (the mechanism of action wasn't fully understood until the 1970s), general anesthesia (still incompletely understood), and bicycle stability (the gyroscopic theory was wrong — it was definitively explained only in 2011).
Practical utility doesn't require theoretical completeness. Engineering can proceed while research continues, which is how it always has.
We didn't stop prescribing aspirin while waiting for the 1971 mechanism paper. We didn't ground planes while debating exactly why wings generate lift. The question isn't whether we understand the mechanism. The question is whether we can build reliable systems with the capabilities that exist, test them rigorously, and improve them as understanding deepens. Kingsbury implicitly assumes that incomplete understanding means we can't build. The entire history of engineering says otherwise.
## What Jepsen would actually find
Here's the irony. Kingsbury's Jepsen methodology is exactly the right approach for AI systems. But it is just applied to the wrong layer.
Jepsen tests databases by injecting failures and checking invariants. It doesn't test the CPU. It doesn't test the operating system. It tests the database — the full system, under stress, against its own claims. When CockroachDB claimed linearizable consistency, Jepsen injected network partitions and checked whether the claim held.
Apply the same methodology to AI systems and the targets are obvious. Don't test whether the model hallucinates! Of course it does. Test whether the system hallucinates:
- Does the harness prevent hallucinated data from reaching the user?
- Does the skill file route the task to deterministic code where precision matters?
- Does the resolver fire for the right inputs?
- Does the entity propagation complete?
These are testable claims. We test them.
- Unit tests for deterministic code.
- Integration tests for pipeline correctness.
- Resolver trigger evals for routing accuracy.
- LLM-as-judge evals for output quality.
- End-to-end tests for the full pipeline.
The testing pyramid for agent systems exists. It's just not what Kingsbury tested. He tested the raw model, which is like running Jepsen against the bare filesystem instead of the database.
## The aesthetics of truth
Let me address Kingsbury's deepest argument, the one that most readers will feel even if they can't articulate it. LLMs don't produce truth. They produce text that looks like truth. The aesthetics of truth-telling without the epistemological grounding. They are, in the philosophical sense that Harry Frankfurt defined, bullshit machines: systems indifferent to the truth-value of their output.
This is correct. And it's precisely why the architecture matters.
A naked model produces plausible text. A harnessed system produces verified text.
The skill file says "check your work against the source data."
The deterministic code says "compare this output to the ground truth and reject if it diverges."
The model produces a draft.
The system produces a verified result.
The gap between plausible and verified is exactly the gap that harness engineering fills.
But here's the part Kingsbury misses entirely: the quality of the verification depends on the quality of the skill file. And the skill file is written by a human, who is the founder, the developer, or the prompt writer.
This is why open source matters so much. A closed-source agent can never let you write the skill that verifies its output, because legalism and safetyism prevent the kind of deep customization that real verification requires. You need to be able to say: "here is my schema, here are my invariants, here is what correct looks like in my domain, now verify against that." You can't do that through an API that guards its system prompt.
The epistemological problem is real. The solution is open harnesses where the user controls the verification layer. Not trust. Engineering.
This is exactly what my YC partner and friend Pete Koomen talks about in AI Horseless Carriages. The user must write their prompt, otherwise we'll be slaves to a system prompt that we can't see.
## Cars
At the end of his essay, Kingsbury compares AI to the automobile. He asks the reader to consider not how fast cars are, but what they did to cities. His answer? Sprawl, lead poisoning, bulldozed communities, car dependency. It's a good metaphor.
But he draws exactly the wrong lesson from it. We didn't solve car problems by being skeptical of engines. We solved them with engineering: seatbelts, crumple zones, catalytic converters, traffic lights, highway design, fuel injection, ABS, airbags, emissions standards. Every decade brought new engineering that made the system safer, more efficient, and more useful — not by making the engine smarter, but by building better systems around it.
Skepticism about engines never saved a life. Engineering the chassis did.
That's the answer to Kingsbury's essay. Not "be careful." Not "these machines are liars."
The answer is:
- Build the system.
- Write the skills.
- Test the code.
- Route with resolvers.
- Make the deterministic parts deterministic.
- Make the latent (model) parts constrained.
- Test the system, not the model.
- And when the system fails — because it will — debug the step that broke, fix the skill, and run it again.
Kingsbury is right that naked models are unreliable. He is right that they confabulate, that they're chaotic, that they produce the aesthetics of truth rather than truth itself. Where he goes wrong is in treating these properties as verdicts rather than constraints. The model is unreliable. The system doesn't have to be.
The model is the engine. The harness is the car. Build the car.
--
I open-sourced the architecture described in this series. GBrain is the knowledge layer to let you build your own personal OpenClaw/Hermes Agent that does work for you. GStack is the skill layer to go fast with Claude Code.
## 相关链接
- [Garry Tan](https://x.com/garrytan)
- [@garrytan](https://x.com/garrytan)
- [4.8K](https://x.com/garrytan/status/2045798603059548364/analytics)
- [Jepsen](https://jepsen.io/)
- ["The Future of Everything is Lies, I Guess: Bullshit About Bullshit Machines."](https://aphyr.com/data/posts/411/the-future-of-everything-is-lies.pdf)
- [first essay in this series](https://x.com/garrytan/status/2042925773300908103)
- [thin harness, fat skills](https://x.com/garrytan/status/2042925773300908103)
- ["Thin Harness, Fat Skills."](https://x.com/garrytan/status/2042925773300908103)
- ["jagged technology frontier"](https://www.hbs.edu/ris/Publication%20Files/24-013_d9b45b68-9e74-42d6-a1c6-c72fb70c7571.pdf)
- [80 lines of markdown](https://x.com/garrytan/status/2044479509874020852)
- [Anthropic's finding](https://www.anthropic.com/research/reasoning-models-dont-say-think)
- [Resolver trigger evals](https://x.com/garrytan/status/2044479509874020852)
- [AI Horseless Carriages](https://koomen.dev/essays/horseless-carriages/)
- [GBrain](https://github.com/garrytan/gbrain)
- [GStack](https://github.com/garrytan/gstack)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [5:36 PM · Apr 19, 2026](https://x.com/garrytan/status/2045798603059548364)
- [4,840 Views](https://x.com/garrytan/status/2045798603059548364/analytics)
- [View quotes](https://x.com/garrytan/status/2045798603059548364/quotes)
---
*导出时间: 2026/4/19 18:22:06*
---
## 中文翻译
# 想象一下，如果裸体的人更蠢。结果发现，裸模型确实更蠢。
**作者**: Garry Tan
**日期**: 2026-04-19T09:36:10.000Z
**来源**: [https://x.com/garrytan/status/2045798603059548364](https://x.com/garrytan/status/2045798603059548364)
---
抱歉，不是那种裸体模型。但你此刻感受到的失望？这正是 Kyle Kingsbury 对 LLM（大型语言模型）的感受。
Kyle Kingsbury 是当今最优秀的系统工程师之一。他的 Jepsen 项目花了十年时间，有条不紊地证明分布式数据库并不像广告中宣传的那样运作。CockroachDB、MongoDB、Redis 等数十个数据库做出了它们无法遵守的一致性保证。他发布了结果，供应商修复了 Bug，整个行业变得更加诚实。Jepsen 是应用怀疑论杰作。
上周，他发表了一篇 32 页的文章，题为《万物的未来都是谎言，我猜：关于废话机器的废话》。文章写得优美，研究深入，确实很幽默，但在最重要的问题上搞错了。
他的观察是正确的。但他的结论是错误的。
关于范围的一点说明：本文针对的是 Kingsbury 的技术主张，即 LLM 是不可靠的“胡扯机器”，无法产生值得信赖的输出。他更广泛的担忧——关于劳动力置换、信息生态和文化影响——是真实存在的，但它们是独立的问题，值得单独撰文探讨。我不在此处讨论这些问题并不意味着我不予理会。我要解决的是架构问题：模型的不可靠性是否意味着有用系统的建立是不可能的，还是说这仅仅是一个工程问题？我认为是后者。Kingsbury 的文章假设是前者。这就是我们分歧的所在。
## 在台架上测试引擎
Kingsbury 的文章结构就像是一本 LLM 失败目录。他让 Gemini 将材质应用到一个 3D 卫生间渲染图中。它忘记了马桶，还改变了房间的形状。他让 Claude 进行图像到图像的转换。它产生了数千行 JavaScript，生成了一堆难以理解的乱码多边形。他让 ChatGPT 在蓝色衬衫上打上白色补丁。它改变了颜色，移动了补丁，甚至删除了它们。他看到一位同事的 LLM 声称下载了股票数据，并生成了一张由随机数字组成的图表。
每一个失败都是真实的。我见过类似的失败。每个使用 LLM 构建应用的人都有同感。Kingsbury 没有编造任何东西。
但在每个例子中实际发生的情况是：一个人坐在原始语言模型面前，用自然语言输入请求，然后看着它失败。
没有技能文件告诉模型如何处理任务。没有确定性工具处理需要精确度的部分。没有解析器将请求路由到正确的功能。没有线束来管理上下文、执行安全约束或限制行为。
他是在台架上测试引擎，并得出结论说汽车是不安全的。
## 架构问题
给没有读过本系列第一篇文章的读者解释一些术语。技能文件是一种可重用的 Markdown 文档，它教模型如何处理任务——它是一个程序，而不是一个提示词。解析器是一个路由表，告诉模型对于哪个任务应该读取哪个文档。确定性代码是指每次都产生相同输出的软件——SQL 查询、API 调用、数学运算——这些是模型永远不应触碰的部分。线束是一个轻量级的导体，它运行模型循环、读取文件并管理上下文。总之：轻量级线束，重量级技能。
Kingsbury 的核心主张是，LLM 是“胡扯机器”。它们会捏造，混乱无序，易受操纵，且不可信任。他通过孤立地测试模型得出这一结论，就像测试函数一样：输入进去，输出来来，评估输出。
遇到 Kingsbury 所描述问题的人，正是那些做着完全相同事情的人：与原始模型聊天并期望可靠的输出。没有遇到这些问题的人构建了线束。这并不是因为线束让模型变得值得信任，而是因为线束让系统变得值得信任，即使其内部的模型并不值得信任。
顺便提一下，Kingsbury 知道线束的存在。他引用了 Claude Code 的源代码泄露：Anthropic 围绕他们自己的模型构建了 512,000 行工程代码。即使是世界上最好的 LLM 制造者也不信任裸模型。他们用实时代码仓库上下文、提示词缓存、专用工具、会话记忆和并行子代理将它包裹起来。这 512,000 行代码证明，模型的可靠性是一个工程问题，而不是哲学问题。
但我想诚实地指出一点：512,000 行代码并不简单。我将描述的模式——技能、解析器、确定性代码、测试——在概念上是清晰的。但稳健的实现是真正的工程挑战。线束会失效。技能文件会编码错误的程序。验证层的效果取决于它所检查的不变量。我们要声称的并不是线束让一切变得简单，而是它们将“模型不可靠性”从一个停止的理由变成了一个待解决的问题。这种区别很重要。
股票数据的例子就是最清楚的说明。一个 LLM 声称下载了股票价格并生成了图表。数据是随机的。Kingsbury 将此作为 LLM 撒谎的证据。但实际发生了什么？有人问一个语言模型（一个文本预测机器）从互联网获取数据。它无法从互联网获取数据。它没有工具。它没有 HTTP 客户端。它没有 API 密钥。所以它做了语言模型会做的事：它生成了看起来像股票数据响应的文本。
解决方案不是更好的模型。解决方案是确定性工具：一个真正调用股票 API、返回真实数字并将其作为上下文传递给模型的函数。模型从不接触数据检索。模型决定查找什么。代码决定如何查找。每次相同的输入，相同的输出。
这就是我在“轻量级线束，重量级技能”中描述的架构。将智能上移至技能层。将执行下移至确定性代码层。保持线束轻量。当你这样做时，Kingsbury 描述的那类失败变得不太可能发生。这不是因为模型变聪明了，而是因为从未要求模型去做它做不到的事情。并非所有错误都会消失。但失败模式从“模型产生幻觉”转变为“技能错了”或“工具有 Bug”。而这些是可调试、可测试、可修复的问题。这就是混乱与工程之间的区别。
## 卫生间问题
以 Kingsbury 的卫生间例子为例。他让 Gemini（一个语言模型）将材质应用到 3D 渲染图中。Gemini 不是图像编辑器。它不是 3D 建模工具。它是一个文本预测系统，图像功能只是作为附加特性加上的。它当然忘记了马桶。它当然改变了房间的形状。它只是在用像素进行即兴创作。
一个配备了适当线束的系统会以不同的方式处理这个问题。技能文件会写道：将任务分解为步骤。
- 第 1 步：识别图像中的每个表面（使用视觉模型）
- 第 2 步：为每个表面选择合适的材质（潜在部分——由模型决定）
- 第 3 步：使用确定性图像处理工具应用材质（代码——Pillow、OpenCV 或 Blender 脚本）
- 第 4 步：验证输出几何形状与输入几何形状匹配（确定性比较）
模型负责判断。代码负责执行。解析器在它们之间进行路由。线束负责编排序列。
这总是能成功吗？不。技能可能会错误地分解任务。视觉模型可能会错误识别表面。确定性比较的容差可能设置不当。这些都是真实的失败模式，我都遇到过。但它们是可调试的失败模式——你可以找到出错的步骤，修复它，然后再次运行。Kingsbury 的失败无法调试，因为没有系统可以调试。只有一个模型在即兴发挥。
## 参差不齐并不意味着损坏
Kingsbury 最好的观察之一是“参差不齐的技术前沿”——即 LLM 具有不规则、不可预测的能力边界。它们能做多元微积分，却解不出简单的单词问题。它们能写文章，却不会数字母。
这是正确的，也很重要。但 Kingsbury 得出了错误的结论。他认为参差不齐的前沿使得 LLM 不适合需要可靠性的任务。这实际意味着你需要路由。
解析器是上下文的路由表。当出现 X 类任务时，加载技能 Y。当任务需要数字母时，将其路由到代码。一个三行的 Python 函数。当任务需要写文章时，将其路由到模型。当任务需要图像编辑时，将其路由到结合了这两者的管道。
参差不齐的前沿是支持线束工程的论据，而不是反对 AI 的论据。不规则性是真实的。解决方案是绕过它！你必须在模型薄弱的地方放置确定性代码，在代码无法推理的地方放置模型判断。解析器描绘了这个领域。
在我自己的系统中，我将 OpenClaw 的大脑解析器设计为 80 行 Markdown，在一个编号的决策树中，将每一条知识路由到正确的目录。当我们没有它时，技能自己做出归档决定，13 个中有 10 个是错的。当我们添加它时，归档错误降为零。模型没有变聪明。路由变得显式了。
## 混乱是为约束提供了论据
Kingsbury 认为 LLM 是混沌系统。它们对微小的输入扰动敏感，易受对抗性操纵，行为不可预测。他是对的。改述一个问题会改变答案。重新排列句子会改变输出。不可见的 Unicode 字符可以劫持行为。
他将此作为一个基本问题提出来。确实如此！如果你向一个裸模型输入原始的、非结构化的数据。
但如果你通过技能文件约束输入，情况就不一定如此了。
技能文件是一个定义程序的结构化 Markdown 文档。它告诉模型读什么，考虑什么，输出什么格式，以及遵守什么约束。模型的输入不是“修复这张图片”——而是一份 200 行的文档，上面写着：这是任务，这是过程，好的输出是什么样的，这是你拥有的工具，这是你没有的工具，如果你不确定该怎么办。
通过技能文件的结构化输入比自由形式的自然语言混乱程度要低得多。技能约束了轨迹。它不会消除混乱（在随机系统中没有什么是可以消除混乱的），但它疏导了混乱，就像河岸疏导河流一样。Kingsbury 说得对，无河岸的河流会泛滥。但这不是反对河流的论据。这是支持河岸的论据。
## 推理的烟雾弹
Kingsbury 关于“推理”模型提出了一个聪明的观点。他指出，思维链痕迹（模型在“思考”时发出的意识流文本）本质上是“LLM 关于它们自己的同人文”。他引用 Anthropic 的发现，Claude 的思维链痕迹并不可靠地反映其实际的推理过程。
这是真的，但这无关紧要。思维链痕迹是草稿纸，而不是产品。当一个人做长除法时，纸上的中间步骤并不是他们神经元活动的真实记录——它们是组织计算的工具。没有人会根据草稿纸的美观程度来评价数学家。
重要的是输出。系统（模型加线束加工具）是否产生了正确的、可验证的结果？这是一个可测试的问题。Kingsbury 不测试这个，因为他不构建系统。
他评估的是草稿纸，而不是答案。
## 我们也不知道阿司匹林为什么起作用
在他文章的结尾，Kingsbury 观察到，“我们真的不知道为什么 Transformer 模型如此成功，也不知道如何让它们变得更好。”这是真的。对于阿司匹林（作用机制直到 1970 年代才被完全理解）、全身麻醉（仍然未被完全理解）和自行车稳定性（陀螺仪理论是错误的——直到 2011 年才得到明确解释）也是如此。
实用性并不要求理论上的完整性。工程可以在研究继续的同时进行，历来如此。
我们并没有在等待 1971 年机制论文的同时停止开阿司匹林。我们也没有在争论机翼究竟为什么产生升力的同时停飞飞机。问题不在于我们是否理解机制。问题在于我们是否能够利用现有的能力构建可靠的系统，对其进行严格测试，并在理解加深时改进它们。Kingsbury 隐含地假设理解不完整意味着我们无法构建。整个工程史证明了并非如此。
## Jepsen 实际上会发现什么
这就是讽刺之处。Kingsbury 的 Jepsen 方法论正是研究 AI 系统的正确方法。但它只是应用到了错误的层面上。
Jepsen 通过注入故障和检查不变量来测试数据库。它不测试 CPU。它不测试操作系统。它测试数据库——完整的系统，在压力下，针对其自身的声明。当 CockroachDB 声称线性一致性时，Jepsen 注入网络分区并检查声明是否成立。
将同样的方法应用于 AI 系统，目标就很明显了。不要测试模型是否产生幻觉！它当然会。测试系统是否产生幻觉：
- 线束是否防止幻觉数据到达用户？
- 技能文件是否将任务路由到需要精度的确定性代码？
- 解析器是否针对正确的输入触发？
- 实体传播是否完成？
这些都是可测试的声明。我们要测试它们。
- 针对确定性代码的单元测试。
- 针对管道正确性的集成测试。
- 针对路由准确性的解析器触发评估。
- 针对输出质量的 LLM 评判评估。
- 针对完整管道的端到端测试。
代理系统的测试金字塔是存在的。只是这不是 Kingsbury 测试的内容。他测试的是原始模型，这就像针对裸文件系统而不是数据库运行 Jepsen。
## 真理的美学
让我回应一下 Kingsbury 最深层的论点，这是大多数读者即使无法表达也能感受到的论点。LLM 不产生真理。它们产生看起来像真理的文本。具有真理叙述的美学，却没有认识论的基础。用 Harry Frankfurt 定义的哲学意义来说，它们是胡扯机器：对其输出的真值漠不关心的系统。
这是正确的。这也正是为什么架构很重要的原因。
裸模型产生看似合理的文本。配备线束的系统产生经过验证的文本。
技能文件说“根据源数据检查你的工作”。
确定性代码说“将此输出与基本真值进行比较，如果存在差异则拒绝”。
模型产生草稿。
系统产生经过验证的结果。
看似合理与经过验证之间的差距，正是线束工程所填补的差距。
但这是 Kingsbury 完全错过的部分：验证的质量取决于技能文件的质量。而技能文件是由人编写的，他是创始人、开发者或提示词编写者。
这就是为什么开源如此重要。闭源代理永远不能让你编写验证其输出的技能，因为法律主义和安全主义阻止了真正验证所需的深度定制。你需要能够说：“这是我的模式，这是我的不变量，这是在我的领域正确是什么样子的，现在对此进行验证。”你不能通过一个保护其系统提示词的 API 来做到这一点。
认识论问题是真实的。解决方案是用户控制验证层的开放线束。不是信任。是工程。
这正是我的 YC 合伙人和朋友 Pete Koomen 在《AI 无马马车》中所谈到的。用户必须编写他们自己的提示词，否则我们将成为无法看到的系统提示词的奴隶。
## 汽车
在他文章的结尾，Kingsbury 将 AI 比作汽车。他要求读者考虑的不是汽车有多快，而是它们对城市做了什么。他的答案？无序扩张、铅中毒、被推平的社区、汽车依赖。这是一个很好的隐喻。
但他从中得出了完全错误的教训。我们没有通过怀疑引擎来解决汽车问题。我们通过工程解决了它们：安全带、溃缩区、催化转化器、红绿灯、高速公路设计、燃油喷射、ABS、安全气囊、排放标准。每十年都带来新的工程，使系统更安全、更高效、更有用——不是通过让引擎更聪明，而是通过围绕它构建更好的系统。
对引擎的怀疑从未挽救过生命。对底盘的工程做到了。
这就是对 Kingsbury 文章的回答。不是“要小心”。不是“这些机器是骗子”。
答案是：
- 构建系统。
- 编写技能。
- 测试代码。
- 使用解析器进行路由。
- 让确定性部分变得确定性。
- 让潜在（模型）部分受到约束。
- 测试系统，而不是模型。
- 当系统失败时——因为它肯定会失败——调试出错的步骤，修复技能，然后再次运行。
Kingsbury 说裸模型不可靠是对的。他说它们会捏造、混乱、产生真理的美学而不是真理本身，也是对的。他错在把这些特性视为裁决而不是约束。模型不可靠。系统不一定非要是不可靠的。
模型是引擎。线束是汽车。去造那辆车吧。
--
我开源了这个系列中描述的架构。GBrain 是知识层，让你构建自己的个人 OpenClaw/Hermes 代理为你工作。GStack 是技能层，让你可以快速使用 Claude Code。
## 相关链接
- [Garry Tan](https://x.com/garrytan)
- [@garrytan](https://x.com/garrytan)
- [Jepsen](https://jepsen.io/)
- [本系列的第一篇文章](https://x.com/garrytan/status/2042925773300908103)
- [“轻量级线束，重量级技能”](https://x.com/garrytan/status/2042925773300908103)
- [“参差不齐的技术前沿”](https://www.hbs.edu/ris/Publication%20Files/24-013_d9b45b68-9e74-42d6-a1c6-c72fb70c7571.pdf)
- [80 行 Markdown](https://x.com/garrytan/status/2044479509874020852)
- [Anthropic 的发现](https://www.anthropic.com/research/reasoning-models-dont-say-think)
- [解析器触发评估](https://x.com/garrytan/status/2044479509874020852)
- [AI 无马马车](https://koomen.dev/essays/horseless-carriages/)
- [GBrain](https://github.com/garrytan/gbrain)
- [GStack](https://github.com/garrytan/gstack)
---
*导出时间: 2026/4/19 18:22:06*