# System Design for Agent Systems (Part 1)
**作者**: Karan
**日期**: 2026-08-20T13:52:08.000Z
**来源**: [https://x.com/kmeanskaran/status/2090436724250026177](https://x.com/kmeanskaran/status/2090436724250026177)
---
Put a frontier model into a badly designed agent system and you get a more articulate failure. Nearly all of the engineering sits outside the model, in the scaffolding that has come to be called the agent harness.
This part covers the five components you design whether or not you notice you are designing them: tools, prompts, memory, orchestration, and where a person sits in the loop. Then the evaluation framework that tells you whether any of it works, because none of this is testable the way ordinary code is.
Building agent systems requires intentional design across three key areas:
1. Understanding the workload: Agent jobs are fundamentally different from web requests
2. Designing the harness: The components you build that aren't the model (tools, prompts, memory, orchestration)
3. Building evaluation frameworks: How to measure quality and catch regressions
This guide covers all three. Everything is drawn from a real system: [DevVoice](https://github.com/kmeanskaran/agent-harness-ops), a five-agent pipeline that turns a GitHub README into a reviewed X thread, LinkedIn post, and [dev.to](//dev.to) article. These are the design decisions that determine whether your agent system is sound.
> Note: Code samples are illustrative pseudocode showing patterns and concepts. Adapt them to your framework (LangChain, LlamaIndex, Anthropic SDK, etc.).
Agent Harness Components
# Part 1: Foundations
Agent Workloads Are Not Web Workloads
Start here, because almost every downstream decision follows from it.
A typical web request is 50–200ms, CPU-bound in bursts, deterministic, and costs approximately nothing per unit. An agent job is none of those things:
Four architectural consequences:
Duration. You cannot answer the request synchronously. Load balancers have idle timeouts. Holding a connection open for two minutes per active job is wasteful.
I/O-bound. Concurrency is not limited by cores. A worker spends 70–80% of its time waiting for the model API. A 2-core container can run dozens of concurrent jobs. Sizing for core count wastes most of what you pay for.
Non-determinism. The same input produces different output, so retries are not free. You cannot assert on output in CI which tests must target the harness, not the model.
Marginal cost. A bug is a bill. An infinite loop in an agent loop burns money at roughly a dollar a minute, silently, until someone notices.
Design for this shape from the first commit, and deployment is configuration. Design for the web shape, and deployment is a rewrite.
# Part 2: The Agent Harness
What You Build That Isn't The Model
The harness is everything you build that isn't the model. Nine boxes of it, in the real system. Four are worth designing early.
## Component 1: Tools
Tools are the agent's only way to affect the world. Design them poorly and you waste model tokens on confusion; design them well and you cut context by 60%.
Tool definition is a contract:
```
@tool
def search_code(
query: str,
repository: str,
max_results: int = 10
) -> list[CodeMatch]:
"""Search for code patterns across a repository.
Args:
query: The code pattern or keyword to search for
repository: GitHub org/repo identifier
max_results: Limit results (default 10, max 100)
Returns:
List of matching code locations with context
"""
```
Three requirements:
1. Typed inputs and outputs. The model sees the schema; ambiguity costs tokens and errors. Use Pydantic models for complex returns.
2. Small surface area. One tool per discrete action. Five focused tools beat one with five optional parameters. Avoid tools that accept arbitrary queries the model will call them wrong.
3. Deterministic and fast. A tool that takes 30 seconds is a context killer. If an operation is expensive, make it async and poll with a status tool, or move it into a task queue.
Tool management:
- Keep tools in a registry, not scattered through agent code
- Version tools separately from agents: agents call search_v2 while others still use search_v1
- Log every tool call with arguments and latency, you'll debug half your issues here
- Add a timeout to every tool; a hung tool stalls the agent
## Component 2: Prompt Engineering
Prompts are not prose, they are state machines. Structure matters as much as words
The basic shape:
```
1. System prompt (static)
- Role and context
- Hard constraints and rules
- Output format specification
2. Static examples (static)
- Few-shot examples showing desired behavior
- Edge cases you've seen fail
3. Tools specification (static)
- Tool definitions and usage examples
- Error handling behavior
4. Dynamic context (dynamic)
- Conversation history
- Per-request data (user input, retrieved documents)
- Job state
5. User turn (dynamic)
- The actual request
```
The cardinal rule: Static content must precede dynamic content. Tool definitions, then system prompt, then examples, then history, then the user turn. Every dynamic value that leaks upward nukes the prompt cache and quietly doubles your token cost.
Versioning prompts:
```
PROMPT_VERSION = "2024-08-19-v3"
def build_system_prompt(version: str) -> str:
if version.startswith("2024-08-19"):
return """You are a code reviewer..."""
elif version.startswith("2024-08-15"):
return """You are a reviewer...""" # Previous version
```
Version your response-cache keys on prompt version. A bad prompt gets cached and served for hours after you fix it. With versioning, rolling back is one env var change.
## Component 3: Memory
Memory Layer
Agents need two kinds of memory, and they are not interchangeable.
Working memory (short-term, in-context):
- Current conversation turn
- Last N messages (retrieved)
- Retrieved context for this job
- Token budget: whatever fits in context
Persistent memory (long-term, storage-backed):
- User preferences and history
- Learned facts from past interactions
- Evaluation results and failures
- Stored indefinitely
The mistake: treating working memory like persistent memory. Adding "remember X" to the system prompt on every turn burns tokens. The right pattern:
```
# At job start: seed working memory
working_memory = {
"user_id": job.user_id,
"past_interactions": fetch_last_n_interactions(job.user_id, n=5),
"learned_preferences": fetch_user_preferences(job.user_id),
"retrieved_documents": retrieve_relevant_docs(job.query),
}
# During job: work from working memory
prompt = build_prompt_with_context(working_memory)
# At job end: update persistent memory
if agent.learned_something_new:
store_user_fact(job.user_id, fact)
```
Avoiding context overflow:
- Truncate working memory intelligently — drop oldest messages, keep most recent and most relevant
- Store summaries of long conversations in persistent memory, retrieve the summary instead of the full history
- Use content-based chunking for documents: break at section boundaries, not token limits
## Component 4: Multi-Agent Orchestration
Agent Orchestration
Agents can coordinate in two ways: parallel (agents work simultaneously, results are merged) and sequential (agents work one after another, output of one feeds into the next).
Parallel agents sound more efficient. They're not. They create a coordination problem: agent A produces output that agent B needs, but agents run at different speeds. You end up polling and merging, which is both slow and complex.
Sequential orchestration with subagents is better:
```
# Bad: parallel agents that might conflict
async def process_parallel(doc: str):
results = await asyncio.gather(
summarizer_agent(doc),
analyzer_agent(doc)
)
return merge_results(results)
# Good: sequential subagents with clear data flow
def process_sequential(doc: str):
summary = subagent_summarize(doc)
analysis = subagent_analyze(doc, context=summary)
final_review = subagent_review(doc, summary, analysis)
return final_review
```
Why subagents win:
- Clear data flow. Agent B explicitly depends on Agent A's output
- Early exit. If Agent A's output is invalid, you stop before starting Agent B
- Simpler state. Working memory is threaded through agents in order; no merging logic
- Easier debugging. Each subagent has one clear input and output
Use parallel agents only when agents genuinely work on independent data and their outputs never interact (e.g., three different APIs for the same query). Even then, consider sequential agents with fallback.
The orchestration surface should be clear:
```
class Orchestrator:
def run(self, job: Job) -> Result:
extracted = self.extract_agent.run(job.content, job.schema)
validated = self.validator_agent.run(extracted, job.rules)
formatted = self.formatter_agent.run(validated, job.format)
return Result(data=formatted, trace=self.trace)
```
Each subagent is built once per process and called many times. State flows through them in order.
## Component 5: Human in the Loop
An agent that can only read is safe and not very interesting. The moment it can send, publish or spend, some of what it does stops being undoable and that is what a human in the loop is for. Not review for its own sake, but a person standing between the model and the actions it cannot take back.
Gate on irreversibility, not importance. The test is mechanical: can another tool call undo this? Reading, retrieving and drafting pass if the model gets them wrong, the next step fixes it. Publishing, sending, deleting and spending do not. Importance is the wrong axis because everything looks important to whoever asked for it.
Over-gating is the failure that looks like caution. A gate firing on everything trains the reviewer to click approve without reading you pay the latency and keep none of the safety. Gates keep their meaning by being rare.
One constraint: the gate is a state the run rests in, not a blocking call. The mechanics are Part 2.
Loop engineering vs graph engineering
Where that gate can sit depends on how the run is structured, and there are two ways to structure it.
- Loop engineering is one agentic node: one agent, one goal, a verifier, a stop condition.
- Graph engineering is the topology around those loops: which nodes exist, which transitions are legal, what state crosses each edge.
They are not rivals. A graph is what you build when one loop stops being enough, and every node in it is still a loop.
This decides what a gate can promise. In a loop, the gate is a tool the model chooses to call so it holds only as well as the model's judgement. In a graph, it is a node on an edge, and the irreversible step has no path around it.
Start with a loop. Move a decision into the graph when its failure is expensive enough that you want it impossible rather than unlikely which is usually the irreversible ones, and usually first.
A model deciding it is finished is the least reliable stop condition you have. Add three more: a step budget, a wall-clock budget, a token budget. The model owns one exit; you own the other three.
A loop that stops making progress does not error. Every call succeeds, and it bills about a dollar a minute until someone notices.
## Part 3: Evaluation and Metrics
Design evaluation before you design agents. Otherwise you're flying blind.
The Evaluation Framework
Agent systems produce output that is plausibly correct but often wrong. You cannot test correctness in CI. You need a framework for evaluating quality, catching regressions, and comparing variants.
Three layers of evaluation:
Layer 1: Harness correctness (deterministic, CI-testable)
- Does the orchestration flow correctly?
- Do tools return the expected schema?
- Does state transition as expected?
- Does the result assemble correctly?
Layer 2: Agent output quality (non-deterministic, sampling-based)
- Does the output match the specification?
- Are the facts accurate?
- Is the reasoning sound?
- Is the format correct?
Layer 3: End-to-end regression (automated, periodic)
- Run a fixed test set through the system weekly
- Compare results to baseline with a judge model
- Flag regressions before they hit production
Standard Metrics
For the orchestration layer:
For output quality:
For cost and efficiency:
## LLM as a Judge
Don't ask humans to score agent output. Use another LLM.
The judge is a specialized agent with one job: evaluate output against a rubric.
```
class JudgeAgent:
def evaluate(self, output: str, rubric: EvaluationRubric) -> Score:
prompt = build_evaluation_prompt(output, rubric)
response = self.model.invoke(prompt)
score = parse_score(response)
return Score(
value=score.value,
reasoning=score.reasoning,
failures=[...],
confidence=score.confidence
)
```
The judge produces structured output:
```
@dataclass
class Score:
value: float # 0-10
reasoning: str # why this score?
failures: list[str] # what went wrong
confidence: float # how sure is the judge?
suggest_fix: str # what would improve it?
```
Use the judge to:
- Score a sample of every day's output (100-200 jobs)
- Flag regressions: if today's P50 judge score < baseline - 1σ, alert
- Compare variants: run both version A and version B on the same 100 test cases, compare judge scores
- Iteratively improve: when the judge finds a failure pattern, add it to the test set
Important: Test the judge itself by:
- Scoring known-good and known-bad outputs , the judge should differentiate them
- Checking judge agreement on edge cases, if two judge instances disagree, the rubric is ambiguous
- Comparing judge scores to human scores, they should correlate
Continuous Evaluation in Production
Weekly eval run:
```
# Every Monday morning
def eval_run():
sample = db.jobs.sample(200)
scores = []
for job in sample:
score = judge.evaluate(job.output, job.rubric)
scores.append(score)
baseline_mean = 7.2
current_mean = np.mean([s.value for s in scores])
if current_mean < baseline_mean - 0.5:
alert("Regression detected", current_mean)
db.metrics.insert(date=today, mean_score=current_mean)
```
A/B testing via judge:
```
def compare_variants(prompt_old, prompt_new, test_set_size=100):
test_jobs = db.jobs.sample(test_set_size)
scores_old = []
scores_new = []
for job in test_jobs:
result_old = orchestrator.run(job, prompt_old)
score_old = judge.evaluate(result_old)
scores_old.append(score_old)
result_new = orchestrator.run(job, prompt_new)
score_new = judge.evaluate(result_new)
scores_new.append(score_new)
mean_old = np.mean(scores_old)
mean_new = np.mean(scores_new)
if mean_new > mean_old + 0.3:
return "ship"
else:
return "revert"
```
# The Scorecard
Not every system builds all of these out. Every system has all of them decided. The only question is whether you decided or defaulted.
1. Tools: few, one action each, tightly typed, with a timeout and a log line
2. Prompts: static before dynamic, versioned, and the cache key versioned with them
3. Memory: working and persistent kept apart, seeded once rather than re-sent every turn
4. Orchestration: sequential subagents over parallel agents; state flows one way
5. Human in the loop: gate on irreversibility, not importance; rare enough that the gate still means something
6. Loop vs graph: start with one loop, move a decision into the graph when its failure should be impossible rather than unlikely
7. Stop conditions: four exits, only one of which the model owns
8. Evaluation: the harness tested on every commit, the output sampled and judged out of band
Notice how little of that is about the model. Swap the model underneath and all eight survive unchanged, which is the argument for treating them as the actual engineering rather than the plumbing around it.
Part 2 is the backend this needs in order to run: the queue, the workers, the two stores, and why an agent workload breaks every instinct you have from web services. Part 3 puts it on AWS.
# Further reading
- [Awesome Harness Engineering](https://github.com/ai-boost/awesome-harness-engineering): curated taxonomy across all of these areas
- [Agent Harness for LLM Agents: A Survey](https://github.com/Gloriaameng/Awesome-Agent-Harness): the 23-system comparison
- [Graph Engineering vs Loop Engineering: What Actually Changed](https://www.louisbouchard.ai/graph-engineering-explained/): the clearest treatment of that section
- [Agentic AI System Design Explained in 27 mins](https://youtu.be/mwN75EiGfCE) by [@aiwithaish](https://x.com/@aiwithaish)
- [3 Years of Graph Engineering with LangGraph](https://www.langchain.com/blog/3-years-of-graph-engineering-with-langgraph): the framework side of the same argument
- [Context Engineering: Memory, Compaction, and Tool Clearing](https://tianpan.co/blog/2026-02-26-context-engineering-memory-compaction-tool-clearing): more on
- [The Agent Harness: Why the LLM Is the Smallest Part](https://www.mongodb.com/company/blog/technical/agent-harness-why-llm-is-smallest-part-of-your-agent-system): the framing this article rests on
- [My PoC: Agent Harness Ops](https://github.com/kmeanskaran/agent-harness-ops)
See you soon.
Follow [@kmeanskaran](https://x.com/@kmeanskaran)
## 相关链接
- [Karan](https://x.com/kmeanskaran)
- [@kmeanskaran](https://x.com/kmeanskaran)
- [2.2K](https://x.com/kmeanskaran/status/2090436724250026177/analytics)
- [DevVoice](https://github.com/kmeanskaran/agent-harness-ops)
- [dev.to](https://dev.to/)
- [Awesome Harness Engineering](https://github.com/ai-boost/awesome-harness-engineering)
- [Agent Harness for LLM Agents: A Survey](https://github.com/Gloriaameng/Awesome-Agent-Harness)
- [Graph Engineering vs Loop Engineering: What Actually Changed](https://www.louisbouchard.ai/graph-engineering-explained/)
- [Agentic AI System Design Explained in 27 mins](https://youtu.be/mwN75EiGfCE)
- [@aiwithaish](https://x.com/@aiwithaish)
- [3 Years of Graph Engineering with LangGraph](https://www.langchain.com/blog/3-years-of-graph-engineering-with-langgraph)
- [Context Engineering: Memory, Compaction, and Tool Clearing](https://tianpan.co/blog/2026-02-26-context-engineering-memory-compaction-tool-clearing)
- [The Agent Harness: Why the LLM Is the Smallest Part](https://www.mongodb.com/company/blog/technical/agent-harness-why-llm-is-smallest-part-of-your-agent-system)
- [My PoC: Agent Harness Ops](https://github.com/kmeanskaran/agent-harness-ops)
- [@kmeanskaran](https://x.com/@kmeanskaran)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [9:52 PM · Aug 20, 2026](https://x.com/kmeanskaran/status/2090436724250026177)
- [2,299 Views](https://x.com/kmeanskaran/status/2090436724250026177/analytics)
---
*导出时间: 2026/8/20 23:01:46*
---
## 中文翻译
# 智能体系统设计（第一部分）
**作者**: Karan
**日期**: 2026-08-20T13:52:08.000Z
**来源**: [https://x.com/kmeanskaran/status/2090436724250026177](https://x.com/kmeanskaran/status/2090436724250026177)
---
把一个前沿模型放入一个设计糟糕的智能体系统中，你得到的只是一个更善于表达的失败。几乎所有的工程工作都在模型之外，存在于那些被称为智能体外壳（agent harness）的脚手架中。
本部分涵盖了无论你是否意识到，你都在设计的五个组件：工具、提示词、记忆、编排，以及人在回路中的位置。然后是评估框架，它告诉你这些组件是否有效，因为这些东西无法像普通代码那样进行测试。
构建智能体系统需要在三个关键领域进行有意识的设计：
1. 理解工作负载：智能体任务与Web请求有着根本的不同
2. 设计外壳：你构建的非模型组件（工具、提示词、记忆、编排）
3. 构建评估框架：如何衡量质量并捕获回归问题
本指南涵盖了以上所有内容。所有内容都源自一个真实系统：[DevVoice](https://github.com/kmeanskaran/agent-harness-ops)，这是一个五智能体流水线，可将GitHub README转化为经过审核的X帖子、LinkedIn帖子和[dev.to](//dev.to)文章。这些设计决策决定了你的智能体系统是否健全。
> 注意：代码示例是说明性伪代码，用于展示模式和概念。请将它们适配到你的框架（LangChain、LlamaIndex、Anthropic SDK等）中。
智能体外壳组件
# 第一部分：基础
智能体工作负载不是Web工作负载
从这里开始，因为几乎所有的后续决策都源于此。
一个典型的Web请求是50-200毫秒，突发性CPU密集，确定性，且每个单元的成本几乎为零。智能体任务则完全不是这样：
四个架构性后果：
**持续时间**。你无法同步响应请求。负载均衡器有空闲超时。为每个活动任务保持连接打开两分钟是浪费的。
**I/O密集型**。并发不受核心数限制。一个工作进程有70-80%的时间在等待模型API。一个2核容器可以运行数十个并发任务。按核心数进行容量规划会浪费你支付的大部分费用。
**非确定性**。相同的输入产生不同的输出，因此重试不是免费的。你不能在CI中对输出进行断言——测试必须针对外壳，而不是模型。
**边际成本**。一个Bug就是一张账单。智能体循环中的无限循环会以大约每分钟一美元的速度悄悄烧钱，直到有人注意到。
从第一次提交开始就为这种形态进行设计，部署就只是配置问题。为Web形态进行设计，部署则是一次重写。
# 第二部分：智能体外壳
你构建的非模型部分
外壳是你构建的一切非模型部分。在真实系统中有九个模块。其中有四个值得早期设计。
## 组件1：工具
工具是智能体影响世界的唯一方式。设计得不好，你会浪费模型的token在困惑上；设计得好，你可以将上下文减少60%。
工具定义是一个契约：
```
@tool
def search_code(
query: str,
repository: str,
max_results: int = 10
) -> list[CodeMatch]:
"""在代码库中搜索代码模式。
参数:
query: 要搜索的代码模式或关键字
repository: GitHub组织/仓库标识符
max_results: 结果数量限制（默认10，最大100）
返回:
带上下文的匹配代码位置列表
"""
```
三个要求：
1. **类型化的输入和输出**。模型看到的是模式定义；歧义会消耗token并导致错误。对于复杂的返回类型，使用Pydantic模型。
2. **较小的表面积**。每个离散操作一个工具。五个专注的工具胜过一个有五个可选参数的工具。避免接受任意查询的工具——模型会错误地调用它们。
3. **确定性和快速**。一个需要30秒的工具是上下文杀手。如果操作成本高昂，使其异步并通过状态工具轮询，或者将其移入任务队列。
工具管理：
- 将工具保存在注册表中，而不是分散在智能体代码中
- 工具与智能体分开版本化：智能体调用search_v2，而其他智能体仍使用search_v1
- 记录每次工具调用及其参数和延迟——你一半的问题都会在这里调试
- 为每个工具添加超时；一个挂起的工具会阻塞智能体
## 组件2：提示词工程
提示词不是散文，它们是状态机。结构与文字同样重要
基本结构：
```
1. 系统提示词（静态）
- 角色和上下文
- 硬性约束和规则
- 输出格式规范
2. 静态示例（静态）
- 展示期望行为的少样本示例
- 你见过的失败边缘案例
3. 工具规范（静态）
- 工具定义和使用示例
- 错误处理行为
4. 动态上下文（动态）
- 对话历史
- 每请求数据（用户输入、检索到的文档）
- 任务状态
5. 用户回合（动态）
- 实际请求
```
首要规则：静态内容必须位于动态内容之前。工具定义、系统提示词、示例、历史记录、用户回合。任何向上泄漏的动态值都会破坏提示词缓存，并悄悄地将你的token成本翻倍。
提示词版本化：
```
PROMPT_VERSION = "2024-08-19-v3"
def build_system_prompt(version: str) -> str:
if version.startswith("2024-08-19"):
return """你是一个代码审查者..."""
elif version.startswith("2024-08-15"):
return """你是一个审查者...""" # 之前的版本
```
将你的响应缓存键与提示词版本一起版本化。一个糟糕的提示词被缓存后，在你修复后仍会持续服务数小时。有了版本化，回滚只需更改一个环境变量。
## 组件3：记忆
记忆层
智能体需要两种记忆，它们不可互换。
**工作记忆**（短期，上下文内）：
- 当前对话轮次
- 最近N条消息（已检索）
- 为此任务检索的上下文
- Token预算：任何适合上下文的内容
**持久记忆**（长期，存储支持）：
- 用户偏好和历史
- 从过去交互中学到的事实
- 评估结果和失败
- 无限期存储
常见错误：将工作记忆当作持久记忆来对待。在每一轮的系统提示词中添加“记住X”会消耗token。正确的模式：
```
# 任务开始时：初始化工作记忆
working_memory = {
"user_id": job.user_id,
"past_interactions": fetch_last_n_interactions(job.user_id, n=5),
"learned_preferences": fetch_user_preferences(job.user_id),
"retrieved_documents": retrieve_relevant_docs(job.query),
}
# 任务期间：从工作记忆中工作
prompt = build_prompt_with_context(working_memory)
# 任务结束时：更新持久记忆
if agent.learned_something_new:
store_user_fact(job.user_id, fact)
```
避免上下文溢出：
- 智能地截断工作记忆——丢弃最旧的消息，保留最近和最相关的
- 将长对话的摘要存储在持久记忆中，检索摘要而不是完整历史
- 对文档使用基于内容的分块：在章节边界处断开，而不是在token限制处
## 组件4：多智能体编排
智能体编排
智能体可以通过两种方式协调：并行（智能体同时工作，结果合并）和顺序（智能体一个接一个地工作，一个的输出作为下一个的输入）。
并行智能体听起来更高效。实际上并非如此。它们会产生协调问题：智能体A产生智能体B需要的输出，但智能体以不同的速度运行。你最终会陷入轮询和合并的困境，这既慢又复杂。
使用子智能体的顺序编排更好：
```
# 不好：可能冲突的并行智能体
async def process_parallel(doc: str):
results = await asyncio.gather(
summarizer_agent(doc),
analyzer_agent(doc)
)
return merge_results(results)
# 好：具有清晰数据流的顺序子智能体
def process_sequential(doc: str):
summary = subagent_summarize(doc)
analysis = subagent_analyze(doc, context=summary)
final_review = subagent_review(doc, summary, analysis)
return final_review
```
为什么子智能体更胜一筹：
- **清晰的数据流**。智能体B明确依赖于智能体A的输出
- **提前退出**。如果智能体A的输出无效，你在启动智能体B之前就停止
- **更简单的状态**。工作记忆按顺序贯穿智能体；无需合并逻辑
- **更易调试**。每个子智能体有一个清晰的输入和输出
只有在智能体真正处理独立数据且其输出永不交互时才使用并行智能体（例如，对同一查询调用三个不同的API）。即便如此，也要考虑带回退的顺序智能体。
编排表面应该清晰：
```
class Orchestrator:
def run(self, job: Job) -> Result:
extracted = self.extract_agent.run(job.content, job.schema)
validated = self.validator_agent.run(extracted, job.rules)
formatted = self.formatter_agent.run(validated, job.format)
return Result(data=formatted, trace=self.trace)
```
每个子智能体在每个进程中构建一次，被多次调用。状态按顺序流过它们。
## 组件5：人在回路中
一个只能读取的智能体是安全的，但不太有趣。一旦它能发送、发布或消费，它所做的某些事情就不再是可撤销的，这就是人在回路中的意义所在。不是为了审查而审查，而是让一个人站在模型和它无法撤销的动作之间。
根据不可逆性而非重要性来设置门控。测试是机械性的：另一个工具调用能否撤销这个操作？读取、检索和起草可以通过——如果模型弄错了，下一步会修复它。发布、发送、删除和消费则不能。重要性是错误的衡量标准，因为对提出请求的人来说，一切看起来都很重要。
过度门控是一种看起来像谨慎的失败。对一切事物都触发的门控会训练审查者不加阅读地点击批准——你付出了延迟的代价，却没有获得任何安全性。门控通过稀缺性来保持其意义。
一个约束：门控是运行暂停的一种状态，而不是阻塞调用。具体机制在第二部分。
循环工程与图工程
门控可以放在哪里取决于运行的结构方式，而有两种方式来构建它。
- **循环工程**是一个智能体节点：一个智能体、一个目标、一个验证器、一个停止条件。
- **图工程**是围绕这些循环的拓扑结构：哪些节点存在、哪些转换是合法的、什么状态穿过每条边。
它们不是对手。图是当一个循环不再足够时你构建的东西，而图中的每个节点仍然是一个循环。
这决定了门控能承诺什么。在循环中，门控是模型选择调用的一个工具，因此它的可靠性取决于模型的判断力。在图中，它是边上的一個节点，不可逆的步骤没有绕过它的路径。
从一个循环开始。当一个决策的失败代价足够高，以至于你希望它不可能发生而不是不太可能发生时，将其移入图中——这通常是指那些不可逆的决策，而且通常是最先被移入的。
模型自行决定“完成”是你拥有的最不可靠的停止条件。再添加三个：步骤预算、墙钟时间预算、token预算。模型拥有一个退出路径；你拥有另外三个。
一个停止取得进展的循环不会报错。每次调用都会成功，并且以大约每分钟一美元的速度计费，直到有人注意到。
## 第三部分：评估与指标
在设计智能体之前先设计评估。否则你就是在盲目飞行。
评估框架
智能体系统产生的输出看似正确，但经常出错。你无法在CI中测试正确性。你需要一个框架来评估质量、捕获回归并比较变体。
三层评估：
**第1层：外壳正确性**（确定性，可在CI中测试）
- 编排流程是否正确？
- 工具是否返回预期的模式？
- 状态转换是否符合预期？
- 结果是否正确组装？
**第2层：智能体输出质量**（非确定性，基于采样）
- 输出是否符合规范？
- 事实是否准确？
- 推理是否合理？
- 格式是否正确？
**第3层：端到端回归**（自动化，定期）
- 每周通过系统运行一组固定的测试集
- 使用评判模型将结果与基线进行比较
- 在回归进入生产环境之前标记它们
标准指标
对于编排层：
对于输出质量：
对于成本和效率：
## LLM作为评判者
不要让人来给智能体输出打分。使用另一个LLM。
评判者是一个专门的智能体，只有一个工作：根据评分标准评估输出。
```
class JudgeAgent:
def evaluate(self, output: str, rubric: EvaluationRubric) -> Score:
prompt = build_evaluation_prompt(output, rubric)
response = self.model.invoke(prompt)
score = parse_score(response)
return Score(
value=score.value,
reasoning=score.reasoning,
failures=[...],
confidence=score.confidence
)
```
评判者产生结构化输出：
```
@dataclass
class Score:
value: float # 0-10
reasoning: str # 为什么是这个分数？
failures: list[str] # 出了什么问题
confidence: float # 评判者有多确定？
suggest_fix: str # 什么可以改进它？
```
使用评判者来：
- 对每天的输出样本进行评分（100-200个任务）
- 标记回归：如果今天的P50评判分数 < 基线 - 1σ，则发出警报
- 比较变体：在相同的100个测试用例上运行版本A和版本B，比较评判分数
- 迭代改进：当评判者发现失败模式时，将其添加到测试集中
重要提示：通过以下方式测试评判者本身：
- 对已知良好和已知不良的输出进行评分——评判者应该能够区分它们
- 检查评判者在边缘案例上的一致性——如果两个评判实例不一致，则评分标准模糊
- 将评判分数与人类分数进行比较——它们应该相关
生产环境中的持续评估
每周评估运行：
```
# 每个周一早上
def eval_run():
sample = db.jobs.sample(200)
scores = []
for job in sample:
score = judge.evaluate(job.output, job.rubric)
scores.append(score)
baseline_mean = 7.2
current_mean = np.mean([s.value for s in scores])
if current_mean < baseline_mean - 0.5:
alert("检测到回归", current_mean)
db.metrics.insert(date=today, mean_score=current_mean)
```
通过评判者进行A/B测试：
```
def compare_variants(prompt_old, prompt_new, test_set_size=100):
test_jobs = db.jobs.sample(test_set_size)
scores_old = []
scores_new = []
for job in test_jobs:
result_old = orchestrator.run(job, prompt_old)
score_old = judge.evaluate(result_old)
scores_old.append(score_old)
result_new = orchestrator.run(job, prompt_new)
score_new = judge.evaluate(result_new)
scores_new.append(score_new)
mean_old = np.mean(scores_old)
mean_new = np.mean(scores_new)
if mean_new > mean_old + 0.3:
return "发布"
else:
return "回滚"
```
# 记分卡
并非每个系统都会构建所有这些组件。但每个系统都已经决定了所有这些组件。唯一的问题是你是主动决定还是被动默认。
1. **工具**：数量少，每个只做一个动作，类型严格，带超时和日志行
2. **提示词**：静态在动态之前，版本化，缓存键与之一同版本化
3. **记忆**：工作记忆和持久记忆分开，一次性初始化而不是每轮重新发送
4. **编排**：顺序子智能体优于并行智能体；状态单向流动
5. **人在回路中**：根据不可逆性而非重要性设置门控；门控足够稀缺以保持其意义
6. **循环与图**：从一个循环开始，当决策的失败应该是不可能而非不太可能时，将其移入图中
7. **停止条件**：四个退出路径，其中只有一个是模型拥有的
8. **评估**：外壳在每次提交时测试，输出被采样并在带外评判
注意这些内容中有多少与模型无关。在底层更换模型，这八项全部保持不变——这就是为什么应该将它们视为真正的工程，而不是围绕模型的管道。
第二部分是运行这一切所需的后端：队列、工作进程、两个存储，以及为什么智能体工作负载会打破你从Web服务中获得的每一种直觉。第三部分将其部署到AWS上。
# 延伸阅读
- [Awesome Harness Engineering](https://github.com/ai-boost/awesome-harness-engineering)：涵盖所有这些领域的精选分类
- [Agent Harness for LLM Agents: A Survey](https://github.com/Gloriaameng/Awesome-Agent-Harness)：23个系统的比较
- [Graph Engineering vs Loop Engineering: What Actually Changed](https://www.louisbouchard.ai/graph-engineering-explained/)：对该部分最清晰的阐述
- [Agentic AI System Design Explained in 27 mins](https://youtu.be/mwN75EiGfCE) by [@aiwithaish](https://x.com/@aiwithaish)
- [3 Years of Graph Engineering with LangGraph](https://www.langchain.com/blog/3-years-of-graph-engineering-with-langgraph)：同一论点的框架侧
- [Context Engineering: Memory, Compaction, and Tool Clearing](https://tianpan.co/blog/2026-02-26-context-engineering-memory-compaction-tool-clearing)：更多相关内容
- [The Agent Harness: Why the LLM Is the Smallest Part](https://www.mongodb.com/company/blog/technical/agent-harness-why-llm-is-smallest-part-of-your-agent-system)：本文所依据的框架
- [My PoC: Agent Harness Ops](https://github.com/kmeanskaran/agent-harness-ops)
下次见。
关注 [@kmeanskaran](https://x.com/@kmeanskaran)
## 相关链接
- [Karan](https://x.com/kmeanskaran)
- [@kmeanskaran](https://x.com/kmeanskaran)
- [2.2K](https://x.com/kmeanskaran/status/2090436724250026177/analytics)
- [DevVoice](https://github.com/kmeanskaran/agent-harness-ops)
- [dev.to](https://dev.to/)
- [Awesome Harness Engineering](https://github.com/ai-boost/awesome-harness-engineering)
- [Agent Harness for LLM Agents: A Survey](https://github.com/Gloriaameng/Awesome-Agent-Harness)
- [Graph Engineering vs Loop Engineering: What Actually Changed](https://www.louisbouchard.ai/graph-engineering-explained/)
- [Agentic AI System Design Explained in 27 mins](https://youtu.be/mwN75EiGfCE)
- [@aiwithaish](https://x.com/@aiwithaish)
- [3 Years of Graph Engineering with LangGraph](https://www.langchain.com/blog/3-years-of-graph-engineering-with-langgraph)
- [Context Engineering: Memory, Compaction, and Tool Clearing](https://tianpan.co/blog/2026-02-26-context-engineering-memory-compaction-tool-clearing)
- [The Agent Harness: Why the LLM Is the Smallest Part](https://www.mongodb.com/company/blog/technical/agent-harness-why-llm-is-smallest-part-of-your-agent-system)
- [My PoC: Agent Harness Ops](https://github.com/kmeanskaran/agent-harness-ops)
- [@kmeanskaran](https://x.com/@kmeanskaran)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [9:52 PM · Aug 20, 2026](https://x.com/kmeanskaran/status/2090436724250026177)
- [2,299 Views](https://x.com/kmeanskaran/status/2090436724250026177/analytics)
---
*导出时间: 2026/8/20 23:01:46* 
