# How to Build a Self-Improving Outbound System on Codex
**作者**: Nicolas Finet
**日期**: 2026-07-19T14:36:13.000Z
**来源**: [https://x.com/nifinet/status/2078851409068654639](https://x.com/nifinet/status/2078851409068654639)
---
Earlier this year, Andrej Karpathy (@karpathy) pointed an agent at his own training code and let it run for two days. It ran 700 experiments, kept the 20 that beat the benchmark, and made the model train 11% faster. Then he said something quite interesting: any metric you can evaluate cheaply can be handed to an agent swarm.
Reply rate is a metric you can evaluate cheaply. I have spent some time since working out what that loop looks like pointed at outbound.
My build:
Codex reads last week's outcomes, edits the scoring and play files the outbound system runs on, runs a test, and opens a pull request. It proposes a change to the playbook with the evidence and the score attached, then waits for a human to approve it. Sending and merging stay outside the loop.
I have built the first loop a few times: sense the market, score the account, write from the signal, check the message, log the outcome, learn from the reply. This article is about the second loop, the one that edits the first.
That is the build: GTM as versioned code that improves from the market.
## The repo
Start with the folder. The shape matters because Codex can only improve what it can read and edit.
```
codex-self-improving-outbound/
AGENTS.md
README.md
config/
scoring.yaml
plays.yaml
prompts/
improve_scoring.md
improve_prompt.md
pr_summary.md
memory/
outcomes.jsonl
evals/
fixtures.yaml
score.py
scripts/
append_outcome.py
run_codex_step.sh
propose_improvement.py
open_pr.sh
weekly_tune.sh
examples/
outcomes.sample.jsonl
weekly-pr.md
```
The repo is intentionally plain. config/scoring.yaml holds the rules that decide which signals matter. prompts/ holds the plays that write the messages. memory/outcomes.jsonl holds what the market did. evals/score.py is the gate that says whether a proposed change helped. AGENTS.md is the law Codex reads before it touches anything.
Run the first version offline. No CRM, no enrichment, no delivery system. The improvement loop should prove itself on local files before it gets anywhere near a real outbound machine.
## Step 1. Write the law first
Before the scoring file, before the prompt files, write AGENTS.md. This is the file that keeps the agent useful and contained.
```
# Self-Improving Outbound Rules
You improve an outbound system from outcomes.
Hard rules:
- Never send messages.
- Never scrape or enrich real people.
- Never self-merge.
- Edit only files in this repo.
- Change one concept at a time.
- Cite outcomes from memory/outcomes.jsonl for every proposed change.
- Improve evals/score.py before a change can become a PR.
- If the eval does not improve, revert your edit and stop.
Allowed edits:
- config/scoring.yaml
- config/plays.yaml
- prompts/*.md
Required output:
- changed files
- reason for each change
- before score
- after score
- pull request summary
```
The law has one job: narrow the work. Without it, Codex will try to help by expanding scope. It will add more data, touch more files, call more tools, or automate a step that should stay under human control. Here the job is smaller: read outcomes, propose one file change, prove it helped, then wait.
What good looks like. You can read the law before approving a PR and know exactly what Codex was allowed to do.
Where it breaks. The law becomes a compliance document. If AGENTS.md needs a table of contents, it is already too large. Keep it operational.
## Step 2. Move judgment into config
Most outbound judgment lives in someone's head. Then the team buys software and expects the software to improve a decision it cannot see.
Move the judgment into a file.
```
signals:
competitor_comparison:
weight: 8
reason: "buyer is comparing alternatives"
implementation_page_visit:
weight: 6
reason: "buyer is checking whether this can be installed"
job_repost:
weight: 5
reason: "role is still open and urgent"
funding_event:
weight: 5
reason: "budget or mandate may have changed"
generic_download:
weight: 1
reason: "content interest, weak buying intent"
thresholds:
draft: 6
human_review: 10
negative_signals:
student_research: -8
vendor_pitch: -6
competitor: -10
```
This file starts as a visible hypothesis. If a generic download should count for zero, the team can point to the exact line and change it. If an implementation page visit is a stronger signal than you thought, Codex can propose the diff and show the outcome rows that justify it.
Do not bury this logic in a Python function. If the rule is visible, the team can review it, argue with it, and improve it without turning a sales judgment into an engineering refactor.
What good looks like. The file is small enough to argue with. Five signals is a good first version.
Where it breaks. The scoring file becomes a junk drawer. Twenty signals, six thresholds, and exception rules for every edge case will make the improver overfit. Start narrow and let the outcomes tell you where the next knob belongs.
## Step 3. Write outcomes as memory
The most important file is memory/outcomes.jsonl.
One line per touch, written when the outcome is known:
```
{"date":"2026-07-01","account":"Northwind Finance","signal":"competitor_comparison","play":"migration_note","score":8,"outcome":"reply","reason":"asked for migration notes"}
{"date":"2026-07-01","account":"Bluepeak Studio","signal":"generic_download","play":"resource_followup","score":1,"outcome":"no_reply","reason":"content-only intent"}
{"date":"2026-07-02","account":"KiteOps","signal":"implementation_page_visit","play":"implementation_angle","score":6,"outcome":"reply","reason":"asked about implementation timeline"}
{"date":"2026-07-02","account":"Atlas Recruiting","signal":"job_repost","play":"hiring_angle","score":5,"outcome":"bad_fit","reason":"student research request"}
```
The reason field is the whole point. no_reply tells you almost nothing. content-only intent tells the next run that this signal might not deserve a draft. bad_fit is useful only when the reason explains why. asked about implementation timeline is the kind of detail that can change a weight.
Build the validator before you build the improver:
```
Build scripts/append_outcome.py.
It accepts:
- date
- account
- signal
- play
- score
- outcome: reply | meeting | no_reply | bad_fit | bounced
- reason
It rejects:
- missing fields
- unknown outcomes
- empty reason
- dates in the future
Append valid rows to memory/outcomes.jsonl.
Print the appended row.
```
This is where the compounding starts. A dashboard can tell you a campaign is down. A clean outcome log can tell Codex which signal, play, or phrase should change before the next run.
What good looks like. After a week, a stranger can read the file and tell which signals created replies, which plays created bad-fit conversations, and which internal favorite the market ignored.
Where it breaks. The team backfills outcomes on Friday from memory. The wins survive, the bad-fit reasons blur, and the system learns from fiction. Write the row when the outcome lands.
## Step 4. Build the eval gate
Before Codex edits anything, it needs a test it cannot explain away.
Create evals/fixtures.yaml:
```
cases:
- account: Northwind Finance
signals: [competitor_comparison, implementation_page_visit]
expected: human_review
note: "two strong signals on one account"
- account: Bluepeak Studio
signals: [generic_download]
expected: ignore
note: "content-only intent"
- account: KiteOps
signals: [implementation_page_visit]
expected: draft
note: "implementation intent should clear draft threshold"
- account: Atlas Recruiting
signals: [job_repost, student_research]
expected: ignore
note: "bad-fit marker cancels the signal"
```
Then create evals/score.py:
```
Build evals/score.py.
Read config/scoring.yaml and evals/fixtures.yaml.
For each case:
1. Sum weights for every signal.
2. Add negative signal penalties.
3. Route the account:
- score >= thresholds.human_review => human_review
- score >= thresholds.draft => draft
- otherwise => ignore
4. Compare route to expected.
Print each prediction.
Print final accuracy as score=0.00 to score=1.00.
Exit 0 only when accuracy is 1.00.
```
The first gate should be small enough to understand and sharp enough to catch a real miss. In my first run, the baseline failed one case:
```
Northwind Finance: predicted=human_review expected=human_review
Bluepeak Studio: predicted=ignore expected=ignore
KiteOps: predicted=ignore expected=draft
Atlas Recruiting: predicted=ignore expected=ignore
score=0.75
```
That was good. The system had implementation intent below the draft threshold, so it ignored an account the fixture said deserved a message. Better to catch that in a test than after a month of missed accounts.
What good looks like. One command gives one number, and every failed case is easy to inspect.
Where it breaks. The fixture only includes obvious wins. Then every reckless change passes. Put ugly cases in the gate: weak intent, bad fit, no reply, stale signals, and the accounts you wish the system had skipped.
## Step 5. Let Codex propose one scoring change
Now Codex can edit.
Create prompts/improve_scoring.md:
```
You improve the outbound scoring system.
Read:
- AGENTS.md
- config/scoring.yaml
- memory/outcomes.jsonl
- evals/fixtures.yaml
Your job:
1. Find one scoring rule that should change.
2. The reason must cite memory/outcomes.jsonl.
3. Change only config/scoring.yaml.
4. Run python3 evals/score.py.
5. If the score improves, keep the change.
6. If the score stays flat or drops, revert your change and stop.
Output:
- the exact line changed
- the outcome rows that caused it
- before score
- after score
- whether the change should become a PR
Do not edit prompts.
Do not add new signals.
Do not touch delivery.
```
Run it through the repo wrapper:
```
scripts/run_codex_step.sh improve_scoring
```
The first version of my improver made a useful mistake. It chased the cleanest-looking reply signal. competitor_comparison had the strongest reply rate in the tiny outcome log, so the improver wanted to increase that weight. The eval stayed at 0.75, so the change was rejected.
That is exactly why the gate exists. A weaker system would have accepted the story because it sounded reasonable. This one asked a better question: did the change fix the known miss?
The second pass found the smallest edit that helped:
```
- implementation_page_visit: 4
+ implementation_page_visit: 6
```
The eval passed:
```
Northwind Finance: predicted=human_review expected=human_review
Bluepeak Studio: predicted=ignore expected=ignore
KiteOps: predicted=draft expected=draft
Atlas Recruiting: predicted=ignore expected=ignore
score=1.00
```
That is the moment the loop becomes useful. It changed one rule, for one reason, and proved the change against a fixture.
What good looks like. The proposed diff is boring and traceable: one line changed, one outcome-backed reason attached, one eval improved.
Where it breaks. Codex changes three weights and two prompts at once. Now nobody can tell which change helped. Keep the law strict: one concept per proposal.
## Step 6. Improve prompt files separately
Scoring is only half the system. The message templates decay too.
A line that worked last month starts sounding familiar. A question that earns replies in one segment gets ignored in another. A phrase that feels sharp internally gets punished by the market. Treat prompt improvement as a separate lane so Codex does not mix scoring and copy in the same PR.
Create config/plays.yaml:
```
plays:
migration_note:
prompt_file: prompts/plays/migration_note.md
use_when:
- competitor_comparison
banned_lines:
- "thought this might be relevant"
- "quick question"
implementation_angle:
prompt_file: prompts/plays/implementation_angle.md
use_when:
- implementation_page_visit
banned_lines:
- "checking out our solution"
- "would love to chat"
```
Then create prompts/improve_prompt.md:
```
You improve one outbound play.
Read:
- AGENTS.md
- config/plays.yaml
- memory/outcomes.jsonl
- the prompt file for the chosen play
Pick one play with at least 10 outcomes.
Find:
- lines or structures that appear in positive outcomes
- lines or structures that appear in no_reply or bad_fit outcomes
- any phrase that should be banned
Make one small edit to that play's prompt.
Rules:
- Do not change scoring.
- Do not create a new play.
- Do not add a new channel.
- Cite outcome rows.
- Write the before and after instruction.
Then run the copy eval if present.
If no copy eval exists, open the PR as review_required.
```
Some improvements can be scored automatically. Others still need taste. If there is no copy eval, Codex can propose the prompt edit, but it should mark the PR for review instead of pretending the edit is proven.
What good looks like. Codex says, "This phrase appeared in seven no-reply outcomes, so I added it to banned_lines," or "positive replies quoted the implementation detail in sentence one, so I tightened the play to require that."
Where it breaks. The improver rewrites the whole voice because one message got a reply. Prompt edits should be smaller than your instinct.
## Step 7. Ship changes as pull requests
This is the control layer. Codex edits files, runs the eval, and writes the PR summary. A human reviews and merges.
Create prompts/pr_summary.md:
```
Write a pull request summary for this outbound improvement.
Include:
1. What changed.
2. Why it changed, citing outcome rows.
3. Before score.
4. After score.
5. Files changed.
6. Risk.
7. What the human reviewer should check.
Keep it short.
Do not claim the change is live.
```
Create `scripts/open_pr.sh`:
```
#!/usr/bin/env bash
set -euo pipefail
branch="codex/weekly-tune-$(date +%Y-%m-%d)"
git checkout -b "$branch"
git add config prompts evals memory
git commit -m "Codex weekly outbound tune"
body="$(cat outputs/pr-summary.md)"
python3 scripts/create_pr.py \
"$branch" \
"Codex weekly outbound tune" \
"$body"
```
The PR should read like a teammate wrote it:
```
Changed:
- Raised implementation_page_visit from 4 to 6.
Why:
- KiteOps had implementation-page intent and replied with implementation timing.
- The previous score routed this account to ignore.
Before:
- eval score 0.75
After:
- eval score 1.00
Reviewer check:
- Make sure implementation intent is specific enough.
- Keep generic downloads low.
- Merge only if this matches actual sales judgment.
```
That is the safety system. Codex does the tedious work. The operator keeps the standard.
What good looks like. One PR a week, small diff, clear reason, passing eval.
Where it breaks. Someone gives Codex permission to merge because review feels like friction. That minute separates a system that improves from a system that drifts.
## Step 8. Put it on a cadence
Do not run this after every reply. That is how a system overfits to one loud account.
Let the week happen, let outcomes accumulate, then tune.
Create scripts/weekly_tune.sh:
```
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
python3 evals/score.py || true
scripts/run_codex_step.sh improve_scoring
python3 evals/score.py
scripts/run_codex_step.sh pr_summary > outputs/pr-summary.md
scripts/open_pr.sh
```
Then cron:
```
0 8 * * MON cd ~/codex-self-improving-outbound && scripts/weekly_tune.sh
```
If you use GitHub Actions, keep the same shape:
```
name: weekly-outbound-tune
on:
schedule:
- cron: "0 8 * * 1"
workflow_dispatch:
jobs:
tune:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install -r requirements.txt
- run: python3 evals/score.py || true
- run: scripts/weekly_tune.sh
```
Run the first two tune-ups by hand. Read every diff. Watch what Codex tries to change when the sample is thin. Once the proposals are boring, put it on a schedule.
What good looks like. A weekly PR appears with the evidence, the diff, and the eval result. You merge, edit, or close it.
Where it breaks. The job runs, nobody reviews, and PRs pile up. A self-improving system still has one human habit: read the diff.
## The clone-and-run version
The repo should ship with four commands:
```
git clone <repo>
cd codex-self-improving-outbound
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python3 evals/score.py
python3 scripts/propose_improvement.py
```
Expected first run:
```
score=0.75
changed config/scoring.yaml
implementation_page_visit: 4 -> 6
score=1.00
open PR for human review
```
The offline demo proves the file contracts. The Codex run proves the editing loop. After that, replace the sample outcomes with your own, rename the signals, add your plays, and build a fixture that reflects the accounts you wish the system had routed differently.
Do not start by wiring delivery. Start by proving the improvement loop.
## The full version: max
This repo is the manual layer. It runs from files, public signals, and your Codex plan. It teaches the shape because every rule is exposed.
yourmax.ai is the same system with the seams hidden.
Instead of a repo you wire together yourself, max is the agent you use directly. It detects movement in the market, decides who is worth contacting and why now, drafts the outreach across email and LinkedIn for your approval, and keeps improving from the outcomes.
The repo shows the self-tuning layer most teams never build: outcomes become proposed rule changes, proposed rule changes run through a gate, and the human merge decides what becomes live. max takes that same operating logic and runs it as a managed system.
If you want the full repo, you can let me know and I will send it your way.
## 相关链接
- [Nicolas Finet](https://x.com/nifinet)
- [@nifinet](https://x.com/nifinet)
- [153K](https://x.com/nifinet/status/2078851409068654639/analytics)
- [@karpathy](https://x.com/@karpathy)
- [yourmax.ai](http://yourmax.ai/)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [10:36 PM · Jul 19, 2026](https://x.com/nifinet/status/2078851409068654639)
- [153.3K Views](https://x.com/nifinet/status/2078851409068654639/analytics)
- [View quotes](https://x.com/nifinet/status/2078851409068654639/quotes)
---
*导出时间: 2026/7/20 23:37:46*
---
## 中文翻译
# 如何在 Codex 上构建自我改进的外联系统
**作者**: Nicolas Finet
**日期**: 2026-07-19T14:36:13.000Z
**来源**: [https://x.com/nifinet/status/2078851409068654639](https://x.com/nifinet/status/2078851409068654639)
---
今年早些时候，Andrej Karpathy (@karpathy) 指向一个代理来处理他自己的训练代码并让它运行了两天。它运行了 700 次实验，保留了 20 个击败基准的实验，并使模型的训练速度提高了 11%。然后他说了一句非常有趣的话：任何你可以低成本评估的指标都可以交给一个代理群组。
回复率是一个你可以低成本评估的指标。从那以后，我花了一些时间弄清楚当这个循环指向外联时是什么样子的。
我的构建方案：
Codex 读取上周的结果，编辑外联系统运行的评分和策略文件，运行测试，并打开一个拉取请求。它提出更改策略的建议，并附上证据和分数，然后等待人工批准。发送和合并保持在循环之外。
我已经构建过好几次第一个循环：感知市场，对账户评分，根据信号撰写内容，检查消息，记录结果，从回复中学习。这篇文章是关于第二个循环的，也就是编辑第一个循环的那个。
这就是构建方案：作为版本化代码的 GTM（进入市场策略），它根据市场的反馈进行改进。
## 仓库
从文件夹开始。结构很重要，因为 Codex 只能改进它能读取和编辑的内容。
```
codex-self-improving-outbound/
AGENTS.md
README.md
config/
scoring.yaml
plays.yaml
prompts/
improve_scoring.md
improve_prompt.md
pr_summary.md
memory/
outcomes.jsonl
evals/
fixtures.yaml
score.py
scripts/
append_outcome.py
run_codex_step.sh
propose_improvement.py
open_pr.sh
weekly_tune.sh
examples/
outcomes.sample.jsonl
weekly-pr.md
```
这个仓库故意保持简洁。config/scoring.yaml 包含决定哪些信号重要的规则。prompts/ 包含撰写消息的策略。memory/outcomes.jsonl 包含市场的实际反应。evals/score.py 是判断建议的更改是否有帮助的关口。AGENTS.md 是 Codex 在接触任何内容之前读取的法律准则。
离线运行第一个版本。没有 CRM，没有数据增强，没有交付系统。改进循环应该在接触到真实的外联机器之前，就在本地文件上证明其价值。
## 步骤 1. 先制定规则
在评分文件之前，在提示词文件之前，先编写 AGENTS.md。这是保持代理有用且受控的文件。
```
# Self-Improving Outbound Rules
You improve an outbound system from outcomes.
Hard rules:
- Never send messages.
- Never scrape or enrich real people.
- Never self-merge.
- Edit only files in this repo.
- Change one concept at a time.
- Cite outcomes from memory/outcomes.jsonl for every proposed change.
- Improve evals/score.py before a change can become a PR.
- If the eval does not improve, revert your edit and stop.
Allowed edits:
- config/scoring.yaml
- config/plays.yaml
- prompts/*.md
Required output:
- changed files
- reason for each change
- before score
- after score
- pull request summary
```
规则只有一个工作：缩小工作范围。没有它，Codex 会试图通过扩大范围来提供帮助。它会添加更多数据，接触更多文件，调用更多工具，或者自动化一个应该由人工控制的步骤。在这里，工作范围更小：读取结果，提出一个文件更改，证明它有帮助，然后等待。
理想状态。你可以在批准 PR 之前阅读规则，并确切知道 Codex 被允许做什么。
失效点。规则变成了合规文档。如果 AGENTS.md 需要一个目录，那就太大了。保持其可操作性。
## 步骤 2. 将判断移入配置
大多数外联判断存在于某人的脑海中。然后团队购买软件，并期望软件改进一个它无法看到的决策。
将判断移入文件中。
```
signals:
competitor_comparison:
weight: 8
reason: "buyer is comparing alternatives"
implementation_page_visit:
weight: 6
reason: "buyer is checking whether this can be installed"
job_repost:
weight: 5
reason: "role is still open and urgent"
funding_event:
weight: 5
reason: "budget or mandate may have changed"
generic_download:
weight: 1
reason: "content interest, weak buying intent"
thresholds:
draft: 6
human_review: 10
negative_signals:
student_research: -8
vendor_pitch: -6
competitor: -10
```
这个文件开始时是一个可见的假设。如果普通下载应该计为零，团队可以指向确切的行并更改它。如果实施页面访问是一个比你想象的更强的信号，Codex 可以提出差异并显示证明其合理的结果行。
不要将此逻辑埋藏在 Python 函数中。如果规则是可见的，团队可以审查它，质疑它，并改进它，而无需将销售判断转化为工程重构。
理想状态。文件足够小，可以争论。五个信号是一个不错的第一个版本。
失效点。评分文件变成了垃圾抽屉。二十个信号，六个阈值，以及针对每个边缘情况的例外规则，将使改进器过拟合。从窄处开始，让结果告诉你下一个旋钮属于哪里。
## 步骤 3. 将结果写入记忆
最重要的文件是 memory/outcomes.jsonl。
每次接触一行，在知道结果时写入：
```
{"date":"2026-07-01","account":"Northwind Finance","signal":"competitor_comparison","play":"migration_note","score":8,"outcome":"reply","reason":"asked for migration notes"}
{"date":"2026-07-01","account":"Bluepeak Studio","signal":"generic_download","play":"resource_followup","score":1,"outcome":"no_reply","reason":"content-only intent"}
{"date":"2026-07-02","account":"KiteOps","signal":"implementation_page_visit","play":"implementation_angle","score":6,"outcome":"reply","reason":"asked about implementation timeline"}
{"date":"2026-07-02","account":"Atlas Recruiting","signal":"job_repost","play":"hiring_angle","score":5,"outcome":"bad_fit","reason":"student research request"}
```
reason 字段是重点。no_reply 几乎告诉不了你什么。content-only intent 告诉下一次运行，这个信号可能不值得起草草稿。bad_fit 只有在原因解释了为什么时才有用。asked about implementation timeline 是那种可以改变权重的细节。
在构建改进器之前构建验证器：
```
Build scripts/append_outcome.py.
It accepts:
- date
- account
- signal
- play
- score
- outcome: reply | meeting | no_reply | bad_fit | bounced
- reason
It rejects:
- missing fields
- unknown outcomes
- empty reason
- dates in the future
Append valid rows to memory/outcomes.jsonl.
Print the appended row.
```
这就是复利开始的地方。仪表板可以告诉你活动效果下降。一个干净的结果日志可以告诉 Codex 在下一次运行之前应该更改哪个信号、策略或短语。
理想状态。一周后，一个陌生人可以阅读该文件并分辨出哪些信号产生了回复，哪些策略产生了不匹配的对话，以及市场忽略了哪个内部 favorites。
失效点。团队在周五根据记忆回填结果。胜利被保留，不匹配的原因变得模糊，系统从虚构中学习。在结果出现时写入该行。
## 步骤 4. 构建评估关口
在 Codex 编辑任何内容之前，它需要一个它无法解释掉的测试。
创建 evals/fixtures.yaml：
```
cases:
- account: Northwind Finance
signals: [competitor_comparison, implementation_page_visit]
expected: human_review
note: "two strong signals on one account"
- account: Bluepeak Studio
signals: [generic_download]
expected: ignore
note: "content-only intent"
- account: KiteOps
signals: [implementation_page_visit]
expected: draft
note: "implementation intent should clear draft threshold"
- account: Atlas Recruiting
signals: [job_repost, student_research]
expected: ignore
note: "bad-fit marker cancels the signal"
```
然后创建 evals/score.py：
```
Build evals/score.py.
Read config/scoring.yaml and evals/fixtures.yaml.
For each case:
1. Sum weights for every signal.
2. Add negative signal penalties.
3. Route the account:
- score >= thresholds.human_review => human_review
- score >= thresholds.draft => draft
- otherwise => ignore
4. Compare route to expected.
Print each prediction.
Print final accuracy as score=0.00 to score=1.00.
Exit 0 only when accuracy is 1.00.
```
第一个关口应该足够小以便理解，并且足够敏锐以捕捉真正的遗漏。在我的第一次运行中，基线失败了一个案例：
```
Northwind Finance: predicted=human_review expected=human_review
Bluepeak Studio: predicted=ignore expected=ignore
KiteOps: predicted=ignore expected=draft
Atlas Recruiting: predicted=ignore expected=ignore
score=0.75
```
那很好。系统将实施意图设定在草稿阈值以下，因此它忽略了一个测试用例认为值得发送消息的账户。最好在测试中抓住这一点，而不是在错过一个月的账户之后。
理想状态。一个命令给出一个数字，每个失败的案例都很容易检查。
失效点。测试用例只包括明显的胜利。然后每一个鲁莽的改变都通过了。将丑陋的案例放入关口：弱意图，不匹配，无回复，陈旧的信号，以及你希望系统跳过的账户。
## 步骤 5. 让 Codex 提出一个评分更改
现在 Codex 可以编辑了。
创建 prompts/improve_scoring.md：
```
You improve the outbound scoring system.
Read:
- AGENTS.md
- config/scoring.yaml
- memory/outcomes.jsonl
- evals/fixtures.yaml
Your job:
1. Find one scoring rule that should change.
2. The reason must cite memory/outcomes.jsonl.
3. Change only config/scoring.yaml.
4. Run python3 evals/score.py.
5. If the score improves, keep the change.
6. If the score stays flat or drops, revert your change and stop.
Output:
- the exact line changed
- the outcome rows that caused it
- before score
- after score
- whether the change should become a PR
Do not edit prompts.
Do not add new signals.
Do not touch delivery.
```
通过仓库包装器运行它：
```
scripts/run_codex_step.sh improve_scoring
```
我的改进器的第一个版本犯了一个有用的错误。它追逐看起来最干净的回复信号。competitor_comparison 在微小的结果日志中拥有最强的回复率，所以改进器想要增加那个权重。评估保持在 0.75，所以更改被拒绝了。
这正是关口存在的原因。一个较弱的系统会因为听起来合理而接受这个故事。这一个问了一个更好的问题：更改是否修复了已知的遗漏？
第二遍找到了有帮助的最小编辑：
```
- implementation_page_visit: 4
+ implementation_page_visit: 6
```
评估通过了：
```
Northwind Finance: predicted=human_review expected=human_review
Bluepeak Studio: predicted=ignore expected=ignore
KiteOps: predicted=draft expected=draft
Atlas Recruiting: predicted=ignore expected=ignore
score=1.00
```
这就是循环变得有用的时刻。它更改了一条规则，出于一个原因，并证明了该更改针对测试用例是有效的。
理想状态。建议的差异是无聊且可追踪的：一行更改，一个附带结果支持的原因，一个改进的评估。
失效点。Codex 一次更改三个权重和两个提示词。现在没人能说哪个更改有帮助。保持法律严格：每个提案一个概念。
## 步骤 6. 单独改进提示词文件
评分只是系统的一半。消息模板也会衰减。
上个月有效的一行开始听起来很熟悉。在一个细分市场赢得回复的问题在另一个市场被忽略。在内部感觉犀利的短语受到市场的惩罚。将提示词改进视为一个单独的通道，以便 Codex 不会在同一个 PR 中混合评分和文案。
创建 config/plays.yaml：
```
plays:
migration_note:
prompt_file: prompts/plays/migration_note.md
use_when:
- competitor_comparison
banned_lines:
- "thought this might be relevant"
- "quick question"
implementation_angle:
prompt_file: prompts/plays/implementation_angle.md
use_when:
- implementation_page_visit
banned_lines:
- "checking out our solution"
- "would love to chat"
```
然后创建 prompts/improve_prompt.md：
```
You improve one outbound play.
Read:
- AGENTS.md
- config/plays.yaml
- memory/outcomes.jsonl
- the prompt file for the chosen play
Pick one play with at least 10 outcomes.
Find:
- lines or structures that appear in positive outcomes
- lines or structures that appear in no_reply or bad_fit outcomes
- any phrase that should be banned
Make one small edit to that play's prompt.
Rules:
- Do not change scoring.
- Do not create a new play.
- Do not add a new channel.
- Cite outcome rows.
- Write the before and after instruction.
Then run the copy eval if present.
If no copy eval exists, open the PR as review_required.
```
一些改进可以自动评分。其他的仍然需要品味。如果没有文案评估，Codex 可以建议提示词编辑，但它应该将 PR 标记为审查而不是假装编辑已得到证明。
理想状态。Codex 说，“这个短语出现在七个 no_reply 结果中，所以我把它添加到了 banned_lines，”或者“正面回复引用了第一句话中的实施细节，所以我收紧了策略以要求这样做。”
失效点。改进器因为一条消息得到了回复而重写了整个语调。提示词编辑应该比你的本能更小。
## 步骤 7. 将更改作为拉取请求发布
这是控制层。Codex 编辑文件，运行评估，并编写 PR 摘要。人工审查并合并。
创建 prompts/pr_summary.md：
```
Write a pull request summary for this outbound improvement.
Include:
1. What changed.
2. Why it changed, citing outcome rows.
3. Before score.
4. After score.
5. Files changed.
6. Risk.
7. What the human reviewer should check.
Keep it short.
Do not claim the change is live.
```
创建 `scripts/open_pr.sh`：
```
#!/usr/bin/env bash
set -euo pipefail
branch="codex/weekly-tune-$(date +%Y-%m-%d)"
git checkout -b "$branch"
git add config prompts evals memory
git commit -m "Codex weekly outbound tune"
body="$(cat outputs/pr-summary.md)"
python3 scripts/create_pr.py \
"$branch" \
"Codex weekly outbound tune" \
"$body"
```
PR 读起来应该像队友写的：
```
Changed:
- Raised implementation_page_visit from 4 to 6.
Why:
- KiteOps had implementation-page intent and replied with implementation timing.
- The previous score routed this account to ignore.
Before:
- eval score 0.75
After:
- eval score 1.00
Reviewer check:
- Make sure implementation intent is specific enough.
- Keep generic downloads low.
- Merge only if this matches actual sales judgment.
```
这就是安全系统。Codex 做乏味的工作。操作员保持标准。
理想状态。每周一个 PR，微小的差异，明确的原因，通过的评估。
失效点。有人给 Codex 合并的权限，因为审查感觉像摩擦。那一分钟区分了一个改进的系统和一个漂移的系统。
## 步骤 8. 将其放入节奏
不要在每次回复后运行此操作。这就是系统过度适应一个吵闹账户的方式。
让一周过去，让结果累积，然后调整。
创建 scripts/weekly_tune.sh：
```
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
python3 evals/score.py || true
scripts/run_codex_step.sh improve_scoring
python3 evals/score.py
scripts/run_codex