# Scoping and curating eval datasets
**作者**: Annabell
**日期**: 2026-08-06T15:05:03.000Z
**来源**: [https://x.com/annabellschfr/status/2085381643687047434](https://x.com/annabellschfr/status/2085381643687047434)
---
Your agent is running and yielding first results, but at some point you realize: 'vibe-fixing my application does not cut it anymore'. You need a structured and strategic way to make sure your agent consistently does what it is supposed to do. You need an eval dataset.
An eval dataset is a repeatable set of examples that represent the scope of your application, and that you use to measure and improve your system. By running your application against the same inputs over time, you can track quality with metrics, compare changes, and catch regressions before they affect production users.
This guide focuses on the design work that happens before and while you create datasets and dataset items. To learn more about the basic building blocks you can check out our academy section on datasets
Dataset design is iterative. A good starting point is a minimally complete dataset: about 15-30 rows that can run through your application, cover the most important input slices, and have an evaluator or review rubric. Run that version early, fix the schema and evaluator, then expand into the gaps you see from those runs or from production input.
Your application will very likely have more than one evaluation dataset. Datasets are often scoped to a specific part of the system or to one sub-step the agent takes.
## 1. Start with the goal of the dataset
Before writing rows, define the smallest useful goal the dataset should support. This keeps the dataset from becoming a vague bucket of interesting examples.
A good early starting point for a dataset is looking at the most common examples end to end. This gives you the foundation and the general understanding from which you can expand.
Over time, teams add datasets for single steps, adversarial cases, red teaming, or specific sub-use cases in their application. We have even seen datasets for checking company name spelling.
The goal you set for your dataset defines boundary and job: end-to-end datasets use the payload the application receives, while step-level datasets use the structured state a step sees in production.
If two jobs need different inputs, evaluators, or release decisions, split them. A stable regression dataset and an adversarial-input dataset can both be useful, but combining them without a clear split makes aggregate scores harder to interpret.
## 2. Inspect available sources
Before selecting examples or writing dataset items, inspect a small sample of the material you could use. The goal is to understand what exists and how it's different from what you expected.
Start with three source types:
- Production traces: show realistic usage, common paths, and observed failures. Scores, user feedback, tickets, and complaints are discovery signals for useful traces; they are not a separate source type.
- Existing assets: old datasets, FAQs, policies, docs, support macros, CSVs, JSON files, and benchmarks can bootstrap known coverage quickly.
- Synthetic cases: expert-written and AI-generated examples can fill gaps and give you an idea of what your application is expected to face.
For each source, review input topics, input and output shapes, and failure modes. You might learn that production traces lack the retrieval context your evaluator needs, that tickets from legacy systems expose better failure labels than traces, or that there is already a good set of end-to-end examples, because support created a 'FAQ' at some point.
Use these sources to choose the distribution of inputs, evaluators, and item schema in the next steps.
## 3. Choose the input distribution
For a first dataset, keep the input distribution simple. Start with the few slices that tell you what to do after a run:
- Scenario type: the main jobs, intents, routes, or task families. For a support-routing dataset, this might be billing, account access, technical issue, and sales request.
- Difficulty or risk: routine, ambiguous, hard, adversarial, or business-critical cases. Include more than happy paths, but do not make the first dataset all edge cases.
- Dataset role: why the row exists: typical case, known regression, observed failure, or synthetic gap-fill.
For a support-routing dataset, the first version might use a simple scenario-by-difficulty matrix:
Input distribution is the deliberate mix of cases in the dataset: which scenarios appear, how difficult they are, and why each row is included. It is the coverage plan that helps you interpret experiment results by slice.
The distribution does not have to mirror production frequency exactly. A regression dataset may intentionally overrepresent failures, edge cases, or high-value paths. What counts is that the mix is deliberate and visible in metadata.
Add more dimensions only when they change behavior or help interpret results. Channel, language, customer segment, region, context availability, and product area can be useful metadata, but they should not all become balancing constraints on day one.
## 4. Decide how evaluation will work
Choose the evaluation style before you write expected outputs. This determines what the dataset item needs to contain and how useful the first results will be.
Use reference-based evaluation when each item has a known target: a correct label, expected tool call, required fact, structured output, reference answer, or expected next action. This is the best fit for regression tests and CI gates because failures are easier to inspect. The trade-off is that references take work to write and can become brittle if they over-specify wording instead of behavior.
Use reference-free evaluation when there is no stable expected output, but every item can be judged against the same rule or rubric. This works for checks such as valid JSON, language match, grounding in provided context, safety, or tone. The trade-off is that the evaluator or rubric carries more weight, so ambiguous rubrics produce ambiguous results.
For example, a docs chatbot dataset can use either approach:
Choose the cheapest evaluator that captures the requirement: code evaluators for deterministic checks, LLM-as-a-judge for language-quality judgments, and manual evaluation while you are still learning what good and bad outputs look like.
At this stage, you only need to decide how each dataset item will be reviewed: by an automated evaluator, manual annotation, or both. For the deeper work of designing the evaluator itself, see the Academy page on evaluation.
If you cannot yet name the evaluator or review rubric that will score each row, go for a review workflow first and use annotation queues to label examples manually.
## 5. Design the item schema
The three dataset item fields are flexible JSON. Here, define them as a concrete contract that your experiment runner, evaluators, and reviewers can all consume.
Define:
- input: the object you will pass into the system boundary
- expectedOutput: only the reference data the evaluator or reviewer needs; omit it for deliberate reference-free evaluation
- metadata: stable slice and provenance fields such as source, scenario type, difficulty, dataset role, and review status
Once you have decided on an item schema, enforce it so your team can effectively collaborate on adding items in the right structure.
For a support-routing dataset, one row could look like this:
The input keeps only the context the router needs. The expectedOutput states the behavior to check: route to billing, ask for the invoice ID, and do not promise a refund before review. The metadata records the row's source, scenario, difficulty, and role.
Keep the schema stable before collecting rows in bulk. Preserve behavior-shaping fields such as conversation history, retrieved context, tool state, routing metadata, or user attributes, but avoid arbitrary per-row fields or natural-language summaries of structured context.
## 6. Draft a first version
Once the goal, distribution, evaluation method, and schema are concrete, start turning selected source examples into dataset items. Use the sources you inspected earlier, and add them in a format that matches the defined schema.
For a minimally complete first version, choose enough rows to test the whole contract along your input distribution:
- common scenarios that should work reliably
- a few ambiguous or high-risk scenarios
- known failures or regressions you want to prevent
- synthetic gap fills only where production traces or existing assets do not cover the distribution
Do not wait until the dataset feels complete. Run the first coherent version in an experiment before adding more rows.
## 7. Run the first experiment and expand deliberately
Use the first run to check whether the input shape works with the real application path, if the evaluator outcomes make sense, and if the expected output shape works for the intended purpose.
After the first run, expand and iterate until you arrive at scope and shape that helps you feel confident about shipping changes.
- Add rows when a failure reveals a missing scenario, difficulty level, or source.
- Edit rows when the expected output, rubric, input shape, or metadata is ambiguous.
- Archive rows when they no longer match current prompts, tools, policies, or product behavior.
## How datasets evolve over time
Datasets have to evolve with your application. By monitoring production and frequently reviewing data through structured error analysis, your datasets can evolve over time to represent the production scope of your system. There are three useful expansion patterns:
- Production-mirroring: add interesting cases from production no matter if good or bad, to expand the coverage of your dataset over time.
- Bad-trace expansion: add a reviewed dataset item whenever you find a serious production failure. This works well once the system is live and you can continuously mine traces. For scoring, use automated evaluators such as LLM-as-a-judge or code evaluators, or send examples to annotation queues for manual review.
- Purpose-specific datasets: build separate datasets for stable regression, adversarial inputs, single-step evaluations.
## Put it into practice
- Start with one release question. Pick a product area and the smallest end-to-end dataset that can answer whether a specific change can ship.
- Use 15-30 rows. Cover the most common scenarios, a few high-risk cases, and one or two known failures.
- Keep every row runnable and evaluable. Each input should pass through the application path and be scorable by the evaluator or review rubric.
- Expand after the first run. Add bad traces for production failures you want to prevent from coming back, create purpose-specific datasets when one dataset starts mixing different jobs, and add synthetic rows only for named gaps.
## 相关链接
- [Annabell](https://x.com/annabellschfr)
- [@annabellschfr](https://x.com/annabellschfr)
- [12K](https://x.com/annabellschfr/status/2085381643687047434/analytics)
- [datasets](https://langfuse.com/academy/datasets)
- [1. Start with the goal of the dataset](https://langfuse.com/academy/datasets/designing-great-datasets#1-start-with-the-goal-of-the-dataset)
- [2. Inspect available sources](https://langfuse.com/academy/datasets/designing-great-datasets#2-inspect-available-sources)
- [3. Choose the input distribution](https://langfuse.com/academy/datasets/designing-great-datasets#3-choose-the-input-distribution)
- [4. Decide how evaluation will work](https://langfuse.com/academy/datasets/designing-great-datasets#4-decide-how-evaluation-will-work)
- [reference-based evaluation](https://langfuse.com/academy/evaluate#reference-based-vs-reference-free)
- [reference-free evaluation](https://langfuse.com/academy/evaluate#reference-based-vs-reference-free)
- [code evaluators](https://langfuse.com/docs/evaluation/evaluation-methods/code-evaluators)
- [LLM-as-a-judge](https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge)
- [manual evaluation](https://langfuse.com/academy/evaluate#how-evaluation-typically-evolves)
- [Academy page on evaluation](https://langfuse.com/academy/evaluate)
- [annotation queues](https://langfuse.com/docs/evaluation/evaluation-methods/annotation-queues)
- [5. Design the item schema](https://langfuse.com/academy/datasets/designing-great-datasets#5-design-the-item-schema)
- [three dataset item fields](https://langfuse.com/academy/datasets#the-dataset-item)
- [in the right structure](https://langfuse.com/docs/evaluation/experiments/datasets#schema-enforcement)
- [6. Draft a first version](https://langfuse.com/academy/datasets/designing-great-datasets#6-draft-a-first-version)
- [7. Run the first experiment and expand deliberately](https://langfuse.com/academy/datasets/designing-great-datasets#7-run-the-first-experiment-and-expand-deliberately)
- [How datasets evolve over time](https://langfuse.com/academy/datasets/designing-great-datasets#how-datasets-evolve-over-time)
- [monitoring production](https://langfuse.com/academy/monitoring)
- [error analysis](https://langfuse.com/academy/monitoring/error-analysis)
- [LLM-as-a-judge](https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge)
- [code evaluators](https://langfuse.com/docs/evaluation/evaluation-methods/code-evaluators)
- [annotation queues](https://langfuse.com/docs/evaluation/evaluation-methods/annotation-queues)
- [Put it into practice](https://langfuse.com/academy/datasets/designing-great-datasets#put-it-into-practice)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [11:05 PM · Aug 6, 2026](https://x.com/annabellschfr/status/2085381643687047434)
- [12K Views](https://x.com/annabellschfr/status/2085381643687047434/analytics)
- [View quotes](https://x.com/annabellschfr/status/2085381643687047434/quotes)
---
*导出时间: 2026/8/7 21:29:17*
---
## 中文翻译
# 规划与策划评估数据集
**作者**: Annabell
**日期**: 2026-08-06T15:05:03.000Z
**来源**: [https://x.com/annabellschfr/status/2085381643687047434](https://x.com/annabellschfr/status/2085381643687047434)
---
你的智能体已经开始运行并产出了初步结果，但在某个时刻你会意识到：“仅凭直觉来修补我的应用已经行不通了”。你需要一种结构化且具有战略性的方法，来确保你的智能体能够始终如一地完成它应有的任务。你需要一个评估数据集。
评估数据集是一组可重复使用的示例，它们代表了你的应用范围，你可以用它来衡量和改进你的系统。通过随着时间的推移对相同的输入运行你的应用，你可以使用指标来跟踪质量、比较变化，并在它们影响到生产环境用户之前发现性能衰退。
本指南重点关注在创建数据集和数据集条目之前及期间所进行的设计工作。要了解有关基础构件的更多信息，你可以查看我们学院关于数据集的章节。
数据集设计是迭代的。一个极简但完整的数据集是一个很好的起点：大约 15-30 行能够贯穿你的应用、覆盖最重要输入切片的数据，并配备一个评估器或审查评分标准。尽早运行该版本，修复数据结构和评估器，然后根据这些运行结果或生产输入中发现的空白进行扩充。
你的应用很可能会有多个评估数据集。数据集通常会被限定在系统的特定部分，或智能体执行的某个子步骤中。
## 1. 从数据集的目标开始
在编写数据行之前，请定义该数据集应支持的最小可用目标。这能防止数据集变成一个装满“看似有趣的示例”却大而全的杂物桶。
数据集早期阶段的一个好起点是查看最常见的端到端示例。这为你提供了扩展的基础和总体理解。
随着时间的推移，团队会为应用中的单个步骤、对抗性场景、红蓝对抗或特定的子用例添加数据集。我们甚至见过专门用于检查公司名称拼写的数据集。
你为数据集设定的目标定义了它的边界和任务：端到端数据集使用应用接收的有效载荷，而步骤级数据集则使用某个步骤在生产环境中看到的结构化状态。
如果两个任务需要不同的输入、评估器或发布决策，请将它们分开。一个稳定的回归测试数据集和一个对抗性输入数据集可能都很有用，但如果不加清晰地区分而将它们混在一起，会使综合得分难以解读。
## 2. 检查可用来源
在选择示例或编写数据集条目之前，请先检查一小部分你可能使用的材料样本。目的是了解现有哪些材料，以及它们与你预期的有何不同。
从三种来源类型开始：
- **生产环境轨迹**：展示真实的使用情况、常见路径和观察到的失败。评分、用户反馈、工单和投诉是发现有价值的轨迹的探索信号；它们本身并不是一种单独的来源类型。
- **现有资产**：旧数据集、常见问题解答（FAQ）、策略、文档、客服话术、CSV、JSON 文件和基准测试可以快速引导建立已知的覆盖率。
- **合成案例**：专家手写和 AI 生成的示例可以填补空白，并让你了解应用预期会面临的情况。
对于每种来源，请审查输入主题、输入和输出的形状（结构）以及失败模式。你可能会发现，生产环境轨迹缺乏评估器所需的检索上下文；遗留系统的工单比轨迹暴露了更好的失败标签；或者由于客服团队在某个时候制作了“FAQ”，其实已经存在一套很好的端到端示例。
在接下来的步骤中，利用这些来源来选择输入的分布、评估器和条目模式（Schema）。
## 3. 选择输入分布
对于第一个数据集，请保持输入分布的简单。从少数几个能在运行后告诉你下一步该怎么做的切片开始：
- **场景类型**：主要任务、意图、路由或任务族。对于客服路由数据集，这可能是账单、账户访问、技术问题和销售请求。
- **难度或风险**：常规、模糊、困难、对抗性或业务关键型案例。除了“正常路径（Happy paths）”外，还应包含更多内容，但不要让第一个数据集全都是边缘案例。
- **数据集角色**：该数据行存在的原因：典型案例、已知回归、观察到的失败或用于填补空白的合成案例。
对于客服路由数据集，第一版可以使用一个简单的“场景 × 难度”矩阵：
输入分布是数据集中各种案例经过深思熟虑的混合：出现了哪些场景，它们的难度如何，以及为什么要包含每一行。它是一个覆盖率计划，能帮助你按切片解读实验结果。
分布不必完全照搬生产环境中的出现频率。一个回归数据集可能会故意增加失败、边缘案例或高价值路径的比例。重要的是，这种比例的混合是经过深思熟虑的，并且在元数据中清晰可见。
只有当更多维度能改变行为或有助于解读结果时，才添加它们。渠道、语言、客户群、地区、上下文可用性和产品领域都可以作为有用的元数据，但它们不应该在第一天就全部成为需要平衡的约束条件。
## 4. 决定评估方式
在编写预期输出之前选择评估风格。这决定了数据集条目需要包含什么内容，以及初步结果会有多大用处。
当每个条目都有一个已知目标时，使用**基于参考的评估**：正确的标签、预期的工具调用、必须包含的事实、结构化输出、参考答案或预期的下一步操作。这最适合用于回归测试和 CI 门禁，因为失败的情况更容易排查。其代价是编写参考内容需要投入精力，而且如果对措辞（而不是行为）规定得过于死板，它们可能会变得脆弱。
当没有稳定的预期输出，但每个条目都可以根据相同的规则或评分标准进行评判时，使用**无参考评估**。这适用于诸如检查 JSON 是否有效、语言是否匹配、是否基于提供的上下文、安全性或语气等场景。代价是评估器或评分标准承担了更大的权重，因此模糊的评分标准会产生模糊的结果。
例如，一个文档聊天机器人的数据集可以采用这两种方法中的任何一种：
选择能够满足要求的最廉价的评估器：代码评估器用于确定性检查，LLM-as-a-judge（大语言模型作为裁判）用于语言质量判断，而在你仍在摸索什么是好的或坏的输出时，则使用人工评估。
在这个阶段，你只需要决定如何审查每个数据集条目：是通过自动化评估器、人工标注，还是两者兼有。关于设计评估器本身的更深入工作，请参阅学院关于评估的页面。
如果你现在还无法确定将用于为每一行打分的评估器或审查评分标准的名称，请先采用审查工作流，并使用标注队列手动为示例打标签。
## 5. 设计条目模式（Schema）
三个数据集条目字段是灵活的 JSON。在这里，将它们定义为一个具体的契约，以便你的实验运行器、评估器和审查人员都可以使用。
定义：
- **input**：你将传递给系统边界的对象。
- **expectedOutput**：仅包含评估器或审查人员所需的参考数据；如果是刻意采用无参考评估，则省略此项。
- **metadata**：稳定的切片和来源字段，如来源、场景类型、难度、数据集角色和审查状态。
一旦确定了条目模式，就要严格执行，这样你的团队才能在添加条目时有效地遵循正确的结构进行协作。
对于客服路由数据集，一行的数据可能如下所示：
输入只保留路由器所需的上下文。预期输出说明了要检查的行为：路由到账单部门、询问发票 ID，以及不承诺在审查前退款。元数据则记录了该行的来源、场景、难度和角色。
在大量收集数据行之前，请保持模式稳定。保留那些能塑造行为的字段，如对话历史、检索到的上下文、工具状态、路由元数据或用户属性，但要避免随意添加的单行字段，或对结构化上下文进行自然语言的总结。
## 6. 起草第一版
一旦目标、分布、评估方法和模式都具体确定下来，就开始将选定的来源示例转化为数据集条目。利用你之前检查过的来源，并按照定义好的模式格式将它们添加进去。
对于一个极简但完整的第一版，请选择足够多的数据行，以便沿着你的输入分布测试整个契约：
- 应该能可靠工作的常见场景。
- 少数模糊或高风险的场景。
- 你想要防止的已知失败或回归。
- 仅在生产轨迹或现有资产未覆盖到分布要求的地方，使用合成数据进行填补。
不要等到数据集感觉已经很完整了才运行。在添加更多行之前，先在实验中运行第一个连贯的版本。
## 7. 运行第一次实验并有意识地扩充
利用第一次运行来检查输入结构是否适用于实际的应用路径，评估器的输出结果是否合理，以及预期的输出结构是否能达到预期目的。
在第一次运行之后，不断扩充和迭代，直到数据集的范围和结构能让你对发布变更充满信心。
- 当一次失败揭示出缺失的场景、难度级别或来源时，添加数据行。
- 当预期输出、评分标准、输入结构或元数据不明确时，修改数据行。
- 当某些数据行不再匹配当前的提示词、工具、策略或产品行为时，将其归档。
## 数据集如何随时间演进
数据集必须随着你的应用一起演进。通过监控生产环境并通过结构化的错误分析频繁审查数据，你的数据集可以随着时间的推移不断演进，以代表你系统的生产环境范围。这里有三种有用的扩充模式：
- **镜像生产环境**：无论好坏，将生产环境中的有趣案例添加进来，随着时间的推移扩大数据集的覆盖率。
- **不良轨迹扩充**：每当你发现严重的生产故障时，就添加一个经过审查的数据集条目。这非常适合在系统上线后，能够持续挖掘轨迹的情况。对于评分，可以使用自动化评估器（如 LLM-as-a-judge 或代码评估器），或者将示例发送到标注队列进行人工审查。
- **特定目的数据集**：为稳定的回归测试、对抗性输入、单步骤评估建立单独的数据集。
## 付诸实践
- 从一个发布问题开始。选择一个产品领域，建立最小的端到端数据集，用来回答“某个特定的变更是否可以发布”。
- 使用 15-30 行数据。覆盖最常见的场景、几个高风险案例以及一两个已知故障。
- 确保每一行都可运行、可评估。每个输入都应能贯穿应用路径，并能被评估器或审查评分标准打分。
- 首次运行后进行扩充。为你想要防止再次发生的生产故障添加不良轨迹；当一个数据集开始混杂不同的任务时，创建特定目的的数据集；并仅针对已明确指出的空白添加合成数据行。
## 相关链接
- [Annabell](https://x.com/annabellschfr)
- [@annabellschfr](https://x.com/annabellschfr)
- [12K](https://x.com/annabellschfr/status/2085381643687047434/analytics)
- [datasets](https://langfuse.com/academy/datasets)
- [1. Start with the goal of the dataset](https://langfuse.com/academy/datasets/designing-great-datasets#1-start-with-the-goal-of-the-dataset)
- [2. Inspect available sources](https://langfuse.com/academy/datasets/designing-great-datasets#2-inspect-available-sources)
- [3. Choose the input distribution](https://langfuse.com/academy/datasets/designing-great-datasets#3-choose-the-input-distribution)
- [4. Decide how evaluation will work](https://langfuse.com/academy/datasets/designing-great-datasets#4-decide-how-evaluation-will-work)
- [reference-based evaluation](https://langfuse.com/academy/evaluate#reference-based-vs-reference-free)
- [reference-free evaluation](https://langfuse.com/academy/evaluate#reference-based-vs-reference-free)
- [code evaluators](https://langfuse.com/docs/evaluation/evaluation-methods/code-evaluators)
- [LLM-as-a-judge](https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge)
- [manual evaluation](https://langfuse.com/academy/evaluate#how-evaluation-typically-evolves)
- [Academy page on evaluation](https://langfuse.com/academy/evaluate)
- [annotation queues](https://langfuse.com/docs/evaluation/evaluation-methods/annotation-queues)
- [5. Design the item schema](https://langfuse.com/academy/datasets/designing-great-datasets#5-design-the-item-schema)
- [three dataset item fields](https://langfuse.com/academy/datasets#the-dataset-item)
- [in the right structure](https://langfuse.com/docs/evaluation/experiments/datasets#schema-enforcement)
- [6. Draft a first version](https://langfuse.com/academy/datasets/designing-great-datasets#6-draft-a-first-version)
- [7. Run the first experiment and expand deliberately](https://langfuse.com/academy/datasets/designing-great-datasets#7-run-the-first-experiment-and-expand-deliberately)
- [How datasets evolve over time](https://langfuse.com/academy/datasets/designing-great-datasets#how-datasets-evolve-over-time)
- [monitoring production](https://langfuse.com/academy/monitoring)
- [error analysis](https://langfuse.com/academy/monitoring/error-analysis)
- [LLM-as-a-judge](https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge)
- [code evaluators](https://langfuse.com/docs/evaluation/evaluation-methods/code-evaluators)
- [annotation queues](https://langfuse.com/docs/evaluation/evaluation-methods/annotation-queues)
- [Put it into practice](https://langfuse.com/academy/datasets/designing-great-datasets#put-it-into-practice)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [11:05 PM · Aug 6, 2026](https://x.com/annabellschfr/status/2085381643687047434)
- [12K Views](https://x.com/annabellschfr/status/2085381643687047434/analytics)
- [View quotes](https://x.com/annabellschfr/status/2085381643687047434/quotes)
---
*导出时间: 2026/8/7 21:29:17* 
