# Designing eval datasets for LLM applications
**作者**: Lotte
**日期**: 2026-05-17T15:20:50.000Z
**来源**: [https://x.com/lotte_verheyden/status/2056032201259831398](https://x.com/lotte_verheyden/status/2056032201259831398)
---
This is one piece of a series we’re publishing as part of the Langfuse Academy, where we walk through the full AI engineering lifecycle. If you’re new to the series,The AI Engineering Loop is the best place to start
## A short recap of the AI Engineering Loop
The AI Engineering Loop is how teams continuously improve AI systems. It connects what’s happening in production (tracing, monitoring) to structured iteration during development (datasets, experiments, evaluation). Each shipped improvement produces new data, and teams loop through this process continuously.
You can read more on this here.
# How datasets fit into the loop
So far, we've covered the first two steps of the AI engineering loop: tracing your application and monitoring its behavior live. Those give you visibility into what your system is actually doing and give you inspiration for improvement.
Now the question becomes: when you spot something worth improving, how do you test a change before deploying it to production? The next three steps of the loop cover exactly this, and it starts with datasets.
A dataset is a collection of test cases that you run your application against each time you make a change ("an experiment"). Instead of deploying and hoping for the best, you get a repeatable, consistent check across a set of inputs that represent real-world usage.
# The dataset item
A dataset is made up of items, each item represents one test case: a situation your application should be able to handle. Generally, an item has three fields:
- Input (required)
- Expected output (optional)
- Metadata (optional)
## The three fields of a dataset item
A good mental model:
## Common expected output patterns
Whether you need an expected output, and what it looks like, depends on which type of evaluator you use.
Reference-based versus reference-free evaluators
Some evaluators check the output against a predefined expected output (reference-based). Others assess the output without needing a ground truth to compare against (reference-free).
Exact match
The expected output is the literal correct answer. For example:
- A classification task where the correct label is "billing_inquiry"
- An extraction task where the expected entities are ["Paris", "Thursday"]
Reference answer
The expected output is a gold-standard response that shows what a good output looks like. The evaluator can compare the test's output against this example, for instance by checking semantic similarity or whether the key points match.
Evaluation criteria
The expected output is a list of checks or requirements the output should satisfy. For example:
- "must mention the refund policy"
- "must include a link to the help center"
The evaluator checks whether the output meets these criteria.
Nothing
Sometimes no expected output is required at all. If you're just checking whether:
- the tone is professional
- the response is safe
- the output follows a required format
Your dataset items don't need anything other than an input as you will use a reference-free evaluator.
Combination of the above
Because you can run a combination of different evaluators on a single dataset item, a dataset item's expected output field can also contain multiple types of reference data. The expected output is a JSON field, so you can store multiple types of reference data without a problem.
# What makes a good dataset
A good dataset mirrors what your system will encounter in production. If passing the dataset gives you confidence before deploying, it's doing its job.
Clear in scope. Each dataset should have a well-defined scope. That can be end-to-end if you treat internal steps as implementation details, or it can target an individual step like retrieval or summarization if that's the part you're trying to improve. You'll likely end up with multiple datasets, each with a clear purpose.
The right size for the workflow. Some datasets are small and fast enough to run on every push as part of your CI/CD pipeline. Others are larger and more comprehensive, and are useful to run periodically but too slow for every minor change.
# Where to start
Start with the most concrete examples you have, then expand coverage once you know what you are trying to test.
1. Pull examples from production traces that you spotted and would like to improve, either as-is or anonymized or transformed by AI.
2. Add hand-written cases based on predefined requirements, edge cases, or behaviors your agent must handle reliably.
3. Generate synthetic examples with AI once you know which dimensions you want to cover more broadly.
# What comes next
Once you have a dataset, the next step is running your system against it to see how changes affect output quality. This is what experiments are for.
## 相关链接
- [Lotte](https://x.com/lotte_verheyden)
- [@lotte_verheyden](https://x.com/lotte_verheyden)
- [27K](https://x.com/lotte_verheyden/status/2056032201259831398/analytics)
- [angfuse Academy](https://langfuse.com/academy)
- [The AI Engineering Loop](https://langfuse.com/academy/ai-engineering-loop)
- [here](https://langfuse.com/academy/ai-engineering-loop)
- [AI engineering loop](https://langfuse.com/academy/ai-engineering-loop)
- [tracing](https://langfuse.com/academy/tracing)
- [monitoring](https://langfuse.com/academy/monitoring)
- [experiments](https://langfuse.com/academy/experiments)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [11:20 PM · May 17, 2026](https://x.com/lotte_verheyden/status/2056032201259831398)
- [27.1K Views](https://x.com/lotte_verheyden/status/2056032201259831398/analytics)
- [View quotes](https://x.com/lotte_verheyden/status/2056032201259831398/quotes)
---
*导出时间: 2026/5/18 12:42:17*
---
## 中文翻译
# 为 LLM 应用设计评估数据集
**作者**: Lotte
**日期**: 2026-05-17T15:20:50.000Z
**来源**: [https://x.com/lotte_verheyden/status/2056032201259831398](https://x.com/lotte_verheyden/status/2056032201259831398)
---
这是我们作为 Langfuse 学院一部分发布的系列文章之一，在该系列中，我们将详细介绍完整的 AI 工程生命周期。如果你是刚接触这个系列，那么“AI 工程循环”是最好的起点。
## AI 工程循环简要回顾
AI 工程循环是团队持续改进 AI 系统的方式。它将生产环境中发生的情况（跟踪、监控）与开发期间的结构化迭代（数据集、实验、评估）连接起来。每次交付的改进都会产生新数据，团队在这个过程中不断循环。
你可以在这里阅读更多相关内容。
# 数据集如何融入循环
到目前为止，我们已经涵盖了 AI 工程循环的前两个步骤：跟踪应用程序和实时监控其行为。这些让你清楚地了解系统实际在做什么，并为你提供改进的灵感。
现在问题变成了：当你发现值得改进的地方时，如何在部署到生产环境之前测试变更？循环的接下来的三个步骤正是解决这个问题，它从数据集开始。
数据集是一组测试用例，每次你做出更改（即“一次实验”）时，都会在这些用例上运行你的应用程序。不再是部署后听天由命，而是通过一组代表真实用法的输入，获得可重复、一致的检查。
# 数据集项
数据集由项目组成，每个项目代表一个测试用例：即你的应用程序应该能够处理的一种情况。通常，一个项目包含三个字段：
- 输入（必需）
- 预期输出（可选）
- 元数据（可选）
## 数据集项的三个字段
一个好的思维模型：
## 常见的预期输出模式
是否需要预期输出以及它看起来像什么，取决于你使用哪种类型的评估器。
基于参考与无参考的评估器
有些评估器会将输出与预定义的预期输出进行核对（基于参考）。其他的则不需要真实值进行比较即可评估输出（无参考）。
完全匹配
预期输出是字面上的正确答案。例如：
- 一个分类任务，其中正确的标签是 "billing_inquiry"
- 一个提取任务，其中预期的实体是 ["Paris", "Thursday"]
参考答案
预期输出是一个黄金标准的响应，展示了好的输出是什么样的。评估器可以将测试的输出与这个示例进行比较，例如通过检查语义相似度或关键点是否匹配。
评估标准
预期输出是输出应该满足的检查清单或要求列表。例如：
- “必须提及退款政策”
- “必须包含帮助中心的链接”
评估器会检查输出是否满足这些标准。
无
有时根本不需要预期输出。如果你只是检查：
- 语气是否专业
- 响应是否安全
- 输出是否遵循了要求的格式
那么你的数据集项目除了输入之外不需要任何东西，因为你将使用无参考评估器。
上述的组合
因为你可以对单个数据集项目运行不同评估器的组合，所以数据集项目的预期输出字段也可以包含多种类型的参考数据。预期输出是一个 JSON 字段，因此你可以毫无问题地存储多种类型的参考数据。
# 什么构成了好的数据集
一个好的数据集反映了你的系统在生产环境中会遇到的情况。如果通过数据集能让你在部署前充满信心，那么它就尽职了。
范围清晰。每个数据集都应有明确的范围。如果你将内部步骤视为实现细节，则可以是端到端的；或者如果你试图改进的只是某个部分（如检索或摘要），则可以针对单个步骤。你最终可能会得到多个数据集，每个都有明确的目的。
适合工作流的大小。有些数据集很小且运行速度足够快，可以作为 CI/CD 管道的一部分在每次推送时运行。其他的则更大更全面，适合定期运行，但对于每次微小的变更来说太慢了。
# 从哪里开始
从你拥有的最具体的例子开始，然后在你知道要测试什么之后扩展覆盖面。
1. 从你发现并希望改进的生产环境跟踪中提取示例，可以原样提取，也可以通过 AI 进行匿名化或转换。
2. 根据预定义的要求、边缘情况或你的代理必须可靠处理的行为，添加手动编写的案例。
3. 一旦你知道了要在哪些维度上进行更广泛的覆盖，就使用 AI 生成合成示例。
# 接下来做什么
一旦你有了数据集，下一步就是针对它运行你的系统，以查看变更如何影响输出质量。这就是实验的目的。
## 相关链接
- [Lotte](https://x.com/lotte_verheyden)
- [@lotte_verheyden](https://x.com/lotte_verheyden)
- [27K](https://x.com/lotte_verheyden/status/2056032201259831398/analytics)
- [angfuse Academy](https://langfuse.com/academy)
- [The AI Engineering Loop](https://langfuse.com/academy/ai-engineering-loop)
- [here](https://langfuse.com/academy/ai-engineering-loop)
- [AI engineering loop](https://langfuse.com/academy/ai-engineering-loop)
- [tracing](https://langfuse.com/academy/tracing)
- [monitoring](https://langfuse.com/academy/monitoring)
- [experiments](https://langfuse.com/academy/experiments)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [11:20 PM · May 17, 2026](https://x.com/lotte_verheyden/status/2056032201259831398)
- [27.1K Views](https://x.com/lotte_verheyden/status/2056032201259831398/analytics)
- [View quotes](https://x.com/lotte_verheyden/status/2056032201259831398/quotes)
---
*导出时间: 2026/5/18 12:42:17*