# What Comes After the Prompt
**作者**: elvis
**日期**: 2026-07-22T17:30:44.000Z
**来源**: [https://x.com/omarsar0/status/2079982491578827214](https://x.com/omarsar0/status/2079982491578827214)
---
Karpathy’s recent post about using long voice sessions as prompts helped me make sense of a prompting technique I now rely on often while building with agents. The visual that accompanies this post, From a prompt to a task, summarizes the idea in one picture.
For lack of a better term, I have been calling the unit a task. A task uses multimodal prompting to give an agent the instruction and as much relevant context as possible in one turn. It covers a larger unit of work than a single prompt, and it leaves behind a stored trace that can later become a reusable skill.
A task can include a long voice explanation, the current screen, precise text, annotations, transcriptions, images, and any other evidence that helps the agent understand the work. Each modality contributes something different. Voice carries reasoning, priorities, examples, and uncertainty. The screen gives the agent the current state and the environment where the work needs to happen. Annotations direct attention to specific details. Text preserves exact requirements, names, and constraints. Together, these signals give the agent a richer representation of the work.
## The interaction
The experience feels closer to guiding an agent through a complex assignment than composing a conventional prompt. I front-load the context that would otherwise emerge across several turns, then give the agent room to complete more of the work in a single pass. In practice, I record a voice note while walking through the work, capture the relevant screen, mark it up with quick annotations, and paste in the exact text the agent needs.
The agent can still ask questions when important information is missing. In my experience, richer tasks reduce the repetitive back-and-forth where I restate context, point out the same details, or correct an assumption that could have been resolved from the beginning. A recent example was scheduling a post on a platform I rarely use. I recorded a short voice note with the goal and constraints, shared the screen with the scheduling page open, and annotated the fields that mattered. The agent completed the setup in one pass, and the usual follow-ups about which fields to fill and which copy to paste never happened.
This has also changed how I think about productivity with agents. A well-formed task gives me more confidence to hand off work and move to something else. That makes parallel work more practical because each agent needs less active supervision while it runs.
## Why it works
Karpathy pointed out that LLMs are remarkably good at reconstructing intent from long, disorganized voice sessions. A ramble contains many weak signals about the goal, the constraints, the examples that matter, and the speaker’s uncertainty. The model can organize those signals into a cleaner representation of the request.
I am extending that idea with more modalities. The voice session provides the reasoning, while the screen, text, annotations, transcriptions, and images provide additional evidence. When one channel is noisy or incomplete, another channel can help resolve the ambiguity.
Complex agent tasks often fail at the boundaries between what I meant, what I explicitly said, and what the agent could observe. Multimodal prompting gives the model more opportunities to close those gaps before it begins the work.
## Cost and payoff
This approach can look like overkill, and sometimes it is. A simple request still deserves a simple prompt. I use richer tasks when the work is long-running, when precision matters, when the agent needs to navigate an unfamiliar interface, or when a mistake would create several rounds of correction.
A multimodal task can also cost more because it contains more context. In my experience, that investment usually pays for itself. I can complete a larger unit of work per turn because the agent begins with more of the context it needs.
This is especially useful for browser use and computer use. The agent can see the environment, hear the reasoning behind the request, follow annotations that identify important elements, and use text for exact details. That combination helps the agent navigate unfamiliar interfaces.
Some of my current examples include scheduling posts on unfamiliar platforms, improving writing and editing, and refining the design of artifacts and web pages. These tasks involve many small decisions that are tedious to encode as a traditional prompt but easy to communicate while showing the work. In a design refinement task, the modalities map naturally. Voice explains what feels off about the layout and what the change should preserve. The screen shows the current state of the artifact. Annotations mark the specific spacing, components, or sections to adjust. Text supplies the exact copy and the constraints that should stay fixed.
## From traces to skills
I store the traces from these tasks and review them for recurring patterns.
The useful patterns usually include the sequence of actions, the constraints I repeat, the quality checks I apply, and the corrections that consistently improve the result. Those patterns can be extracted into reusable skills so the next agent starts with a stronger workflow.
This connection to automation is important. A task gives me a practical unit that I can inspect, improve, and eventually place inside a larger loop. The richer initial trace helps me understand which parts can be automated reliably and where human guidance still adds value.
The process usually starts with a manual task. Repeated use produces traces, the traces reveal patterns, and the patterns become a reusable skill. Over time, the workflow requires less explanation because the important guidance has been captured.
If you are curious to learn more, I will be demoing, sharing, and writing more about this with our academy here: https://academy.dair.ai/
## Toward omnimodels
Omnimodels, models built to consume voice, vision, images, and text natively, should make this style of interaction feel natural. We will be able to speak, show, point, type, and provide examples within the same session, while the model integrates those signals directly.
I feel like I am rehearsing for that interaction now. The current tools already make it possible, even if the experience still feels stitched together across voice, browser state, images, and text.
The term task is provisional, but the underlying idea has become clear through repeated use. Give the agent a richer trace of the work, let it reconstruct the intent, store what happened, and reuse the patterns that work.
This came from a practical need. I wanted fewer correction loops, stronger handoffs, and more dependable long-running agent workflows. Multimodal prompting has moved me steadily in that direction, and it has become my default way of handing agents real work.
## 相关链接
- [elvis](https://x.com/omarsar0)
- [@omarsar0](https://x.com/omarsar0)
- [15K](https://x.com/omarsar0/status/2079982491578827214/analytics)
- [https://academy.dair.ai/](https://academy.dair.ai/)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:30 AM · Jul 23, 2026](https://x.com/omarsar0/status/2079982491578827214)
- [15.7K Views](https://x.com/omarsar0/status/2079982491578827214/analytics)
- [View quotes](https://x.com/omarsar0/status/2079982491578827214/quotes)
---
*导出时间: 2026/7/23 11:43:04*
---
## 中文翻译
# 提示词之后是什么
**作者**: elvis
**日期**: 2026-07-22T17:30:44.000Z
**来源**: [https://x.com/omarsar0/status/2079982491578827214](https://x.com/omarsar0/status/2079982491578827214)
---
Karpathy 最近关于使用长语音会话作为提示词的帖子，帮我理清了一种提示词技术，我现在在构建智能体时经常依赖它。这篇帖子配套的图片“从提示词到任务”，用一张图总结了这个想法。
由于没有更好的术语，我一直把这个单元称为“任务”。任务使用多模态提示词，在一个回合内给智能体提供指令以及尽可能多的相关上下文。它涵盖了比单个提示词更大的工作单元，并留下一个存储的追踪记录，该记录以后可以成为可重用的技能。
一个任务可以包含长语音解释、当前屏幕、精确文本、标注、转录、图像，以及任何其他能帮助智能体理解工作的证据。每种模态都有其独特的贡献。语音承载着推理、优先级、示例和不确定性。屏幕为智能体提供当前状态以及工作需要发生的环境。标注引导注意力到具体的细节。文本保留确切的要求、名称和约束。这些信号共同为智能体提供了工作更丰富的表示。
## 交互
这种体验感觉更像是引导智能体完成一项复杂的任务，而不是撰写传统的提示词。我将那些原本需要多轮对话才会显现的上下文前置加载，然后给智能体留出空间，让它一次性完成更多的工作。在实践中，我在梳理工作时录制语音备忘录，捕捉相关屏幕，用快速标注进行标记，并粘贴智能体所需的精确文本。
当重要信息缺失时，智能体仍然可以提问。根据我的经验，更丰富的任务减少了那些重复的来回拉扯，比如我需要重申上下文、指出相同的细节，或者纠正一个本可以从一开始就解决的假设。最近的一个例子是在一个我很少使用的平台上安排发布帖子。我录制了一个包含目标和约束的简短语音备忘录，打开了调度页面的屏幕共享，并标注了相关的字段。智能体一次性完成了设置，而那些关于填写哪些字段、粘贴哪些文案的常规后续追问再也没有发生。
这也改变了我对智能体生产力的看法。一个结构良好的任务让我更有信心移交工作并转向其他事情。这使得并行工作更加切实可行，因为每个智能体在运行时需要更少的主动监督。
## 为什么有效
Karpathy 指出，大型语言模型（LLM）非常擅长从漫长且杂乱的语音会话中重构意图。一段漫谈包含了许多关于目标、约束、重要示例以及说话者不确定性的微弱信号。模型可以将这些信号组织成对请求更清晰的表示。
我将这一想法扩展到了更多的模态。语音会话提供推理，而屏幕、文本、标注、转录和图像提供额外的证据。当一个通道有噪音或不完整时，另一个通道可以帮助消除歧义。
复杂的智能体任务通常失败于“我意指的”、“我明确说出的”和“智能体能观察到的”这些边界之间。多模态提示词为模型提供了更多机会，让其在开始工作之前弥合这些差距。
## 成本与回报
这种方法看起来可能有些大材小用，有时确实如此。一个简单的请求仍然值得用一个简单的提示词。当工作是长期的、当精度很重要、当智能体需要导航不熟悉的界面，或者当一个错误会导致多轮修正时，我会使用更丰富的任务。
多模态任务也可能成本更高，因为它包含更多的上下文。根据我的经验，这种投资通常是物有所值的。我每轮可以完成更大的工作单元，因为智能体从一开始就拥有它所需的更多上下文。
这对于浏览器使用和计算机使用特别有用。智能体可以看到环境，听到请求背后的推理，跟随识别重要元素的标注，并使用文本获取确切的细节。这种组合帮助智能体在不熟悉的界面中导航。
我目前的一些例子包括在不熟悉的平台上安排帖子、改进写作和编辑，以及优化工件和网页的设计。这些任务涉及许多微小的决策，将其编码为传统提示词既繁琐又困难，但在展示工作的同时进行沟通则很容易。在设计优化任务中，模态的映射非常自然。语音解释了布局哪里感觉不对，以及变更应该保留什么。屏幕显示工件的当前状态。标注标记了需要调整的具体间距、组件或部分。文本提供确切的文案和应保持不变的约束。
## 从追踪到技能
我存储这些任务的追踪记录，并审查其中的重复模式。
有用的模式通常包括操作序列、我重复的约束、我应用的质量检查，以及那些能始终改进结果的修正。这些模式可以被提取成可重用的技能，这样下一个智能体就能以更强大的工作流程开始。
这种与自动化的联系很重要。一个任务给了我一个实用的单元，我可以检查、改进它，并最终将其放入更大的循环中。更丰富的初始追踪帮助我理解哪些部分可以可靠地自动化，以及人类的指导在何处仍然能增加价值。
这个过程通常从手动任务开始。重复使用产生追踪，追踪揭示模式，模式变成可重用的技能。随着时间的推移，工作流程需要更少的解释，因为重要的指导已经被捕获了。
如果你想了解更多，我将在这里通过我们的学院进行演示、分享和撰写更多相关内容：https://academy.dair.ai/
## 迈向全能模型
全能模型（Omnimodels），即原生设计用于消费语音、视觉、图像和文本的模型，应该会让这种交互风格感觉自然。我们将能够在同一个会话中说话、展示、指点、打字并提供示例，而模型将直接整合这些信号。
我觉得我现在正在为这种交互做排练。目前的工具已经使其成为可能，即使这种体验在语音、浏览器状态、图像和文本之间仍然感觉像是拼凑起来的。
“任务”这个术语是暂时的，但通过重复使用，其潜在想法已经变得清晰。给智能体更丰富的工作追踪，让它重构意图，存储发生的事情，并重用有效的模式。
这源于一个实际的需求。我想要更少的修正循环、更强的交接，以及更可靠的长期智能体工作流程。多模态提示词已经让我稳步朝着这个方向迈进，并且这已成为我向智能体移交实际工作的默认方式。
## 相关链接
- [elvis](https://x.com/omarsar0)
- [@omarsar0](https://x.com/omarsar0)
- [15K](https://x.com/omarsar0/status/2079982491578827214/analytics)
- [https://academy.dair.ai/](https://academy.dair.ai/)
- [升级到 Premium](https://x.com/i/premium_sign_up)
- [2026年7月23日 上午1:30](https://x.com/omarsar0/status/2079982491578827214)
- [1.57万次查看](https://x.com/omarsar0/status/2079982491578827214/analytics)
- [查看引用](https://x.com/omarsar0/status/2079982491578827214/quotes)
---
*导出时间: 2026/7/23 11:43:04* 
