如何对视频进行 Grep 检索

Amy Xiao 2026-04-24 21.8 KB 11K 150 𝕏 文章列表

文章探讨了如何让 LLM 理解和处理海量视频数据。作者指出单纯依赖上下文窗口并非最佳方案，因为成本高昂且无法进行确定性检索。借鉴编程中的 Unix 模式，文章提出构建“中间表示层”，提前提取并结构化对齐转录、视觉、情感等多模态数据，从而实现对视频内容类似 Grep 的高效搜索与组合分析。

多模态视频分析 Agent 设计模式 RAG 数据结构上下文工程非结构化数据

# How to grep video **作者**: Amy Xiao **日期**: 2026-04-23T16:04:17.000Z **来源**: [https://x.com/amyxst/status/2047345827128164860](https://x.com/amyxst/status/2047345827128164860) --- ![Image](/doc/1053/image/img_15bb162d8448.jpg) Notes on context engineering and agent harnesses for video libraries: designing the structured representations to make media legible to LLMs. Authored with @kdrwins Since we started joining meetings from our computers, video has become the default way that organizations capture what happens at work. We’re at the point now where recording things has become so easy that we’ve created a new problem: we have more videos than we can ever watch. Part of the reason LLMs were so transformative was because they gave us a way to synthesize text right when we were drowning in documents. Now we’re at that same point with video and audio, but the synthesis layer hasn’t arrived yet. If you work with AI agents for coding, you may have noticed that the good ones are beginning to converge on the Unix design pattern. Read files → grep for patterns → edit in place → use the shell as your orchestrator. Each tool does one thing and does it well. Claude Code does this. LlamaIndex is building on it. The pattern is powerful because text has a useful property: text is its own searchable representation. A Python file is simultaneously the program being run and the document being searched. A markdown file is simultaneously the content and the index. This property has shaped much of the current thinking around agent design: context windows, tool use, and file-based memory assume that all the material an agent needs can be read and searched in its native form. To be fair, this is true for a lot of the information we work with everyday, like code, logs, config files, and documents. But it’s not true for video, images, or similar non-textual data. An MP4 is a binary container of compressed frames and audio packets. There is no grep that finds a concept inside it. So if filesystem idioms don’t translate to video, what replaces them? ## The context window isn't the answer Context windows are getting huge. Gemini now offers a 1M token context window that accepts video natively. It seems plausible that, with a big enough context window, you’d be able to work with video the same way you work with text. But when you try, you bump up against some limitations pretty quickly. Gemini tokenizes video at roughly 300 tokens per second at high resolution, or about 100 at low resolution. A 1M token window fits a small batch of short clips (Gemini caps requests at 10 videos) or a single meeting recording. That's useful for one-off tasks working with individual recordings. But it's not a general solution for working across a library. Imagine your sales team did 120 calls last quarter, each about 35 minutes long. That's roughly 70 hours of footage, so somewhere around 25 million tokens at low-res tokenization. Twenty-five times more than fits in a single context window. The query a sales leader actually wants to run looks something like: “Across all 120 Q3 calls, find every moment where the prospect pushed back on pricing. For each, return the call ID, what the prospect said, the timestamp, who was on the call, and how the rep responded.” You can't fit the data in to ask the question. Suppose you could fit a whole video library, though. You’d still have two problems: - You can't search inside a context window. You can only prompt over it, which is not quite the same thing. "Find every mention of pricing across these 20 recordings" is a retrieval problem. You want, targeted results over a known dataset. What you get instead is a best guess re-derived from frames on every run, with no guarantee the next run will agree with the last. That's not good enough when completeness matters. - The economics are strange. If the answer to "who was the speaker at 14:32?" lives in a 30-second segment, the structured representation that captures the speaker's name and timestamp might be 100 tokens. Without that structured representation you'd send 50,000 tokens of video context to answer the same question (500× the cost). Understanding the video is still a model call (a one-time index doesn't make it deterministic) but you pay that cost up front at extraction time instead of on every query that comes in afterward. The probabilistic cost and the token cost only get paid once and stay paid. Context windows will keep growing. But unless the economics radically change, you’ll need to structure and index videos ahead of time to know where to look and avoid starting from scratch every time. ## Every video is many videos When we extract data from video today, we typically just pull the transcript, which does give us a lot to work with. But when you introduce the possibility of querying all types of data within a video, a 30-minute recording turns out to be deceptively dense with information: - Ask for a meeting summary and you get speaker timelines, discussion topics, action items, key decisions. Normal stuff. - But ask for sales coaching from the exact same recording and you get something entirely different: talk-to-listen ratios, filler word frequency, how objections were handled, energy and pace through the call. - Ask for customer sentiment and you’ll get new information again: an analysis of tone shifts when the conversation turned to pricing, facial expressions during the demo, and the beat of silence before an answer. The signs that actually tell you whether a deal is alive usually don't make it into the transcript. Same video. Same binary blob on disk. Completely different knowledge, determined entirely by what you asked for. Underneath any of these questions, the same baseline gets cached once per segment: Prompt-specific extractions layer on top. Earlier in the same call, here's what a coaching query adds: And here's what a sentiment query adds for the segment we started with: The transcript said the prospect agreed. The prosody and facial data disagree. No single modality catches this on its own; aligned on a shared timeline, they do. The original sales-leader query becomes a trivial filter over the coaching schema, joined to the sentiment schema. No need to rewatch or re-prompt the raw video. In video, there's the speech transcript, obviously, but also speaker identity, visual scene descriptions like on-screen text, slides and product walkthroughs, entity references, chapter boundaries, laughter, cross-talk, camera shot boundaries, tone shifts, and more. Each of these is a different representation of the same source material. None of them is "the" representation. All of them are lossy. And all of them are useful, for different tasks and at different times. The job then becomes deciding which projections to compute, how to align them temporally, and how to make the results composable and searchable. This is context engineering for video libraries: designing structured intermediate representations that make opaque media legible to an LLM. We studied this question in our research on aligned video captions: time-aligned compact representations that combine signals across modalities (speech, visual, textual) consistently outperform both single-modality approaches and context window stuffing, across 1.8 million clips. The representation layer is what gives an agent something like grep-level access to video. It does this by creating the searchable object that the original format never was. ## Making grep work for video To create a grep-like pattern for video, you need to run the process in reverse. ![Image](/doc/1053/image/img_2ccbc62eb250.jpg) For text, you search first and read second. grep finds the line and cat shows the file. For video, the searchable form doesn't exist yet. So you go the other direction: describe first, cache the result, then search. This means a video agent needs three tiers of lookup: - Local keyword search over cached enrichments. Instant and free. This is the equivalent of grep over a file you've already written. Against the Q3 collection, a query like objections.topic = "pricing" returns every relevant moment across all 120 calls, with citations. - Semantic search across an indexed collection. A question like "find moments where the prospect hesitated before answering" can't be answered by keyword matching. It needs embeddings that map a text query to the relevant spans of time across your enriched representations, matching on meaning rather than words. - Full re-analysis: Send the video back through the enrichment pipeline. Expensive, and a last resort for when existing cached representations don't contain what you need. The enrichment cache is the key primitive. It needs prompt-aware keys. The same video processed with different extraction schemas produces different cache entries that coexist. And it needs multiple output fidelities: a compact representation (~1–4KB per video) for routing and quick lookups, and a full structured output (50K+ tokens) for deep analysis. This architecture also gets cheaper over time. Each interaction enriches the cache, which makes the next query faster and less expensive. It solves the inefficiency problem of the context window approach, where every question starts from scratch, with the full cost of the raw media. ## The other 90% of our information Translating every type of information encoded in a video into structured data LLMs can understand opens up a class of work transcripts can’t support. Pulling the transcripts from a sales call helps an account executive review the topics and action items they covered with a prospect. But with video context engineering, a sales leader working with the same Q3 collection could analyze a quarter’s worth of calls and generate a coaching plan for their team. They could query for and extract talk-to-listen ratios, filler-word frequency, objections and their responses, energy levels, pricing negotiations, and phrases that progressed deals. This is just one example. As with LLMs, I think we’ll discover hundreds more as we get accustomed to doing work across huge libraries of video. We’re building in this space at Cloudglue now. I’ll share more soon on what video context engineering makes possible. If you want to read this on the Cloudglue blog, and to follow for more technical pieces from our team: https://blog.cloudglue.dev/how-to-grep-video/ ## 相关链接 - [Amy Xiao](https://x.com/amyxst) - [@amyxst](https://x.com/amyxst) - [6.2K](https://x.com/amyxst/status/2047345827128164860/analytics) - [@kdrwins](https://x.com/@kdrwins) - [is building on it](https://www.llamaindex.ai/blog/files-are-all-you-need) - [in our research on aligned video captions](https://dl.acm.org/doi/10.1145/3701716.3716890) - [Cloudglue](https://cloudglue.dev/) - [https://blog.cloudglue.dev/how-to-grep-video/](https://blog.cloudglue.dev/how-to-grep-video/) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [12:04 AM · Apr 24, 2026](https://x.com/amyxst/status/2047345827128164860) - [6,275 Views](https://x.com/amyxst/status/2047345827128164860/analytics) --- *导出时间: 2026/4/24 17:52:11* --- ## 中文翻译 # 如何 grep 视频 **作者**： Amy Xiao **日期**： 2026-04-23T16:04:17.000Z **来源**： [https://x.com/amyxst/status/2047345827128164860](https://x.com/amyxst/status/2047345827128164860) --- ![Image](/doc/1053/image/img_15bb162d8448.jpg) 关于视频库的上下文工程与智能体框架的笔记：设计结构化表示，让媒体内容对大语言模型变得可读。与 @kdrwins 合著自从我们开始在电脑上参加会议以来，视频已成为组织记录工作情况默认方式。如今录制变得如此容易，以至于我们制造了一个新问题：我们的视频多到永远看不完。 LLM 之所以具有如此大的变革性，部分原因在于，当我们在文档的海洋中溺亡时，它们给了我们一种综合文本的方法。现在我们在视频和音频领域也到了同样的境地，但综合层还没有到来。如果你使用 AI 智能体进行编程，你可能已经注意到，优秀的智能体正在向 Unix 设计模式靠拢。读取文件 → grep 查找模式 → 原地编辑 → 用 shell 作为编排器。每个工具只做一件事，并把它做好。Claude Code 就是这样做的。LlamaIndex 也在基于此构建。这种模式之所以强大，是因为文本有一个有用的特性：文本本身就是它自己的可搜索表示。一个 Python 文件既是正在运行的程序，又是正在被搜索的文档。一个 markdown 文件既是内容，又是索引。这一特性塑造了当前智能体设计的许多思路：上下文窗口、工具使用和基于文件的记忆，都假设智能体所需的所有材料都能以其原生形式被读取和搜索。平心而论，对于我们日常处理的很多信息来说确实如此，比如代码、日志、配置文件和文档。但对于视频、图像或类似的非文本数据，情况并非如此。一个 MP4 是压缩帧和音频数据包的二进制容器。没有任何 grep 能在其中找到某个概念。那么，如果文件系统的惯用手法无法适用于视频，用什么来替代它们呢？ ## 上下文窗口不是答案上下文窗口正在变得巨大。Gemini 现在提供 1M token 的上下文窗口，可以原生接受视频。看起来似乎只要上下文窗口足够大，你就能像处理文本一样处理视频。但当你真正尝试时，很快就会碰到一些限制。Gemini 以高分辨率处理视频时，每秒大约消耗 300 个 token，低分辨率约为 100 个。一个 1M token 的窗口只能容纳一小批短片段（Gemini 将每次请求限制为 10 个视频）或单场会议录像。这对于处理单个录像的一次性任务是有用的。但它并不是跨整个视频库工作的通用解决方案。设想你的销售团队上个季度进行了 120 通电话，每通大约 35 分钟。那就是大约 70 小时的素材，按低分辨率 token 化计算，约 2500 万个 token。是单个上下文窗口容量的 25 倍。销售负责人真正想运行的查询大概是这样的： “在这 120 通 Q3 电话中，找出客户对定价提出异议的每一个时刻。对每一处，返回通话 ID、客户说了什么、时间戳、通话中的人员，以及销售代表如何回应。” 你无法把数据塞进去来提出这个问题。不过，假设你能装下整个视频库，你仍然会遇到两个问题： - 你无法在上下文窗口内部进行搜索。你只能在其之上进行提示，这并不完全是一回事。“在这 20 段录像中找出所有提到定价的地方”是一个检索问题。你想要的是针对已知数据集的定向结果。而你得到的却是每次运行都从帧中重新推导出的最佳猜测，且无法保证下一次运行与上一次一致。当完整性很重要时，这还不够好。 - 经济账很奇怪。如果“14:32 时的发言人是谁？”这个答案存在于一个 30 秒的片段中，那么捕捉发言人姓名和时间戳的结构化表示可能只需 100 个 token。没有这个结构化表示，你就得发送 50,000 个 token 的视频上下文来回答同一个问题（成本是 500 倍）。理解视频仍然需要模型调用（一次性的索引并不能使它变为确定性），但你在提取时预先支付了这笔成本，而不是在之后的每一次查询中支付。概率性成本和 token 成本只需支付一次，就已付清。上下文窗口会继续增长。但除非经济模式发生根本性改变，否则你仍需要提前对视频进行结构化和索引，以便知道去哪里查找，避免每次都从零开始。 ## 每个视频都是多个视频如今我们提取视频数据时，通常只是拉取文字转录稿，这确实给我们提供了不少可用信息。但当你引入查询视频内所有类型数据的可能性时，一段 30 分钟的录像所蕴含的信息密度远超想象： - 请求会议摘要，你会得到发言时间线、讨论话题、行动项、关键决策。常规内容。 - 但从同一段录像请求销售辅导，你会得到完全不同的东西：说与听的比例、口头禅的频率、异议处理方式、通话中的精力和节奏。 - 请求客户情绪分析，你又会得到新的信息：当话题转向定价时语气变化的分析、演示期间的面部表情，以及回答前那一拍沉默。真正能告诉你一笔交易是否还有希望的那些信号，通常不会出现在转录稿里。同一段视频。磁盘上同一个二进制文件。却有着完全不同的知识，完全由你的请求内容决定。在所有这些问题之下，同一套基线数据每个片段只缓存一次：针对特定提示的提取则在其上叠加。在同一次通话中较早的部分，以下是辅导查询额外添加的内容：而以下是我们最开始那个片段的情感查询额外添加的内容：文字记录显示潜在客户表示同意。韵律和面部数据却表明并非如此。任何单一模态都无法独自捕捉到这一点；但当它们对齐在同一时间线上时就可以。销售主管最初的查询变成了对教练模式（coaching schema）的一个简单过滤，并与情感模式（sentiment schema）进行联接。无需重新观看或重新提示原始视频。在视频方面，显然有语音文字记录，还有说话人身份、屏幕文字等视觉场景描述、幻灯片和产品演示、实体引用、章节边界、笑声、交叉对话、镜头切换边界、语气变化等等。这些中的每一项都是同一源素材的不同表示。没有哪一种是“唯一”的表示。它们都是有损的。而且它们都是有用的，只是适用于不同的任务和不同的时间。接下来的工作就变成了决定要计算哪些投影、如何按时间对齐它们，以及如何使结果可组合、可搜索。这就是面向视频库的上下文工程：设计结构化的中间表示，让不透明的媒体对 LLM 变得可读。我们在关于对齐视频字幕的研究中研究了这个问题：时间对齐的紧凑表示融合了跨模态（语音、视觉、文本）的信号，在 180 万个片段上持续优于单一模态方法和上下文窗口堆塞。表示层是让智能体获得类似 grep 级别视频访问能力的关键。它通过创建原始格式从未具备的可搜索对象来实现这一点。 ## 让 grep 适用于视频要为视频创建类似 grep 的模式，你需要反向运行这一过程。 ![Image](/doc/1053/image/img_2ccbc62eb250.jpg) 对于文本，先搜索后阅读。grep 找到行，cat 显示文件。对于视频，可搜索的形式尚不存在。所以你要反其道而行之：先描述，缓存结果，然后搜索。这意味着视频智能体需要三层查找机制： - 对缓存富化内容的本地关键词搜索。即时且免费。这相当于对已写好的文件进行 grep。针对 Q3 合集，像 objections.topic = "pricing" 这样的查询会返回全部 120 通通话中所有相关时刻，并附带引用。 - 跨索引集合的语义搜索。像“找出潜在客户在回答前犹豫不决的时刻”这样的问题无法通过关键词匹配来回答。它需要嵌入（embedding）将文本查询映射到富化表示中相关的时间跨度，按含义而非字词进行匹配。 - 完整的重新分析：将视频重新送回富化流水线。成本高昂，是现有缓存表示不包含所需内容时的最后手段。富化缓存是关键原语。它需要具备提示词感知的键。同一视频用不同的提取模式处理会产生共存的不同缓存条目。它还需要多个输出保真度级别：用于路由和快速查找的紧凑表示（每个视频约 1–4KB），以及用于深度分析的完整结构化输出（50K+ token）。这种架构随着时间推移还会变得更便宜。每次交互都会丰富缓存，使下一次查询更快、成本更低。它解决了上下文窗口方法的低效问题——在那里每个问题都从零开始，并承担原始媒体的全部成本。 ## 我们信息中另外的 90% 将视频中编码的每一类信息转换为 LLM 可以理解的结构化数据，开启了一类文字记录无法支持的工作。从销售通话中提取文字记录可以帮助客户经理回顾他们与潜在客户讨论的话题和行动项。但借助视频上下文工程，一位面对同一 Q3 合集的销售主管可以分析一个季度的通话记录，并为团队生成教练辅导计划。他们可以查询并提取说话与倾听比例、口头禅频率、异议及其应对、能量水平、价格谈判，以及推动成交的话语。这只是其中一个例子。正如 LLM 一样，我认为当我们习惯于在庞大的视频库上开展工作时，还会发现数百种更多用途。我们现在正在 Cloudglue 于这个领域进行建设。我很快会分享更多关于视频上下文工程所能实现的可能。如果你想阅读 Cloudglue 博客上的这篇文章，并关注我们团队的更多技术文章：https://blog.cloudglue.dev/how-to-grep-video/ ## 相关链接 - [Amy Xiao](https://x.com/amyxst) - [@amyxst](https://x.com/amyxst) - [6.2K](https://x.com/amyxst/status/2047345827128164860/analytics) - [@kdrwins](https://x.com/@kdrwins) - [is building on it](https://www.llamaindex.ai/blog/files-are-all-you-need) - [in our research on aligned video captions](https://dl.acm.org/doi/10.1145/3701716.3716890) - [Cloudglue](https://cloudglue.dev/) - [https://blog.cloudglue.dev/how-to-grep-video/](https://blog.cloudglue.dev/how-to-grep-video/) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [12:04 AM · Apr 24, 2026](https://x.com/amyxst/status/2047345827128164860) - [6,275 Views](https://x.com/amyxst/status/2047345827128164860/analytics) --- *导出时间: 2026/4/24 17:52:11*