How we built our knowledge base ✍ Cerebras🕐 2026-07-17📦 236.3 KB 🟢 已读 𝕏 文章列表 Cerebras 介绍了其内部知识库的构建过程，该知识库每天处理超过15,000个问题。系统通过从Slack、GitHub等平台提取数据，利用Postgres存储嵌入和元数据，结合全文搜索、向量搜索、IDF和衰减时间等混合搜索技术，实现了高效的信息检索。 知识库向量搜索SlackPostgres数据 ingestion混合搜索嵌入式实时更新WebSocket信息检索 # How we built our knowledge base **作者**: Cerebras **日期**: 2026-07-16T18:27:55.000Z **来源**: [https://x.com/cerebras/status/2077822555159945507](https://x.com/cerebras/status/2077822555159945507) --- Authors: @hi_im_isaac_, @learnwdaniel, @gaozenghao note: the interactive version of full technical blog available: https://www.cerebras.ai/blog/how-we-built-our-knowledge-base Employees ask our internal knowledge base more than 15,000 questions every day. It's become one of the most widely adopted internal tools at the company since launching 3 months ago. Used by humans, automations and agents. At Cerebras, our teams work across data center operations, chip design, hardware, training, inference, cloud platform, and more. With hundreds of new employees joining every year, our communication channels were filling up with the same questions: - “Where can I find X?” - “Who is the expert in Y?” - “What is Z?” We built Cerebras Knowledge to help people connect people and systems to useful information. # Meeting data where it lives Finding information inside an organization is hard. The data is scattered across tools, and every quarter or so someone proposes the same brilliant fix: let’s record everything in one platform so that all information is in a single place. The dream of a single source of truth, of course, rarely works in practice. Information is generated wherever it is convenient and ergonomic: suggested edits in a document, threads in Slack, code references in GitHub, and status metadata in Jira. These platforms are tailor-made for their specific domains, optimized through years of product engineering and analytics. Discussing a pull request in Google Docs would be a terrible experience. So we set out to design a system that required minimal change to existing behavior. On the data collection side, this meant extracting data from each platform directly. ## Anatomy of a knowledge base Our knowledge base provides three things: - A platform for collecting and storing internal data. - A platform for querying that data. - A layer that enforces authentication and authorization, with auditing and analytics. At the core is a single Postgres table that holds embeddings, raw summaries, and metadata from many sources. The system continually ingests data from across the company and maintains a query-ready datastore. We wanted a data interface that was simple but could work with most forms of data. We also wanted other developers at Cerebras to be able to build custom connectors. The result is deliberately simple: every source, from Slack threads to netlists, lands in the same embeddings table, and anything in that table is immediately queryable through the same interface: Each data source defines what the data is, how to connect to it, and how often it should be fetched. Each resulting embedding row follows the same interface regardless of whether it came from Slack, a code repository, a document system, or a custom database. # How we process unstructured Slack conversations Slack was the most important data source we needed to design for. It is where the most up-to-date engineering discussions happen across the company. We initially tested whether simple embeddings over raw text performed well enough. We quickly realized that vector search alone was insufficient for matching all relevant data. Slack messages present several challenges: - Information density varies enormously: “hey yeah sure mike” and a detailed kernel explanation are both messages. - Message lengths vary, and shorter messages frequently beat longer, more detailed messages in cosine similarity. - The meaning of a message often depends on the surrounding conversation. We needed a hybrid approach. We built Slack ingestion so every thread is retrievable through several search techniques at once, where each technique makes up for the weaknesses of the others: - Full-text search catches the exact tokens that embeddings blur together: error strings, flag names, host names. When an engineer pastes a literal error message, an exact lexical match is almost always the best evidence, and no amount of semantic similarity should outrank it. - Embedding search catches paraphrase. The person asking “restore hangs after manifest load” and the person who answered “checkpoint stalls on the NFS mount” may never share vocabulary. Vector similarity is what connects a question to an answer written in different words.(1) - Inverse document frequency separates signal from filler. A short message built around rare tokens, such as an obscure config flag, deserves to rank. “sounds good, thanks!” sits close to many queries in embedding space but scores near zero once term rarity is taken into account. - Age decay encodes that Slack answers expire. Two threads can answer the same question, and the one from six months ago may describe infrastructure that no longer exists. When relevance is otherwise equal, the newer thread wins. No single scorer is trusted on its own. Each technique produces its own ranked view of the same corpus, and those views are fused at query time (see Reranking). ## Socket Mode To collect data in real time, we installed a Slack bot into our workspace and ran it in Socket Mode. Slack pushes every message event to us over a persistent WebSocket, so we get real-time updates without polling the Web API and burning through its rate limits. When an event arrives, we immediately acknowledge it, deduplicate it using the stable event ID, and mark the message for the ingest consumer. The ingest consumer does not save a new message in isolation. It resolves the thread that the message belongs to and re-fetches the entire conversation, including the parent and every reply, from the Slack API. It then writes the whole thread back as one row. A reply to an existing thread therefore re-pulls the parent and all siblings, so the stored content, participant list, and last-activity timestamp always reflect the complete conversation. Every Slack channel in our system has its own data source. This provides fine-grained tuning for data freshness. A team may choose to ingest a busy incident channel more frequently, for example. ## Threads and messages Raw Slack text is keyword-searchable as soon as it lands because we maintain a Postgres full-text (GIN) index over the raw content. To enable useful vector search, however, we do some additional processing.(8) During distillation, an LLM extracts structured data from the full thread: - A one-line question that an engineer would actually search for. - A short summary. - The resolution. - The systems and code references mentioned. We embed these data points and write them into the shared embeddings table. The original transcript is not embedded directly. In our experiments, accuracy increased significantly when the thread was normalized into a consistent format.(7,9) The additional metadata also gives the semantic match more useful signal. ## Bursting At this point Slack search was good, but we kept encountering the same problem: important messages inside long threads were not always represented in the thread-level summary. To boost the signal from individual messages, we use bursting. A burst is a run of consecutive messages from the same author. We embed individual bursts with the thread topic prepended as context(2) because sometimes the answer lives in one tangent message whose vocabulary never makes it into the thread summary. Burst embeddings make that message findable on its own. To prevent low-signal data from reaching the database, each burst is scored against a weighted combination of signals and must clear a threshold before it is embedded: - It contains a relatively rare token across the corpus, with IDF of at least 4.0. - The combined burst is at least 200 characters. - One or more messages in the burst contain reactions, providing a social boost. After distillation, qualifying bursts are embedded and stored in the embeddings table alongside the thread-level record. # Code repositories Initially we debated whether embedding code repositories was necessary. With the rise of Claude Code and other command-line tools, creating code embeddings felt counterintuitive when it seemed like “grep is all you need.” After talking with others in the industry and reading Cursor’s findings on semantic search in large codebases, we decided to try. We have many internal repositories, some larger than 40 GB. Our main concern was how to keep them current efficiently. ## Using @cocoindex_io to maintain code embeddings After several experiments, we landed on CocoIndex, an open-source document embedding framework that specializes in vectorizing codebases. For each repository, we split the code using language-specific regex boundaries ordered from coarse to fine. The splitter tries higher-level boundaries, such as classes, first. If a resulting chunk is still too large, it falls back to method boundaries and then smaller blocks. We embed the resulting chunks and write the vectors to Postgres. A single file may generate multiple embeddings at different levels of specificity, such as file-level and function-level records. CocoIndex tracks synchronization metadata in Postgres. On each commit, it re-embeds and re-exports only the changed code chunks instead of recomputing the whole repository. This worked especially well for us because the synchronization state and embedding store live in the same database. As the number of codebases grew, we moved repository onboarding into configuration files that teams can submit themselves, including allowlists and denylists at the file-path level. ## Custom data sources Some teams already had their own databases and did not want to move data into Slack or a document system just to participate in the knowledge base. They wanted the same query surface over their existing tables. To support this, we treat custom sources as plugin scripts. A team opens a pull request with a small Python module that knows how to read from its system and emit rows shaped like our embeddings table, plus a matching data source entry. As long as the script writes into the shared database using the same schema as every other embedding row, the rest of the stack works unchanged. The data becomes queryable alongside Slack, code, and documents, with no special handling elsewhere in the system. ## Planning and tool fan-out For every query, we first run a short planning pass where an LLM decides which tools and data sources are likely to matter. The main tools: - subsystem_index: per-file LLM summaries. - search: the unified vector pipeline across Slack, wiki, code, and other indexed sources, merged and reranked internally. - search_slack: direct Slack retrieval. - search_code: ripgrep over source repositories. - recent_prs: recent pull requests relevant to the question. - who_knows: people with demonstrated expertise on a topic. The planner works over a compact description of what we have indexed: which projects exist, which sources are available in each project, and what each source is good at answering. Given the user’s query and active scope, it emits tool selections that the executor fans out in parallel, normalizes into a common evidence format, and passes to a final synthesis LLM.(4) ## Reranking A document can surface near the top simply because it shares vocabulary with the query while answering a different question. Before reranking, we combine the retrievers’ incompatible result lists with reciprocal rank fusion, or RRF. For every document, we add weight / (60 + rank) for each list in which it appears, with a default weight of 1.0 and a smoothing constant of 60. The smoothing constant makes consensus matter more than a single strong vote: a document that shows up near the top across several retrievers can beat one that ranks first in only one of them. We then merge duplicate chunks back to one source, cap how many results each file can contribute, and end up with a more diverse top twenty. We send the original query and those candidates to a small reranker model. It gives each document a score from zero to ten, and we keep the top ten.(6) Once the ranking is final, we add context back to the winners. For example, if we match a wiki section we pull in the two neighboring sections so the heading, preconditions, and caveats that chunking split apart aren’t lost. This gives readers a complete snippet instead of a lonely paragraph that’s missing important context. So the output of search is a rich packet of evidence: results fused from different retrievers, deduplicated at the source level, reranked against the actual question, and only then expanded with surrounding context. ## MCP In the MCP integration, we expose retrieval building blocks as direct tools instead of hiding them behind one “answer this question” endpoint. These tools are intentionally simple and as LLM-free as possible so clients can query them quickly and cheaply.(5) Each MCP tool corresponds to one underlying retrieval primitive, such as search_slack, search_code, search, or who_knows. Tool inputs and outputs are narrow, structured, and stable, making them easy to call from any client or agent without embedding additional orchestration logic inside the tool itself. Most tools run one query pipeline, such as vector search, lexical search, or ripgrep, apply lightweight scoring heuristics, and return raw evidence rows. Claude Code, or any MCP-compatible agent, becomes the orchestration engine. It decides which tools to call, in what order, and how to assemble the results into a final answer or code edit. The retrieval layer itself does not depend on those LLM decisions in order to serve requests. ## Web UI In the web UI, the same tools exist, but they are connected to a complete query pipeline that runs end to end for every user question. The UI agent owns the planner and executor steps. - Planner: A lightweight LLM pass inspects the query and active project, then chooses which retrieval tools to invoke, such as search, search_slack, and subsystem_index. - Executor: The system fans those tool calls out in parallel, gathers the results, and normalizes them into a shared evidence schema with scores, recency, and source hints. - Synthesis: A final LLM pass takes the typed evidence bundle and original question, then produces the answer shown in the UI, including citations, caveats, and cross-source synthesis. From the user’s perspective, the web UI is simply “ask a question and get an answer.” Under the hood, it runs the same planner → executor → synthesizer pattern that MCP clients can recreate explicitly. # Organization As the corpus grew, “search everything everywhere” rapidly stopped being useful. Engineers on compiler teams did not want infrastructure runbooks in their results, and vice versa. Projects are how we make search relevant by default. ## Projects and scoped search We introduced projects as the primary way to organize the workspace that a query runs over. A project is a named bundle of data sources: specific Slack channels, code repositories, internal databases, and document spaces relevant to a team or initiative. Projects are intentionally lightweight. The same data source, such as a shared incidents channel or central platform repository, can be referenced by multiple projects instead of being duplicated. ## Onboarding and defaults During onboarding, users are prompted to select or create a default project that matches how they work, such as ML training infrastructure, Compiler, or Data Center Operations. That default project is stored on the user profile and scopes queries automatically. A new engineer gets high-signal answers without first having to learn which Slack channels, repositories, or document spaces matter. # Final Thoughts In the end, the knowledge base works because it meets people where the information already lives, instead of forcing everything into one rigid system. By combining various search techniques, we can surface evidence quickly. The result is a search experience that stays flexible enough for real company data, but structured enough to remain useful as Cerebras keeps growing. If you read this far and this is interesting to you, the ai/growth team is hiring reach out if you are interested @learnwdaniel # References 1. Malkov and Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs, arXiv:1603.09320 / IEEE TPAMI 2018. 2. Anthropic, Introducing Contextual Retrieval, 2024. 3. Cormack, Clarke, and Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, SIGIR 2009. 4. Li et al., Search-o1: Agentic Search-Enhanced Large Reasoning Models, arXiv:2501.05366, 2025. 5. Anthropic, Code Execution with MCP, 2025. 6. Liu et al., Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172, 2023. 7. Anthropic, Use XML Tags. 8. Salesforce/Slack Engineering, How Slack AI Processes Billions of Messages. 9. Improving Agents, Best Nested Data Format. 10. Cursor, Improving Agent with Semantic Search, 2025. ## 相关链接 - [Cerebras](https://x.com/cerebras) - [@cerebras](https://x.com/cerebras) - [277K](https://x.com/cerebras/status/2077822555159945507/analytics) - [@hi_im_isaac_](https://x.com/@hi_im_isaac_) - [@learnwdaniel](https://x.com/@learnwdaniel) - [@gaozenghao](https://x.com/@gaozenghao) - [https://www.cerebras.ai/blog/how-we-built-our-knowledge-base](https://www.cerebras.ai/blog/how-we-built-our-knowledge-base) - [@cocoindex_io](https://x.com/@cocoindex_io) - [@learnwdaniel](https://x.com/@learnwdaniel) - [Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs](https://arxiv.org/abs/1603.09320) - [Introducing Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval) - [Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods](https://dl.acm.org/doi/10.1145/1571941.1572114) - [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://arxiv.org/abs/2501.05366) - [Code Execution with MCP](https://www.anthropic.com/engineering/code-execution-with-mcp) - [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172) - [Use XML Tags](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags) - [Improving Agent with Semantic Search](https://cursor.com/blog/semsearch) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [2:27 AM · Jul 17, 2026](https://x.com/cerebras/status/2077822555159945507) - [277.2K Views](https://x.com/cerebras/status/2077822555159945507/analytics) - [View quotes](https://x.com/cerebras/status/2077822555159945507/quotes) --- *导出时间: 2026/7/17 11:25:37* --- ## 中文翻译 # 我们如何构建我们的知识库 **作者**: Cerebras **日期**: 2026-07-16T18:27:55.000Z **来源**: [https://x.com/cerebras/status/2077822555159945507](https://x.com/cerebras/status/2077822555159945507) --- 作者：@hi_im_isaac_, @learnwdaniel, @gaozenghao 注：完整技术博客的交互版本可在以下链接获取：https://www.cerebras.ai/blog/how-we-built-our-knowledge-base 员工每天向我们的内部知识库提问超过 15,000 次。自三个月前推出以来，它已成为公司内部使用最广泛的工具之一。使用者包括人类、自动化系统和智能体。 在 Cerebras，我们的团队工作涉及数据中心运营、芯片设计、硬件、训练、推理、云平台等众多领域。随着每年有数百名新员工加入，我们的沟通渠道充斥着相同的问题： - “我在哪里可以找到 X？” - “谁是 Y 领域的专家？” - “Z 是什么？” 我们构建了 Cerebras Knowledge（Cerebras 知识库）来帮助人们、系统与有用信息建立连接。 # 在数据所在的场景中对接数据 在组织内部查找信息很难。数据分散在各种工具中，每隔一个季度左右，就会有人提出同样的绝妙修复方案：让我们把所有内容都记录在一个平台上，以便所有信息都集中在一个地方。当然，单一事实来源（Single Source of Truth）的梦想在实践中很少行得通。 信息是在任何方便和符合人体工学的地方生成的：文档中的建议编辑、Slack 中的讨论串、GitHub 中的代码引用以及 Jira 中的状态元数据。这些平台都是为其特定领域量身定制的，经过多年的产品工程和分析进行了优化。在 Google Docs 中讨论拉取请求（pull request）将是一场糟糕的体验。 因此，我们着手设计一个几乎不需要改变现有行为的系统。在数据收集方面，这意味着直接从每个平台提取数据。 ## 知识库的剖析 我们的知识库提供三样东西： - 用于收集和存储内部数据的平台。 - 用于查询该数据的平台。 - 强制执行身份验证和授权的层，并包含审计和分析功能。 核心是一张单一的 Postgres 表，其中保存了来自许多来源的嵌入（embeddings）、原始摘要和元数据。系统不断从整个公司摄取数据，并维护一个可供查询的数据存储。 我们想要一个简单但能适用于大多数形式数据的数据接口。我们还希望 Cerebras 的其他开发者能够构建自定义连接器。其结果故意设计得非常简单：每个来源，从 Slack 讨论串到网表（netlists），都落入同一个嵌入表中，该表中的任何内容都可以通过相同的接口立即进行查询： 每个数据源都定义了数据是什么、如何连接到它以及应该多久获取一次。每个生成的嵌入行都遵循相同的接口，无论它是来自 Slack、代码仓库、文档系统还是自定义数据库。 # 我们如何处理非结构化的 Slack 对话 Slack 是我们需要设计的最重要的数据源。这是全公司进行最新工程讨论的地方。 我们最初测试了针对原始文本的简单嵌入是否表现良好。我们很快意识到，仅靠向量搜索不足以匹配所有相关数据。 Slack 消息带来了几个挑战： - 信息密度差异巨大：“嘿，是的，没问题 Mike”和详细的内核解释都是消息。 - 消息长度不一，在余弦相似度计算中，较短的消息经常击败较长、更详细的消息。 - 消息的含义通常取决于周围的对话。 我们需要一种混合方法。我们构建了 Slack 摄取功能，使每个讨论串都可以通过多种搜索技术同时检索，其中每种技术都能弥补其他技术的不足： - 全文搜索捕捉嵌入模糊处理的精确标记：错误字符串、标志名称、主机名。当工程师粘贴字面错误消息时，精确的词汇匹配几乎总是最好的证据，任何语义相似度都不应凌驾于其上。 - 嵌入搜索捕捉改写（paraphrase）。提问“清单加载后恢复挂起”的人和回答“检查点在 NFS 挂载上停滞”的人可能从未共享过词汇。向量相似度是将问题与用不同词语撰写的的答案联系起来的关键。(1) - 逆文档频率（IDF）将信号与填充词区分开来。一条围绕罕见标记（例如晦涩的配置标志）构建的短消息值得排在前面。在嵌入空间中，“听起来不错，谢谢！”靠近许多查询，但一旦考虑了术语稀有度，其得分接近于零。 - 时间衰减编码了 Slack 答案会过期这一事实。两个讨论串可以回答同一个问题，但六个月前的那个可能描述了不再存在的基础设施。当其他相关性相同时，较新的讨论串获胜。 没有单一评分器是被独立信任的。每种技术都会对同一语料库产生自己的排名视图，这些视图在查询时被融合（参见重排序）。 ## Socket 模式 为了实时收集数据，我们在工作区中安装了一个 Slack 机器人，并以 Socket 模式运行它。Slack 通过持久的 WebSocket 将每条消息事件推送给我们要，因此我们可以获得实时更新，而无需轮询 Web API 并消耗其速率限制。 当事件到达时，我们立即确认它，使用稳定的事件 ID 对其进行去重，并将消息标记为供摄取消费者处理。 摄取消费者不会孤立地保存新消息。它会解析该消息所属的讨论串，并从 Slack API 重新获取包括父消息和每个回复在内的整个对话。然后，它将整个讨论串作为一行写回。因此，对现有讨论串的回复会重新拉取父消息和所有兄弟消息，以便存储的内容、参与者列表和最后活动时间戳始终反映完整的对话。 我们系统中的每个 Slack 频道都有自己的数据源。这为数据新鲜度提供了细粒度的调整。例如，团队可以选择更频繁地摄取繁忙的故障频道。 ## 讨论串和消息 由于我们在原始内容上维护了 Postgres 全文（GIN）索引，原始 Slack 文本在存入后即可进行关键字搜索。然而，为了实现有用的向量搜索，我们会进行一些额外的处理。(8) 在蒸馏（distillation）期间，LLM 从完整的讨论串中提取结构化数据： - 工程师实际会搜索的单行问题。 - 简短摘要。 - 解决方案。 - 提及的系统和代码引用。 我们嵌入这些数据点并将它们写入共享的嵌入表。原始逐字稿不是直接嵌入的。在我们的实验中，当讨论串被标准化为一致的格式时，准确性显著提高。(7,9) 额外的元数据也为语义匹配提供了更有用的信号。 ## 突发处理（Bursting） 此时，Slack 搜索已经做得很好，但我们一直遇到同样的问题：长讨论串中的重要消息并不总是反映在讨论串级别的摘要中。 为了提升来自单个消息的信号，我们使用了突发处理（Bursting）。突发是指来自同一作者的一系列连续消息。我们将单个突发与讨论串主题作为前缀上下文一起嵌入(2)，因为有时答案位于一条切题的消息中，其词汇从未出现在讨论串摘要中。突发嵌入使该消息本身能够被找到。 为了防止低信号数据进入数据库，每个突发都会根据信号的加权组合进行评分，并且必须清除阈值才能被嵌入： - 它在整个语料库中包含相对罕见的标记，IDF 至少为 4.0。 - 组合后的突发至少包含 200 个字符。 - 突发中的一条或多条消息包含反应（reactions），提供了社会信号提升。 蒸馏后，符合条件的突发被嵌入并存储在嵌入表中，与讨论串级别的记录并列。 # 代码仓库 最初，我们争论是否有必要嵌入代码仓库。随着 Claude Code 和其他命令行工具的兴起，创建代码嵌入感觉是反直觉的，因为似乎“grep 就是你所需要的”。在与行业内的其他人交谈并阅读了 Cursor 关于大型代码库中语义搜索的发现后，我们决定尝试一下。 我们有许多内部仓库，有些大于 40 GB。我们主要关心的是如何高效地保持它们最新。 ## 使用 @cocoindex_io 维护代码嵌入 经过几次实验，我们选择了 CocoIndex，这是一个专门用于向量化代码库的开源文档嵌入框架。 对于每个仓库，我们使用按从粗到细排序的语言特定正则边界来拆分代码。拆分器首先尝试更高级别的边界，例如类。如果生成的块仍然太大，则回退到方法边界，然后是更小的块。我们对生成的块进行嵌入并将向量写入 Postgres。单个文件可能会生成不同粒度级别的多个嵌入，例如文件级别和函数级别的记录。 CocoIndex 在 Postgres 中跟踪同步元数据。在每次提交时，它仅重新嵌入和重新导出更改的代码块，而不是重新计算整个仓库。这对我们特别有效，因为同步状态和嵌入存储位于同一个数据库中。 随着代码库数量的增长，我们将仓库入驻移入了配置文件，团队可以自行提交，包括文件路径级别的允许列表和拒绝列表。 ## 自定义数据源 有些团队已经拥有自己的数据库，并且不想为了参与知识库而将数据移动到 Slack 或文档系统中。他们希望在其现有表上拥有相同的查询表面。 为了支持这一点，我们将自定义源视为插件脚本。团队通过一个小的 Python 模块打开一个拉取请求，该模块知道如何从其系统读取数据并发出形状类似于我们嵌入表的行，外加一个匹配的数据源条目。 只要脚本使用与其他嵌入行相同的架构写入共享数据库，堆栈的其余部分就可以不变地工作。数据可以与 Slack、代码和文档一起查询，而无需在系统的其他地方进行特殊处理。 ## 规划和工具分发 对于每个查询，我们首先运行一个简短的规划步骤，其中 LLM 决定哪些工具和数据源可能重要。主要工具包括： - subsystem_index：每个文件的 LLM 摘要。 - search：跨 Slack、wiki、代码和其他索引源的统一向量管道，在内部合并和重排序。 - search_slack：直接 Slack 检索。 - search_code：源代码仓库上的 ripgrep。 - recent_prs：与问题相关的最近拉取请求。 - who_knows：在某个主题上具有公认专业知识的人员。 规划器基于我们所索引内容的紧凑描述进行工作：存在哪些项目、每个项目中可用哪些源，以及每个源擅长回答什么。给定用户的查询和活动范围，它会发出工具选择，执行器将这些选择并行分发，规范化为通用的证据格式，并传递给最终的合成 LLM。(4) ## 重排序 文档可能仅仅因为它与查询共享词汇但回答的是不同问题而出现在顶部附近。在重排序之前，我们使用倒数排名融合（RRF）结合检索器不兼容的结果列表。对于每个文档，我们为它出现的每个列表添加 weight / (60 + rank)，默认权重为 1.0，平滑常数为 60。 平滑常数使共识比单一强力投票更重要：在多个检索器中出现在顶部附近的文档可以击败仅在一个检索器中排名第一的文档。然后，我们将重复的块合并回一个源，限制每个文件可以贡献的结果数量，最终得到更多样化的前二十名结果。 我们将原始查询和这些候选文档发送给一个小型重排序模型。它给出每个文档从零到十的分数，我们保留前十名。(6) 一旦排名最终确定，我们将上下文添加回获胜者。例如，如果我们匹配了一个 wiki 部分，我们会拉入相邻的两个部分，以便分块拆分的标题、先决条件和警告不会丢失。这为读者提供了一个完整的片段，而不是缺少重要上下文的孤立段落。 因此，搜索的输出是一个丰富的证据包：结果来自不同的检索器融合，在源级别去重，根据实际问题重新排序，只有在那时才用周围的上下文进行扩展。 ## MCP 在 MCP 集成中，我们将检索构建块作为直接工具公开，而不是将它们隐藏在一个“回答这个问题”的端点后面。这些工具故意设计得简单，并尽可能不涉及 LLM，以便客户端可以快速且廉价地查询它们。(5) 每个 MCP 工具对应一个底层的检索原语，例如 search_slack、search_code、search 或 who_knows。工具输入和输出是狭窄的、结构化的和稳定的，使得任何客户端或智能体都可以轻松调用它们，而无需在工具本身内部嵌入额外的编排逻辑。 大多数工具运行一个查询管道，例如向量搜索、词汇搜索或 ripgrep，应用轻量级评分启发式算法，并返回原始证据行。 Claude Code 或任何兼容 MCP 的智能体都成为编排引擎。它决定调用哪些工具、以什么顺序调用，以及如何将结果组装成最终答案或代码编辑。检索层本身不依赖那些 LLM 决策来服务请求。 ## Web UI 在 Web UI 中，存在相同的工具，但它们连接到一个完整的查询管道，该管道为每个用户问题端到端运行。UI 智能体拥有规划器和执行器步骤。 - 规划器：一个轻量级的 LLM 传递，检查查询和活动项目，然后选择调用哪些检索工具，例如 s
H How we built our knowledge base Cerebras 分享了构建其内部知识库的经验。该知识库每天处理超过 15,000 个问题，集成了 Slack、GitHub 等多种数据源。文章详细介绍了如何通过 Postgres 存储 embeddings，并采用混合搜索策略（全文搜索、向量搜索、IDF 加权、时间衰减）来高效检索非结构化对话数据，解决了信息分散和匹配精度的问题。 技术 › DevOps ✍ Cerebras🕐 2026-07-18 知识库向量搜索EmbeddingsSlack混合搜索Postgres数据工程检索增强生成企业工具架构设计
H How we built our knowledge base Cerebras 分享了如何构建内部知识库，该系统每天处理超过 15,000 个查询。文章介绍了其数据采集架构，特别是如何通过混合搜索技术（全文搜索、Embedding 搜索、IDF 和时间衰减）来处理非结构化的 Slack 对话，以提高信息检索的准确性和实时性。 技术 › 后端 ✍ Cerebras🕐 2026-07-17 知识库SlackEmbedding检索搜索
C ChatGPT Agent Loop 优化技术解析 本文深入解析了 ChatGPT 如何通过 Harness、API 和 Inference 三层架构优化 Agent 循环，重点介绍了持久化 WebSocket、增量 Token 化、KV 缓存管理和推测解码等技术，以降低成本并提升效率。 技术 › Harness Engineering ✍ Bytebytego🕐 2026-07-30 Agent优化LLM架构ChatGPTOpenAI性能成本控制WebSocketTokenization
如 如何构建能在你睡觉时运行的公司大脑 本文介绍如何超越个人知识库，构建一个“公司大脑”。通过将文件系统与 AI 智能体结合，设定文件夹作用域、定时任务和自动汇报，让智能体自动处理营销、文件归档等工作，实现无需人工干预的全自动企业级知识管理。 技术 › Agent ✍ Hila Shmuel🕐 2026-07-29 AI知识库自动化工作流LLMCabinet
J Jack Dorsey's Buzz: Clearly Explained Buzz 是一款集成了智能体的团队聊天工具，允许用户切换底层模型并保留对话记忆，支持本地共享计算。文章探讨了其 Agents 团队协作、Git 工作流及数据控制优势，适合小团队快速迭代。 技术 › Agent ✍ The Startup Ideas Podcast (SIP)🕐 2026-07-29 BuzzJack DorseySlackAgent本地模型团队协作开源
如 如何在5分钟内构建你的公司大脑 文章介绍了如何利用Supermemory快速构建一个“公司大脑”，即基于知识库和Agent的综合系统。通过邀请机器人加入Slack、连接工具、自动化任务等步骤，企业可以拥有一个具备全公司知识的高级AI员工，用于销售、工程、研发等多种场景。 技术 › Agent ✍ Dhravya Shah🕐 2026-07-28 公司大脑SupermemoryAgentSlack自动化知识管理AI员工工具集成
一 一个暑假，我侄子用两个 Skill 搞明白了 Agent 最难的部分 作者讲述侄子利用暑假开发两个健康类 Agent Skill 的经历。从“养生谣言粉碎机”到“体检行动助手”，解决了大模型幻觉、信息检索权威性及流程设计等难点。文章重点介绍了如何利用豆包搜索 API 的结构化数据与权威过滤功能来增强 Agent 能力，并分享了 Skill 开发优于独立 Agent 的实践经验和建议。 技术 › Agent ✍ 宝玉🕐 2026-07-28 AgentSkill豆包搜索AI开发大模型实战经验信息检索健康类目工作流开源
用 用好 Obsidian Skill，让你的 WorkBuddy 对知识库的理解提升 80% 文章介绍了如何在 WorkBuddy 中安装 Obsidian skill，通过自动识别本地知识库、补充 CLI 能力及验证读写操作，实现对 Obsidian 知识库的高效管理。相比直接操作本地文件，使用 skill 能更好地理解双链结构和知识库逻辑。 技术 › Skill ✍ 阿蔺A-Lin🕐 2026-07-28 WorkBuddyObsidian知识库CLI教程工具笔记双链
如 如何利用AI构建和扩展单人企业 文章介绍了利用AI构建和扩展单人企业的完整蓝图。通过使用AI员工（如Viktor）在内容、项目、拓展、财务和广告五个领域实现自动化，只需保留决策环节。文章探讨了两种构建方式：快速路径和自定义构建，强调业务知识库和操作规则的重要性。 技术 › Agent ✍ Machina🕐 2026-07-26 AI单人企业自动化Agent生产力Viktor商业模式知识库LLM效率
用 用飞书多维表格打造你的数字分身 文章介绍了如何利用飞书多维表格结合 AI 技术打造知识库“数字分身”。通过三级进化：将知识结构化存库、利用 AI 字段自动化处理知识、接入飞书机器人实现对话交互，解决专业知识沉淀与复用难题，提升业务效率。 技术 › 工具与效率 ✍ 黄小木🕐 2026-07-24 飞书多维表格AI知识库数字分身自动化Agent低代码
构 构建改变生活的自托管 Agent 知识库 文章介绍了如何利用 Hermes Agent 构建一个集中的个人知识库，解决 AI 使用中上下文分散的问题。通过四个提示词，作者展示了如何建立结构化存储、自动更新归档技能、基于证据的检索机制以及大脑倾倒处理流程，从而让 Agent 全面了解用户并大幅提升生活质量。 技术 › Agent ✍ EP🕐 2026-07-24 AgentHermes知识库自托管个人助理
《 《指挥 AI，做出一个企业级 Agent》08：企业 RAG 不是上传几个 PDF 文章指出企业 RAG 不仅仅是上传 PDF 回答问题，更需关注文档版本、权限、来源追溯及停用管理。作者通过模拟企业真实文档环境，构建了包含版本控制和过滤机制的评测体系，强调了可靠检索与诚实无答案的重要性，并分享了针对小规模知识库的技术选型经验。 技术 › Agent ✍ jager🕐 2026-07-22 RAG企业级Agent版本控制向量检索评测Embedding知识库技术选型