How we built our knowledge base Cerebras 2026-07-17 261.6 KB ☆ 收藏 𝕏 文章列表 Cerebras 分享了如何构建内部知识库，该系统每天处理超过 15,000 个查询。文章介绍了其数据采集架构，特别是如何通过混合搜索技术（全文搜索、Embedding 搜索、IDF 和时间衰减）来处理非结构化的 Slack 对话，以提高信息检索的准确性和实时性。 知识库SlackEmbedding检索搜索 # How we built our knowledge base **作者**: Cerebras **日期**: 2026-07-16T18:27:55.000Z **来源**: [https://x.com/cerebras/status/2077822555159945507](https://x.com/cerebras/status/2077822555159945507) ---  Authors: @hi_im_isaac_, @learnwdaniel, @gaozenghao note: the interactive version of full technical blog available: https://www.cerebras.ai/blog/how-we-built-our-knowledge-base Employees ask our internal knowledge base more than 15,000 questions every day. It's become one of the most widely adopted internal tools at the company since launching 3 months ago. Used by humans, automations and agents. At Cerebras, our teams work across data center operations, chip design, hardware, training, inference, cloud platform, and more. With hundreds of new employees joining every year, our communication channels were filling up with the same questions: - “Where can I find X?” - “Who is the expert in Y?” - “What is Z?”  We built Cerebras Knowledge to help people connect people and systems to useful information. # Meeting data where it lives Finding information inside an organization is hard. The data is scattered across tools, and every quarter or so someone proposes the same brilliant fix: let’s record everything in one platform so that all information is in a single place. The dream of a single source of truth, of course, rarely works in practice. Information is generated wherever it is convenient and ergonomic: suggested edits in a document, threads in Slack, code references in GitHub, and status metadata in Jira. These platforms are tailor-made for their specific domains, optimized through years of product engineering and analytics. Discussing a pull request in Google Docs would be a terrible experience. So we set out to design a system that required minimal change to existing behavior. On the data collection side, this meant extracting data from each platform directly. ## Anatomy of a knowledge base Our knowledge base provides three things: - A platform for collecting and storing internal data. - A platform for querying that data. - A layer that enforces authentication and authorization, with auditing and analytics. At the core is a single Postgres table that holds embeddings, raw summaries, and metadata from many sources. The system continually ingests data from across the company and maintains a query-ready datastore. We wanted a data interface that was simple but could work with most forms of data. We also wanted other developers at Cerebras to be able to build custom connectors. The result is deliberately simple: every source, from Slack threads to netlists, lands in the same embeddings table, and anything in that table is immediately queryable through the same interface:  Each data source defines what the data is, how to connect to it, and how often it should be fetched. Each resulting embedding row follows the same interface regardless of whether it came from Slack, a code repository, a document system, or a custom database. # How we process unstructured Slack conversations Slack was the most important data source we needed to design for. It is where the most up-to-date engineering discussions happen across the company.  We initially tested whether simple embeddings over raw text performed well enough. We quickly realized that vector search alone was insufficient for matching all relevant data. Slack messages present several challenges: - Information density varies enormously: “hey yeah sure mike” and a detailed kernel explanation are both messages. - Message lengths vary, and shorter messages frequently beat longer, more detailed messages in cosine similarity. - The meaning of a message often depends on the surrounding conversation. We needed a hybrid approach. We built Slack ingestion so every thread is retrievable through several search techniques at once, where each technique makes up for the weaknesses of the others: - Full-text search catches the exact tokens that embeddings blur together: error strings, flag names, host names. When an engineer pastes a literal error message, an exact lexical match is almost always the best evidence, and no amount of semantic similarity should outrank it. - Embedding search catches paraphrase. The person asking “restore hangs after manifest load” and the person who answered “checkpoint stalls on the NFS mount” may never share vocabulary. Vector similarity is what connects a question to an answer written in different words.(1) - Inverse document frequency separates signal from filler. A short message built around rare tokens, such as an obscure config flag, deserves to rank. “sounds good, thanks!” sits close to many queries in embedding space but scores near zero once term rarity is taken into account. - Age decay encodes that Slack answers expire. Two threads can answer the same question, and the one from six months ago may describe infrastructure that no longer exists. When relevance is otherwise equal, the newer thread wins.  No single scorer is trusted on its own. Each technique produces its own ranked view of the same corpus, and those views are fused at query time (see Reranking). ## Socket Mode To collect data in real time, we installed a Slack bot into our workspace and ran it in Socket Mode. Slack pushes every message event to us over a persistent WebSocket, so we get real-time updates without polling the Web API and burning through its rate limits. When an event arrives, we immediately acknowledge it, deduplicate it using the stable event ID, and mark the message for the ingest consumer. The ingest consumer does not save a new message in isolation. It resolves the thread that the message belongs to and re-fetches the entire conversation, including the parent and every reply, from the Slack API. It then writes the whole thread back as one row. A reply to an existing thread therefore re-pulls the parent and all siblings, so the stored content, participant list, and last-activity timestamp always reflect the complete conversation. Every Slack channel in our system has its own data source. This provides fine-grained tuning for data freshness. A team may choose to ingest a busy incident channel more frequently, for example. ## Threads and messages Raw Slack text is keyword-searchable as soon as it lands because we maintain a Postgres full-text (GIN) index over the raw content. To enable useful vector search, however, we do some additional processing.(8) During distillation, an LLM extracts structured data from the full thread: - A one-line question that an engineer would actually search for. - A short summary. - The resolution. - The systems and code references mentioned.  We embed these data points and write them into the shared embeddings table. The original transcript is not embedded directly. In our experiments, accuracy increased significantly when the thread was normalized into a consistent format.(7,9) The additional metadata also gives the semantic match more useful signal. ## Bursting At this point Slack search was good, but we kept encountering the same problem: important messages inside long threads were not always represented in the thread-level summary. To boost the signal from individual messages, we use bursting. A burst is a run of consecutive messages from the same author. We embed individual bursts with the thread topic prepended as context(2) because sometimes the answer lives in one tangent message whose vocabulary never makes it into the thread summary. Burst embeddings make that message findable on its own. To prevent low-signal data from reaching the database, each burst is scored against a weighted combination of signals and must clear a threshold before it is embedded: - It contains a relatively rare token across the corpus, with IDF of at least 4.0. - The combined burst is at least 200 characters. - One or more messages in the burst contain reactions, providing a social boost.  After distillation, qualifying bursts are embedded and stored in the embeddings table alongside the thread-level record. # Code repositories Initially we debated whether embedding code repositories was necessary. With the rise of Claude Code and other command-line tools, creating code embeddings felt counterintuitive when it seemed like “grep is all you need.” After talking with others in the industry and reading Cursor’s findings on semantic search in large codebases, we decided to try. We have many internal repositories, some larger than 40 GB. Our main concern was how to keep them current efficiently. ## Using @cocoindex_io to maintain code embeddings After several experiments, we landed on CocoIndex, an open-source document embedding framework that specializes in vectorizing codebases. For each repository, we split the code using language-specific regex boundaries ordered from coarse to fine. The splitter tries higher-level boundaries, such as classes, first. If a resulting chunk is still too large, it falls back to method boundaries and then smaller blocks. We embed the resulting chunks and write the vectors to Postgres. A single file may generate multiple embeddings at different levels of specificity, such as file-level and function-level records.  CocoIndex tracks synchronization metadata in Postgres. On each commit, it re-embeds and re-exports only the changed code chunks instead of recomputing the whole repository. This worked especially well for us because the synchronization state and embedding store live in the same database. As the number of codebases grew, we moved repository onboarding into configuration files that teams can submit themselves, including allowlists and denylists at the file-path level. ## Custom data sources Some teams already had their own databases and did not want to move data into Slack or a document system just to participate in the knowledge base. They wanted the same query surface over their existing tables. To support this, we treat custom sources as plugin scripts. A team opens a pull request with a small Python module that knows how to read from its system and emit rows shaped like our embeddings table, plus a matching data source entry. As long as the script writes into the shared database using the same schema as every other embedding row, the rest of the stack works unchanged. The data becomes queryable alongside Slack, code, and documents, with no special handling elsewhere in the system. ## Planning and tool fan-out For every query, we first run a short planning pass where an LLM decides which tools and data sources are likely to matter. The main tools: - subsystem_index: per-file LLM summaries. - search: the unified vector pipeline across Slack, wiki, code, and other indexed sources, merged and reranked internally. - search_slack: direct Slack retrieval. - search_code: ripgrep over source repositories. - recent_prs: recent pull requests relevant to the question. - who_knows: people with demonstrated expertise on a topic. The planner works over a compact description of what we have indexed: which projects exist, which sources are available in each project, and what each source is good at answering. Given the user’s query and active scope, it emits tool selections that the executor fans out in parallel, normalizes into a common evidence format, and passes to a final synthesis LLM.(4)  ## Reranking A document can surface near the top simply because it shares vocabulary with the query while answering a different question. Before reranking, we combine the retrievers’ incompatible result lists with reciprocal rank fusion, or RRF. For every document, we add weight / (60 + rank) for each list in which it appears, with a default weight of 1.0 and a smoothing constant of 60.  The smoothing constant makes consensus matter more than a single strong vote: a document that shows up near the top across several retrievers can beat one that ranks first in only one of them. We then merge duplicate chunks back to one source, cap how many results each file can contribute, and end up with a more diverse top twenty. We send the original query and those candidates to a small reranker model. It gives each document a score from zero to ten, and we keep the top ten.(6) Once the ranking is final, we add context back to the winners. For example, if we match a wiki section we pull in the two neighboring sections so the heading, preconditions, and caveats that chunking split apart aren’t lost. This gives readers a complete snippet instead of a lonely paragraph that’s missing important context. So the output of search is a rich packet of evidence: results fused from different retrievers, deduplicated at the source level, reranked against the actual question, and only then expanded with surrounding context. ## MCP In the MCP integration, we expose retrieval building blocks as direct tools instead of hiding them behind one “answer this question” endpoint. These tools are intentionally simple and as LLM-free as possible so clients can query them quickly and cheaply.(5) Each MCP tool corresponds to one underlying retrieval primitive, such as search_slack, search_code, search, or who_knows. Tool inputs and outputs are narrow, structured, and stable, making them easy to call from any client or agent without embedding additional orchestration logic inside the tool itself. Most tools run one query pipeline, such as vector search, lexical search, or ripgrep, apply lightweight scoring heuristics, and return raw evidence rows. Claude Code, or any MCP-compatible agent, becomes the orchestration engine. It decides which tools to call, in what order, and how to assemble the results into a final answer or code edit. The retrieval layer itself does not depend on those LLM decisions in order to serve requests. ## Web UI In the web UI, the same tools exist, but they are connected to a complete query pipeline that runs end to end for every user question. The UI agent owns the planner and executor steps. - Planner: A lightweight LLM pass inspects the query and active project, then chooses which retrieval tools to invoke, such as search, search_slack, and subsystem_index. - Executor: The system fans those tool calls out in parallel, gathers the results, and normalizes them into a shared evidence schema with scores, recency, and source hints. - Synthesis: A final LLM pass takes the typed evidence bundle and original question, then produces the answer shown in the UI, including citations, caveats, and cross-source synthesis. From the user’s perspective, the web UI is simply “ask a question and get an answer.” Under the hood, it runs the same planner → executor → synthesizer pattern that MCP clients can recreate explicitly.  # Organization As the corpus grew, “search everything everywhere” rapidly stopped being useful. Engineers on compiler teams did not want infrastructure runbooks in their results, and vice versa. Projects are how we make search relevant by default. ## Projects and scoped search We introduced projects as the primary way to organize the workspace that a query runs over. A project is a named bundle of data sources: specific Slack channels, code repositories, internal databases, and document spaces relevant to a team or initiative. Projects are intentionally lightweight. The same data source, such as a shared incidents channel or central platform repository, can be referenced by multiple projects instead of being duplicated.  ## Onboarding and defaults During onboarding, users are prompted to select or create a default project that matches how they work, such as ML training infrastructure, Compiler, or Data Center Operations. That default project is stored on the user profile and scopes queries automatically. A new engineer gets high-signal answers without first having to learn which Slack channels, repositories, or document spaces matter. # Final Thoughts In the end, the knowledge base works because it meets people where the information already lives, instead of forcing everything into one rigid system. By combining various search techniques, we can surface evidence quickly. The result is a search experience that stays flexible enough for real company data, but structured enough to remain useful as Cerebras keeps growing. If you read this far and this is interesting to you, the ai/growth team is hiring reach out if you are interested @learnwdaniel # References 1. Malkov and Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs, arXiv:1603.09320 / IEEE TPAMI 2018. 2. Anthropic, Introducing Contextual Retrieval, 2024. 3. Cormack, Clarke, and Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, SIGIR 2009. 4. Li et al., Search-o1: Agentic Search-Enhanced Large Reasoning Models, arXiv:2501.05366, 2025. 5. Anthropic, Code Execution with MCP, 2025. 6. Liu et al., Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172, 2023. 7. Anthropic, Use XML Tags. 8. Salesforce/Slack Engineering, How Slack AI Processes Billions of Messages. 9. Improving Agents, Best Nested Data Format. 10. Cursor, Improving Agent with Semantic Search, 2025. ## 相关链接 - [Cerebras](https://x.com/cerebras) - [@cerebras](https://x.com/cerebras) - [278K](https://x.com/cerebras/status/2077822555159945507/analytics) - [@hi_im_isaac_](https://x.com/@hi_im_isaac_) - [@learnwdaniel](https://x.com/@learnwdaniel) - [@gaozenghao](https://x.com/@gaozenghao) - [https://www.cerebras.ai/blog/how-we-built-our-knowledge-base](https://www.cerebras.ai/blog/how-we-built-our-knowledge-base) - [@cocoindex_io](https://x.com/@cocoindex_io) - [@learnwdaniel](https://x.com/@learnwdaniel) - [Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs](https://arxiv.org/abs/1603.09320) - [Introducing Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval) - [Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods](https://dl.acm.org/doi/10.1145/1571941.1572114) - [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://arxiv.org/abs/2501.05366) - [Code Execution with MCP](https://www.anthropic.com/engineering/code-execution-with-mcp) - [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172) - [Use XML Tags](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags) - [Improving Agent with Semantic Search](https://cursor.com/blog/semsearch) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [2:27 AM · Jul 17, 2026](https://x.com/cerebras/status/2077822555159945507) - [278.5K Views](https://x.com/cerebras/status/2077822555159945507/analytics) - [View quotes](https://x.com/cerebras/status/2077822555159945507/quotes) --- *导出时间: 2026/7/17 11:27:38* --- ## 中文翻译 # 我们如何构建自己的知识库 **作者**: Cerebras **日期**: 2026-07-16T18:27:55.000Z **来源**: [https://x.com/cerebras/status/2077822555159945507](https://x.com/cerebras/status/2077822555159945507) ---  作者：@hi_im_isaac_, @learnwdaniel, @gaozenghao 注：完整技术博客的交互式版本可在以下网址查看：https://www.cerebras.ai/blog/how-we-built-our-knowledge-base 员工们每天向我们的内部知识库提出超过 15,000 个问题。自 3 个月前上线以来，它已成为公司内部使用最广泛的工具之一。使用者包括人类、自动化流程和智能体。 在 Cerebras，我们的团队分布在数据中心运营、芯片设计、硬件、训练、推理、云平台等多个领域。随着每年数百名新员工加入，我们的沟通渠道里充斥着同样的问题： - “我在哪里可以找到 X？” - “Y 方面的专家是谁？” - “Z 是什么？”  我们构建了 Cerebras Knowledge，帮助人们将人和系统与有用的信息连接起来。 # 到数据所在之处获取数据 在一个组织内部查找信息很难。数据散落在各种工具中，每隔一个季度左右就有人提出同样“绝妙”的解决方案：我们把所有东西都记录在一个平台上，这样所有信息就集中在一个地方了。当然，单一事实来源的梦想在实践中很少能实现。 信息总是在最方便、最符合使用习惯的地方产生：文档中的批注建议、Slack 中的讨论串、GitHub 中的代码引用，以及 Jira 中的状态元数据。这些平台都是为各自特定领域量身定制的，经过多年的产品工程和分析优化。在 Google Docs 里讨论 pull request 将会是一种糟糕的体验。 因此，我们着手设计一个对现有工作方式改动最小的系统。在数据收集方面，这意味着直接从每个平台提取数据。 ## 知识库的解剖结构 我们的知识库提供三样东西： - 一个收集和存储内部数据的平台。 - 一个查询这些数据的平台。 - 一个执行身份验证和授权的层，并带有审计和分析功能。 其核心是一张单一的 Postgres 表，保存来自众多来源的嵌入向量、原始摘要和元数据。系统持续从全公司各个角落摄取数据，并维护一个随时可供查询的数据存储。 我们想要一个简单但能兼容大多数数据形式的数据接口。我们还希望 Cerebras 的其他开发者能够构建自定义连接器。最终的结果刻意保持了简单：所有来源的数据——从 Slack 讨论串到网表——都落入同一张嵌入表中，而表中的任何内容都可以立即通过同一个接口进行查询：  每个数据源都定义了数据是什么、如何连接到它，以及获取频率。每一条生成的嵌入记录都遵循相同的接口，无论它来自 Slack、代码仓库、文档系统还是自定义数据库。 # 我们如何处理非结构化的 Slack 对话 Slack 是我们最需要精心设计的数据来源。全公司最前沿的工程讨论都发生在这里。  我们最初测试了直接对原始文本做简单嵌入是否效果足够好。我们很快意识到，仅靠向量搜索不足以匹配所有相关数据。 Slack 消息带来了几个挑战： - 信息密度差异巨大：“hey yeah sure mike”和一段详细的内核解释都是消息。 - 消息长度不一，较短的消息在余弦相似度上经常胜过更长、更详细的消息。 - 一条消息的含义往往取决于其周围的对话上下文。 我们需要一种混合方法。我们构建的 Slack 摄取系统能让每个讨论串同时通过多种搜索技术被检索到，而每种技术可以弥补其他技术的不足： - 全文检索能够捕捉那些被向量嵌入模糊掉的精确标记：错误字符串、标志名称、主机名。当工程师粘贴一条字面的错误信息时，精确的词法匹配几乎总是最好的证据，任何程度的语义相似度都不应超越它。 - 向量嵌入检索捕捉的是同义改写。提问“restore 在加载清单后挂起”的人和回答“checkpoint 在 NFS 挂载上停滞”的人可能永远不会使用相同的词汇。向量相似度正是把一个问题与用不同措辞写成的答案连接起来的关键。(1) - 逆文档频率将信号与噪音区分开来。围绕罕见标记（例如一个冷门的配置标志）构建的短消息应当获得高排名。“听起来不错，谢谢！”这类消息在嵌入空间中与许多查询都很接近，但一旦考虑词项稀有度，其得分就接近于零。 - 时间衰减编码了 Slack 答案会过时这一事实。两个帖子可能回答同一个问题，而六个月前的那个可能描述的是已不存在的基础设施。当相关性相同时，较新的帖子胜出。  没有任何单一的打分器可以单独被信任。每种技术都会对同一个语料库生成自己的排序视图，这些视图在查询时进行融合（见“重排序”）。 ## Socket Mode 为了实时收集数据，我们在工作区中安装了一个 Slack 机器人，并以 Socket Mode 运行。Slack 通过持久化的 WebSocket 将每条消息事件推送给我们，因此我们无需轮询 Web API、消耗其速率限制，就能获得实时更新。 当事件到达时，我们立即确认它，使用稳定的事件 ID 进行去重，并将该消息标记给摄取消费者。 摄取消费者不会孤立地保存一条新消息。它会解析该消息所属的会话线程，并从 Slack API 重新获取整个对话，包括父消息和所有回复。然后，它将整个会话线程作为一行写回。因此，对现有线程的一次回复会重新拉取父消息和所有同级回复，从而保证存储的内容、参与者列表和最后活动时间戳始终反映完整的对话。 我们系统中的每个 Slack 频道都有其自己的数据源。这为数据新鲜度提供了细粒度的调节能力。例如，一个团队可以选择更频繁地摄取一个繁忙的事故频道。 ## 会话线程与消息 原始 Slack 文本在落地后即可进行关键词检索，因为我们在原始内容上维护了 Postgres 全文（GIN）索引。然而，为了实现有用的向量检索，我们还需要一些额外的处理。(8) 在蒸馏过程中，LLM 从完整的会话线程中提取结构化数据： - 工程师实际会搜索的一行式问题。 - 一段简短的摘要。 - 解决方案。 - 提及的系统和代码引用。  我们对这些数据点进行嵌入，并写入共享的嵌入表。原始会话记录本身不直接嵌入。在我们的实验中，当会话线程被规范化为一致的格式后，准确率显著提升。(7,9) 这些额外的元数据也为语义匹配提供了更有用的信号。 ## 消息爆发 到这一步，Slack 检索已经不错了，但我们不断遇到同一个问题：长会话线程中的重要消息并不总是体现在线程级的摘要中。 为了提升单条消息的信号，我们使用了消息爆发（bursting）。一次爆发是同一作者连续发送的一串消息。我们为单次爆发进行嵌入，并将线程主题作为上下文前置(2)，因为有时答案存在于某条偏题消息中，而其词汇从未进入线程摘要。爆发嵌入使这条消息本身也能被检索到。 为防止低信号数据进入数据库，每次爆发都会根据多个信号的加权组合进行打分，必须超过阈值才会被嵌入： - 它包含在语料库中相对罕见的标记，IDF 至少为 4.0。 - 组合后的爆发内容至少 200 个字符。 - 爆发中有一条或多条消息包含表情回应，从而获得社交加分。  蒸馏之后，符合条件的爆发会被嵌入，并与会话线程级记录一起存储在嵌入表中。 # 代码仓库 起初我们曾争论是否有必要对代码仓库进行嵌入。随着 Claude Code 和其他命令行工具的兴起，当“grep 就足够了”看起来成立时，创建代码嵌入显得有些反直觉。在与业界同行交流并阅读了 Cursor 关于大型代码库语义搜索的研究结果后，我们决定尝试一下。 我们有许多内部代码仓库，有些超过 40 GB。我们主要关心的是如何高效地保持它们是最新的。 ## 使用 @cocoindex_io 维护代码嵌入 经过多次实验，我们选择了 CocoIndex，一个专注于代码库向量化的开源文档嵌入框架。 对于每个代码仓库，我们使用从粗到细排序的语言特定正则边界来切分代码。切分器会优先尝试更高层级的边界，例如类。如果切出的代码块仍然过大，则退回到方法边界，然后再退回到更小的块。我们对切出的代码块进行嵌入，并将向量写入 Postgres。单个文件可能会生成多个不同粒度的嵌入，例如文件级和函数级记录。  CocoIndex 在 Postgres 中跟踪同步元数据。每次提交时，它只对发生变化的代码块进行重新嵌入和重新导出，而不是重新计算整个代码仓库。这对我们来说效果尤其好，因为同步状态和嵌入存储位于同一个数据库中。 随着代码库数量的增长，我们将代码仓库的接入迁移到了团队可以自行提交的配置文件中，包括文件路径级别的白名单和黑名单。 ## 自定义数据源 一些团队已经有自己的数据库，不想为了加入知识库而把数据迁移到 Slack 或某个文档系统中。他们希望能在自己现有的表上使用同样的查询接口。 为支持这一点，我们将自定义数据源视为插件脚本。团队通过拉取请求提交一个小的 Python 模块，该模块知道如何从其系统中读取数据，并输出与我们嵌入表结构一致的行，同时添加一个对应的数据源条目。 只要脚本按照与其他所有嵌入行相同的 schema 写入共享数据库，技术栈的其余部分无需任何改动即可正常工作。这些数据就能与 Slack、代码和文档一起被查询，系统其他地方无需任何特殊处理。 ## 规划与工具扇出 对于每个查询，我们首先运行一个简短的规划步骤，由 LLM 决定哪些工具和数据源可能相关。主要工具包括： - subsystem_index：按文件生成的 LLM 摘要。 - search：跨 Slack、wiki、代码及其他已索引来源的统一向量检索管道，内部进行合并和重排序。 - search_slack：直接检索 Slack。 - search_code：对源代码仓库执行 ripgrep。 - recent_prs：与问题相关的近期拉取请求。 - who_knows：在某主题上展现出专业能力的人。 规划器基于对我们已索引内容的简要描述进行工作：存在哪些项目，每个项目有哪些可用数据源，以及每个数据源擅长回答什么问题。给定用户查询和当前作用范围，它会输出工具选择，由执行器并行发起这些调用，将结果规范化为统一的证据格式，然后传递给最终的综合 LLM。(4)  ## 重排序 一篇文档可能仅仅因为与查询共享了词汇，而实际上回答的是另一个问题，却被排到了靠前的位置。在重排序之前，我们使用倒数排名融合（reciprocal rank fusion，RRF）来合并各检索器不兼容的结果列表。对每篇文档，在其出现的每个列表中，我们累加权重 / (60 + 排名)，默认权重为 1.0，平滑常数为 60。  平滑常数让“共识”比单一的强票更重要的：一篇在多个检索器中都靠前出现的文档，可以胜过只在其中一个检索器中排名第一的文档。随后，我们将重复的块合并回单一来源，限制每个文件最多能贡献的结果数量，最终得到一个更多样化的前二十名。 我们将原始查询和这些候选文档发送给一个小型重排序模型。它为每篇文档打出一个 0 到 10 的分数，我们保留前十名。(6) 排名确定后，我们会为胜出的文档补充上下文。例如，如果匹配到某个 wiki 小节，我们会把相邻的两个小节一并拉入，这样分块时被拆开的标题、前提条件和注意事项就不会丢失。这为读者提供了一个完整的片段，而不是缺失重要上下文的孤立段落。 因此，search 的输出是一份内容丰富的证据包：融合了不同检索器的结果，在来源层面去重，针对实际问题重新排序，最后才扩展周边上下文。 ## MCP 在 MCP 集成中，我们将检索的基本构件作为直接工具暴露出来，而不是把它们隐藏在一个“回答这个问题”的端点之后。这些工具特意设计得简单，并尽可能不依赖 LLM，以便客户端能够快速、低成本地调用。(5) 每个 MCP 工具对应一个底层检索原语，例如 search_slack、search_code、search 或 who_knows。工具的输入和输出都窄小、结构化且稳定，任何客户端或智能体都能轻松调用，无需在工具本身内部嵌入额外的编排逻辑。 大多数工具只运行一个查询管道，例如向量检索、词法检索或 ripgrep，应用轻量级的评分启发式规则，然后返回原始的证据行。 Claude Code 或任何兼容 MCP 的智能体则成为编排引擎。它决定调用哪些工具、以什么顺序调用，以及如何将结果组装成最终答案或代码修改。检索层本身提供服务时并不依赖这些 LLM 决策。 ## Web UI 在 Web UI 中，存在同样的工具，但它们连接到一个完整的查询管道，对每个用户问题端到端地运行。UI 智能体负责规划器和执行器步骤。 - 规划器：一个轻量级的 LLM 步骤会检查查询和当前项目，然后选择要调用哪些检索工具，例如 search、search_slack 和 subsystem_index。 - 执行器：系统并行发起这些工具调用，收集结果，并将它们规范化为包含分数、时效性和来源提示的统一证据 schema。 - 综合：最后一个 LLM 步骤接收结构化的证据包和原始问题，然后生成显示在 UI 中的答案，包括引用、注意事项和跨来源的综合。 从用户的角度看，Web UI 只是“提问并得到答案”。而在底层，它运行的是同样的规划器 → 执行器 → 综合器模式，MCP 客户端也可以显式地复现这一模式。  # 组织方式 随着语料库的增长，“到处搜索一切”很快变得不再实用。编译器团队的工程师不希望在结果中看到基础设施运维手册，反之亦然。项目是我们让搜索默认就具备相关性的方式。 ## 项目与作用域搜索 我们引入项目作为组织查询工作区的主要方式。项目是一个命名的数据源集合：特定于某个团队或计划的 Slack 频道、代码仓库、内部数据库和文档空间。 项目有意设计得非常轻量。同一个数据源，例如共享的事故频道或中央平台仓库，可以被多个项目引用，而不需要复制。  ## 入门引导与默认设置 在入职引导过程中，用户会被提示选择或创建一个符合其工作方式的默认项目，例如机器学习训练基础设施、编译器或数据中心运营。 该默认项目会存储在用户个人资料中，并自动限定查询范围。新工程师无需先了解哪些 Slack 频道、代码仓库或文档空间是重要的，就能获得高相关性的答案。 # 结语 归根结底，这个知识库之所以有效，是因为它在信息已经存在的地方服务于人们，而不是强迫所有内容都纳入一个僵化的系统中。通过结合多种搜索技术，我们可以快速呈现证据。最终结果是一个足够灵活、能适应真实公司数据的搜索体验，同时又具备足够的结构性，能在 Cerebras 持续发展的过程中保持实用性。 如果你读到了这里并且对此感兴趣，ai/growth 团队正在招聘，如果你有兴趣请联系 @learnwdaniel # 参考文献 1. Malkov 和 Yashunin，《Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs》（使用分层可导航小世界图的高效且稳健的近似最近邻搜索），arXiv:1603.09320 / IEEE TPAMI 2018。 2. Anthropic，《Introducing Contextual Retrieval》（引入上下文检索），2024 年。 3. Cormack、Clarke 和 Büttcher，《Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods》（倒数排名融合优于 Condorcet 及个体排名学习方法），SIGIR 2009。 4. Li 等人，《Search-o1: Agentic Search-Enhanced Large Reasoning Models》（Search-o1：智能体搜索增强的大型推理模型），arXiv:2501.05366，2025 年。 5. Anthropic，《Code Execution with MCP》（使用 MCP 执行代码），2025 年。 6. Liu 等人，《Lost in the Middle: How Language Models Use Long Contexts》（迷失在中间：语言模型如何使用长上下文），arXiv:2307.03172，2023 年。 7. Anthropic，《Use XML Tags》（使用 XML 标签）。 8. Salesforce/Slack Engineering，《How Slack AI Processes Billions of Messages》（Slack AI 如何处理数十亿条消息）。 9. Improving Agents，《Best Nested Data Format》（最佳嵌套数据格式）。 10. Cursor，《Improving Agent with Semantic Search》（利用语义搜索改进智能体），2025 年。 ## 相关链接 - [Cerebras](https://x.com/cerebras) - [@cerebras](https://x.com/cerebras) - [27.8万](https://x.com/cerebras/status/2077822555159945507/analytics) - [@hi_im_isaac_](https://x.com/@hi_im_isaac_) - [@learnwdaniel](https://x.com/@learnwdaniel) - [@gaozenghao](https://x.com/@gaozenghao) - [https://www.cerebras.ai/blog/how-we-built-our-knowledge-base](https://www.cerebras.ai/blog/how-we-built-our-knowledge-base) - [@cocoindex_io](https://x.com/@cocoindex_io) - [@learnwdaniel](https://x.com/@learnwdaniel) - [使用分层可导航小世界图实现高效且稳健的近似最近邻搜索](https://arxiv.org/abs/1603.09320) - [介绍上下文检索](https://www.anthropic.com/news/contextual-retrieval) - [倒数排序融合优于 Condorcet 及个体排序学习方法](https://dl.acm.org/doi/10.1145/1571941.1572114) - [Search-o1：智能体搜索增强的大型推理模型](https://arxiv.org/abs/2501.05366) - [使用 MCP 进行代码执行](https://www.anthropic.com/engineering/code-execution-with-mcp) - [迷失在中间：语言模型如何利用长上下文](https://arxiv.org/abs/2307.03172) - [使用 XML 标签](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags) - [通过语义搜索改进智能体](https://cursor.com/blog/semsearch) - [升级到高级版](https://x.com/i/premium_sign_up) - [2026年7月17日 凌晨2:27](https://x.com/cerebras/status/2077822555159945507) - [27.85万次浏览](https://x.com/cerebras/status/2077822555159945507/analytics) - [查看引用](https://x.com/cerebras/status/2077822555159945507/quotes) *导出时间：2026/7/17 11:27:38*
杀 杀死 2023 RAG 的，是更好的 RAG：9 位检索专家的前沿实战共识「Beyond Naive RAG」 本文介绍 Hamel Husain 主编的演讲合集《Beyond Naive RAG》，汇集 9 位一线检索专家的共识：朴素 RAG（切块+单向量+余弦检索）已失效，但检索增强生成本身远未死亡。全书论证三条主线：长上下文不能替代检索（Context Rot 证明上下文越长模型越不可靠）；评估先于优化（公开基准已污染，须建自有黄金数据集）；复杂性不是目的（慎用图数据库等过度工程）。涵盖 ColBERT 后期交互、推理增强检索（Promptriever、Rank1）、多重表示与智能路由、FreshStack 评估指标等前沿方法，提出'评估驱动+多表示+推理增强+克制架构'的实践观。 技术 › LLM meng shao 2026-10-12 RAG检索embedding评估长上下文Context RotColBERT信息检索LLM
B Build a business second brain: the 13-lesson guide 本文是 Nick Spisak 的 13 节课教程指南，讲解如何为企业搭建'第二大脑'（second brain）知识库，并结合 AI Agent 使用。作者以自己的公司 Capstone Brands 为例，完整展示了从选择业务问题、创建 GitHub 组织与仓库、用 GitHub Desktop 克隆本地副本、在 Codex 中使用知识库模板发起构建请求，到按销售、营销、情报、财务、运营五大业务职能组织知识的全过程。最终目标是让 Slack 中的 Agent 能阅读公司知识库、回答真实业务问题并引用来源文件。教程还涵盖版本历史管理、访问权限控制、标准操作流程转化为 Agent Skill 等实践要点。 技术 › Agent Nick Spisak 2026-09-07 AI Agent第二大脑CodexGitHub知识库企业知识管理教程Claude CodeSlack业务自动化
H How to build an agent for your product 本文面向工程负责人，讲解如何为产品构建 AI Agent 并部署到三个渠道：产品内聊天界面、作为 Claude/ChatGPT 的连接器、以及 Slack 工作区。作者提出 v0 Agent 由模型、工具和指令三部分组成，并强调应自己掌控 Agent 编排而非简单暴露 API。随后介绍 v1 的改进：会话连续性、上下文精简、接入产品知识库、用户学习与历史会话搜索。最后通过 Agno 的 AgentOS 将 Agent 封装为 REST API 产品，支持流式输出、会话管理、数据隔离与后台执行，帮助产品解锁新的分发和收入机会。 技术 › Agent Ashpreet Bedi 2026-09-02 AgentClaudeChatGPTAgentOSAgno产品工程Slack会话管理知识库上下文工程
B Bypassing Figma & Building a Knowledge Product 本文介绍了Little Plains团队如何将多年积累的工作室资料转化为可搜索的知识中心，并在此基础上构建了首个产品。文章详细阐述了知识层的构建过程，包括归档、索引、检索系统的设计，以及如何通过视觉搜索、动态聊天和语音三种方式提供公共访问。作者强调了组织知识的重要性，并分享了从传统设计转向代码驱动设计的经验，以及如何区分快速知识与深度知识以优化检索效率。 技术 › 工具与效率 Emmett 2026-08-20 知识管理知识库搜索AI设计工程内容检索数据索引产品开发用户体验工作流
《 《指挥 AI，做出一个企业级 Agent》08：企业 RAG 不是上传几个 PDF 文章指出企业 RAG 不仅仅是上传 PDF 回答问题，更需关注文档版本、权限、来源追溯及停用管理。作者通过模拟企业真实文档环境，构建了包含版本控制和过滤机制的评测体系，强调了可靠检索与诚实无答案的重要性，并分享了针对小规模知识库的技术选型经验。 技术 › Agent jager 2026-07-22 RAG企业级Agent版本控制向量检索评测Embedding知识库技术选型
H How we built our knowledge base Cerebras 分享了构建其内部知识库的经验。该知识库每天处理超过 15,000 个问题，集成了 Slack、GitHub 等多种数据源。文章详细介绍了如何通过 Postgres 存储 embeddings，并采用混合搜索策略（全文搜索、向量搜索、IDF 加权、时间衰减）来高效检索非结构化对话数据，解决了信息分散和匹配精度的问题。 技术 › DevOps Cerebras 2026-07-18 知识库向量搜索EmbeddingsSlack混合搜索Postgres数据工程检索增强生成企业工具架构设计
H How we built our knowledge base Cerebras 介绍了其内部知识库的构建过程，该知识库每天处理超过15,000个问题。系统通过从Slack、GitHub等平台提取数据，利用Postgres存储嵌入和元数据，结合全文搜索、向量搜索、IDF和衰减时间等混合搜索技术，实现了高效的信息检索。 技术 › DevOps Cerebras 2026-07-17 知识库向量搜索SlackPostgres数据 ingestion混合搜索嵌入式实时更新WebSocket信息检索
如 如何把聊天记录变成可检索知识库（2026版） 文章探讨了如何将微信、Telegram等平台的聊天记录转化为可用的个人知识库。作者从数据获取、清洗降噪（语义分块、去重、上下文修复）到存储检索（Embedding选型、GraphRAG）提供了全流程的技术指导，并提出了数字分身、知识传承等多种应用场景。 技术 › LLM AI最严厉的父亲 2026-05-12 RAG知识库数据处理语义搜索Embedding微信记录个人成长工具与效率数据清洗GraphRAG
如 如何精通 RAG 并构建拥有完美记忆的 AI Agent (全流程教程) 本文是一篇关于检索增强生成（RAG）技术的完整教程。文章指出，大多数 AI Agent 难以落地的原因在于模型无法获取企业私有数据。RAG 通过将外部知识库与大模型结合，解决了这一痛点。作者详细拆解了构建 RAG 系统的五个核心阶段：文档摄取与分块、文本嵌入、向量存储、信息检索以及最终生成，并提供了关于分块策略、嵌入模型选择及混合检索的实战建议。 技术 › LLM Khairallah AL-Awady 2026-05-01 RAGAgent教程向量数据库Embedding知识库ClaudeOpenAI大模型开发指南
开 开源微信流 APP：手机微信聊天记录直接给 WorkBuddy 和 Obsidian 作者苍何介绍了开源项目 WeChatBridge（微信流）的 iOS 版本发布。该工具可将微信聊天记录批量转发给 Codex、Claude Code 等各家 Agent 进行总结分析，也可沉淀到 Obsidian 作为个人知识库。作者分享了借助 Claude Code 和 Kimi K3.1 Pro 开发 iOS 版的过程：先让模型制定详细计划和技术文档，再照文档执行，6 个多小时完成了原计划 18 个工作日的工作。文章强调"别跟它聊需求，跟它定验收"的 Agent 开发心得，并详细介绍了 iOS 版的收、存、投功能，包括分批收集超过 100 条消息、场景化提示词、隐私安全设计等。 技术 › Claude Code 苍何 2026-10-27 WeChatBridge开源KimiiOS开发Agent编程Obsidian微信知识库验收驱动开发提示词
H Harness engineering for a new hire: the AI employee's first hours on my motion workflow 作者将一名名为 Viktor 的 AI 员工接入工作室的 Slack，并把自己的 motion 工作流交付包交给它检查。首次任务是检查 harness ZIP 包（8 个 Markdown 指令文件和 4 个辅助文本），为读者准备快速上手指南和交接清单。Viktor 返回四份文档，并发现七个在 README 中被引用但未包含在交接中的 QA 和溯源记录——这是作者自己的遗漏。作者随后纠正：指定 LAUNCH-PROMPT.txt 为主路线，补充七个 QA 记录，并要求将读者说明控制在 80 字内。文章详细记录了任务下达、首轮审查、纠正与修订、说明批准和最终结果，展示了如何通过明确的上下文、输出定义和人工审核步骤来有效管理 AI 协作者。 技术 › Harness Engineering beamnxw 2026-10-27 AI员工工作流Slackharness交接清单QA记录快速上手人机协作
G Grok Bot Memory Engineering: How One Agent Learns With a Team 文章围绕 Grok Bot 新发布的 Team Bots 展开，深入分析团队级 AI Agent 的记忆工程。核心观点：团队 Bot 需要在共享学习与私有上下文之间划清边界。作者提出将上下文分为五类：实时源数据、团队技能、团队记忆、私有笔记和任务状态，并强调'可复用判断放记忆，易变数据留在源头'的原则。文章还提出记忆准入卡与审查机制的设计方案，包括准入门槛、外部账本、审查日期和评估流程，避免 Bot 在错误位置记住错误信息或把过时决策套用到新场景。作者同时说明这些流程是其提议方案，而非已验证的产品功能。 技术 › Agent rari🈀 2026-10-26 GrokTeam Bots记忆工程共享记忆私有上下文Agent架构团队协作Slack插件数据边界
