下一代 LLM 推理网络：ZCube 如何缓解网络瓶颈 ✍ Z.ai🕐 2026-05-21📦 18.8 KB 🟢 已读 𝕏 文章列表 文章探讨了随着长上下文推理和 Prefill-Decode 解耦成为主流，网络如何成为 LLM 推理集群吞吐量和延迟的关键瓶颈。Z.ai、Harnets.AI 和清华大学联合开发了 ZCube 网络架构，通过扁平化拓扑和混合轨道设计，有效解决了传统 ROFT 架构下的流量负载不均衡问题。生产环境基准测试显示，ZCube 仅通过架构优化便实现了交换机成本降低 33%、吞吐量提升 15% 以及 TTFT 延迟降低 40.6% 的显著收益。 LLM推理网络架构ZCube性能优化Prefill-Decode负载均衡ROFT基础设施清华大学 # Next-generation LLM Inference Network: How ZCube Alleviates Network Bottlenecks? **作者**: Z.ai **日期**: 2026-05-20T21:47:33.000Z **来源**: [https://x.com/Zai_org/status/2057216685040443743](https://x.com/Zai_org/status/2057216685040443743) ---  LLM inference is reshaping AI infrastructure. The network used to be the least interesting part of an inference cluster. That isn't true anymore. With long-context inference and Prefill-Decode disaggregation now standard, the network sits on the critical path of throughput, tail latency, and per-token serving cost. To address the increasingly severe topology-induced congestion in Prefill-Decode disaggregated deployments, Z.ai, Harnets.AI, and Tsinghua University jointly developed and deployed the ZCube network architecture in an online production environment. The deployment shows that system-level innovation at the network architecture layer can unlock hardware potential in a highly cost-effective way. In production benchmarking for the GLM-5.1 coding workload, ZCube delivered significant gains through architectural optimization alone: - Cost optimization: GPUs, the software stack, and applications remained unchanged, while switch and optical module CapEx was reduced by 33%. - Throughput improvement: Average GPU inference throughput increased by 15%. - Latency improvement: TTFT P99 was reduced by 40.6%. The root cause of the congestion lies in the shift of inference traffic patterns. As PD disaggregation becomes mainstream, cross-node KV Cache transfers make inference traffic highly asymmetric, with dynamically changing sources, destinations, and traffic volumes. In traditional ROFT (Rail-Optimized Fat-Tree) architectures, static topology and port mappings can easily concentrate traffic on a limited set of switches and links, causing local hotspots, queue buildup, and PFC backpressure. This leads to a structural issue where aggregate bandwidth appears sufficient, yet localized congestion occurs frequently. ZCube addresses this issue by using a fully flattened network topology together with a hybrid single-rail / multi-rail access design. At the network architecture layer, it decouples and distributes PD traffic across a broader path space, reducing the probability of topology-induced congestion at its source. This provides a more efficient networking foundation for next-generation hyperscale inference clusters.  Figure 1: ZCube Effectively Avoids Topology-Induced Network Congestion Compared with ROFT # Network Becoming a Bottleneck for Effective Inference When thousands of GPUs serve online inference requests concurrently, every KV Cache transfer and every data synchronization operation traverses the inter-GPU network. As long-context inference and Prefill-Decode disaggregated inference gradually become mainstream, data exchange between Prefill and Decode nodes continues to grow. Network bandwidth, and more importantly the ability to use it effectively, has begun to affect cluster-level throughput and latency directly. To quantify the impact of networking on inference performance, we first conducted an ablation study on a 512-GPU cluster. We kept GPU compute, the software stack, the model, and application logic unchanged, and only adjusted the available NIC bandwidth cap. We then measured changes in overall cluster throughput and Time to First Token (TTFT).   Figure 2: Overall cluster throughput and TTFT under different network bandwidth settings For example, when network bandwidth was increased from 100Gbps to 200Gbps, overall inference throughput improved by approximately 19%, while Time to First Token, or TTFT, decreased by approximately 22%. This indicates that, in LLM inference, network bandwidth has become one of the key factors constraining service performance. # 1. Network Congestion in Inference Today, AI clusters commonly use Clos, or Fat-Tree, architectures. The basic idea is to scale the network by stacking multiple layers of switches. However, the performance of Clos networks depends heavily on ideal load balancing across switches, which is difficult to achieve in practice due to routing policies and real traffic patterns. For example, in many two-tier Fat-Tree deployments, which consist of Spine and Leaf layers, traffic across Spine switches can become severely imbalanced. As a result, upper-layer applications often fail to obtain the expected network performance. To reduce the overhead of cross-layer forwarding, the industry often adopts ROFT (Rail-Optimized Fat-Tree) architectures [1]. As shown in Figure 3, ROFT groups GPUs by index ("rail"), and connects GPUs with the same index to the same Leaf switch, reducing the communication cost across Spine switches.  Figure 3: In ROFT, traffic among Leaf switches can easily become imbalanced ROFT works well for certain training traffic patterns. However, in Prefill-Decode disaggregated inference, we observed a more prominent issue: KV Cache transfers exhibit strong source-destination asymmetry. Different GPUs and different NICs carry highly uneven communication loads, as shown in Figure 4. As a result, ROFT’s rail mapping no longer naturally translates into load balancing. Instead, traffic can become concentrated on a small number of Leaf switches and links, leading to link congestion and degraded transfer performance. This manifests in several ways: - Some Leaf switches become persistent load hotspots, increasing the probability that multiple KV Cache transfer flows compete on the same links. As a result, actual transfer throughput can fall far below the NIC bandwidth capacity. - Certain egress queues on some Leaf switches remain at high depth for extended periods and frequently trigger PFC backpressure, as shown in Figure 5. - Link congestion further amplifies tail latency, affecting both TTFT and overall throughput.  Figure 4: KV Cache transfer load imbalance across different NICs on the same machine  Figure 5: Frequent PFC Pause events on selected Leaf switch ports It is important to distinguish between the two types of network congestion, as illustrated in Figure 6: - Unavoidable congestion: For example, when multiple GPUs send data to the same destination at the same time, contention on the final-hop link is inevitable. - Avoidable congestion: This is caused by topology design, traffic mapping, or imbalanced multipath utilization. Fundamentally, it is an architecture-level design problem.  Figure 6: Illustration of two types of network congestion For the first type of congestion, we typically rely on congestion control, traffic shaping, and related mechanisms to mitigate its impact. For the second type, new network transport mechanisms such as adaptive routing [2], packet spraying [3,4], and MRC [5] can help. However, a more effective approach is to prevent network conflicts that should not occur in the first place through innovation at the network architecture layer. Prefill-Decode disaggregated inference is a typical example. If the network topology cannot match the traffic pattern, the system will repeatedly generate load hotspots and link conflicts. Solving this problem requires rethinking the inference network architecture itself. # 2. ZCube Network Architecture To address the above issues, we deployed a new ZCube network architecture [6]. ZCube breaks away from the traditional Clos design philosophy of hierarchical switch stacking and instead introduces a fully flattened GPU server interconnect. The ZCube routing strategy, designed specifically for the ZCube architecture, fully leverages the structural properties of the flattened topology. It can achieve near-ideal load balancing across all switches in the network, thereby significantly improving overall cluster network bandwidth. Compared with Clos, ZCube has a natural advantage in load balancing. This advantage benefits both training clusters and inference clusters. Importantly, ZCube achieves these performance gains while reducing switch and optical module costs by approximately one third compared with Clos. Based on current mainstream switch and NIC configurations, ZCube can support flattened networking for tens of thousands, or even hundreds of thousands, of GPUs. ## 2.1 ZCube Core Architecture As shown in Figure 7, the core ideas of ZCube are: 1. Remove the Spine switch layer. 2. Divide Leaf switches into two groups of equal size, typically odd-numbered switches and even-numbered switches. 3. Establish a complete bipartite interconnect between the two switch groups. 4. Connect the two ports of each GPU NIC to the corresponding switches in the two groups using single-rail and multi-rail access patterns.  Figure 7: ZCube architecture overview  Suppose each GPU has a corresponding NIC with two ports, i.e., p=2. There are n GPUs in total, and GPUs and NICs share the same indices: 1,2,…,n. Let k denote the number of GPUs connected to each switch. The total number of switches is 2n/k, numbered 1,2,…,2n/k. For GPU i, where 1≤i≤n: - The first port connects to the odd-numbered switch: ((i−1)mod(n/k))×2+1 - The second port connects to the even-numbered switch: ⌈i/k⌉×2 The two switch groups are connected as a complete bipartite graph: every odd-numbered switch connects to every even-numbered switch. A ZCube topology under dual-port NIC configuration, withp=2,n=32, and k=8, is shown in Figure 7. ## 2.2 Key Properties of ZCube Network Diameter ZCube has a network diameter of two switch hops, meaning any pair of GPUs can reach each other through two switches. This sits between a one-layer switch network, which has one switch hop but limited scale, and a conventional two-layer switch network, which supports a larger scale but typically requires three switch hops and incurs higher latency. Load Balancing First, the ZCube routing strategy ensures that each GPU pair has a unique optimal path, avoiding traffic conflicts caused by multipath route selection. Second, ZCube uses two complementary GPU-to-switch connection patterns. One switch group connects to GPUs in a single-rail pattern, where each switch connects to a contiguous range of GPU IDs. The other switch group connects to GPUs in a multi-rail pattern, where each switch connects to GPUs with the same relative index across groups. This design enables ZCube to achieve highly effective load balancing across the entire switch fabric under both typical AI training traffic patterns, such as AllReduce and All-to-All, and typical AI inference traffic patterns, where source-destination relationships are uncertain, and NIC loads can be highly imbalanced. As a result, ZCube can avoid the second type of network congestion described earlier at the architecture layer. As shown in Figure 8, traffic flows that would conflict under ROFT can obtain dedicated network paths under ZCube, thereby avoiding congestion.  Figure 8: Load balancing under the ZCube architecture Scalability ZCube provides strong scalability while preserving its favorable performance characteristics. For example, using one layer of 51.2T switches, each with 128 × 400Gbps ports, ZCube can construct a network connecting 16,384 400Gbps NICs. If higher-capacity switches are used, or if the ZCube network is divided into more planes, the architecture can scale further to support interconnection among tens of thousands or even hundreds of thousands of GPUs. Cost At the same cluster scale, ZCube can reduce switch and optical module costs by approximately one third compared with traditional Clos / ROFT architectures. For example, in a 10,000-GPU AI cluster, ZCube can save roughly 210 million RMB to 640 million RMB in network hardware investment. These characteristics show that ZCube can achieve better load balancing and performance while requiring lower network hardware cost. ## 2.3 Real-World Cluster Testing: Boosting Inference Performance While Cutting Network Costs We upgraded the network architecture of a thousand-GPU cluster running GLM-5.1 coding inference services from the original ROFT to the ZCube architecture. Since the ZCube architecture eliminates the Spine-layer switches found in traditional Clos architectures, the legacy cabling patterns, IP addressing schemes, routing policies, and switch configuration methods established under the Clos framework could not be reused directly, necessitating a complete redesign tailored to ZCube. To tackle these challenges, the Harnets.AI Network Team designed a comprehensive network solution centered on the ZCube architecture. They developed a suite of automation tools, including the ZCube Controller, a data center layout design tool, and a cabling correctness verification program. This enabled capabilities such as data center deployment planning, cabling validation, automated configuration generation, and batch deployment, effectively resolving numerous hurdles in ZCube deployment. This suite of tools was the critical factor enabling the successful transformation of a large-scale production cluster within an exceptionally tight timeframe. Following the seamless network architecture migration, we conducted real-world testing on the ZCube architecture by running the GLM-5.1 coding inference services on this cluster. By comparing the cluster's inference performance before and after the upgrade, we found that ZCube boosted the average GPU inference throughput by over 15% compared to the ROFT architecture (as shown in Figure 9), while dropping the P99 tail latency of TTFT by 40.6%.   Figure 9: Throughput and TTFT Comparison Between ZCube and ROFT Architectures in the Same Cluster In summary, for GPU and server hardware of the same scale and configuration, and without modifying any applications, upgrading the networking architecture to ZCube allowed us to not only save 1/3 of the optical modules and switch hardware, but also enable the cluster to serve 15% more inference requests per second. Against the current backdrop of exploding inference workloads and severe shortage of compute resources, this approach proves to be highly pragmatic and valuable. Currently, this ZCube cluster has been running stably for over two weeks, playing a vital role in powering the GLM-5.1 coding inference services. # 3. Conclusion LLM inference is moving from point-wise optimization toward system-level co-design. The coupling between the network and the inference engine is becoming increasingly tight, making networking a critical component of the inference system. The production deployment of ZCube shows that network architecture innovation can directly unlock the effective capacity of inference systems. By better aligning the network architecture with KV Cache transfers and PD traffic patterns, ZCube reduces the probability of topology-induced congestion at the source, improving throughput and latency while enhancing cluster cost efficiency. Looking ahead to next-generation LLM infrastructure, network design will evolve from general-purpose interconnects toward model-traffic-driven system co-design. Long-context inference, PD disaggregation, MoE, and integrated training-inference workloads are reshaping intra-cluster communication patterns, requiring network topology, communication libraries, and scheduling policies to be jointly optimized around real model traffic. Looking ahead, we will continue pioneering novel AI network architectures for larger-scale inference and training clusters ─ upgrading the network from a foundational GPU connection layer into a core driver of token generation efficiency, system resilience, and cost-effectiveness. # Acknowledgements ZCube was published at ACM SIGCOMM 2025, and was recognized as “significantly change the way we think about and understand networking.” This is the first large-scale deployment of the technology in a production inference cluster. We thank the Harnets.AI team for their professional support and close collaboration throughout this network architecture upgrade and optimization effort. ## Reference [1] NVIDIA. 2023. SuperPOD: Next Generation Scalable Infrastructure for AI Leadership. https://docs.nvidia.com/https:/docs.nvidia.com/dgx-superpod-reference-architecture-dgx-h100.pdf [2] NVIDIA. 2025. https://developer.nvidia.com/blog/accelerating-ai-storage-by-up-to-48-with-nvidia-spectrum-x-networking-platform-and-partners/ [3] Ultra Ethernet Consortium. Ultra Ethernet specification v1.0.1, 2025. [4] Tommaso Bonato, Abdul Kabbani, Ahmad Ghalayini, Michael Papamichael, Mohammad Dohadwala, Lukas Gianinazzi, Mikhail Khalilov, Elias Achermann, Daniele De Sensi, and Torsten Hoefler. REPS: Recycled entropy packet spraying for adaptive load balancing and failure mitigation, 2026. [5] Araujo, J., Chow, A., Handley, M., Lewis, R., Paasch, C., Padhye, J., … & Sur, S. (2026). Resilient AI Supercomputer Networking using MRC and SRv6. arXiv preprint arXiv:2605.04333. [6] Yan, Z., Li, D., Chen, L., Xiong, D., Gao, K., Zhang, Y., … & Lin, H. (2025, September). From ATOP to ZCube: Automated topology optimization pipeline and a highly cost-effective network topology for large model training. In Proceedings of the ACM SIGCOMM 2025 Conference (pp. 861-881). ## 相关链接 - [Z.ai](https://x.com/Zai_org) - [@Zai_org](https://x.com/Zai_org) - [12K](https://x.com/Zai_org/status/2057216685040443743/analytics) - [Z.ai](http://z.ai/) - [Harnets.AI](http://harnets.ai/) - [Harnets.AI](http://harnets.ai/) - [Harnets.AI](https://harnets.ai/) - [https://docs.nvidia.com/https:/docs.nvidia.com/dgx-superpod-reference-architecture-dgx-h100.pdf](https://docs.nvidia.com/https:/docs.nvidia.com/dgx-superpod-reference-architecture-dgx-h100.pdf) - [https://developer.nvidia.com/blog/accelerating-ai-storage-by-up-to-48-with-nvidia-spectrum-x-networking-platform-and-partners/](https://developer.nvidia.com/blog/accelerating-ai-storage-by-up-to-48-with-nvidia-spectrum-x-networking-platform-and-partners/) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [5:47 AM · May 21, 2026](https://x.com/Zai_org/status/2057216685040443743) - [12.6K Views](https://x.com/Zai_org/status/2057216685040443743/analytics) - [View quotes](https://x.com/Zai_org/status/2057216685040443743/quotes) --- *导出时间: 2026/5/21 09:57:07* --- ## 中文翻译 # 下一代 LLM 推理网络：ZCube 如何缓解网络瓶颈？ **作者**: Z.ai **日期**: 2026-05-20T21:47:33.000Z **来源**: [https://x.com/Zai_org/status/2057216685040443743](https://x.com/Zai_org/status/2057216685040443743) ---  LLM 推理正在重塑 AI 基础设施。网络曾经是推理集群中最不起眼的部分，但这已不再是事实。随着长上下文推理和 Prefill-Decode（预填充-解码）解耦逐渐成为标准，网络已经成为吞吐量、尾延迟和每 token 服务成本的关键路径。 为了应对在 Prefill-Decode 解耦部署中日益严重的拓扑引发的拥塞，Z.ai、Harnets.AI 与清华大学联合开发并在线上生产环境中部署了 ZCube 网络架构。部署表明，在网络架构层进行系统级创新，可以以高性价比的方式充分释放硬件潜力。 在 GLM-5.1 编码任务的生产基准测试中，仅通过架构优化，ZCube 就取得了显著收益： - 成本优化：在 GPU、软件栈和应用保持不变的情况下，交换机与光模块的资本支出减少了 33%。 - 吞吐量提升：平均 GPU 推理吞吐量提升了 15%。 - 延迟改善：TTFT P99 降低了 40.6%。 拥塞的根本原因在于推理流量模式的变化。随着 PD 解耦成为主流，跨节点 KV Cache 传输使得推理流量呈现高度不对称，并且源、目标和流量量均动态变化。在传统 ROFT（Rail-优化的 Fat-Tree）架构中，静态拓扑与端口映射容易将流量集中在少数交换机与链路上，导致局部热点、队列堆积和 PFC 反压。这会导致一个结构性问题：尽管聚合带宽看似充足，但局部拥塞却频繁发生。 ZCube 通过采用完全扁平化的网络拓扑，并结合单轨/多轨混合访问设计来解决这一问题。在网络架构层，它将 PD 流量解耦并分发到更广阔的路径空间，从源头降低拓扑引发拥塞的概率。这为下一代超大规模推理集群提供了更高效的网络基础。  图 1：ZCube 与 ROFT 相比有效避免拓扑引发的拥塞 # 网络成为高效推理的瓶颈 当数千块 GPU 并发服务在线推理请求时，每一次 KV Cache 传输和每一次数据同步操作都需跨越 GPU 互联网络。随着长上下文推理和 Prefill-Decode 解耦推理逐渐成为主流，Prefill 与 Decode 节点之间的数据交换持续增长。网络带宽，更重要的是有效使用带宽的能力，已开始直接影响集群级的吞吐量与延迟。 为了量化网络对推理性能的影响，我们在一个 512 GPU 的集群上进行了消融实验。我们保持 GPU 计算、软件栈、模型与应用逻辑不变，仅调整可用网卡带宽上限，随后测量整体集群吞吐量与 Time to First Token（TTFT）的变化。   图 2：不同网络带宽设置下的整体集群吞吐量与 TTFT 例如，当网络带宽从 100Gbps 提升到 200Gbps 时，整体推理吞吐量提升了约 19%，而 Time to First Token（TTFT）降低了约 22%。这表明，在 LLM 推理中，网络带宽已成为制约服务性能的关键因素之一。 # 1. 推理中的网络拥塞 目前，AI 集群普遍采用 Clos 或 Fat-Tree 架构。其基本思想是通过堆叠多层交换机来扩展网络规模。然而，Clos 网络的性能高度依赖于交换机间的理想负载均衡，而在实践中，由于路由策略和实际流量模式，这种均衡很难实现。 例如，在许多由 Spine 与 Leaf 层构成的两层 Fat-Tree 部署中，Spine 交换机之间的流量会出现严重不均衡。结果，上层应用往往无法获得预期的网络性能。 为减少跨层转发的开销，业界常采用 ROFT（Rail-优化的 Fat-Tree）架构[1]。如图 3 所示，ROFT 按“轨”索引对 GPU 进行分组，并将相同索引的 GPU 连接到同一台 Leaf 交换机，从而减少 Spine 交换机之间的通信代价。  图 3：在 ROFT 中，Leaf 交换机之间的流量易出现不均衡 ROFT 对于某些训练流量模式效果良好。但在 Prefill-Decode 解耦推理中，我们观察到一个更为突出的问题：KV Cache 传输表现出强烈的源-目标不对称性。不同 GPU 与不同网卡所承载的通信负载极不均衡，如图 4 所示。因此，ROFT 的“轨”映射不再自然转化为负载均衡。相反，流量可能集中在少数 Leaf 交换机与链路上，导致链路拥塞并降低传输性能。 这具体表现在以下方面： - 部分 Leaf 交换机成为持续的负载热点，增加了多个 KV Cache 传输流在同一条链路上竞争的概率。结果，实际传输吞吐量可能远低于网卡带宽能力。 - 某些 Leaf 交换机的出方向队列在较长时间内保持较高深度，并频繁触发 PFC 反压，如图 5 所示。 - 链路拥塞进一步放大尾延迟，影响 TTFT 与整体吞吐量。  图 4：同一机器上不同网卡之间的 KV Cache 传输负载不均衡  图 5：特定 Leaf 交换机端口上频繁的 PFC 暂停事件 区分图 6 所示的两种网络拥塞至关重要： - 不可避免的拥塞：例如，当多个 GPU 同时向同一目标发送数据时，最后一跳链路的竞争在所难免。 - 可避免的拥塞：由拓扑设计、流量映射或多路径利用不均导致。从根本上看，这是架构层的设计问题。  图 6：两类网络拥塞的示意 对于第一类拥塞，我们通常依赖拥塞控制、流量整形等相关机制来减轻影响。对于第二类拥塞，自适应路由[2]、数据包喷洒[3,4]和 MRC[5]等新型网络传输机制可提供帮助。然而，更有效的方式是在网络架构层进行创新，从一开始就避免不应发生的网络冲突。 Prefill-Decode 解耦推理就是典型案例。如果网络拓扑与流量模式不匹配，系统将反复产生负载热点与链路冲突。解决这一问题需要重新思考推理网络架构本身。 # 2. ZCube 网络架构 为解决上述问题，我们部署了全新的 ZCube 网络架构[6]。ZCube 摒弃了传统 Clos 的分层交换机堆叠设计理念，引入了完全扁平化的 GPU 服务器互联。 专为 ZCube 架构设计的 ZCube 路由策略充分利用扁平化拓扑的结构特性，可在所有交换机之间实现接近理想的负载均衡，从而显著提升集群整体网络带宽。 相较于 Clos，ZCube 在负载均衡方面具有天然优势，这对训练集群与推理集群均有所助益。重要的是，在取得这些性能收益的同时，ZCube 的交换机与光模块成本比 Clos 低约三分之一。基于当前主流交换机与网卡配置，ZCube 可支持数万乃至数十万 GPU 的扁平化组网。 ## 2.1 ZCube 核心架构 如图 7 所示，ZCube 的核心思路包括： 1. 移除 Spine 交换机层。 2. 将 Leaf 交换机分为规模相等的两组，通常为奇数编号交换机与偶数编号交换机。 3. 在这两组交换机之间建立完全二部互连。 4. 采用单轨与多轨混合的访问模式，将每块 GPU 网卡的两个端口分别连接到两组中的对应交换机。  图 7：ZCube 架构概览  假设每块 GPU 对应一块拥有两个端口的网卡，即 p=2。共有 n 个 GPU，且 GPU 与网卡共享相同的索引：1,2,…,n。设 k 为每台交换机连接的 GPU 数量。交换机总数为 2n/k，编号为 1,2,…,2n/k。对于 GPU i（1≤i≤n）： - 第一个端口连接到奇数编号交换机： ((i−1)mod(n/k))×2+1 - 第二个端口连接到偶数编号交换机： ⌈i/k⌉×2 两组交换机之间以完全二部图连接：每一台奇数编号交换机与所有偶数编号交换机相连。 图 7 展示了双端口网卡配置下的 ZCube 拓扑，其中 p=2，n=32，k=8。 ## 2.2 ZCube 的关键特性 网络直径 ZCube 的网络直径为两跳，即任意一对 GPU 之间均可通过两台交换机互通。这介于仅一跳但规模有限的单层交换机网络，以及通常需要三跳且延迟更高的传统两层交换机网络之间。 负载均衡 首先，ZCube 路由策略确保每一对 GPU 都有一条唯一的最优路径，避免多径路由选择导致的流量冲突。 其次，ZCube 采用两种互补的 GPU-交换机连接模式：一组交换机以单轨模式连接 GPU，每台交换机连接一段连续的 GPU ID；另一组交换机以多轨模式连接 GPU，每台交换机连接各组内具有相同相对索引的 GPU。 该设计使 ZCube 无论是在典型的 AI 训练流量（如 AllReduce、All-to-All），还是在源-目标关系不确定、网卡负载高度不均衡的典型 AI 推理流量下，均能在整个交换机平面内实现高效负载均衡。 因此，ZCube 能够在架构层避免前述第二类网络拥塞。如图 8 所示，在 ROFT 下会产生冲突的流量，在 ZCube 下可获得独立网络路径，从而避免拥塞。  图 8：ZCube 架构下的负载均衡 扩展性 ZCube 在保持良好性能特征的同时，提供强大的扩展能力。例如，使用一层 51.2T 交换机（每台 128 × 400Gbps 端口），ZCube 可构建一个连接 16,384 个 400Gbps 网卡的网络。若采用更高容量交换机，或将 ZCube 网络划分为更多平面，该架构还可进一步扩展，以支持数万乃至数十万 GPU 的互联。 成本 在相同集群规模下，与传统 Clos / ROFT 架构相比，ZCube 可将交换机与光模块成本降低约三分之一。以 10,000 GPU 的 AI 集群为例，ZCube 可在网络硬件投资上节省约 2.1 亿至 6.4 亿元人民币。这些特性表明，ZCube 可以在更低的网络硬件成本下实现更佳的负载均衡与性能。 ## 2.3 真实集群测试：在降低网络成本的同时提升推理性能 我们将运行 GLM-5.1 编码推理服务的一个千卡集群的网络架构从原有 ROFT 升级为 ZCube 架构。由于 ZCube 架构移除了传统 Clos 架构中的 Spine 层交换机，Clos 框架下原有的布线模式、IP 地址规划、路由策略以及交换机配置方法无法直接复用，需要针对 ZCube 进行全面重新设计。 为应对这些挑战，Harnets.AI 网络团队设计了以 ZCube 架构为中心的整套网络解决方案，开发了包括 ZCube 控制器、数据中心布局设计工具和布线正确性验证程序在内的自动化工具集，实现了数据中心部署规划、布线验证、配置自动生成与批量部署，有效解决了 ZCube 部署中的诸多难题。这套工具是能够在极短时间内完成大规模生产集群成功升级的关键因素。 在网络架构平滑迁移完成后，我们在该集群上运行 GLM-5.1 编码推理服务，对 ZCube 架构进行真实测试。通过对比升级前后的推理性能，我们发现：与 ROFT 架构相比，ZCube 将平均 GPU 推理吞吐量提升了 15% 以上，并将 TTFT 的 P99 尾延迟降低了 40.6%（如图 9 所示）。   图 9：同一集群下 ZCube 与 ROFT 架构的吞吐量与 TTFT 对比 总之，在相同规模与配置的 GPU 与服务器硬件、且不修改任何应用的前提下，将网络架构升级到 ZCube，不仅节省了 1/3 的光模块与交换机硬件，还能使集群每秒多处理 15% 的推理请求。在当前推理需求爆发、算力资源严重短缺的背景下，这种方法务实且价值显著。目前，该 ZCube 集群已稳定运行超过两周，为 GLM-5.1 编码推理服务提供了关键支撑。 # 3. 结论 LLM 推理正在从单点优化转向系统级协同设计。网络与推理引擎的耦合愈发紧密，网络已成为推理系统的关键组成部分。ZCube 的生产部署表明，网络架构创新可以直接释放推理系统的有效容量。通过使网络架构更契合 KV Cache 传输与 PD 流量模式，ZCube 从源头减少拓扑引发拥塞的概率，在提升吞吐量、降低延迟的同时，提高集群成本效率。 展望下一代 LLM 基础设施，网络设计将从通用互联向……
L LLM 推理原理详解：从 Prefill 到 Decode 本文深入解析了大语言模型（LLM）推理的计算流程。文章指出，LLM 推理主要包含 Prefill（处理提示词，计算密集型）和 Decode（逐词生成，显存带宽密集型）两个阶段。作者详细阐述了分词、Embedding、Transformer 层及 KV Cache 的工作原理，并分析了 KV Cache 带来的内存挑战及 vLLM 的优化方案。最后，探讨了 DeepSeek-V4 等新架构通过重新设计注意力机制来从根本上压缩 KV Cache 的创新思路。 技术 › LLM ✍ Avi Chawla🕐 2026-06-29 LLM推理KV CachePrefillDecodeDeepSeekTransformer性能优化
L LLM 优化面试笔记：训练与推理核心技术 这是一份针对 AI 实验室面试准备的笔记，涵盖了高效训练和部署大语言模型（LLM）的核心优化策略。内容主要分为三部分：内存优化（如 Flash Attention、MQA/GQA、激活检查点）、计算优化（序列打包、高效变体 Transformer）以及推理优化（KV 缓存、投机解码、量化技术）。文章旨在总结在大规模模型开发中应对算力和内存瓶颈的关键技术。 技术 › LLM ✍ Gauri Gupta🕐 2026-05-06 LLM优化推理训练Flash AttentionKV Cache量化面试Transformer性能优化
大 大语言模型训练与服务背后的数学原理 本文基于 Reiner Pope 的播客内容，深入剖析了大模型在集群中的运行机制。文章通过 Roofline 分析和 KV Cache 等核心概念，解释了推理延迟、Batch Size、API 定价策略及 MoE 架构背后的物理与经济逻辑，揭示了硬件限制如何塑造 AI 进展。 技术 › LLM ✍ Saito🕐 2026-05-01 LLM推理架构设计数学原理MoEAPI定价性能优化Transformer
K K3 部署推理引擎指南 vLLM 宣布对 Kimi K3 模型提供首日支持。K3 是拥有 2.8 万亿参数的 MoE 模型，支持 100 万上下文窗口。vLLM 通过 DSpark 推测解码将吞吐量提升至 370 tok/s，并支持架构创新的混合缓存管理、预填充/解码分离及智能体服务，确保生产环境下的高性能与低延迟。 技术 › 后端 ✍ vLLM🕐 2026-07-28 vLLMKimi K3MoE推理引擎性能优化DSpark部署指南大模型基础设施
构 构建优秀的垂直领域 Agent：以 Shortcut 为例 本文探讨了如何构建一个在实际应用中表现卓越的垂直领域 Agent。作者结合开发 Shortcut 电子表格 Agent 的经验，提出核心原则是“对任务分布的忠实压缩”。文章详细介绍了“分层缓存”架构，将上下文分为 L1（常驻核心）、L2（按需规格）和 L3（兜底参考），以平衡效率与准确性。同时强调了“单工具优于多工具”的策略，主张通过代码执行来统一能力，降低模型决策复杂度，从而在长尾任务中实现高准确率。 技术 › Agent ✍ Peter Wang🕐 2026-06-12 AgentLLM架构设计上下文管理垂直领域性能优化Tool Use代码执行缓存策略
为 为什么你的agent项目难以落地：论Agent Harness的重要性 本文探讨了Agent项目难以落地的核心原因。作者指出，许多团队忽视了模型与应用之间的基础设施层——Agent Harness。文章通过生理学类比和工程细节分析，阐述了为什么仅靠LLM无法构建高效Agent，并预测AI行业将分化为模型层、Harness平台层和Agent应用层，建议业务团队应专注于业务逻辑而非底层基础设施。 技术 › Agent ✍ North@CreaoAI🕐 2026-05-25 AgentHarness工程架构LLM落地实践行业观察基础设施CreaoAI
L LLM 原理与实践指南 (2026 版) 这是一份关于大语言模型（LLM）的实践指南。文章从基础循环机制出发，详细解释了文本如何转化为 Token，以及 Transformer、注意力机制、KV Cache 和 RoPE 等核心概念的工作原理。作者强调理解模型内部的推理机制（如 Prefill 和 Decode 阶段）对于硬件选型、显存估算和本地部署的重要性。文章涵盖了 Token 化、上下文窗口、量化、模型服务及本地 AI 硬件数学计算等关键主题，旨在为深入理解 LLM 提供直观的基础。 技术 › LLM ✍ Ahmad🕐 2026-05-23 LLMTransformer本地部署Tokenization推理Attention硬件教程KV Cache
单 单智能体与多智能体解决方案的选择指南 文章探讨了在大语言模型时代，如何权衡使用单智能体与多智能体系统。引用斯坦福及Google/MIT的研究指出，多智能体系统并非总是更优，往往伴随着更高的计算成本、延迟和通信开销，且在“思考预算”相当的情况下，优化后的单智能体常能胜出。文章建议将单智能体作为默认起点，仅当任务涉及大量工具、上下文退化严重、存在自然解耦边界或需严格合规验证时，才考虑引入多智能体架构。 技术 › Agent ✍ AlphaSignal AI🕐 2026-05-07 AgentLLM系统架构多智能体性能优化StanfordGoogleMIT决策框架
光 光互联深度：当算力撞上物理边界，人类又一次问光 文章深入探讨了 AI 时代算力发展面临的物理瓶颈，指出光互联是解决 GPU 之间、机柜之间高速通信的关键路径。随着 AI 进入推理时代，大模型长上下文和 Agent 任务对带宽和延迟提出了极高要求，英伟达的 NVLink 协议虽然建立了深厚护城河，但铜缆技术已达极限，产业正加速向光互连（如 CPO、硅光子）转型。文中还详细对比了 Scale-up 与 Scale-out 架构，以及 UALink 开放联盟对英伟达的挑战。 投资 › 宏观产业 ✍ 大宇🕐 2026-05-04 AI光互联芯片投资英伟达半导体技术趋势Scale-up推理网络架构NVLink
软 软件的终结与新生：动态软件时代的到来 文章提出了软件正在经历从静态到动态的范式转变。过去的五十年，软件基于硬编码的控制流，追求确定性；而随着 2024 年 AI 的发展，软件变得“活着”了。动态软件像现场演出而非录音，具有上下文感知能力和非确定性。作者探讨了这一变革打破了关于状态、会话、时间和确定性等旧有假设，并指出当前的挑战在于缺乏类似 Django 或 Kubernetes 的成熟“运行时”基础设施。 技术 › LLM ✍ Ashpreet Bedi🕐 2026-05-01 AI Agent软件架构基础设施动态软件行业观察LLM编程范式Agno软件工程未来趋势
家 家庭实验室进化为私有云：基于 Proxmox 的架构实践 本文详细介绍了作者如何将原本杂乱的家庭实验室改造为具备云原生特性的私有控制平面。文章指出，核心差异在于引入了访问层、网关、路由策略、可搜索状态和计算通道，而非仅仅依赖本地硬件。作者利用 Proxmox 作为底层基础架构，通过 WireGuard 实现安全的 VPN 入口，结合 SNAT、私有 DNS/DHCP 等技术，构建了一个包含自动化平面、状态存储和计算织带的综合系统，实现了对数十个 LXC 容器和 VM 的统一、高效管理。 技术 › DevOps ✍ Ahmad🕐 2026-05-01 HomelabProxmoxWireGuard私有云网络架构DevOpsVPN基础设施LinuxSNAT
T The Harness Is the Backend 文章探讨了当前 AI 架构中 Harness（编排层）与后端基础设施分离带来的问题，指出这种分离导致了 Agent 的随机性与后端的确定性之间的调试复杂性。作者提出将后端抽象为 Worker、Trigger 和 Function 三个基本元素，介绍了名为 'iii' 的架构，使 Agent 作为 Worker 直接参与后端系统，而非通过外部集成层调用，从而实现真正的统一基础设施。 技术 › Harness Engineering ✍ Mike Piccolo🕐 2026-04-29 Agent架构后端LLM分布式系统Harness基础设施