# Inference Engines for LLMs & Local AI Hardware (2026 Edition)
**作者**: Ahmad
**日期**: 2026-05-20T19:37:06.000Z
**来源**: [https://x.com/TheAhmadOsman/status/2057183854444843202](https://x.com/TheAhmadOsman/status/2057183854444843202)
---
> You don't pick an inference engine first. You pick a hardware strategy, a workload shape, and a serving model. The engine follows.
That is the most useful way to think about LLM inference engines.
Series note: This is Part 3 in my series teaching Self-hosted LLMs / Local AI.
- Part 1: GPU Memory Math for LLMs (2026 Edition).
- Part 2: Memory Bandwidth for Local AI Hardware (2026 Edition).
Those two pieces explain the hardware capacity and bandwidth math.
This one explains the software layer that turns that hardware into usable inference.
## Engines
These tools serve different purposes / occupy different layers
- Local portability
- Consumer CUDA
- Apple unified-memory workflows
- Quantized inference
- Production serving
- Distributed orchestration
- Vendor-optimized datacenter execution
A useful mental model:
The inference engine is not "the model." It is the traffic cop, memory manager, kernel dispatcher, scheduler, cache accountant, parallelism planner, API surface, and sometimes the deployment framework.
The best engine matches your memory hierarchy, interconnect, quantization format, latency and throughput targets, model architecture, and operational maturity.
## The one-page decision guide
- Laptop / edge / odd hardware → llama.cpp
- Mac-first workflows → MLX / MLX-LM
- Single RTX local inference → ExLlamaV2
- 2-4+ NVIDIA / CUDA GPUs → ExLlamaV3
- General production serving → vLLM
- Long-context / MoE / routing → SGLang
- NVIDIA max performance → TensorRT-LLM
- Cluster orchestration → NVIDIA Dynamo
The rest of this guide explains why.
## What an inference engine actually does
An inference engine loads weights, tokenizes input, runs the forward pass, samples tokens, maintains the KV cache, and streams results. Serious engines also handle batching, scheduling, prefix caching, quantization, parallel execution, API serving, metrics, and distributed execution.
The workload has two phases:
Prefill reads the prompt and builds the initial KV cache. It is compute-intensive.
Decode generates one token at a time, repeatedly reading weights and KV cache. It is memory-bandwidth-bound. Decode speed tracks memory bandwidth more than peak compute.
That distinction explains almost everything:
- Short prompt, long answer: decode dominates → memory bandwidth and batching matter.
- Long prompt, short answer: prefill dominates → attention kernels and chunked prefill matter.
- Many users: scheduler quality matters → continuous batching, cache paging, fairness.
- Long context: KV cache dominates → paged attention, KV quantization, offload.
- MoE: expert routing dominates → expert parallelism, interconnect, grouped GEMMs.
- Multi-node: interconnect dominates → NVLink, RDMA, pipeline parallelism, disaggregation.
PagedAttention tackled KV cache fragmentation. FlashAttention used IO-aware tiling to cut HBM (High Bandwidth Memory) traffic. Speculative decoding drafts cheap tokens and verifies them in parallel. The recurring theme: inference performance is memory movement plus scheduling.
## The real bottlenecks
1. Memory bandwidth, not just VRAM size. VRAM determines fit. Bandwidth determines decode speed. Apple's M3 Ultra offers up to 819 GB/s unified-memory bandwidth. NVIDIA's H100 SXM lists 3.35 TB/s GPU memory bandwidth. Unified memory lets you fit models that would not fit in consumer VRAM. HBM lets you serve them faster when the model fits. Fit is not speed. Capacity is not bandwidth.
2. KV cache growth. KV cache grows with batch size and context length. Long-context workloads can run out of memory even when weights fit. PagedAttention partitions the KV cache into blocks, increasing utilization and supporting larger batches.
3. Interconnect. The moment a model crosses GPU boundaries (multi-GPUs), you pay communication cost. Tensor parallelism needs frequent all-reduce collectives. Pipeline parallelism communicates at stage boundaries. Expert parallelism needs all-to-all traffic for MoE. vLLM's docs note that without NVLink, pipeline parallelism can outperform tensor parallelism.
4. Scheduler quality. A good scheduler decides which requests enter the batch, how prefill and decode share the accelerator, whether long prompts block short decodes, and how to avoid starvation. Supporting batching is not the same as behaving like a production-ready scheduler.
5. Runtime overhead. CUDA graphs, kernel fusion, sampling overhead, tokenizer overhead, HTTP overhead, LoRA switching, and structured decoding all matter. At high scale, the annoying 2% overheads form a union and demand attention (no punt intended).
## The engine families
There are four broad families:
Portable local runtimes: llama.cpp, MLC LLM, ONNX Runtime GenAI, OpenVINO, Ollama-style tools. These care about "make it run here."
Apple/unified-memory runtimes: MLX and MLX-LM. These care about "use big shared memory and Apple's stack well."
Consumer CUDA quant engines: ExLlamaV2 and ExLlamaV3. These care about "make my 3090/4090/5090 box scream with low-bit weights."
Production serving engines: vLLM, SGLang, TensorRT-LLM, TGI, LMDeploy. These care about concurrent users, KV cache, batching, parallelism, observability, and cost per token.
Then there are orchestration layers like Dynamo that sit above engines and coordinate fleets, disaggregated prefill/decode, routing, and autoscaling.
## llama.cpp: the portability king
llama.cpp is the answer when the hardware is weird, constrained, offline, CPU-heavy, edge-oriented, or not a tidy NVIDIA datacenter node.
It supports Apple Silicon via ARM NEON, Accelerate, and Metal; x86 via AVX/AVX2/AVX512/AMX; RISC-V; low-bit quantization; CUDA; AMD via HIP; MUSA; Vulkan; SYCL; and CPU+GPU hybrid offload. That is why llama.cpp owns the "just make it run" lane.
The HTTP server is more capable than a "toy local runner". llama-server provides OpenAI-compatible routes, Anthropic Messages API compatibility, reranking, continuous batching, multimodal support, JSON schema constraints, function calling, speculative decoding, and a web UI.
The critical limitation: llama.cpp is not for serious multi-node production serving. Its RPC backend is explicitly documented as proof-of-concept, fragile, and insecure.
Verdict: Use llama.cpp when portability, offline operation, GGUF, or hybrid offload matter more than fleet-scale serving.
DO NOT use with Multi-GPUs
## MLX and MLX-LM: the Apple Silicon weapon
MLX is Apple's array framework for Apple Silicon, and MLX-LM is the LLM package built on it. It is a Mac-first ML stack.
The key hardware fact is unified memory. Apple Silicon gives the CPU and GPU direct access to the same memory pool. MLX arrays live in unified memory, and you choose the device when running the operation rather than moving arrays between separate memory spaces.
This changes the local inference tradeoff. On a discrete GPU system, the question is "does it fit in VRAM?" On an M-series Mac with large unified memory, the question becomes "does it fit in memory, and can the memory system feed the GPU fast enough?" Large quantized models can fit on machines where the same model would be impossible on a 24 GB consumer GPU.
However, it is also slower.
MLX-LM adds Hugging Face Hub integration, quantization, LoRA and full fine-tuning, distributed inference, and a large MLX Community model ecosystem. MLX is no longer Mac-only: it offers CUDA and CPU-only packages for Linux. Distributed communication supports MPI, Ring over TCP, JACCL for RDMA over Thunderbolt, and NCCL for CUDA.
MLX-LM's server itself warns that it is not recommended for production because it only implements basic security checks.
Verdict: Use MLX for Mac-first ML and LLM workflows. For high-concurrency public serving, start with a real serving stack.
## ExLlamaV2 and V3: consumer CUDA, tuned and fast
ExLlamaV2 is the local CUDA quantization engine for people who want a consumer NVIDIA GPU to punch above its weight. It supports paged attention, dynamic batching, prompt caching, KV cache deduplication, batched generation, streaming, and speculative decoding. The word to remember is local. It makes quantized models fast on modern CUDA GPUs, especially consumer cards.
Best fits: one RTX 3090/4090/5090 box, local coding assistant, local chat, EXL2 quantized models, and prosumer workstation use.
ExLlamaV3 extends the philosophy toward multi-GPU and MoE-local inference. It adds the EXL3 quantization format based on QTIP, flexible tensor-parallel and expert-parallel inference for consumer hardware, an OpenAI-compatible server through TabbyAPI, continuous dynamic batching, and multimodal support.
V3 is compelling when you have 2-4+ consumer NVIDIA GPUs or want local MoE. Expect caveats: some models do not support tensor or expert parallelism in ExLlamaV3.
Verdict: ExLlamaV2 is the enthusiast's local CUDA engine. ExLlamaV3 is the frontier for multi-GPU (2-4) local setups. Expect rougher edges for better capability.
## vLLM: the default open-source production server
vLLM is the first engine most teams should evaluate for serious opensource LLM serving.
It offers PagedAttention-based KV memory management, continuous batching, chunked prefill, prefix caching, CUDA/HIP graphs, extensive quantization (FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF), optimized attention and GEMM/MoE kernels, speculative decoding, torch.compile, and disaggregated prefill/decode/encode.
It is also flexible: tensor/pipeline/data/expert/context parallelism, streaming, structured outputs, tool calling, OpenAI-compatible and Anthropic Messages APIs, gRPC, multi-LoRA, and support for NVIDIA, AMD, x86/ARM/PowerPC CPUs, plus plugins for TPUs, Gaudi, Ascend, Apple Silicon, and more.
vLLM's docs note that multi-node deployments typically use Ray, and without NVLink, pipeline parallelism may beat tensor parallelism. The trap is assuming vLLM removes the need for systems thinking. You still need to tune batching, context length, GPU memory utilization, parallelism layout, and routing. vLLM gives you a very good engine; it still requires good System Design.
Verdict: If someone says "we need to serve open models in production," vLLM is the default starting point.
## SGLang: vLLM's systems-brained cousin
SGLang is what you reach for when the serving workload is ugly: structured outputs, long context, MoE, disaggregation, and routing.
It offers RadixAttention prefix caching, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, tensor/pipeline/expert/data parallelism, structured outputs, chunked prefill, and multi-LoRA batching. It supports NVIDIA, AMD, Intel Xeon, Google TPUs, Ascend NPUs, and more.
SGLang's differentiator is serving architecture. Its prefill-decode disaggregation separates compute-intensive prefill from memory-intensive decode into specialized instances, transferring KV cache between them. This prevents long prefill batches from interrupting decode and spiking token latency.
Verdict: SGLang is for teams whose bottleneck is no longer "can we run the model?" but "can we run it under hostile traffic without torching latency, memory, and cost?"
## TensorRT-LLM: maximum NVIDIA performance
TensorRT-LLM is the NVIDIA-max-performance stack. It is optimized, specialized, powerful, and not pretending to be portable.
It provides Python APIs to build TensorRT engines with state-of-the-art optimizations, plus Python and C++ runtimes. It includes custom kernels for attention, GEMMs, and MoE; prefill-decode disaggregation, Wide Expert Parallelism, speculative decoding; and a high-level Python API integrated with NVIDIA Dynamo and Triton Inference Server.
B200 GPUs can load FP4 weights with optimized kernels. H100 and later support FP8 quantization that can double performance and halve memory consumption versus 16-bit with minimal accuracy loss.
Where it shines: H100/H200/B200/GB200/GB300-class fleets, NVIDIA-only datacenters, FP8/FP4 deployment, multi-node serving, and MoE at scale. Where it is awkward: AMD, Apple, or Intel portability; fast-changing experimental models; small local setups; and teams that need "works on everything."
Verdict: If you are committed to NVIDIA and care about absolute performance, TensorRT-LLM belongs in the bake-off. You trade portability for performance. Tuned specialization but less features.
## The rest of the field
TGI is Hugging Face's production server with tracing, metrics, tensor parallelism, and continuous batching. Use it when HF integration and simplicity matter.
MLC LLM is the compiler-first universal deployment engine with OpenAI-compatible APIs across REST, Python, JavaScript, iOS, and Android. Best for "ship LLMs everywhere," especially browser, mobile, and native apps.
ONNX Runtime GenAI implements the full generative loop over ONNX Runtime and powers Foundry Local, Windows ML, and the VS Code AI Toolkit. It supports CPU, CUDA, DirectML, TensorRT-RTX, OpenVINO, QNN, WebGPU, and AMD GPU. Best for app deployment and ONNX workflows.
OpenVINO GenAI is the Intel-optimized story for Xeon CPUs, Arc GPUs, Core Ultra, and NPUs. It offers OpenAI-compatible serving with continuous batching and paged attention. Best for Intel hardware.
LMDeploy is a CUDA-focused toolkit with TurboMind for performance and PyTorch for accessibility. Most interesting for CUDA users who want an alternative to vLLM/SGLang/TensorRT-LLM.
NVIDIA Dynamo is a distributed orchestration layer above engines like vLLM, SGLang, and TensorRT-LLM, supporting disaggregation, intelligent routing, and multi-tier KV caching. Use it when single-engine serving is no longer enough.
Note: DO NOT USE Ollama.
## Hardware strategy recipes
CPU-only server: llama.cpp first. OpenVINO for Intel Xeon. ONNX Runtime GenAI for app/ONNX deployment.
MacBook / Mac Studio: MLX / MLX-LM for Mac-native workflows. llama.cpp for GGUF portability.
Single RTX 3090 / 4090 / 5090: ExLlamaV2 for EXL2 local inference. llama.cpp for GGUF or portability. vLLM if serving multiple users.
Dual or quad consumer RTX box: ExLlamaV3 for multi-GPU quantized inference or MoE. vLLM if serving behavior matters. SGLang if testing routing or long-context patterns.
8×H100 / H200 node: Start with vLLM or SGLang. Benchmark TensorRT-LLM if NVIDIA-only and performance justifies tuning. Use Dynamo when multi-node orchestration becomes necessary.
B200 / GB200 / GB300-class infrastructure: Benchmark TensorRT-LLM, SGLang, and vLLM. Add Dynamo for fleet-level orchestration, KV-aware routing, and autoscaling.
AMD MI300 / MI325 / MI350 / MI355: Start with vLLM or SGLang on ROCm. Avoid assuming NVIDIA benchmarks transfer cleanly.
Intel Xeon / Core Ultra / Arc: OpenVINO GenAI or OpenVINO Model Server. ONNX Runtime GenAI if app embedding matters.
Browser, mobile, app-native: MLC LLM / WebLLM or ONNX Runtime GenAI.
## Benchmarking: what to measure
Bad benchmark: "I got 180 tok/s."
Good benchmark includes:
Model: exact model, architecture, parameter count, active MoE params.
Weights: dtype, quant format, group size, calibration.
Engine: version, commit, backend, flags.
Hardware: GPU SKU, memory capacity, bandwidth, interconnect, CPU, RAM.
Workload: input/output length distributions, concurrency, streaming, shared prefixes, structured output.
Metrics: TTFT, TPOT, end-to-end latency, p50/p95/p99, tokens per second, requests per second, GPU memory usage, KV cache hit rate, prefill throughput, decode throughput, cost per 1M tokens.
Benchmarking Rules:
1. Never compare engines using only single-user tokens per second.
2. Test your actual prompt and output distribution.
3. Test with realistic concurrency.
4. Separate prefill from decode.
5. Track p95 and p99, not only averages.
6. Measure memory headroom at target context length.
7. Test cache reuse if your app has repeated prefixes.
8. Benchmark structured output separately; grammar adds overhead.
9. Benchmark LoRA and multi-LoRA separately.
10. Re-test after driver, CUDA, ROCm, model, or engine upgrades.
## Common mistakes
Choosing by VRAM capacity alone. VRAM determines fit. Bandwidth and scheduler determine speed. A large unified-memory machine can fit huge models, but an H100 decodes faster when the model fits due to much higher HBM bandwidth.
Using tensor parallelism on weak interconnect. Without NVLink or NVSwitch, test pipeline parallelism. vLLM's docs call this out for L40S-like setups.
Ignoring KV cache. Long context and concurrency can make KV cache the limiting factor. PagedAttention, prefix caching, KV quantization, and disaggregation are not optional at scale.
Treating local engines as production servers. llama.cpp server is capable. MLX-LM server is convenient. Ollama is pleasant yet SHOULD NOT BE USED.
However, production means security, observability, backpressure, routing, autoscaling, and SLA behavior. MLX-LM itself warns that its server is not recommended for production.
Assuming every quantization format is portable. GGUF, EXL2, EXL3, AWQ, GPTQ, FP8, FP4, MLX formats, and ONNX are not interchangeable. The right format is the one your engine has optimized kernels for.
Ignoring model architecture. Dense models, MoE, hybrid attention, multimodal models, and long-context variants stress different parts of the engine. Broad support does not mean every optimization works equally.
Trusting benchmark charts without workload shape. A chart for Llama 3.1 8B at 1K input / 128 output says little about a coding agent with 80K context running on Qwen 3.6 27B / Gemma 4 26B-A4B, or a RAG service with 500 concurrent users.
## The opinionated final map
Local AI user: LM Studio or Harbor for convenience. llama.cpp for control. MLX on Mac. ExLlamaV2/V3 for CUDA local performance.
Building a local agent: Any should work, but given what most people use; llama.cpp for portability. MLX if users are on Apple Silicon. vLLM if simulating production serving locally.
Serving an internal team: Start with vLLM. Use SGLang if structured outputs, long context, multi-LoRA, MoE, or routing matter.
Serving customers at scale: Benchmark vLLM, SGLang, and TensorRT-LLM. If routing and disaggregation matter, SGLang and Dynamo deserve attention.
NVIDIA datacenter: TensorRT-LLM for max performance. vLLM for flexibility. SGLang for complex serving. Dynamo for fleet orchestration.
Apple Silicon: MLX for native development. llama.cpp for GGUF. Unified memory is a capacity superpower with bandwidth tradeoffs, not HBM.
Edge, app, browser, or Windows-native: llama.cpp, MLC LLM, ONNX Runtime GenAI, or OpenVINO, depending on stack.
## Final principle
Inference Engines have consequences.
Pick the engine after answering these:
1. What hardware do I actually have?
2. Does the model fit in fast memory, or only in system/unified memory?
3. Is decode or prefill the bottleneck?
4. What context length and concurrency matter?
5. Are prompts shared enough for prefix caching?
6. Is the model dense, MoE, multimodal, or hybrid?
7. Do I need local convenience, production serving, or fleet orchestration?
8. What quantization format has optimized kernels on my target engine?
9. Is my interconnect PCIe, NVLink, NVSwitch, Ethernet, RDMA, or Thunderbolt?
10. Am I optimizing latency, throughput, cost, privacy, portability, or developer speed?
The engine follows the answers.
Until next time.
-Ahmad
## 相关链接
- [Ahmad](https://x.com/TheAhmadOsman)
- [@TheAhmadOsman](https://x.com/TheAhmadOsman)
- [29K](https://x.com/TheAhmadOsman/status/2057183854444843202/analytics)
- [GPU Memory Math for LLMs (2026 Edition)](https://x.com/TheAhmadOsman/status/2040103488714068245)
- [Memory Bandwidth for Local AI Hardware (2026 Edition)](https://x.com/TheAhmadOsman/status/2041331757329285589)
- [Multi-GPUs](https://www.ahmadosman.com/blog/do-not-use-llama-cpp-or-ollama-on-multi-gpus-setups-use-vllm-or-exllamav2/)
- [Harbor](https://github.com/av/harbor)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [3:37 AM · May 21, 2026](https://x.com/TheAhmadOsman/status/2057183854444843202)
- [29.7K Views](https://x.com/TheAhmadOsman/status/2057183854444843202/analytics)
- [View quotes](https://x.com/TheAhmadOsman/status/2057183854444843202/quotes)
---
*导出时间: 2026/5/21 10:26:52*
---
## 中文翻译
# 面向 LLM 与本地 AI 硬件的推理引擎（2026 版）
**作者**: Ahmad
**日期**: 2026-05-20T19:37:06.000Z
**来源**: [https://x.com/TheAhmadOsman/status/2057183854444843202](https://x.com/TheAhmadOsman/status/2057183854444843202)
---
> 不要先选推理引擎。你应该先选硬件策略、工作负载形态和服务模式。引擎是随之确定的。
这是思考 LLM 推理引擎最有用的方式。
系列说明：这是我的自托管 LLM / 本地 AI 教程系列的第 3 部分。
- 第 1 部分：LLM 的 GPU 显存计算（2026 版）。
- 第 2 部分：本地 AI 硬件的内存带宽（2026 版）。
前两篇文章解释了关于硬件容量和带宽的计算原理。
这一篇则解释将上述硬件转化为可用推理能力的软件层。
## 引擎
这些工具服务于不同的目的 / 占据不同的层级
- 本地可移植性
- 消费级 CUDA
- Apple 统一内存工作流
- 量化推理
- 生产级服务
- 分布式编排
- 厂商优化的数据中心执行
一个有用的心智模型：
推理引擎不是“模型”。它是交通指挥、内存管理器、内核调度器、任务调度器、缓存会计、并行规划器、API 接口，有时甚至是部署框架。
最好的引擎应与你的内存层级、互连方式、量化格式、延迟和吞吐量目标、模型架构以及运营成熟度相匹配。
## 一页决策指南
- 笔记本 / 边缘设备 / 奇怪硬件 → llama.cpp
- Mac 优先的工作流 → MLX / MLX-LM
- 单卡 RTX 本地推理 → ExLlamaV2
- 2-4+ 张 NVIDIA / CUDA GPU → ExLlamaV3
- 通用生产服务 → vLLM
- 长上下文 / MoE / 路由 → SGLang
- NVIDIA 极致性能 → TensorRT-LLM
- 集群编排 → NVIDIA Dynamo
本指南的其余部分将解释原因。
## 推理引擎实际在做什么
推理引擎负责加载权重、对输入进行分词、运行前向传播、采样 token、维护 KV 缓存以及流式输出结果。成熟的引擎还需要处理批处理、调度、前缀缓存、量化、并行执行、API 服务、指标监控和分布式执行。
工作负载分为两个阶段：
Prefill（预填充）读取提示词并构建初始 KV 缓存。它是计算密集型的。
Decode（解码）一次生成一个 token，反复读取权重和 KV 缓存。它是内存带宽密集型的。解码速度更多取决于内存带宽，而非峰值算力。
这种区别几乎解释了一切：
- 短提示、长回答：解码占主导 → 内存带宽和批处理很重要。
- 长提示、短回答：预填充占主导 → 注意力内核和分块预填充很重要。
- 多用户：调度器质量很重要 → 连续批处理、缓存分页、公平性。
- 长上下文：KV 缓存占主导 → 分页注意力、KV 量化、卸载。
- MoE：专家路由占主导 → 专家并行、互连、分组 GEMM。
- 多节点：互连占主导 → NVLink、RDMA、流水线并行、解聚。
PagedAttention 解决了 KV 缓存碎片化问题。FlashAttention 使用感知 I/O 的分块来减少 HBM（高带宽内存）流量。投机解码通过廉价 token 草稿并并行验证。 recurring 主题是：推理性能 = 内存移动 + 调度。
## 真正的瓶颈
1. 内存带宽，不仅仅是 VRAM 大小。VRAM 决定能否装下。带宽决定解码速度。Apple M3 Ultra 提供高达 819 GB/s 的统一内存带宽。NVIDIA H100 SXM 的 GPU 内存带宽为 3.35 TB/s。统一内存让你能装下在消费级 VRAM 中装不下的模型。HBM 让你在模型装得下时更快地服务它。装得下不等于速度快。容量不等于带宽。
2. KV 缓存增长。KV 缓存随批大小和上下文长度增长。长上下文工作负载即使在权重装得下的情况下也可能耗尽内存。PagedAttention 将 KV 缓存划分为块，提高利用率并支持更大的批次。
3. 互连。当模型跨越 GPU 边界（多 GPU）时，你需要付出通信成本。张量并行需要频繁的 all-reduce 集合通信。流水线并行在阶段边界通信。专家并行需要针对 MoE 的全对全通信。vLLM 文档指出，在没有 NVLink 的情况下，流水线并行可能优于张量并行。
4. 调度器质量。一个好的调度器决定哪些请求进入批次，预填充和解码如何共享加速器，长提示是否会阻塞短解码，以及如何避免饥饿。支持批处理不等于具备了生产就绪的调度器行为。
5. 运行时开销。CUDA 图、内核融合、采样开销、分词器开销、HTTP 开销、LoRA 切换和结构化解码都很重要。在高规模下，那些恼人的 2% 开销会结盟并要求关注（此处双关）。
## 引擎家族
主要有四大类：
便携式本地运行时：llama.cpp, MLC LLM, ONNX Runtime GenAI, OpenVINO, Ollama 类工具。它们关注“让它在这里跑起来”。
Apple/统一内存运行时：MLX 和 MLX-LM。它们关注“好好利用大容量共享内存和 Apple 的技术栈”。
消费级 CUDA 量化引擎：ExLlamaV2 和 ExLlamaV3。它们关注“让我的 3090/4090/5090 在低位重量下跑满”。
生产服务引擎：vLLM, SGLang, TensorRT-LLM, TGI, LMDeploy。它们关注并发用户、KV 缓存、批处理、并行、可观测性和每 token 成本。
还有像 Dynamo 这样的编排层，它们位于引擎之上，负责协调集群、解聚的预填充/解码、路由和自动扩缩容。
## llama.cpp：可移植性之王
当硬件怪异、受限、离线、重 CPU、面向边缘或不是整洁的 NVIDIA 数据中心节点时，llama.cpp 就是答案。
它通过 ARM NEON、Accelerate 和 Metal 支持 Apple Silicon；通过 AVX/AVX2/AVX512/AMX 支持 x86；RISC-V；低位量化；CUDA；通过 HIP 支持 AMD；MUSA；Vulkan；SYCL；以及 CPU+GPU 混合卸载。这就是为什么 llama.cpp 占据了“让它跑起来”这一赛道。
它的 HTTP 服务器比一个“玩具级本地运行器”更强。llama-server 提供兼容 OpenAI 的路由、Anthropic Messages API 兼容、重排序、连续批处理、多模态支持、JSON 模式约束、函数调用、投机解码以及 Web UI。
关键局限性：llama.cpp 不适用于严肃的多节点生产服务。它的 RPC 后端在文档中明确标注为概念验证、脆弱且不安全。
结论：当可移植性、离线运行、GGUF 或混合卸载比集群级服务更重要时，使用 llama.cpp。
不要在多 GPU 环境下使用。
## MLX 和 MLX-LM：Apple Silicon 的利器
MLX 是 Apple 用于 Apple Silicon 的数组框架，而 MLX-LM 是基于此构建的 LLM 包。这是一个 Mac 优先的 ML 栈。
关键的硬件事实是统一内存。Apple Silicon 让 CPU 和 GPU 直接访问同一内存池。MLX 数组存在于统一内存中，你在运行操作时选择设备，而不是在独立的内存空间之间移动数组。
这改变了本地推理的权衡。在独立 GPU 系统上，问题是“装得下 VRAM 吗？”在拥有大容量统一内存的 M 系列 Mac 上，问题变成了“装得下内存吗？内存系统能足够快地供给 GPU 吗？”大型量化模型可以在某些机器上运行，而同样的模型在 24 GB 的消费级 GPU 上是不可能的。
然而，它也较慢。
MLX-LM 增加了 Hugging Face Hub 集成、量化、LoRA 和全量微调、分布式推理以及庞大的 MLX 社区模型生态系统。MLX 不再仅限于 Mac：它为 Linux 提供了 CUDA 和仅 CPU 的包。分布式通信支持 MPI、基于 TCP 的 Ring、用于 Thunderbolt 上 RDMA 的 JACCL，以及用于 CUDA 的 NCCL。
MLX-LM 的服务器本身警告说，它不推荐用于生产环境，因为它仅实现了基本的安全检查。
结论：使用 MLX 进行 Mac 优先的 ML 和 LLM 工作流。对于高并发公共服务，请从一个真正的服务栈开始。
## ExLlamaV2 和 V3：消费级 CUDA，调优且快速
ExLlamaV2 是面向希望消费级 NVIDIA GPU 发挥超常性能的人的本地 CUDA 量化引擎。它支持分页注意力、动态批处理、提示缓存、KV 缓存去重、批处理生成、流式输出和投机解码。关键词是本地。它使量化模型在现代 CUDA GPU（尤其是消费级显卡）上运行快速。
最佳适配：单台 RTX 3090/4090/5090 机器、本地编程助手、本地聊天、EXL2 量化模型以及专业发烧友工作站使用。
ExLlamaV3 将这一理念扩展到了多 GPU 和 MoE 本地推理。它增加了基于 QTIP 的 EXL3 量化格式、面向消费级硬件的灵活张量并行和专家并行推理、通过 TabbyAPI 提供的兼容 OpenAI 的服务器、连续动态批处理和多模态支持。
当你拥有 2-4+ 张消费级 NVIDIA GPU 或想要本地 MoE 时，V3 很有吸引力。但要注意：某些模型在 ExLlamaV3 中不支持张量或专家并行。
结论：ExLlamaV2 是发烧友的本地 CUDA 引擎。ExLlamaV3 是多 GPU（2-4）本地设置的前沿。预期为了更好的能力会有更多粗糙的边缘。
## vLLM：默认的开源生产服务器
vLLM 是大多数团队在认真评估开源 LLM 服务时应首先考虑的引擎。
它提供基于 PagedAttention 的 KV 内存管理、连续批处理、分块预填充、前缀缓存、CUDA/HIP 图、广泛的量化（FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF）、优化的注意力和 GEMM/MoE 内核、投机解码、torch.compile 以及解聚的预填充/解码/编码。
它也很灵活：张量/流水线/数据/专家/上下文并行、流式输出、结构化输出、工具调用、兼容 OpenAI 和 Anthropic Messages API、gRPC、多 LoRA，并支持 NVIDIA、AMD、x86/ARM/PowerPC CPU，以及针对 TPU、Gaudi、Ascend、Apple Silicon 等的插件。
vLLM 文档指出，多节点部署通常使用 Ray，而在没有 NVLink 的情况下，流水线并行可能胜过张量并行。陷阱在于假设 vLLM 消除了对系统思考的需求。你仍然需要调优批处理、上下文长度、GPU 内存利用率、并行布局和路由。vLLM 给了你一个非常好的引擎；它仍然需要优秀的系统设计。
结论：如果说有人说“我们需要在生产环境中服务开源模型”，vLLM 是默认的起点。
## SGLang：vLLM 的系统思维表亲
当服务负载变得棘手时——比如结构化输出、长上下文、MoE、解聚和路由——你会选择 SGLang。
它提供 RadixAttention 前缀缓存、预填充-解码解聚、投机解码、连续批处理、分页注意力、张量/流水线/专家/数据并行、结构化输出、分块预填充和多 LoRA 批处理。它支持 NVIDIA、AMD、Intel Xeon、Google TPUs、Ascend NPUs 等。
SGLang 的差异化在于服务架构。它的预填充-解码解聚将计算密集型的预填充与内存密集型的解码分离到专门的实例中，并在它们之间传输 KV 缓存。这防止了长预填充批次中断解码并导致 token 延迟激增。
结论：SGLang 适用于那些瓶颈不再是“我们能运行模型吗？”而是“我们能否在充满敌意的流量下运行模型而不牺牲延迟、内存和成本？”的团队。
## TensorRT-LLM：NVIDIA 极致性能
TensorRT-LLM 是 NVIDIA 极致性能技术栈。它是优化的、专用的、强大的，并不假装具有可移植性。
它提供 Python API 以构建具有最先进优化的 TensorRT 引擎，以及 Python 和 C++ 运行时。它包含针对注意力、GEMM 和 MoE 的自定义内核；预填充-解码解聚、Wide Expert Parallelism（宽专家并行）、投机解码；以及与 NVIDIA Dynamo 和 Triton Inference Server 集成的高级 Python API。
B200 GPU 可以使用优化内核加载 FP4 权重。H100 及更高版本支持 FP8 量化，与 16 位相比，这可以使性能翻倍并减半内存消耗，而精度损失最小。
它的闪光点：H100/H200/B200/GB200/GB300 级别的集群、仅限 NVIDIA 的数据中心、FP8/FP4 部署、多节点服务以及大规模 MoE。它的尴尬之处：AMD、Apple 或 Intel 的可移植性；快速变化的实验性模型；小型本地设置；以及需要“到处都能用”的团队。
结论：如果你致力于 NVIDIA 并关心绝对性能，TensorRT-LLM 值得参与对比。你用可移植性换取了性能。调优的专门化但功能较少。
## 其他领域的选手
TGI 是 Hugging Face 的生产服务器，具有追踪、指标、张量并行和连续批处理。当 HF 集成和简单性很重要时使用它。
MLC LLM 是编译器优先的通用部署引擎，通过 REST、Python、JavaScript、iOS 和 Android 提供兼容 OpenAI 的 API。最适合“将 LLM 部署到各处”，尤其是浏览器、移动端和原生应用。
ONNX Runtime GenAI 在 ONNX Runtime 之上实现了完整的生成循环，并为 Foundry Local、Windows ML 和 VS Code AI Toolkit 提供支持。它支持 CPU、CUDA、DirectML、TensorRT-RTX、OpenVINO、QNN、WebGPU 和 AMD GPU。最适合应用部署和 ONNX 工作流。
OpenVINO GenAI 是面向 Xeon CPU、Arc GPU、Core Ultra 和 NPU 的 Intel 优化方案。它提供兼容 OpenAI 的服务，具有连续批处理和分页注意力。最适合 Intel 硬件。
LMDeploy 是一个以 CUDA 为中心的工具包，拥有用于性能的 TurboMind 和用于可访问性的 PyTorch。对于那些想要 vLLM/SGLang/TensorRT-LLM 替代方案的 CUDA 用户来说最有趣。
NVIDIA Dynamo 是位于 vLLM、SGLang 和 TensorRT-LLM 等引擎之上的分布式编排层，支持解聚、智能路由和多层 KV 缓存。当单引擎服务不再足够时使用它。
注意：不要使用 Ollama。
## 硬件策略方案
纯 CPU 服务器：首选 llama.cpp。Intel Xeon 选 OpenVINO。应用/ONNX 部署选 ONNX Runtime GenAI。
MacBook / Mac Studio：Mac 原生工作流选 MLX / MLX-LM。GGUF 可移植性选 llama.cpp。
单卡 RTX 3090 / 4090 / 5090：EXL2 本地推理选 ExLlamaV2。GGUF 或可移植性选 llama.cpp。如果服务多用户则选 vLLM。
双卡或四卡消费级 RTX 机器：多 GPU 量化推理或 MoE 选 ExLlamaV3。如果服务行为很重要则选 vLLM。如果测试路由或长上下文模式则选 SGLang。
8×H100 / H200 节点：从 vLLM 或 SGLang 开始。如果是仅 NVIDIA 且性能证明调优合理，则对 TensorRT-LLM 进行基准测试。当多节点编排变得必要时，使用 Dynamo。
B200 / GB200 / GB300 级别基础设施：对 TensorRT-LLM、SGLang 和 vLLM 进行基准测试。添加 Dynamo 以进行集群级编排、KV 感知路由和自动扩缩容。
AMD MI300 / MI325 / MI350 / MI355：在 ROCm 上从 vLLM 或 SGLang 开始。避免假设 NVIDIA 基准测试能直接平移。
Intel Xeon / Core Ultra / Arc：OpenVINO GenAI 或 OpenVINO Model Server。如果应用嵌入很重要，选 ONNX Runtime GenAI。
浏览器、移动端、应用原生：MLC LLM / WebLLM 或 ONNX Runtime GenAI。
## 基准测试：测量什么
糟糕的基准：“我跑到了 180 tok/s。”
好的基准包括：
模型：确切的模型、架构、参数量、活跃 MoE 参数。
权重：数据类型、量化格式、组大小、校准方式。
引擎：版本、提交号、后端、启动参数。
硬件：GPU 型号、内存容量、带宽、互连、CPU、RAM。
工作负载：输入/输出长度