LLM 原理与实践指南 (2026 版) ✍ Ahmad🕐 2026-05-23📦 56.1 KB 🟢 已读 𝕏 文章列表 这是一份关于大语言模型（LLM）的实践指南。文章从基础循环机制出发，详细解释了文本如何转化为 Token，以及 Transformer、注意力机制、KV Cache 和 RoPE 等核心概念的工作原理。作者强调理解模型内部的推理机制（如 Prefill 和 Decode 阶段）对于硬件选型、显存估算和本地部署的重要性。文章涵盖了 Token 化、上下文窗口、量化、模型服务及本地 AI 硬件数学计算等关键主题，旨在为深入理解 LLM 提供直观的基础。 LLMTransformer本地部署Tokenization推理Attention硬件教程KV Cache # LLMs 101: A Practical Guide (2026 Edition) **作者**: Ahmad **日期**: 2026-05-21T22:31:52.000Z **来源**: [https://x.com/TheAhmadOsman/status/2057590224729911346](https://x.com/TheAhmadOsman/status/2057590224729911346) ---  Start with the loop. Text becomes tokens. Tokens move through a Transformer. Attention decides which earlier tokens matter. The runtime keeps a KV cache so the model does not recompute the whole conversation every time. Then the model picks the next token and does it again. > A practical guide to how LLMs work, how models think 1 token at a time, and how to run them locally. Once that loop clicks, the hardware and software choices become easier to reason about. VRAM, quantization, context length, chat templates, decoding, RAG, serving engines, and model selection all fall out of the same mechanics. > Start with the loop: Tokens in, probabilities out, one next token at a time.Weights tell the model what patterns it learned. Context tells it what it is looking at now. The KV cache is the working memory that keeps the loop usable.Hardware, runtimes, and model selection only make sense after you understand the memory, context, and formatting rules the model is obeying. The goal is to make local LLM mechanics intuitive first, then give you a practical path into hardware, runtimes, serving, and current LLM research as of May 21, 2026. ## Focus This is a model-first guide. It starts with the mechanics: inference, tokens, Transformers, attention, KV cache, prefill, decode, decoding controls, model packages, chat templates, model types, long context, RAG, agents, fine-tuning, and multimodal models. After that, it moves into the local deployment layer: what local really means, quantization, VRAM math, hardware tiers, runtime choices, serving modes, licenses, model selection, privacy, troubleshooting, benchmarks, setup paths, and practical use cases. That order matters. You should understand why a long prompt costs memory before choosing a GPU. You should understand why chat templates matter before judging a model. You should understand whydecode is sequential before caring about tokens per second. For the deeper hardware and software path, I have a three-part series teaching self-hosted LLMs / local AI: - Part 1: GPU Memory Math for LLMs (2026 Edition). - Part 2: Memory Bandwidth for Local AI Hardware (2026 Edition). - Part 3: Inference Engines for LLMs & Local AI Hardware (2026 Edition). The first two pieces explain the hardware capacity and bandwidth math. The third explains the software layer that turns that hardware into usable inference. This article gives you the model-side foundation first, then points back to those deployment layers once the mechanics are clear. ## What An LLM Actually Does  Running a model is called inference. For a standard decoder-only LLM, inference is the same loop repeated over and over: 1. Convert your text into tokens. 2. Feed those tokens into the model. 3. Compute scores for every possible next token. 4. Choose one token with a decoding policy. 5. Append that token to the sequence. 6. Repeat until the model stops, the user stops it, or a token limit is reached. The model is not writing a whole answer in one shot. It is generating one token at a time. Every new token becomes part of the sequence that influences the next token. Mathematically, the model is a learned function: > f(theta, sequence) -> probability distribution over next_token Where: - theta means the model weights. - sequence means the prompt plus generated tokens so far. - Logits are the raw scores before softmax. - Probabilities are the normalized scores after softmax. - Decoding turns those probabilities into one selected token. This is why local generation speed is measured in tokens per second. Your system repeatedly runs a forward pass, picks or samples a token, updates the KV cache, and continues. Perception matters here. A long prefill means a long pause before the first word appears. Slow decodemeans the answer streams slowly. Local builders often obsess over decode speed because it is what users feel, but prefill time is what hurts when you paste a 10K-token document. ## Tokens  LLMs do not see raw text as words. They see tokens: Small chunks of text represented internally as integer IDs. A token might be: - A whole word: "hello" - A word fragment: "inter", "national", "ization" - A punctuation mark - A whitespace-prefixed string - A byte-level fallback - A special control marker such as <|user|>, <|assistant|>, , or The tokenizer maps text to token IDs and token IDs back to text. Common tokenizer families include BPE-style tokenizers and SentencePiece-style tokenizers. Different model families use different tokenizers, and that matters. A 4,000-word document may be 5,000 tokens in one tokenizer and 7,500 tokens in another. Vocabulary size matters too. A tokenizer with a larger vocabulary can compress some text into fewer tokens, but it also changes embedding and output-projection size. This is one reason tokens per second is not perfectly comparable across model families. Tokens matter because they determine: - How much text fits in the context window. - How large the KV cache becomes. - How much latency you pay during prompt processing. - Whether multilingual or code-heavy text is efficient. - Whether the model sees special chat markers correctly. A model's context window is the maximum number of tokens it can attend to at once. In 2026, common local-capable models range from 8K and 32K contexts to 128K, 256K, and even 1M-token contexts in server-class systems. But supported context length is not the same as cheap, fast, or equally accurate context. A model that can technically handle 128K tokens may slow to a crawl at 64K and lose coherence at 100K. Always test the context lengths you actually plan to use. Tokens are the unit of work. Once you understand that, long context stops looking magical and starts looking like a bill you can estimate. Helpful Exercise: Try my Tokenizer demo app to see how text gets broken into tokens in real time. ## Transformers  Most modern LLMs are based on the Transformer architecture. Most local chat LLMs are decoder-only Transformers: They predict the next token while looking back at previous tokens. Everything above this point, including tokens, weights, config, and chat templates, is setup for the real engine underneath. The Transformer is the skeleton that moves the numbers around. A simplified Transformer layer contains: 1. Token embeddings: Token IDs become vectors. 2. Positional information: The model needs token order. Many modern LLMs use RoPE (Rotary Position Embeddings), which encodes position by rotating representations. 3. Self-attention: Each token representation looks back at prior token representations and decides what matters. 4. MLP / feed-forward block: A dense nonlinear computation that expands and compresses representations. A large fraction of parameters live here. 5. Layer normalization and residual connections: These stabilize deep networks and help information flow through many layers. 6. Output projection: The final hidden state becomes logits over the vocabulary. Stack this recipe dozens or hundreds of times and you get a language model. Transformer recap: Tokens become vectors, attention connects the sequence, MLPs reshape the representation, RoPE keeps position straight, and the final projection turns the last hidden state into next-token logits. ## Attention Attention is how a token decides which earlier tokens matter for the next prediction. It is also one of the reasons local inference is so memory-sensitive. Classic MHA (multi-head attention) stores separate key/value state for many heads. It gives the model flexibility, but it makes the KV cache large. Modern local models often use more efficient attention designs: - MQA: Multiple query heads share one key/value head. It is memory-efficient, but can be less expressive. - GQA: Groups of query heads share key/value heads. It is the common middle ground in many current local models. - MHA: Full multi-head attention. It can be strong, but long context gets expensive quickly. Modern kernels such as FlashAttention and SDPA-style implementations reduce attention memory traffic and keep the GPU busier. A runtime with good attention kernels can be dramatically faster than one without, even on the same model and hardware. This is why two 7B models can behave very differently at long context. Parameter count is not the whole story. A 7B MHA model at 128K context can exhaust a 24 GB GPU, while a 7B GQA model with the same advertised context may fit with room to spare. When comparing models, look at attention type, KV heads, context length, and runtime support, not just parameter count. ## KV Cache  The KV cache is the model's working memory during generation. It stores key/value attention states for previous tokens so the model does not recompute the entire history from scratch on every generated token. Without a KV cache, generation would be brutally inefficient. With a KV cache, generation is usable, but the cache consumes memory proportional to: > tokens x layers x kv_heads x head_dim x precision x 2 The x 2 is for keys and values. A useful rule of thumb for older Llama-like 7B MHA models is roughly 0.5 MiB per token in FP16 KV cache. That means 4K tokens can cost around 2 GiB just for KV cache. At 32K tokens, you may be looking at 16 GiB of KV cache alone. Newer GQA/MQA models reduce this substantially. Some runtimes also support FP8 or INT8 KV cache. That is often the practical compression floor I would recommend for local users in 2026. Do not treat sub-8-bit KV cache as a default. Research systems such as KIVI, KVQuant, and newer compressed-cache kernels show that 2-bit to 4-bit KV can work with careful algorithms, calibration, and custom kernels. That is not the same as casually flipping a Q4 KV toggle in a desktop runtime. Below 8-bit, benchmark hard, especially for coding, tool calls, JSON, long-context retrieval, and tasks where exact earlier tokens matter. Also do not confuse KV-cache quantization with speculative decoding. DFlash and DDTree, often shortened informally to DTree, attack decode latency by drafting future tokens and verifying them. They can improve speed, but they do not erase the KV-cache memory bill. This is why a model can fit at an empty prompt but crash when you load a long document. The weights fit. The working memory did not. ## Prefill And Decode LLM inference has two different performance regimes: prefill and decode.  Prefill processes the prompt you gave the model. If you paste a 20,000-token document, the model must process those 20,000 tokens before it can produce the first answer token. Prefill is relatively parallelizable, so GPUs can handle it efficiently, but it can still be expensive. The time you spend waiting for the first token to appear is usually prefill time. Decode generates new tokens one at a time. Each generated token depends on the sequence so far, so decode is much more sequential. This is where the streaming typing effect comes from, and it is usually the phase that determines whether a model feels fast or slow. Long prompts punish prefill. Long answers punish decode. Long conversations punish both because the KV cache grows. In a chat session, every turn adds to the cache. If you let a conversation run to 16K tokens, you are paying the memory cost for all 16K tokens on every new token generated. This is why chat UIs that keep infinite history eventually slow down or crash. ## Decoding  After the model produces logits, it has not written anything yet. It has only scored every possible next token. Decoding is the policy that turns those scores into one actual token, appends that token to the context, and repeats the loop. The runtime, or inference engine, can choose tokens in several ways. It can pick the highest-probability token every time. It can sample from a narrowed set of likely tokens. It can penalize repetition. It can stop at a delimiter. It can use a fixed seed so the same prompt behaves reproducibly. These choices do not change the model weights, but they change the model's voice, determinism, creativity, risk profile, and tendency to loop. The important knobs answer three practical questions: - Randomness: How much variation is allowed? - Tail reach: How far into lower-probability tokens can the sampler go? - Boundaries: What prevents loops, rambling, schema breaks, or runaway output? For precise work, start narrow: low temperature, short max-token limits, explicit stop sequences, and constrained decoding when output must match JSON or a schema. For creative work, give the sampler more room with higher temperature, top-p, and multiple candidates ranked afterward. For coding, keep the first pass conservative, then sample alternatives only when you are intentionally exploring. Greedy decoding is not always more accurate. It is often brittle. A greedy decoder can get stuck in loops or produce generic answers because it never explores alternatives. For evals, use deterministic settings. For ideation, let the model breathe. ## What A Model Package Contains A runnable local LLM is more than one big weight file. A model package usually includes: - Architecture/config: Layer count, hidden size, attention type, RoPE settings, vocabulary size, special tokens, and context length. - Weights: The learned parameters, often stored as safetensors, GGUF, GPTQ, AWQ, EXL2, or another runtime-specific format. - Tokenizer: The rules that turn text into token IDs and token IDs back into text. - Chat template: The exact markup for system, user, assistant, tool, and reasoning messages. - Generation config: Defaults for temperature, top-p, stop tokens, repetition penalties, and max tokens. - License and model card: The legal and operational instructions for how the model can be used. The weights are the largest file, but they are not the whole model. If the tokenizer, config, or chat template is wrong, the same weights can feel broken. The package section tells you what has to travel together. The next section explains why the chat template is the part people most often break. ## Chat Templates  A chat model was trained with a specific conversation format. For example, it may expect something like: > <|system|> You are a helpful assistant. <|user|> Explain KV cache. <|assistant|> Another model may expect: > [BOS] [INST] Explain KV cache. [/INST] Another may use ChatML-style markers. Another may require special reasoning tokens. Another may need tool-call XML or JSON wrappers. Using the wrong format can cause gibberish, role confusion, ignored system prompts, repeated prompts, refusal weirdness, broken tool calls, bad benchmark results, and conclusions that the model is dumb when the template is the actual bug. Best practice: - Use the tokenizer's apply_chat_template when using Transformers. - Use model-specific templates in Harbor-backed frontends, llama.cpp, LM Studio, vLLM, or SGLang. - Check whether the model is base, instruct, chat, reasoning, or tool-tuned. - Ensure BOS/EOS tokens are correct. - Keep system prompts short unless they need to be long. - For tool use, follow the exact schema expected by the model/runtime. If you are building an application that lets users switch models, you need template switching too. Hardcoding one template format and then loading a model that expects another is a common source of bad local-model evals. Treat the template like an API contract. If you get it wrong, you are not really testing the model you think you are testing. ## Model Types  Not all LLMs are tuned for the same behavior. For most users, the default starting point should be a recent instruct/chat-tuned model in a size that fits comfortably in memory. Do not start with a base model unless you know why. Base models complete your prompt rather than answer it. They are useful for researchers, fine-tuners, and people building custom pipelines. They are frustrating for everyone else. If you ask a base model What is the capital of France?, it might continue with and what is the population of Paris? instead of answering Paris. The practical split is simple: - Base model: Good for pretraining research, fine-tuning, and custom pipelines. - Instruct model: Good for direct instruction following. - Chat model: Good for multi-turn dialogue with role formatting. - Reasoning model: Good when the task benefits from extra thinking tokens and verification. - Tool-tuned model: Good when structured calls, JSON, or function use matters. ## What Local Really Means  A local LLM is a model whose weights and inference runtime are under your control. You decide what model runs, how it runs, what data it sees, and what happens to the outputs. That freedom comes with work. You are now the ops team. You handle downloads, updates, compatibility, memory limits, and security. When something breaks, there is no support ticket to file. There is only you, the logs, and the documentation. Local can mean: - A 2B parameter model running on a phone. - A 7B to 14B model running on a consumer GPU. - A 30B to 70B model running on a high-end workstation. - A sparse MoE model running on one or more datacenter GPUs. - A private deployment using vLLM, SGLang, TensorRT-LLM, llama.cpp, Harbor, LM Studio, or a custom PyTorch stack. The key point: local does not automatically mean offline, private, safe, cheap, or opensource. It only means you are running the model yourself. A local app can still phone home. A model can be open-weight but not opensource. A model can be local but unsafe to load. A quantized model can fit in memory but answer poorly. The tradeoff is worth it when you need privacy, low latency, custom behavior, offline operation, or cost control at scale. It is not worth it when you need the absolute best model quality and do not have the hardware to match. In that case, a hosted API is the right tool. Local LLMs are practical when you understand one equation: > Local LLM success = model fit + correct prompt format + good runtime + realistic evals. Everything else is details. The details matter. ## Quantization  Quantization stores weights in lower precision to reduce memory and sometimes improve throughput. The 2026 rule of thumb for local users: - FP16/BF16: Best quality when memory is abundant. Use it as a baseline for evaluation. - Q8 / INT8: Near-lossless for many tasks, but still large. Good when you have VRAM and want minimal quality loss. - Q6 / Q5: Excellent quality with moderate savings. This is a strong middle ground. - Q4: The default consumer sweet spot for many chat and document workflows. - Q3 / Q2: Only when you must fit a bigger model. Math, code, structured output, and tool use degrade first. Weight quantization is not the same as KV-cache quantization. Weight quantization shrinks the model. KV-cache quantization shrinks the live context memory. For KV cache, treat FP16/BF16 as the clean baseline and FP8/INT8 as the practical local compression floor. Below 8-bit is research-heavy and workload-sensitive. Use it only after measuring quality on your actual prompts. Quantization failure shows up first in math, multi-step reasoning, code correctness, tool-use reliability, JSON/schema adherence, subtle instruction following, and long-context retrieval. A smaller model at higher precision can beat a larger model crushed into too few bits. Do not worship parameter count. A 7B model at Q6 can beat a 13B model at Q2 on reasoning tasks while using less memory and running faster. ## File Formats And Load Safety  safetensors is a safe tensor serialization format designed to store tensors without Python pickle behavior. Use safetensors when possible, especially for PyTorch/Transformers models. Avoid random .bin files from untrusted sources. PyTorch pickle-based loading can execute arbitrary code during deserialization. Local AI security rule number one: do not let a stranger's model file become a stranger's code execution. GGUF is the llama.cpp ecosystem's binary model format. Use GGUF when you want llama.cpp, CPU inference, Apple Silicon inference, simple local servers, portable quantized models, or desktop tools like LM Studio. ONNX is useful for standardized deployment and hardware-specific acceleration, especially outside the usual PyTorch stack. If you are deploying to Intel NPUs, ARM devices, or custom accelerators, ONNX is often the path of least resistance. TensorRT-LLM is NVIDIA's high-performance inference path for production GPU deployments. It is powerful, but more complex than llama.cpp or Harbor. You typically convert a checkpoint to TensorRT engines, which takes time and GPU memory, but yields excellent throughput once built. EXL2 / GPTQ / AWQ formats are common in GPU-focused local inference communities, especially for squeezing larger models onto single GPUs. The file format choice is not cosmetic. It determines which runtimes can load the model, what quantization you can use, and how fast it runs. ## Runtimes And Serving Modes  A runtime is the software that loads the model and performs inference. In 2026, the local LLM runtime ecosystem is mature, useful, and fragmented. For a single person experimenting locally, start with Harbor, LM Studio, or llama.cpp. Harbor is the best fit when you want a complete local stack with frontends, backends, and supporting services wired together. LM Studio is the easiest desktop-first path. llama.cpp is the portable low-level workhorse. For a team or private service, look at vLLM or SGLang. For maximum NVIDIA production performance, investigate TensorRT-LLM. For browser or mobile deployment, look at MLC or WebLLM. The runtime choice often locks you into a format ecosystem. llama.cpp means GGUF. vLLM and SGLang usually mean safetensors or Hugging Face checkpoints. TensorRT-LLM means ONNX or optimized engines. Pick the runtime first, then find models in the right format.  There are three practical serving modes. Single-user local means a desktop app, CLI stack, or command-line server for one person. Harbor, LM Studio, llama.cpp server, ExLlama/TabbyAPI, and small Transformers scripts all fit here. The goal is quick iteration: Compare behavior, speed, memory use, and prompt formats without building an ops platform. Team or private API means an OpenAI-compatible endpoint on a workstation or server. vLLM, SGLang, TensorRT-LLM, and llama.cpp server all show up here depending on model size and throughput needs. Once multiple people or jobs share a model, you need monitoring, prompt/version management, routing, and realistic latency measurements. Production serving is a different job. Now the conversation includes continuous batching, prefix caching, speculative decoding, paged attention, tensor parallelism, pipeline parallelism, quantized serving, structured outputs, load balancing, GPU utilization, latency percentiles, prompt caching, admission control, logging, failover, privacy, and cost controls. At production scale, can I load the model? is the easy question. The hard question is can I serve it reliably under real traffic? ## VRAM Math For Local Models  There are three main memory consumers: 1. Model weights 2. KV cache 3. Runtime overhead The rough weight memory formula is: > weight_memory ~= parameters x bytes_per_parameter Useful approximations: - FP16/BF16: About 2 bytes per parameter. - INT8/Q8: About 1 byte per parameter. - Q4: About 0.5 bytes per parameter, plus format overhead. Then add: - Runtime overhead: Framework buffers, CUDA overhead, memory fragmentation, and temporary tensors. - KV cache: Grows with every token in the active context. - Batch/concurrency memory: Each concurrent request needs its own cache. - Vision encoder memory: Images become tokens too. - Speculative decoding memory: Draft models, draft heads, or extra verification structures are not free. - Adapter memory: LoRA adapters are small, but still real. MoE models add one more wrinkle. A model may activate only a fraction of its parameters per token, but the inactive experts usually still need to live somewhere in memory. Active parameters affect compute cost. Total parameters still affect loading and capacity planning. A realistic estimate looks like: > total_memory = quantized_weightsKV_cache_for_context > runtime_overhead > batch_or_concurrency_overhead > safety_margin Here is the trap: A 13B model in Q4 may fit easily at 8K context, then fail at 32K because the KV cache quadrupled. The weights did not change. The context did. Leave 10 to 20 percent headroom. Running at 99 percent VRAM utilization is begging for out-of-memory errors and fragmentation failures. ## Hardware Tiers In Practice  These are practical 2026 rules of thumb, assuming quantized inference and sane context lengths. Exact results depend on runtime, quantization, model architecture, attention type, context length, and OS/driver overhead. For most serious local users in 2026, 16 GB is the minimum comfortable GPU tier, 24 GB is the best value enthusiast tier, and 48 GB+ is where the stronger local world opens up. Performance depends on memory bandwidth, GPU FLOPs, VRAM capacity, KV-cache size, attention implementation, quantization, batch size, prompt length, generated length, and runtime maturity. Decode is often memory-bandwidth-bound: The GPU streams weights repeatedly while doing relatively little compute per byte. Prefill is more compute-bound because it can process the prompt in parallel. That is why two cards with the same VRAM capacity can have very different token speeds if one has much higher memory bandwidth. The most painful local setup is one where the model almost fits and spills layers to CPU. It may technically run, but token speed can collapse. CPU offload is acceptable for experimentation. It is not a performance strategy. ## Choose A Model That Fits The practical question is not what is the best model? It is what is the smallest model that wins your real workload on your hardware? Start with a recent instruct/chat model that fits comfortably with the context length you actually need. If you are on 8 GB to 12 GB of VRAM or unified memory, start small. If you are on 16 GB to 24 GB, test 7B to 14B class models first. If you have 48 GB or more, larger dense models and MoE models become realistic. Use this memory gate before you fall in love with a checkpoint: > weights + KV cache + runtime overhead <= 80 to 90 percent of available memory Then run the same 20 to 50 prompts across candidates. Include your real tasks: Coding edits, document Q&A, JSON output, summaries, tool calls, long context, or whatever you actually need. Measure answer quality, latency, memory use, template reliability, and failure modes. A practical model choice usually comes down to five checks: - Task fit: Chat, coding, documents, agents, multimodal, edge, or fine-tuning. - Memory fit: Weights, KV cache, runtime overhead, and safety margin. - Interface fit: Tokenizer, chat template, stop tokens, tool schema, and reasoning mode. - Runtime fit: Does your runtime support this architecture, quantization, context length, and serving mode well? - License fit: Can you actually use it where you plan to use it? Leaderboards are useful for discovery. They are not a substitute for your own evals. Your workload is the benchmark that matters.  For a simple local assistant, pick a recent 7B to 14B instruct model, Q4/Q5 quantization, the correct chat template, 8K to 32K context, and Harbor, LM Studio, or llama.cpp. Prioritize responsiveness over giant size. For a local coding assistant, pick a code-capable 14B to 32B model if you have enough VRAM. Use low temperature, repository retrieval, test execution, and a patch-based workflow. A code model without tools is half a product. For a private document assistant, pick a strong instruct model, a local embedding model, a reranker, a RAG pipeline, citation enforcement, and moderate-to-long context. Do not paste a 200-page PDF and hope. For a reasoning setup, pick a reasoning-tuned model, budget extra tokens, use low-to-medium temperature, add verification, and use tools for math, code, or search. Reasoning models spend more tokens. Budget accordingly. For a low-resource setup, pick a 1B to 4B model, Q4/Q5, short prompts, structured tasks, retrieval or tools, and a tight output schema. Small models become useful when the task is constrained. ## What Controls Speed  Tokens per second is not controlled by one thing. It is the result of model size, memory bandwidth, compute, attention kernels, context length, quantization, batching, and runtime quality. The main levers are: - Memory bandwidth: Decode often streams model weights repeatedly, so bandwidth dominates single-user token speed. - GPU FLOPs: Prefill and large batches use more parallel compute, so FLOPs matter more there. - VRAM capacity: If the model or KV cache spills to CPU, performance can collapse. - Attention implementation: FlashAttention, SDPA, paged attention, and runtime-specific kernels change both speed and memory behavior. - Quantization: Smaller weights reduce memory movement, but aggressive quantization can hurt quality and sometimes add dequantization overhead. - Batch size and concurrency: Batching improves throughput, but each active sequence needs KV cache. - Prompt length: Long prompts increase prefill time. - Generated length: Long answers expose decode speed. - Speculative decoding: EAGLE-style methods, MTP, DFlash, and DDTree can verify more than one drafted token per target pass when supported. The painful setup is the almost-fit setup. A model that spills layers or cache to CPU may technically run, but token speed can drop from usable to miserable. Benchmark the exact runtime, quantization, context length, prompt shape, and workload you plan to use. A BF16 leaderboard number does not tell you what your Q4 local stack will feel like. ## Long Context Long context sounds magical: 128K, 256K, or even 1M tokens in one prompt. It is useful, but it has real costs. More context means more KV cache memory, slower prompt processing, more attention work, harder evaluation, and more ways for irrelevant text to distract the model. Quality can also decay across distance. A model may handle the end of a long document well while missing critical details buried near the beginning. Use long context for whole-document analysis, codebase slices, legal or technical review, transcript summarization, multi-file reasoning, and RAG fallback when retrieval misses context. Do not treat long context as a replacement for retrieval. It is a complement. Use RAG for large corpora and long context for the final selected evidence. Practical habits help: - Put critical instructions near the beginning and near the end. - Use section headers and delimiters. - Ask for citations tied to source chunks. - Compress irrelevant history. - Use summary memory instead of infinite chat history. Think of long context as expensive attention, not a free notebook. ## Multimodality Multimodal local models accept images, and sometimes audio or video, in addition to text. Modern open-weight ecosystems increasingly include these models. The hidden cost is that non-text input becomes tokens too. Vision encoders add memory. Image patches consume context. Audio and video can explode the input budget. Multimodal templates are also easier to get wrong than text-only templates. A single high-resolution image can consume thousands of tokens in the context window. If you are running a multimodal model locally, count image tokens the same way you count text tokens. They come from the same budget. Small VLMs can hallucinate visual details. OCR reliability varies. Charts and tables are still hard. For serious document or image workflows, evaluate with real samples. Do not trust a demo of a simple photo to prove invoice extraction quality. ## The 2026 Local Model Scene  The model scene changes fast. As of May 21, 2026, local LLM users should think in terms of families and ecosystems, not one best model. Qwen 3.5 / Qwen 3.6 is a major open-weight family because it covers the whole stack: Small models for laptops, dense mid-size models for workstations, MoE models for multi-GPU serving, FP8 variants, long context, multilingual work, coding, tools, and agentic workflows. The practical takeaway is simple: Qwen is a strong default family when you want one ecosystem that spans laptop experiments and serious local serving. Gemma 4 matters because Google DeepMind is pushing the family toward useful local deployment: Efficient edge models, larger dense and MoE options, multimodality, long context on the larger models, broad language support, stronger coding/agent behavior, and Apache 2.0 licensing. That combination makes it worth testing when commercial use and device-side deployment matter. Kimi / Moonshot AI, GLM / Z.ai, DeepSeek, MiniMax, and Mistral are also core families to track. Kimi is relevant for long-horizon coding, multimodal reasoning, tool use, and agent workflows. GLM is important for coding agents, long-horizon tasks, MoE systems, and deployment-oriented model releases. DeepSeek remains influential because of large MoE systems, Multi-head Latent Attention, DeepSeekMoE, FP8 serving paths, sparse attention, and high-throughput self-hosting. MiniMax is worth watching for practical agent workloads and inference-efficient MoE models. Mistral still matters because its lineup covers generalist, coding, reasoning, multimodal, and specialist use cases with strong deployment support. Nemotron 3 is NVIDIA's open model family for production-grade agent systems on NVIDIA hardware. The family includes Nano, Super, and Ultra sizes, uses hybrid Mamba-Transformer MoE designs, and is tied closely to TensorRT-LLM, NIM, Dynamo, Blackwell NVFP4/FP8 paths, and enterprise agent deployment. Treat it less like a casual desktop-chat family and more like a signal for where NVIDIA wants open-weight serving stacks to go. Open-weight AI is no longer just Llama vs. everything else. You are choosing an ecosystem: Weights, license, tokenizer, template, quantizations, runtime support, serving path, community tools, and failure modes. The Qwen 27B Dense Model Qwen 3.5 / 3.6 27B (Dense) is one of the most practical public-weight options for local users who care about coding, multilingual work, tool use, thinking/non-thinking modes, and long context. The Qwen 3.5 27B and Qwen 3.6 27B model cards describe OpenAI-compatible serving paths, thinking mode defaults, tool use, and context lengths up to 262,144 tokens, with longer-context extension through YaRN in supported frameworks. Qwen is a strong default for a 2x RTX 3090 setup when the runtime is configured correctly for coding, agents, or multilingual coverage. Inference Research The 2026 frontier is not only model quality. It is also inference efficiency. PagedAttention attacks KV-cache memory waste in serving. FP8 KV cache is now a practical runtime feature in systems such as vLLM. DFlash and DDTree explore speculative decoding with block diffusion draft models and draft trees. NVFP4 is also worth watching on NVIDIA hardware because it changes the practical deployment conversation for supported stacks. Some of this is production-ready. Some of it is still research. Some of it only matters if your runtime supports it cleanly. Do not treat paper speedups as a checkbox in a desktop app. ## Failure Modes And Fixes Most local LLM failures are not mysterious. They usually come from memory fit, formatting, runtime support, decoding settings, or retrieval quality.  Out of memory: The weights, KV cache, runtime overhead, or batch size do not fit. Use a smaller model, reduce context, lower batch/concurrency, choose a better quantization, or leave more headroom. Gibberish or role confusion: The chat template, tokenizer, BOS/EOS token, reasoning-mode switch, or tool schema is wrong. Verify the model card and runtime template before blaming model quality. Slow first token: Prefill is expensive. Shorten the prompt, use prefix caching, improve retrieval, reduce context, or use a faster runtime. Slow streaming: Decode is the bottleneck. Check memory bandwidth, quantization, CPU spill, attention backend, speculative decoding support, and whether the model is simply too large for the hardware. Bad document answers: Retrieval probably failed. Inspect parsed text, chunk boundaries, metadata, top-k retrieval, reranking, and citation grounding. Bad JSON or tool calls: Use lower temperature, constrained decoding, stricter schemas, better examples, and a model tuned for tool use. Repeating loops: Reduce temperature or top-p, add repetition penalties, check stop tokens, and make sure the template is not causing the model to see its own answer as a new prompt. Start with the boring checks. They fix more problems than model swapping does. ## How To Grow The Stack  Beginner: Easiest Useful Setup Use Harbor or LM Studio, a recent 4B to 9B instruct model, Q4 quantization, 8K to 32K context, and a built-in chat UI. Download two or three models in the same size class and compare them on the same prompts. Goal: Learn prompting, compare models, understand speed and memory, and avoid custom code at first. Intermediate: Developer Setup Use llama.cpp or Transformers, GGUF or safetensors, an OpenAI-compatible local server, a simple RAG pipeline, and a small eval set. Call your local server from a real application or script instead of only using a chat UI. Goal: Build local apps, test retrieval, measure quality, and serve from localhost. Advanced: Private Serving Setup Use vLLM or SGLang, one or more GPUs, an OpenAI-compatible API, monitoring, prompt/version management, an evaluation suite, RAG with reranking, and tool sandboxing. Goal: Serve real users or internal workflows, optimize throughput and latency, and maintain safety and observability. Expert: Custom Optimization Use TensorRT-LLM, custom kernels, specialized runtimes, quantization experiments, speculative decoding, multi-GPU parallelism, fine-tuning, distillation, and production evals. Goal: Trade engineering time for inference efficiency, lower cost, and higher quality at scale. ## Privacy Is Not Automatic  Local LLMs improve privacy because prompts and outputs can remain on your hardware. But local does not automatically mean secure. Threats include malicious model files, pickle-based weight loading, untrusted trust_remote_code, prompt injection in retrieved documents, tool-call abuse, secret leakage through logs, telemetry from desktop apps, browser extensions or plugins, model hallucinations in high-stakes settings, license violations, and data contamination during fine-tuning. A workable local AI security baseline has four habits: - Load carefully: Prefer safetensors or GGUF from reputable sources, avoid untrusted .bin files, and do not enable trust_remote_code casually. - Run with boundaries: Use an unprivileged user, containers or sandboxes for agents, and disabled network access when offline privacy matters. - Protect secrets: Keep credentials out of prompts and RAG indexes, review desktop app telemetry settings, and validate tool calls before execution. - Version what matters: Track model, prompt, adapter, runtime, and quantization versions, and log enough for debugging without creating a privacy disaster. Local AI security is mostly boring operational discipline. That is also how you avoid downloading a random checkpoint, running it as root, and turning local AI into local compromise. ## Benchmarks That Matter  Benchmark the stack you will actually run. A model's BF16 leaderboard score is not your Q4 local reality. Measure quality, latency, memory, reliability, and operating fit: - Quality: Correctness on your real tasks, not only generic benchmarks. - Latency: Time to first token, decode tokens per second, and end-to-end time. - Memory: Weight memory, KV-cache growth, peak VRAM, and headroom under load. - Formatting: Chat template correctness, JSON/schema success, tool-call reliability, and stop-token behavior. - Retrieval: Citation faithfulness, answer grounding, missing-evidence behavior, and reranker impact. - Operations: Startup time, warmup behavior, crash recovery, logging, privacy, and version tracking. Create a small eval set with 30 to 100 representative prompts. Include expected answers or scoring criteria, latency and memory measurements, failure categories, RAG-specific grounding checks, JSON compliance checks if relevant, and human review for ambiguous tasks. Then compare models. Do not let a leaderboard choose your local stack for you. ## Coding With Local Models  Coding is one of the best local LLM use cases because prompts often include private code, latency matters, iteration is frequent, API costs can grow quickly, and local models can integrate with editors, shells, grep, test runners, and patch workflows. The strongest local coding setup is not a naked chatbot. It is a code-capable instruct model connected to targeted repository context, retrieval over the codebase, file paths, relevant snippets, test execution, and a patch loop. Keep decoding deterministic or low-temperature. Ask for patches instead of vague advice. Run tests automatically. Keep a small eval set of real bugs and tasks so you can tell when a new model is actually better. Do not let a local model rewrite a large codebase without review. Local does not make a coding agent wise. It only makes the context private, the loop cheaper, and the integration easier to control. ## Local Agents Need Guardrails  A local LLM becomes much more useful when it can use tools: File search, shell commands, browser automation, databases, code execution, calendars, ticket systems, internal APIs, vector databases, home automation, robotics, or edge devices. Tool use changes the safety model. A chatbot that hallucinates is annoying. An agent with filesystem access can delete things. An agent with browser access can leak secrets. An agent with shell access can damage the machine faster than you can read the logs. Local agent safety has four layers. Scope the agent tightly by giving it only the directories, APIs, network access, and credentials it actually needs. Constrain execution with sandboxes, containers, least-privilege users, confirmations for destructive actions, and schema-validated tool arguments. Treat inputs as hostile because retrieved documents, web pages, tickets, and emails can contain prompt injection. Keep an audit trail by logging tool calls, model versions, prompts, and approvals without dumping secrets into logs. Structured outputs help, but they are not a security boundary. JSON schemas, constrained decoding, and function signatures make tool calls easier to validate. They do not prove the model understood the request, chose the safe action, or avoided injected instructions. For serious tool use, put policy checks outside the model. ## RAG Beats Giant Prompts RAG means Retrieval-Augmented Generation. Instead of stuffing all information into the prompt, you retrieve relevant chunks from a knowledge base and give only those chunks to the model. A good local RAG system usually has document ingestion, parsing, chunking, embeddings, a vector index, retrieval, reranking, prompt construction, answer generation, grounding checks, and evaluation. Each stage is a failure point. Bad parsing turns tables into garbage. Bad chunking splits the answer across boundaries. Bad retrieval returns irrelevant paragraphs. Bad reranking buries the right answer at rank 20. A good model cannot reliably answer from evidence it never received. Most bad RAG systems are not bad because of the LLM. They are bad because of chunking, retrieval, reranking, and evaluation. Chunking strategy is the silent killer. Fixed-size chunks with no overlap can split sentences and lose context. Semantic chunking or hierarchical chunking with parent-document retrieval often works better, but there is no universal answer. You have to evaluate chunk size, overlap, and splitting rules on your actual documents. A good reranker can rescue mediocre retrieval. No reranker can fix chunks that lost the answer during ingestion. ## Documents And Knowledge Work For private documents, local LLMs shine: Meeting transcript summaries, contract review, technical documentation Q&A, research note synthesis, email drafting, policy search, internal support assistants, and compliance workflows all benefit from keeping source material close to the machine or organization that owns it. The workflow is simple but unforgiving. Parse documents carefully, preserve page and section metadata, chunk semantically, use embeddings and rerankers, ask for citations, separate answer from sources from general reasoning, and evaluate citation faithfulness. Do not assume the model knows what is in your documents. It only knows what you put into the prompt or retrieve into context. For meeting transcripts, preserve speaker labels and timestamps. For contract review, chunk by clause or section instead of arbitrary token count. For technical documentation Q&A, include page numbers or section anchors in retrieved chunks so the model can cite sources accurately. For document work, your parser and retriever matter as much as the model. ## Edge Deployment  Small models are increasingly useful on phones, laptops, robots, IoT gateways, factory devices, vehicles, medical devices, offline field equipment, and browser apps. The edge is not only a smaller version of the workstation. It has a different set of constraints. Edge deployment is ruled by low memory, low power, thermal limits, intermittent connectivity, privacy requirements, real-time latency, small context windows, and predictable fallback behavior. On those devices, a small reliable model beats a large fragile one. A practical edge setup often uses a 0.5B to 4B model, aggressive weight quantization, tiny prompts, fixed schemas, tool-assisted workflows, local embeddings, caching, and no unnecessary chat history. When connectivity drops, a local model that keeps working is more valuable than a larger model that fails. The future of local AI is not only giant workstation models. It is also small models doing useful work close to the data. ## A Local LLM Runbook Use this as the final gate before trusting a local model for real work. Choose and fit: Pick a model family suited to the task, read the license, confirm hardware requirements, choose a quantization level, and estimate the full memory bill. Do not stop at weight size. Include KV cache, runtime overhead, batch/concurrency, and safety margin. Load and format: Prefer safetensors or GGUF from reputable sources, avoid untrusted pickle-based files, verify tokenizer and chat template, set context length intentionally, and choose decoding parameters for the task. If the template is wrong, the eval is invalid. Evaluate and operate: Test with representative prompts, measure time to first token and decode speed, track peak memory, evaluate retrieval before adding RAG, sandbox tools before adding agents, and fine-tune only after simpler methods fail. Version everything that matters: Model, quantization, runtime, prompt, chat template, adapter, embedding model, reranker, eval set, and hardware profile. Local systems are easier to control only when you can reproduce what you ran. ## Fine-Tuning Fine-tuning changes model behavior by training on additional data. For local users, the most important methods are LoRA and QLoRA. LoRA freezes the base model and trains small low-rank adapter weights. That reduces trainable parameters and lets you maintain multiple lightweight adapters. QLoRA extends this by fine-tuning through a frozen 4-bit quantized model into LoRA adapters. Fine-tune when you need a consistent writing style, domain-specific output format, repetitive classification or extraction behavior, tool-call format reliability, a specialized assistant persona, domain adaptation that RAG cannot solve, or better small-model performance on a narrow task. Do not fine-tune first. Try this order: correct chat template, better prompting, better model, better decoding, RAG, reranking, few-shot examples, then fine-tuning. Most problems that look like the model does not understand my domain are actually my prompt is vague, my template is wrong, or my retrieval is broken. A good fine-tuning plan includes clean data, train/validation/test splits, baseline evals, clear target behavior, safety review, overfitting checks, regression evals, adapter versioning, license review, and a rollback plan. ## Open-Weight Does Not Mean Opensource In 2026, the phrase open model is often used sloppily. You should distinguish open-weight, source-available, opensource, and local-compatible. Open-weight usually means you can download the weights. It does not automatically mean you can use the model commercially, modify it freely, train on its outputs, deploy it at any scale, or ignore attribution requirements. Source-available means code or weights are visible. It does not necessarily mean the license is opensource. Opensource AI model is a stronger claim. OSI's Opensource AI Definition treats an AI system as including architecture, parameters/weights, inference code, and enough data information and code used to derive the parameters. That is a much higher bar than the weights are on Hugging Face. Some licenses look permissive but contain restrictions: No competitive use, no training on outputs, no deployment above a certain scale, geographic exclusions, attribution requirements, patent clauses, or copyleft-like obligations on derivatives. Rule: read the model card and license before using any model commercially. A model can be excellent, downloadable, and locally runnable while still being a bad fit for your legal or deployment constraints. ## Glossary Model And Tuning Terms - Active Parameters: In an MoE model, only some parameters are used for a given token. A model may have hundreds of billions of total parameters but far fewer active parameters per token. - Adapter: A small trainable module added to a base model, often through LoRA. - Base Model: A pretrained model not specifically tuned for chat or instruction following. - Fine-Tuning: Additional training that changes model behavior for a target domain or output style. - Instruct Model: A model tuned to follow instructions. - LoRA / QLoRA: Efficient fine-tuning methods using low-rank adapters, with QLoRA training through quantized base models. - MoE: Mixture of Experts. A sparse architecture where only selected expert subnetworks activate per token. - Weights / Parameters: The learned numerical values inside the model. Inference Mechanics - BOS / EOS: Beginning-of-sequence and end-of-sequence tokens. - Chat Template: The formatting used to represent system, user, assistant, and tool messages. - Context Window: The maximum number of tokens the model can process at once. - Decode: The phase where the model generates new tokens one by one. - DFlash: A 2026 speculative decoding approach that uses block diffusion for parallel drafting. - DDTree / DTree: A speculative decoding method that builds a draft tree from block diffusion distributions and verifies it efficiently. - GQA / MQA: Attention variants that reduce KV-cache size and improve inference efficiency. - Inference: Running the model to produce outputs. - KV Cache: Stored key/value attention states for previous tokens. - Prefill: The phase where the model processes the input prompt before generating. - RoPE: Rotary Position Embeddings, a positional encoding method common in modern LLMs. - Speculative Decoding: A speed technique where a cheaper drafter proposes tokens and the target model verifies them. - Tokenizer: The component that converts text into token IDs and back. - Top-p / Top-k / Temperature: Sampling controls for token generation. Retrieval, Files, And Serving - AWQ: Activation-aware weight quantization. - Embedding Model: A model that converts text into vectors for search/retrieval. - FP8 KV Cache: A practical 8-bit KV-cache compression mode supported in some runtimes. - GGUF: A model file format used heavily by llama.cpp. - PagedAttention: A KV-cache memory-management technique used by vLLM-style serving. - Quantization: Reducing numeric precision to save memory and improve efficiency. - RAG: Retrieval-Augmented Generation. Retrieve relevant external context and give it to the model. - Reranker: A model that reorders retrieved passages by relevance. - Safetensors: A safer tensor serialization format that avoids pickle-based execution risks. ## Final Words The local LLM ecosystem includes compact edge models, strong 7B to 32B consumer models, large MoE open-weight systems, multimodal models, long-context models, local reasoning models, mature inference runtimes, and increasingly capable private serving stacks. But the fundamentals have not changed: the model predicts one token at a time, tokens are not words, weights are not the whole model, chat templates matter, KV cache is the hidden memory bill, quantization is a tradeoff, long context is not free, RAG quality depends on retrieval, fine-tuning needs evals, and local privacy still requires security discipline. You do not need mythology to run local models well. You need to know what fits in memory, which template the model expects, how the runtime behaves, and whether your evals match the work you care about. Local LLMs are mostly memory math plus formatting plus evaluation. Get those right, and the rest of the stack becomes much easier to reason about. Until next time. -Ahmad ## 相关链接 - [Ahmad](https://x.com/TheAhmadOsman) - [@TheAhmadOsman](https://x.com/TheAhmadOsman) - [171K](https://x.com/TheAhmadOsman/status/2057590224729911346/analytics) - [GPU Memory Math for LLMs (2026 Edition)](https://x.com/TheAhmadOsman/status/2040103488714068245) - [Memory Bandwidth for Local AI Hardware (2026 Edition)](https://x.com/TheAhmadOsman/status/2041331757329285589) - [Inference Engines for LLMs & Local AI Hardware (2026 Edition)](https://x.com/TheAhmadOsman/status/2057183854444843202) - [Tokenizer demo app](https://ahmadosman.com/tokenizer) - [KIVI](https://arxiv.org/abs/2402.02750) - [KVQuant](https://arxiv.org/abs/2401.18079) - [DFlash](https://arxiv.org/abs/2602.06036) - [DDTree](https://arxiv.org/abs/2604.12989) - [inference engine](https://x.com/TheAhmadOsman/status/2057183854444843202) - [Harbor](https://github.com/av/harbor) - [Z.ai](https://z.ai/) - [Qwen 3.5 27B](https://huggingface.co/Qwen/Qwen3.5-27B) - [Qwen 3.6 27B](https://huggingface.co/Qwen/Qwen3.6-27B) - [PagedAttention](https://arxiv.org/abs/2309.06180) - [vLLM](https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [6:31 AM · May 22, 2026](https://x.com/TheAhmadOsman/status/2057590224729911346) - [171.8K Views](https://x.com/TheAhmadOsman/status/2057590224729911346/analytics) - [View quotes](https://x.com/TheAhmadOsman/status/2057590224729911346/quotes) --- *导出时间: 2026/5/23 09:05:35* --- ## 中文翻译 # LLM 入门：实战指南（2026 年版） **作者**: Ahmad **日期**: 2026-05-21T22:31:52.000Z **来源**: [https://x.com/TheAhmadOsman/status/2057590224729911346](https://x.com/TheAhmadOsman/status/2057590224729911346) ---  从循环开始讲起。文本变成 Tokens（词元）。Tokens 穿过 Transformer。Attention（注意力机制）决定哪些之前的 Tokens 重要。运行时会保留一个 KV 缓存，这样模型就不会每次都重新计算整个对话。然后，模型挑选下一个 Token，并重复这一过程。 > 一篇关于 LLM 如何工作、模型如何一次一个 Token 地思考以及如何在本地运行它们的实战指南。 一旦你理解了这个循环，硬件和软件的选择就会变得更容易推理。VRAM、量化、上下文长度、聊天模板、解码、RAG、服务引擎和模型选择，都源于相同的机制。 > 从循环开始：Token 进，概率出，每次一个下一个 Token。权重告诉模型它学到了哪些模式。上下文告诉模型它现在正在看什么。KV 缓存是保持循环可用的工作内存。硬件、运行时和模型选择只有在你理解了模型所遵守的内存、上下文和格式规则之后才有意义。 目标是首先让本地 LLM 的机制变得直观，然后为你提供一条通往硬件、运行时、服务以及截至 2026 年 5 月 21 日的最新 LLM 研究的实战路径。 ## 焦点 这是一本以模型为核心的指南。它从机制开始：推理、Tokens、Transformers、注意力、KV 缓存、预填充、解码、解码控制、模型包、聊天模板、模型类型、长上下文、RAG、智能体、微调和多模态模型。 之后，它进入本地部署层：本地到底意味着什么、量化、VRAM 数学计算、硬件层级、运行时选择、服务模式、许可证、模型选择、隐私、故障排除、基准测试、设置路径和实战用例。 这个顺序很重要。在选择 GPU 之前，你应该了解为什么长提示词会消耗内存。在评判一个模型之前，你应该了解为什么聊天模板很重要。在关心每秒 Tokens 数之前，你应该了解为什么解码是顺序的。 对于更深入的硬件和软件路径，我有一个三部分系列，专门介绍自托管 LLM / 本地 AI： - 第一部分：LLM 的 GPU 内存数学计算（2026 年版）。 - 第二部分：本地 AI 硬件的内存带宽（2026 年版）。 - 第三部分：LLM 和本地 AI 硬件的推理引擎（2026 年版）。 前两篇文章解释了硬件容量和带宽的数学计算。第三篇解释了将硬件转化为可用推理的软件层。本文首先为你奠定模型方面的基础，然后在机制清晰之后，再回指那些部署层。 ## LLM 到底在做什么  运行模型被称为推理。对于标准的仅解码器 LLM，推理就是一遍又一遍重复的相同循环： 1. 将你的文本转换为 Tokens。 2. 将这些 Tokens 输入模型。 3. 计算每个可能的下一个 Token 的分数。 4. 使用解码策略选择一个 Token。 5. 将该 Token 追加到序列中。 6. 重复直到模型停止、用户打断或达到 Token 限制。 模型并不是一次性写出整个答案。它是一次生成一个 Token。每一个新的 Token 都会成为序列的一部分，进而影响下一个 Token。 从数学上讲，模型是一个学习到的函数： > f(theta, sequence) -> 关于 next_token 的概率分布 其中： - theta 指的是模型权重。 - sequence 指的是提示词加上迄今为止生成的 Tokens。 - Logits 是 Softmax 之前的原始分数。 - Probabilities 是 Softmax 之后的归一化分数。 - Decoding（解码）将这些概率转化为一个选定的 Token。 这就是为什么本地生成速度以每秒 Tokens 数来衡量。你的系统反复运行前向传播，挑选或采样一个 Token，更新 KV 缓存，然后继续。 感知在这里很重要。长的预填充意味着在第一个词出现之前有很长的暂停。慢的解码意味着答案流式输出的速度很慢。本地构建者经常痴迷于解码速度，因为这是用户能感受到的，但当你粘贴一个 10K Token 的文档时，预填充时间才是真正让人头疼的地方。 ## Tokens  LLM 不把原始文本看作单词。它们看到的是 Tokens：内部表示为整数 ID 的小块文本。 一个 Token 可能是： - 一个完整的单词："hello" - 单词片段："inter", "national", "ization" - 标点符号 - 带空格前缀的字符串 - 字节级别的回退 - 特殊控制标记，如 <｜begin of sentence｜>, <｜end of sentence｜>, <｜reserved｜>, 或 <｜user｜> 分词器将文本映射为 Token ID，并将 Token ID 映射回文本。常见的分词器家族包括 BPE 风格和 SentencePiece 风格的分词器。不同的模型家族使用不同的分词器，这很重要。一篇 4000 字的文档在一种分词器中可能是 5000 个 Tokens，而在另一种中可能是 7500 个 Tokens。 词表大小也很重要。词表更大的分词器可以将某些文本压缩成更少的 Tokens，但它也会改变嵌入和输出投影的大小。这就是为什么每秒 Tokens 数在不同模型家族之间不能完全比较的原因之一。 Tokens 很重要，因为它们决定了： - 上下文窗口能容纳多少文本。 - KV 缓存会变得多大。 - 在提示词处理期间你需要支付多少延迟成本。 - 多语言或重代码文本是否高效。 - 模型是否能正确看到特殊的聊天标记。 模型的上下文窗口是它一次能关注的最大 Tokens 数。在 2026 年，常见的本地可用模型的上下文范围从 8K 和 32K 到 128K，服务器级系统中甚至有 256K 和 100 万 Tokens 的上下文。 但是，支持的上下文长度并不等同于便宜、快速或同样准确的上下文。一个技术上可以处理 128K Tokens 的模型在 64K 时可能会慢得像蜗牛，在 100K 时可能会失去连贯性。始终测试你实际计划使用的上下文长度。 Token 是工作的单位。一旦你明白了这一点，长上下文就不再看起来像魔法，而开始像一张你可以估算的账单。 有用的练习：试试我的分词器演示应用，看看文本是如何实时分解成 Tokens 的。 ## Transformers  大多数现代 LLM 基于 Transformer 架构。大多数本地聊天 LLM 是仅解码器的 Transformers：它们在回顾之前 Tokens 的同时预测下一个 Token。 在此之前的一切，包括 Tokens、权重、配置和聊天模板，都是为下面真正的引擎做准备的。Transformer 是移动数字的骨架。 一个简化的 Transformer 层包含： 1. Token 嵌入：Token ID 变成向量。 2. 位置信息：模型需要 Token 顺序。许多现代 LLM 使用 RoPE（旋转位置嵌入），通过旋转表示来编码位置。 3. 自注意力：每个 Token 表示回顾之前的 Token 表示，并决定哪些是重要的。 4. MLP / 前馈块：一种密集的非线性计算，用于扩展和压缩表示。很大一部分参数位于这里。 5. 层归一化和残差连接：这些稳定了深层网络，并帮助信息流经许多层。 6. 输出投影：最终的隐藏状态变成词表上的 logits。 将这个配方堆叠几十或几百次，你就得到了一个语言模型。 Transformer 回顾：Tokens 变成向量，注意力连接序列，MLP 重塑表示，RoPE 保持位置正确，最终投影将最后的隐藏状态变成下一个 Token 的 logits。 ## Attention Attention 是一个 Token 决定哪些早期的 Token 对下一次预测重要的机制。这也是本地推理如此敏感内存的原因之一。 经典的 MHA（多头注意力）为许多头存储独立的键/值状态。它赋予了模型灵活性，但也使 KV 缓存变得很大。 现代本地模型经常使用更高效的注意力设计： - MQA：多个查询头共享一个键/值头。它节省内存，但可能表现力较弱。 - GQA：成组的查询头共享键/值头。这是许多当前本地模型中常见的折中方案。 - MHA：完整的多头注意力。它可能很强大，但长上下文的成本会迅速增加。 现代内核如 FlashAttention 和 SDPA 风格的实现减少了注意力的内存流量，并让 GPU 保持忙碌。拥有良好注意力内核的运行时可能比没有的快得多，即使是在相同的模型和硬件上。 这就是为什么两个 7B 模型在长上下文下的表现可能非常不同。参数数量并不是全部。一个 7B MHA 模型在 128K 上下文下可能会耗尽 24 GB 的 GPU，而具有相同宣传上下文的 7B GQA 模型可能还有富余空间。 比较模型时，要看注意力类型、KV 头数、上下文长度和运行时支持，而不仅仅是参数数量。 ## KV Cache  KV 缓存是模型在生成期间的工作内存。它存储之前 Tokens 的键/值注意力状态，这样模型就不会在每个生成的 Token 上从头重新计算整个历史。 没有 KV 缓存，生成将是极其低效的。有了 KV 缓存，生成是可用的，但缓存消耗的内存与以下成正比： > tokens x layers x kv_heads x head_dim x precision x 2 x 2 是因为键和值。 对于类似旧版 Llama 的 7B MHA 模型，一个有用的经验法则是 FP16 KV 缓存中每个 Token 大约 0.5 MiB。这意味着 4K Tokens 仅 KV 缓存就可能消耗大约 2 GiB。在 32K Tokens 时，你可能仅 KV 缓存就需要 16 GiB。 较新的 GQA/MQA 模型大大减少了这一点。一些运行时也支持 FP8 或 INT8 KV 缓存。这通常是我会向 2026 年的本地用户推荐的实用压缩底线。 不要将低于 8 位的 KV 缓存视为默认设置。KIVI、KVQuant 和较新的压缩缓存内核等研究系统表明，通过仔细的算法、校准和自定义内核，2 位到 4 位的 KV 是可行的。这与在桌面运行时随意打开 Q4 KV 开关是不一样的。在 8 位以下，特别是对于编码、工具调用、JSON、长上下文检索以及精确早期 Token 很重要的任务，要进行严格的基准测试。 另外不要混淆 KV 缓存量化和投机解码。DFlash 和 DDTree（通常简称为 DTree）通过起草未来的 Tokens 并验证它们来攻击解码延迟。它们可以提高速度，但它们不能消除 KV 缓存的内存账单。 这就是为什么模型可以在空提示词下装得下，但在加载长文档时崩溃的原因。权重装得下。工作内存装不下。 ## Prefill And Decode LLM 推理有两种不同的性能机制：预填充和解码。  Prefill 处理你给模型的提示词。如果你粘贴一个 20,000 Token 的文档，模型必须处理这 20,000 个 Tokens 才能产生第一个回答 Token。Prefill 相对容易并行化，所以 GPU 可以高效处理它，但它仍然可能很昂贵。 你等待第一个 Token 出现的时间通常是预填充时间。 Decode 一次生成一个新的 Token。每个生成的 Token 都取决于迄今为止的序列，所以 Decode 要顺序得多。这就是流式打字效果的来源，也是通常决定模型感觉快还是慢的阶段。 长提示词惩罚预填充。长回答惩罚解码。长对话惩罚两者，因为 KV 缓存会增长。 在聊天会话中，每一轮都会增加缓存。如果你让对话运行到 16K Tokens，你每生成一个新的 Token，都要为所有 16K Tokens 支付内存成本。这就是为什么保持无限历史的聊天 UI 最终会变慢或崩溃。 ## Decoding  在模型产生 logits 之后，它还没有写任何东西。它只是给每个可能的下一个 Token 打了分。解码是将这些分数转化为一个实际的 Token，将该 Token 追加到上下文中，并重复循环的策略。 运行时或推理引擎可以通过多种方式选择 Tokens。它可以每次都选择概率最高的 Token。它可以从缩小的可能 Token 集合中采样。它可以惩罚重复。它可以在分隔符处停止。它可以使用固定的种子，以便相同的提示词产生可重现的行为。 这些选择不会改变模型权重，但它们会改变模型的声音、确定性、创造力、风险概况和循环倾向。 重要的旋钮回答三个实际问题： - 随机性：允许多大的变化？ - 尾部触及：采样器可以深入到低概率 Tokens 的程度如何？ - 边界：什么防止循环、漫谈、模式中断或失控输出？ 对于精确的工作，开始时要窄：低温度、短的最大 Token 限制、明确的停止序列，以及当输出必须匹配 JSON 或模式时的约束解码。对于创造性工作，通过更高的温度、top-p 和事后排名的多个候选给采样器更多的空间。对于编码，保持第一遍保守，然后仅在故意探索时才采样替代方案。 贪婪解码并不总是更准确。它往往是脆弱的。贪婪解码器可能会陷入循环或产生通用的答案，因为它从不探索替代方案。对于评估，使用确定性设置。对于构思，让模型呼吸。 ## 模型包包含什么 一个可运行的本地 LLM 不仅仅是一个大的权重文件。模型包通常包括： - 架构/配置：层数、隐藏大小、注意力类型、RoPE 设置、词表大小、特殊 Tokens 和上下文长度。 - 权重：学习的参数，通常存储为 safetensors、GGUF、GPTQ、AWQ、EXL2 或其他特定于运行时的格式。 - 分词器：将文本转换为 Token ID 并将 Token ID 转换回文本的规则。 - 聊天模板：系统、用户、助手、工具和推理消息的确切标记。 - 生成配置：温度、top-p、停止 Tokens、重复惩罚和最大 Tokens 的默认值。 - 许可证和模型卡：关于如何使用模型的法律和操作说明。 权重是最大的文件，但它们不是整个模型。如果分词器、配置或聊天模板错误，相同的权重可能会感觉像是坏的。 包这一节告诉你哪些东西必须一起旅行。下一节解释为什么聊天模板是人们最容易搞乱的部分。 ## Chat Templates  聊天模型是用特定的对话格式训练的。例如，它可能期望类似这样的格式： > <｜begin of sentence｜> 你是一个乐于助人的助手。 <｜user｜> 解释 KV 缓存。 <｜end of sentence｜> 另一个模型可能期望： > [BOS] [INST] 解释 KV 缓存。 [/INST] 另一个可能使用 ChatML 风格的标记。另一个可能需要特殊的推理 Tokens。另一个可能需要工具调用的 XML 或 JSON 包装器。 使用错误的格式会导致乱码、角色混淆、忽略系统提示词、重复提示词、奇怪的拒绝、损坏的工具调用、糟糕的基准测试结果，以及当模板才是真正的错误时得出模型很愚蠢的结论。 最佳实践： - 使用 Transformers 时，使用分词器的 apply_chat_template。 - 在支持 Harbor 的前端、llama.cpp、LM Studio、vLLM 或 SGLang 中使用特定于模型的模板。 - 检查模型是 base、instruct、chat、reasoning 还是 tool-tuned。 - 确保 BOS/EOS Tokens 正确。 - 保持系统提示词简短，除非需要很长。 - 对于工具使用，遵循模型/运行时期望的确切模式。 如果你正在构建……
F From GPT2 to Kimi3, Explained 文章回顾了从 GPT-2 (2019) 到 KimiK3 (2026) 的大语言模型架构演进，重点对比了参数规模从 1.24 亿到 2.8 万亿的巨大跨越。深入解析了 GPT-2 的 Transformer 解码器结构、KV 缓存机制，并探讨了线性注意力机制在降低计算复杂度方面的原理与权衡。 技术 › LLM ✍ ali🕐 2026-07-28 LLMTransformerKimiK3GPT-2AttentionKV Cache架构演进线性注意力
L LLM 推理原理详解：从 Prefill 到 Decode 本文深入解析了大语言模型（LLM）推理的计算流程。文章指出，LLM 推理主要包含 Prefill（处理提示词，计算密集型）和 Decode（逐词生成，显存带宽密集型）两个阶段。作者详细阐述了分词、Embedding、Transformer 层及 KV Cache 的工作原理，并分析了 KV Cache 带来的内存挑战及 vLLM 的优化方案。最后，探讨了 DeepSeek-V4 等新架构通过重新设计注意力机制来从根本上压缩 KV Cache 的创新思路。 技术 › LLM ✍ Avi Chawla🕐 2026-06-29 LLM推理KV CachePrefillDecodeDeepSeekTransformer性能优化
S Step-By-Step LLM Engineering Projects (2026 Edition) 文章提出了一个基于项目的 2026 年 LLM 工程学习路线图。作者主张通过从零构建系统来掌握技术栈，路径涵盖从基础的 Tokenizer、Embedding、Positional Encoding、Attention 机制，到 Transformer 模块构建、训练循环、KV Cache、MoE、量化、服务部署、RAG 及 Agent 等高级系统。该指南强调“实现、绘制、破坏、解释”的学习闭环，旨在帮助读者深入理解大模型原理并具备工程构建能力。 技术 › LLM ✍ Ahmad🕐 2026-05-25 LLMTransformer工程实践AI架构TokenizationAttentionRoPE模型训练Deep Learning学习路线
L LLM 优化面试笔记：训练与推理核心技术 这是一份针对 AI 实验室面试准备的笔记，涵盖了高效训练和部署大语言模型（LLM）的核心优化策略。内容主要分为三部分：内存优化（如 Flash Attention、MQA/GQA、激活检查点）、计算优化（序列打包、高效变体 Transformer）以及推理优化（KV 缓存、投机解码、量化技术）。文章旨在总结在大规模模型开发中应对算力和内存瓶颈的关键技术。 技术 › LLM ✍ Gauri Gupta🕐 2026-05-06 LLM优化推理训练Flash AttentionKV Cache量化面试Transformer性能优化
M Math Behind Large Language Model 本文介绍了大语言模型背后的数学原理，涵盖Attention机制、缩放因子、反向传播、梯度下降、交叉熵损失、RoPE位置编码和RMSNorm归一化等核心概念。 技术 › LLM ✍ Amit Shekhar🕐 2026-07-30 LLM数学原理AttentionTransformer机器学习深度学习梯度下降反向传播RoPERMSNorm
3 30 分钟给你的 Agent 搭好永久记忆 本文介绍了一种为编码 Agent 添加持久记忆的轻量级方案。针对 Agent 上下文窗口有限且容易遗忘历史信息的问题，作者选择了开源项目 EverOS。与传统的黑箱向量数据库不同，EverOS 将记忆存储为本地 Markdown 文件，透明且可编辑。文章详细讲解了从环境准备、安装初始化到验证记忆写入与检索的全过程，帮助开发者在半小时内赋予 Agent 跨会话的长期记忆能力。 技术 › Agent ✍ AYi🕐 2026-06-24 Agent记忆系统EverOS教程向量库MarkdownLLM开发工具本地部署CLI
从 从零构建 LLM 架构并转化为高薪职业的指南 本文详细介绍了如何从零开始学习大语言模型（LLM）架构。内容涵盖了从掌握 Python 基础、理解神经网络和 Transformer 原理，到深入掌握 Tokenization、Embedding、Attention 机制及 RAG 技术等核心概念。文章还提供了具体的学习路线图，并探讨了通过自由职业、构建 SaaS 产品、远程工作及自动化代理等方式将 LLM 技能转化为高美元收入的职业路径。 技术 › LLM ✍ Shabnam Parveen🕐 2026-05-18 LLMTransformerRAG职业发展PythonDeepLearningAI工程创业教程SaaS
大 大语言模型训练与服务背后的数学原理 本文基于 Reiner Pope 的播客内容，深入剖析了大模型在集群中的运行机制。文章通过 Roofline 分析和 KV Cache 等核心概念，解释了推理延迟、Batch Size、API 定价策略及 MoE 架构背后的物理与经济逻辑，揭示了硬件限制如何塑造 AI 进展。 技术 › LLM ✍ Saito🕐 2026-05-01 LLM推理架构设计数学原理MoEAPI定价性能优化Transformer
L LLM 中的 Prompt Caching 技术详解：以 Claude 为例的高效缓存策略 本文深入探讨了 LLM 中的 Prompt Caching 技术，解释了其背后的 KV Cache 机制及静态/动态上下文分离原理。文章通过 Claude Code 的案例分析，展示了如何通过保持 92% 的缓存命中率来将计算成本降低 81%，并总结了哈希敏感性和工程化落地的关键约束。 技术 › LLM ✍ Avi Chawla🕐 2026-04-20 Prompt CachingLLMAgentClaudeKV Cache成本优化系统架构Transformer
图 图解 Transformer 架构：从原理到细节的全面解析 本文深入浅出地解析了 Transformer 架构，它是现代大语言模型（LLM）的核心。文章从宏观视角出发，通过解码的方式，逐一拆解了 Transformer 的关键组件，包括编码器与解码器、分词与嵌入、位置编码、注意力机制（Attention）以及多头注意力等。作者详细解释了该架构如何解决传统模型的“遗忘”和“慢速”问题，实现了并行处理和长距离依赖捕捉，并最后阐述了其强大的原因。 技术 › LLM ✍ Amit Shekhar🕐 2026-04-18 Transformer架构解析注意力机制深度学习AILLM编码器解码器机器学习教程
如 如何成为一名机器学习工程师：完全指南 本文是一份详细的机器学习（ML）工程师学习路径指南。作者首先纠正了被动观看视频的学习误区，提出“两遍学习法”以建立深刻的技术理解。文章明确了ML工程师与数据科学家及研究员的区别，强调工程实现能力。随后，作者将学习路径分为两个阶段：第一阶段利用 3Blue1Brown 的视频构建数学直觉，涵盖神经网络、梯度下降、反向传播及 Transformer 架构；第二阶段（文中截断处未完）预告将通过 Andrej Karpathy 的课程来提升代码实现能力。 技术 › 机器学习 ✍ Arman Hezarkhani🕐 2026-01-21 机器学习学习路径神经网络Transformer深度学习职业发展LLM3Blue1BrownKarpathy教程
C ChatGPT Agent Loop 优化技术解析 本文深入解析了 ChatGPT 如何通过 Harness、API 和 Inference 三层架构优化 Agent 循环，重点介绍了持久化 WebSocket、增量 Token 化、KV 缓存管理和推测解码等技术，以降低成本并提升效率。 技术 › Harness Engineering ✍ Bytebytego🕐 2026-07-30 Agent优化LLM架构ChatGPTOpenAI性能成本控制WebSocketTokenization