Step-By-Step LLM Engineering Projects (2026 Edition) ✍ Ahmad🕐 2026-05-25📦 24.3 KB 🟢 已读 𝕏 文章列表 文章提出了一个基于项目的 2026 年 LLM 工程学习路线图。作者主张通过从零构建系统来掌握技术栈，路径涵盖从基础的 Tokenizer、Embedding、Positional Encoding、Attention 机制，到 Transformer 模块构建、训练循环、KV Cache、MoE、量化、服务部署、RAG 及 Agent 等高级系统。该指南强调“实现、绘制、破坏、解释”的学习闭环，旨在帮助读者深入理解大模型原理并具备工程构建能力。 LLMTransformer工程实践AI架构TokenizationAttentionRoPE模型训练Deep Learning学习路线 # Step-By-Step LLM Engineering Projects (2026 Edition) **作者**: Ahmad **日期**: 2026-05-25T03:01:53.000Z **来源**: [https://x.com/TheAhmadOsman/status/2058745340895870985](https://x.com/TheAhmadOsman/status/2058745340895870985) ---  At some point, reading about LLMs stops being enough. You need to build the stack yourself: Tokenizer first, then embeddings, position, attention, Transformer blocks, objectives, decoding, cache, long context, routing, data, post-training, serving, evaluation, tools, and alignment / safety. > A project-based roadmap for turning LLM fundamentals into working systems you can build, measure, break, and explain.  The Local AI series gives you the hardware, runtime, and model-mechanics foundation. This article is what comes after that foundation clicks: A build order for learning the field by reproducing the important pieces, watching them fail, and writing down what the failure taught you. > Tokenization, attention, KV cache, MoE, quantization, serving, RAG, agents, evaluation, and safety are not separate buzzwords. They are parts of one system.Start with the project loop: Implement the primitive, plot the behavior, break it on purpose, then explain what changed.Products and frameworks make more sense when you have already touched the model, memory, data, inference, and evaluation rules underneath them. The goal is to move from "I understand the concepts" to "I can build it." As of May 2026, that means learning the model architecture, the training loop, the inference path, the data pipeline, the serving layer, and the evaluation habits together. # Focus This is a projects-first guide. It starts with the mechanics: Tokenizers, embeddings, positional methods, attention, multi-head attention, Transformer blocks, training loops, objectives, decoding, speculative decoding, KV cache, MQA, GQA, MLA, long context, efficient attention, and hardware-aware modeling. After that, it moves into the wider systems and research layer: Mixture of Experts, sparse-model trade-offs, state-space and linear-attention alternatives, diffusion-style language models, data pipelines, synthetic data, scaling laws, post-training, RLHF, PPO, GRPO, RLVR, quantization, serving stacks, evaluation harnesses, RAG, tools, agents, multimodal adapters, interpretability, red-team suites, and a complete capstone model system. That order matters. You should understand why tokenization changes context length before training a model. You should understand why attention becomes memory-bound before choosing a serving stack. You should understand why evaluation and failure galleries matter before trusting an empty product demo. For the foundation this roadmap builds on, I have a four-part series teaching self-hosted LLMs / local AI: - Part 1: GPU Memory Math for LLMs (2026 Edition). - Part 2: Memory Bandwidth for Local AI Hardware (2026 Edition). - Part 3: Inference Engines for LLMs & Local AI Hardware (2026 Edition). - Part 4: LLMs 101 (2026 Edition): How Models Think One Token At A Time. The first two pieces explain the hardware capacity and bandwidth math. The third explains the software layer that turns hardware into usable inference. The fourth explains the model-side mechanics: Tokens, Transformers, attention, KV cache, decoding, context, RAG, agents, and local deployment mechanics. This article assumes that foundation and turns it into a sequence of projects you can build, test, and publish. ## The Principle: Each Project Teaches One Hard-Earned Concept For every project below, follow the same loop: Build it. Implement the core idea yourself before using the library version. Plot it. Make loss curves, latency curves, memory curves, attention heatmaps, routing histograms, entropy plots, or failure galleries. Break it. Ablate the thing you just built. Remove positional encodings. Disable causal masks. Quantize too aggressively. Starve the data. Collapse the router. Overload the KV cache. Explain it. Write a short technical note: What you expected, what happened, what surprised you, and what you'd try next. Ship the artifact. A repo, notebook, blog post, small demo, benchmark chart, or reproducible experiment beats vague theory. The stack below moves from tokens to systems, then from systems to research. # Part I: Text Becomes Tensors  ## 1. Build a tokenizer from scratch Before you train a model, you decide how the world becomes symbols. That decision affects compression, multilingual behavior, rare words, code, math, emojis, latency, context usage, and model quality. Start by implementing byte-pair encoding. Train a tiny subword vocabulary on a small corpus. Then compare it with unigram/SentencePiece-style tokenization. BPE became central because it handles rare words by composing them from subword units, while SentencePiece made tokenization trainable directly from raw text without assuming pre-tokenized whitespace-separated words. ## 2. One-hot vectors, learned embeddings, and semantic geometry A token ID by itself has no meaning. Meaning starts when IDs become vectors. Build the simplest possible embedding table and learn it on a next-token objective. # Part II: Position Gives Tokens Order  ## 3. Implement sinusoidal, learned, RoPE, and ALiBi positional methods Attention alone is permutation-invariant. Without position, "dog bites man" and "man bites dog" look too similar. The original Transformer used sinusoidal positional encodings; later systems often use learned positions, rotary position embeddings, ALiBi, or RoPE-scaling variants. RoPE encodes relative position through rotations in embedding space, while ALiBi biases attention scores based on distance and was designed to improve length extrapolation. # Part III: Attention Makes Context Useful  ## 4. Hand-wire scaled dot-product attention for one token Before implementing a Transformer block, implement attention for a single query. Compute Q, K, V manually. Take the dot product. Scale it. Softmax it. Use it to form a weighted sum of values. The Transformer replaced recurrence with attention, making it possible to parallelize training over sequences while letting every token attend to other tokens. That design choice is still the foundation of most LLMs. ## 5. Scale attention to multi-head attention Multi-head attention lets different subspaces learn different relational patterns. Some heads may specialize in local syntax, induction-like copying, delimiter tracking, or long-range dependencies. # Part IV: The Transformer Block  ## 6. Build a single Transformer decoder block Now stack the pieces: Token embedding, positional method, masked multi-head attention, residual connection, normalization, feed-forward network, and output projection. Modern decoder blocks are typically pre-norm, use RMSNorm or LayerNorm variants, and often use gated MLPs such as SwiGLU rather than the oldest ReLU-style feed-forward block. RMSNorm simplifies LayerNorm by focusing on rescaling, and GLU/SwiGLU-style feed-forward layers have been shown to improve Transformer performance in several settings. ## 7. Stack blocks into a "mini-former" Train a tiny decoder-only model on toy text. The point is not to build a useful chatbot. The point is to understand the training loop. # Part V: Objectives Define What The Model Learns  ## 8. Compare causal LM, masked LM, prefix LM, and denoising objectives Different objectives produce different capabilities. BERT used masked language modeling to build bidirectional representations. GPT-style models use causal next-token prediction. T5 framed many NLP tasks as text-to-text generation with denoising objectives. UL2 mixed several denoising modes to cover causal, prefix, and bidirectional behavior. # Part VI: Decoding Turns Probabilities Into Text  ## 9. Build a sampling dashboard A model outputs logits. A product outputs text. Decoding is the bridge. ## 10. Implement speculative decoding Autoregressive decoding is sequential: One token depends on the previous token. Speculative decoding accelerates generation by using a smaller draft model to propose tokens that a larger model verifies, preserving the target distribution when implemented correctly. Newer variants reduce or remove the need for a separate draft model. Medusa adds extra decoding heads to propose multiple future tokens, while lookahead decoding explores parallel candidate n-grams without an auxiliary model. # Part VII: KV Cache And Memory-Bound Inference  ## 11. Build a KV cache During autoregressive inference, past keys and values do not need to be recomputed every step. KV caching stores them. This is one of the most important practical differences between training and inference. ## 12. Implement MQA, GQA, and study MLA Standard multi-head attention gives every query head its own key and value heads. Multi-query attention shares keys and values across query heads to reduce memory bandwidth. Grouped-query attention is a middle ground that keeps several KV heads rather than one, often preserving more quality while improving inference efficiency. By 2026, KV-cache reduction is a major architecture design axis. DeepSeek-V2 introduced Multi-head Latent Attention, a low-rank KV compression approach, and DeepSeek-V3 continued combining MLA with sparse MoE scaling. # Part VIII: Long Context Is A Systems Problem  ## 13. Build sliding-window attention and attention-sink experiments Long context is not solved by increasing a number in a config file. Models can lose information, over-attend to irrelevant tokens, collapse on long sequences, or become too expensive to serve. Mistral 7B popularized a practical combination of sliding-window attention and grouped-query attention in an open model. StreamingLLM showed that preserving early "attention sink" tokens can stabilize streaming generation over very long token streams. ## 14. Extend context with RoPE scaling, YaRN-style interpolation, and memory mechanisms Context extension often combines positional interpolation, fine-tuning, efficient attention, and evaluation. YaRN showed that RoPE-based context extension can be much more data- and compute-efficient than some previous approaches. Infini-attention introduced a compressive memory mechanism designed to scale Transformer-like models to unbounded input lengths with bounded memory and computation. # Part IX: Efficient Attention And Hardware-Aware Modeling  ## 15. Compare naive attention, PyTorch SDPA, and FlashAttention The same mathematical operation can have radically different runtime depending on memory access patterns. FlashAttention made attention faster by reducing high-bandwidth-memory reads and writes. FlashAttention-3 further optimized attention for NVIDIA Hopper GPUs using hardware-aware scheduling, asynchronous operations, and FP8 support. ## 16. Learn the hardware budget: FLOPs, bandwidth, memory, and precision By 2026, LLM engineering is deeply hardware-constrained. NVIDIA Blackwell systems emphasize FP4/FP8 inference and large NVLink domains; DGX B200-class systems expose massive HBM capacity and bandwidth. Google's Ironwood TPU generation is positioned heavily around inference. AMD's MI300X carries 192GB HBM3 per accelerator for memory-heavy workloads. # Part X: Mixture Of Experts  ## 17. Build a two-expert router Sparse Mixture-of-Experts models increase parameter count without activating every parameter for every token. Switch Transformer showed that sparse expert routing could scale model capacity efficiently. Mixtral demonstrated a practical open sparse MoE design where each token routes to a subset of experts, and DeepSeek's MoE work explored finer-grained experts and shared experts. ## 18. Reproduce modern sparse-model trade-offs Recent frontier and open-weight systems use sparse activation heavily. DeepSeek-V3 reports 671B total parameters with 37B activated per token, using MLA, DeepSeekMoE, auxiliary-loss-free load balancing, and multi-token prediction. Llama 4 introduced Meta's first open-weight natively multimodal MoE model family. Qwen3 open-weighted two MoE models, Qwen3-235B-A22B and Qwen3-30B-A3B, plus six dense models. Moonshot's Kimi K2 line continued the large sparse-model trend. The Kimi K2.6 model card describes a native multimodal agentic MoE model with 1T total parameters, 32B activated parameters, 256K context window, and a MoonViT vision encoder. # Part XI: Beyond Vanilla Transformers  ## 19. Implement a toy state-space or linear-attention model Transformers dominate, but they are not the only serious sequence architecture. Mamba introduced selective state-space models with linear-time sequence scaling and strong throughput claims. Mamba-2 connected state-space models and attention more directly through state-space duality. RetNet proposed retention mechanisms that can train in parallel while supporting recurrent-style inference. ## 20. Build a diffusion-style language model toy Autoregressive generation is not the only possible decoding paradigm. LLaDA trained a diffusion-style language model from scratch using masking and denoising, challenging the assumption that high-quality language modeling must be autoregressive. Dream explored open diffusion language modeling with parallel denoising. Inception's Mercury models marketed diffusion LLMs around very high generation speed. # Part XII: Data Is The Real Pretraining Substrate  ## 21. Build a pretraining data pipeline Data quality is one of the highest-impact parts of the stack. FineWeb released a 15-trillion-token dataset built from Common Crawl snapshots with filtering and deduplication choices aimed at strong open pretraining. Dolma provided the open corpus behind OLMo. DataComp-LM showed how standardized data curation experiments can substantially improve model quality at fixed compute. ## 22. Synthetic data: Generate, filter, and prove it helped Synthetic data can help when it is targeted, filtered, and evaluated honestly. The phi-1 work showed that small models trained on high-quality "textbook-like" and synthetic code data could perform surprisingly well on coding benchmarks. But synthetic data can also amplify mistakes, collapse diversity, or contaminate evaluations if used carelessly. # Part XIII: Scaling Laws And Capacity  ## 23. Train tiny, small, and medium models and fit scaling curves Scaling laws quantify how loss changes with model size, data size, and compute. Kaplan-style scaling laws showed smooth power-law relationships over several orders of magnitude. Chinchilla later argued that many large models were undertrained and that compute-optimal training should scale parameters and tokens together more carefully. # Part XIV: Post-Training Turns A Base Model Into An Assistant  ## 24. Fine-tune, instruction-tune, and preference-tune A base model predicts text. An assistant follows instructions. InstructGPT showed that human-feedback fine-tuning can make models more aligned with user intent than simply scaling pretrained models. Direct Preference Optimization later simplified preference training by avoiding an explicit reward model and RL loop in many settings. ## 25. Build a toy RLHF, PPO, GRPO, and RLVR lab Reinforcement learning became especially important for reasoning models. OpenAI's o1 system card emphasized that performance improves with more reinforcement learning and more inference-time thinking. DeepSeek-R1 popularized an open discussion of reasoning-oriented RL, including a pure-RL R1-Zero stage and a multi-stage pipeline for the final model. DeepSeekMath introduced Group Relative Policy Optimization as a PPO-style alternative that reduces memory overhead by avoiding a separate critic model. # Part XV: Quantization And Compression  ## 26. Quantize a model and measure the damage Quantization is not just "make it smaller." It changes numerical behavior, latency, memory bandwidth, and sometimes model quality. GPTQ demonstrated one-shot post-training quantization for very large language models. AWQ showed that protecting a small fraction of salient weights can improve low-bit quantization. GGUF became a widely used file format in the llama.cpp ecosystem for local inference. # Part XVI: Serving Systems  ## 27. Serve the same model through multiple inference stacks LLM serving is its own discipline. vLLM introduced PagedAttention to reduce KV-cache memory waste and support efficient serving. TensorRT-LLM includes features such as in-flight batching, paged attention, quantization, and multi-GPU execution. SGLang emphasizes fast structured generation with features such as RadixAttention, prefix caching, scheduling, speculative decoding, and disaggregated serving. # Part XVII: Evaluation Stops Guessing  ## 28. Build an evaluation harness If you cannot measure a model, you cannot improve it. HELM argued for broad evaluation across accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. MMLU became a standard benchmark for broad knowledge and reasoning across many subjects. The EleutherAI language-model evaluation harness became a common open tool for standardized evaluation. # Part XVIII: Retrieval, Tools, And Agents As Capstones  ## 29. Build retrieval-augmented generation from scratch RAG combines parametric memory in the model with non-parametric memory in an external corpus. The original RAG work showed that retrieval can improve knowledge-intensive tasks and make outputs easier to update or ground. ## 30. Build tool use and agent loops only after you understand the stack Agents are not fake, but they are fragile when treated as magic. ReAct showed that interleaving reasoning traces with actions can improve task solving and interpretability. Toolformer showed that models can learn to use external APIs from self-supervised signals. DSPy reframed LM pipelines as programs whose prompts and weights can be optimized. # Part XIX: Multimodal LLMs  ## 31. Build a tiny vision-language adapter Modern LLMs increasingly consume text, images, audio, video, and structured tool outputs. CLIP showed the power of contrastive image-text pretraining. Flamingo connected frozen vision and language models for few-shot multimodal learning. LLaVA demonstrated visual instruction tuning by connecting a vision encoder to an LLM. By 2026, systems such as Llama 4 and Kimi K2.6 emphasize native multimodality rather than treating vision as a side feature. # Part XX: Interpretability And Failure Modes  ## 32. Study circuits, probes, and sparse autoencoders Mechanistic interpretability tries to understand what models compute internally. Transformer-circuits work decomposed attention and MLP behavior into more interpretable components. Anthropic's sparse-autoencoder work showed that learned features can be more interpretable than individual neurons and can sometimes isolate concepts across large models. ## 33. Build a red-team and safety evaluation suite As models become more capable, safety evaluation becomes part of engineering. Constitutional AI explored using a written set of principles plus AI feedback to reduce reliance on direct human labels. Red-teaming research has shown how manual and automated adversarial attacks can uncover harmful behaviors. Broader AI safety reports synthesize risks across misuse, malfunction, systemic impacts, and loss of control. # Part XXI: The Final Capstone Builds A Complete Small LLM System  ## 34. Train, tune, quantize, serve, evaluate, and document one model Your final project should connect the entire stack. # A realistic lock-in plan A few weeks is enough to change your instincts. A few months is enough to build a serious foundation. Weeks 1-2: Representations and attention Build the tokenizer, embeddings, positional encodings, attention, masking, multi-head attention, and a one-block Transformer. Weeks 3-4: Training and objectives Train a mini-former, compare objectives, build sampling tools, study normalization and activations, and run your first ablations. Weeks 5-6: Inference systems Implement KV cache, MQA/GQA, speculative decoding, quantization, and serving benchmarks. Weeks 7-8: Long context, MoE, and data Build sliding-window attention, context-extension tests, a toy MoE, and a real data pipeline. Weeks 9-10: Post-training and evaluation Run SFT, LoRA/QLoRA, DPO, toy RL, benchmark harnesses, and safety evals. Weeks 11-12: Capstone Train or fine-tune one small model, quantize it, serve it, add RAG/tools, evaluate it, red-team it, and publish the full write-up. The exact schedule matters less than the loop: Build, plot, break, explain. # What to publish after every project For each project, produce five artifacts: 1. Implementation: clean code with tests. 2. Notebook: one reproducible experiment. 3. Plots: at least three charts showing behavior. 4. Failure gallery: examples where the system breaks. 5. Short write-up: what you learned and what changed your mind. Do not just post "I implemented attention." Post the heatmaps. Post the mask bug. Post the entropy curve. Post the latency chart. Post the weird repetition loop. Post the expert-routing collapse. Post the quantization regression. Post the safety failure you did not expect. That is how fundamentals compound. # The mindset Do not get stuck in theory forever. But also do not mistake demos for understanding. An agent framework can hide the model. A serving framework can hide the cache. A benchmark can hide leakage. A product can hide the failure modes. A wrapper can hide the fact that the model is doing all the work. Fundamentals remove the hiding places. Learn tokenization. Learn embeddings. Learn position. Learn attention. Learn residual streams. Learn normalization. Learn objectives. Learn sampling. Learn KV cache. Learn long context. Learn MoE. Learn quantization. Learn data. Learn scaling. Learn post-training. Learn serving. Learn evaluation. Learn safety. Learn failure modes. Then build agents. Build products. Build companies. Build labs. But build them on bedrock. Your future self will thank you. Until next time. -Ahmad ## 相关链接 - [Ahmad](https://x.com/TheAhmadOsman) - [@TheAhmadOsman](https://x.com/TheAhmadOsman) - [3.3K](https://x.com/TheAhmadOsman/status/2058745340895870985/analytics) - [Local AI](https://x.com/TheAhmadOsman/status/2058617968628551718) - [GPU Memory Math for LLMs (2026 Edition)](https://x.com/TheAhmadOsman/status/2040103488714068245) - [Memory Bandwidth for Local AI Hardware (2026 Edition)](https://x.com/TheAhmadOsman/status/2041331757329285589) - [Inference Engines for LLMs & Local AI Hardware (2026 Edition)](https://x.com/TheAhmadOsman/status/2057183854444843202) - [LLMs 101 (2026 Edition): How Models Think One Token At A Time](https://x.com/TheAhmadOsman/status/2057590224729911346) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [11:01 AM · May 25, 2026](https://x.com/TheAhmadOsman/status/2058745340895870985) - [3,337 Views](https://x.com/TheAhmadOsman/status/2058745340895870985/analytics) - [View quotes](https://x.com/TheAhmadOsman/status/2058745340895870985/quotes) --- *导出时间: 2026/5/25 11:59:09* --- ## 中文翻译 # 循序渐进的大语言模型工程项目（2026 版） **作者**: Ahmad **日期**: 2026-05-25T03:01:53.000Z **来源**: [https://x.com/TheAhmadOsman/status/2058745340895870985](https://x.com/TheAhmadOsman/status/2058745340895870985) ---  在某个阶段，仅靠阅读关于大语言模型（LLM）的文章已经不够了。你需要亲自构建整个技术栈：首先是分词器，然后是嵌入、位置编码、注意力机制、Transformer 块、目标函数、解码、缓存、长上下文、路由、数据、训练后处理、服务、评估、工具以及对齐/安全。 > 一份基于项目的路线图，旨在将 LLM 基础知识转化为你可以构建、衡量、破坏和解释的可用系统。  “本地 AI”系列为你提供了硬件、运行时和模型机制的基础。这篇文章则是建立在该基础融会贯通之后的进阶内容：通过复现重要组件、观察其失效并记录从失败中学到的教训来学习这一领域的构建顺序。 > 分词、注意力、KV 缓存、MoE、量化、服务、RAG、智能体、评估和安全并不是彼此割裂的流行词。它们是同一个系统的组成部分。从项目循环开始：实现原语，绘制行为曲线，故意破坏它，然后解释发生了什么变化。只有当你已经接触过底层的模型、内存、数据、推理和评估规则时，各类产品和框架才更有意义。 目标是从“我理解这些概念”转变为“我可以构建它”。截至 2026 年 5 月，这意味着要一起学习模型架构、训练循环、推理路径、数据流水线、服务层和评估习惯。 # 重点 这是一个项目优先的指南。它从机制开始：分词器、嵌入、位置方法、注意力、多头注意力、Transformer 块、训练循环、目标函数、解码、投机解码、KV 缓存、MQA、GQA、MLA、长上下文、高效注意力以及硬件感知建模。 之后，它进入更广泛的系统和研究层：混合专家、稀疏模型的权衡、状态空间和线性注意力的替代方案、扩散风格的语言模型、数据流水线、合成数据、缩放定律、训练后处理、RLHF、PPO、GRPO、RLVR、量化、服务栈、评估工具、RAG、工具、智能体、多模态适配器、可解释性、红队测试套件以及一个完整的顶点模型系统。 这个顺序很重要。在训练模型之前，你应该理解为什么分词会改变上下文长度。在选择服务栈之前，你应该理解为什么注意力会受限于内存。在信任一个空洞的产品演示之前，你应该理解为什么评估和失败案例集很重要。 关于本路线图所依赖的基础，我有一个四部分的系列教程，专门讲解自托管 LLM / 本地 AI： - 第 1 部分：LLM 的 GPU 内存数学（2026 版）。 - 第 2 部分：本地 AI 硬件的内存带宽（2026 版）。 - 第 3 部分：LLM 与本地 AI 硬件的推理引擎（2026 版）。 - 第 4 部分：LLM 入门 101（2026 版）：模型如何逐个 Token 进行思考。 前两部分解释了硬件容量和带宽的数学原理。第三部分解释了将硬件转化为可用推理的软件层。第四部分解释了模型侧的机制：Token、Transformer、注意力、KV 缓存、解码、上下文、RAG、智能体和本地部署机制。本文假设你已经具备了这些基础，并将其转化为你可以构建、测试和发布的一系列项目。 ## 原则：每个项目教授一个来之不易的概念 对于下面的每个项目，都遵循相同的循环： 构建它。在使用库版本之前，先自己实现核心思想。 绘制它。制作损失曲线、延迟曲线、内存曲线、注意力热图、路由直方图、熵图或失败案例集。 破坏它。消融你刚刚构建的东西。移除位置编码。禁用因果掩码。过度量化。削减数据。使路由器崩溃。超载 KV 缓存。 解释它。写一篇简短的技术笔记：你预期什么，发生了什么，什么让你惊讶，以及你接下来会尝试什么。 发布成果。一个仓库、笔记本、博客文章、小型演示、基准测试图表或可复现的实验，胜过模糊的理论。 下面的技术栈从 Token 到系统，然后从系统到研究。 # 第一部分：文本变成张量  ## 1. 从零开始构建分词器 在训练模型之前，你需要决定世界如何转化为符号。这个决定会影响压缩率、多语言行为、生僻词、代码、数学、表情符号、延迟、上下文使用和模型质量。 从实现字节对编码（BPE）开始。在一个小型语料库上训练一个微小的子词表。然后将其与 unigram/SentencePiece 风格的分词进行比较。BPE 之所以成为核心，是因为它通过子词单元组合来处理生僻词，而 SentencePiece 使分词可以直接从原始文本中训练，而不需要假设预先分词的空格分隔的单词。 ## 2. 独热向量、学习的嵌入和语义几何 Token ID 本身没有意义。当 ID 变成向量时，意义才开始。构建最简单的嵌入表，并在“下一个 Token”的目标上学习它。 # 第二部分：位置赋予 Token 顺序  ## 3. 实现正弦、学习、RoPE 和 ALiBi 位置方法 仅凭注意力机制是排列不变的。没有位置信息，“狗咬人”和“人咬狗”看起来太相似了。最初的 Transformer 使用正弦位置编码；后来的系统通常使用学习的位置、旋转位置嵌入、ALiBi 或 RoPE 缩放变体。RoPE 通过嵌入空间中的旋转来编码相对位置，而 ALiBi 基于距离对注意力分数施加偏差，旨在改善长度外推能力。 # 第三部分：注意力使上下文有用  ## 4. 为单个 Token 手动连接缩放点积注意力 在实现 Transformer 块之前，先为单个查询实现注意力。手动计算 Q、K、V。进行点积。缩放它。对它进行 Softmax。使用它来形成值的加权和。 Transformer 用注意力取代了循环，使得可以在序列上并行化训练，同时让每个 Token 关注其他 Token。这个设计选择仍然是大多数 LLM 的基础。 ## 5. 将注意力扩展为多头注意力 多头注意力让不同的子空间学习不同的关系模式。某些头可能专精于局部语法、类似归纳复制的模式、定界符跟踪或长距离依赖。 # 第四部分：Transformer 块  ## 6. 构建单个 Transformer 解码器块 现在将各个部分堆叠起来：Token 嵌入、位置方法、掩码多头注意力、残差连接、归一化、前馈网络和输出投影。 现代解码器块通常是预归一化的，使用 RMSNorm 或 LayerNorm 变体，并且通常使用门控 MLP（如 SwiGLU），而不是最古老的 ReLU 风格的前馈块。RMSNorm 通过专注于重新缩放简化了 LayerNorm，而 GLU/SwiGLU 风格的前馈层已被证明在多种设置下能提高 Transformer 的性能。 ## 7. 将块堆叠成“mini-former” 在玩具文本上训练一个微小的仅解码器模型。重点不是构建一个有用的聊天机器人。重点是理解训练循环。 # 第五部分：目标函数定义模型学习的内容  ## 8. 比较因果 LM、掩码 LM、前缀 LM 和去噪目标函数 不同的目标函数产生不同的能力。BERT 使用掩码语言建模来构建双向表示。GPT 风格的模型使用因果性的下一个 Token 预测。T5 将许多 NLP 任务构建为具有去噪目标的文本到文本生成。UL2 混合了几种去噪模式以覆盖因果、前缀和双向行为。 # 第六部分：解码将概率转化为文本  ## 9. 构建采样仪表板 模型输出 Logits。产品输出文本。解码是桥梁。 ## 10. 实现投机解码 自回归解码是顺序的：一个 Token 依赖于前一个 Token。投机解码通过使用较小的草稿模型提议 Token，然后由较大的模型进行验证来加速生成，如果实现正确，可以保留目标分布。 较新的变体减少或消除了对单独草稿模型的需求。Medusa 添加了额外的解码头以提议多个未来的 Token，而前瞻解码则在无需辅助模型的情况下探索并行的候选 n-gram。 # 第七部分：KV 缓存与内存受限推理  ## 11. 构建 KV 缓存 在自回归推理期间，过去的键和值不需要每一步都重新计算。KV 缓存存储它们。这是训练和推理之间最重要的实际区别之一。 ## 12. 实现 MQA、GQA 并研究 MLA 标准的多头注意力为每个查询头提供自己的键和值头。多查询注意力在查询头之间共享键和值以减少内存带宽。分组查询注意力是一个折中方案，它保留几个 KV 头而不是一个，通常在提高推理效率的同时保留更多质量。 到 2026 年，KV 缓存减少已成为一个主要的架构设计轴线。DeepSeek-V2 引入了多头潜在注意力，这是一种低秩 KV 压缩方法，而 DeepSeek-V3 继续将 MLA 与稀疏 MoE 扩展相结合。 # 第八部分：长上下文是一个系统问题  ## 13. 构建滑动窗口注意力和注意力汇实验 长上下文不能仅仅通过增加配置文件中的一个数字来解决。模型可能会丢失信息、过度关注无关的 Token、在长序列上崩溃，或者变得服务成本过高。 Mistral 7B 在一个开放模型中普及了滑动窗口注意力和分组查询注意力的实用组合。StreamingLLM 表明，保留早期的“注意力汇”Token 可以在非常长的 Token 流上稳定流式生成。 ## 14. 使用 RoPE 缩放、YaRN 风格的插值和内存机制扩展上下文 上下文扩展通常结合位置插值、微调、高效注意力和评估。YaRN 表明，基于 RoPE 的上下文扩展比以前的一些方法在数据和计算效率上要高得多。Infini-attention 引入了一种压缩内存机制，旨在将类 Transformer 模型扩展到具有有限内存和计算的无界输入长度。 # 第九部分：高效注意力与硬件感知建模  ## 15. 比较朴素注意力、PyTorch SDPA 和 FlashAttention 相同的数学运算根据内存访问模式的不同，运行时可能会有根本性的差异。FlashAttention 通过减少高带宽内存的读写使注意力变得更快。FlashAttention-3 使用硬件感知调度、异步操作和 FP8 支持进一步针对 NVIDIA Hopper GPU 优化了注意力。 ## 16. 了解硬件预算：FLOPs、带宽、内存和精度 到 2026 年，LLM 工程深受硬件限制。NVIDIA Blackwell 系统强调 FP4/FP8 推理和大型 NVLink 域；DGX B200 级系统展示了巨大的 HBM 容量和带宽。Google 的 Ironwood TPU 代系主要围绕推理进行定位。AMD 的 MI300X 每个加速器配备 192GB HBM3，用于内存密集型工作负载。 # 第十部分：混合专家  ## 17. 构建一个双专家路由器 稀疏混合专家模型增加了参数数量，而不会为每个 Token 激活每个参数。Switch Transformer 表明，稀疏专家路由可以有效地扩展模型容量。Mixtral 展示了一种实用的开放稀疏 MoE 设计，其中每个 Token 路由到专家的一个子集，而 DeepSeek 的 MoE 工作探索了更细粒度的专家和共享专家。 ## 18. 复现现代稀疏模型的权衡 最近的前沿和开放权重系统大量使用稀疏激活。DeepSeek-V3 报告称总参数为 671B，每个 Token 激活 37B，使用 MLA、DeepSeekMoE、无辅助损失的负载平衡和多 Token 预测。Llama 4 推出了 Meta 首个开放权重的原生多模态 MoE 模型系列。Qwen3 开放了两个 MoE 模型 Qwen3-235B-A22B 和 Qwen3-30B-A3B，外加六个密集模型。 Moonshot 的 Kimi K2 系列延续了大型稀疏模型的趋势。Kimi K2.6 模型卡描述了一个原生多模态智能体 MoE 模型，总参数为 1T，激活参数为 32B，上下文窗口为 256K，并配备 MoonViT 视觉编码器。 # 第十一部分：超越普通 Transformer  ## 19. 实现一个玩具状态空间或线性注意力模型 Transformer 占据主导地位，但它们并非唯一严肃的序列架构。Mamba 引入了具有线性时间序列缩放和强大吞吐量声称的选择性状态空间模型。Mamba-2 通过状态空间二元性更直接地连接了状态空间模型和注意力。RetNet 提出了保留机制，可以并行训练同时支持循环风格的推理。 ## 20. 构建一个扩散风格的语言模型玩具 自回归生成并非唯一的可能解码范式。LLaDA 从头开始训练了一个扩散风格的语言模型，使用掩码和去噪，挑战了高质量语言建模必须自回归的假设。Dream 使用并行去噪探索了开放扩散语言建模。Inception 的 Mercury 模型围绕极高的生成速度推销扩散 LLM。 # 第十二部分：数据是真正的预训练基底  ## 21. 构建预训练数据流水线 数据质量是堆栈中影响最大的部分之一。FineWeb 发布了一个基于 Common Crawl 快照构建的 15 万亿 Token 数据集，其过滤和去重选择旨在实现强大的开放预训练。Dolma 提供了 OLMo 背后的开放语料库。DataComp-LM 表明，标准化的数据筛选实验可以在固定计算下大幅提高模型质量。 ## 22. 合成数据：生成、过滤并证明它有帮助 当合成数据有针对性、经过过滤并得到诚实的评估时，它会有所帮助。phi-1 的工作表明，在高质量的“教科书式”和合成代码数据上训练的小模型可以在编码基准测试中表现出惊人的好成绩。但如果使用不当，合成数据也可能放大错误、导致多样性崩溃或污染评估。 # 第十三部分：缩放定律与容量  ## 23. 训练微型、小型和中型模型并拟合缩放曲线 缩放定律量化了损失如何随模型大小、数据大小和计算而变化。Kaplan 风格的缩放定律显示了跨越几个数量级的平滑幂律关系。Chinchilla 后来辩称，许多大型模型训练不足，计算最优的训练应该更仔细地一起扩展参数和 Token。 # 第十四部分：训练后处理将基础模型转变为助手  ## 24. 微调，
M Math Behind Large Language Model 本文介绍了大语言模型背后的数学原理，涵盖Attention机制、缩放因子、反向传播、梯度下降、交叉熵损失、RoPE位置编码和RMSNorm归一化等核心概念。 技术 › LLM ✍ Amit Shekhar🕐 2026-07-30 LLM数学原理AttentionTransformer机器学习深度学习梯度下降反向传播RoPERMSNorm
L LLM 原理与实践指南 (2026 版) 这是一份关于大语言模型（LLM）的实践指南。文章从基础循环机制出发，详细解释了文本如何转化为 Token，以及 Transformer、注意力机制、KV Cache 和 RoPE 等核心概念的工作原理。作者强调理解模型内部的推理机制（如 Prefill 和 Decode 阶段）对于硬件选型、显存估算和本地部署的重要性。文章涵盖了 Token 化、上下文窗口、量化、模型服务及本地 AI 硬件数学计算等关键主题，旨在为深入理解 LLM 提供直观的基础。 技术 › LLM ✍ Ahmad🕐 2026-05-23 LLMTransformer本地部署Tokenization推理Attention硬件教程KV Cache
F From GPT2 to Kimi3, Explained 文章回顾了从 GPT-2 (2019) 到 KimiK3 (2026) 的大语言模型架构演进，重点对比了参数规模从 1.24 亿到 2.8 万亿的巨大跨越。深入解析了 GPT-2 的 Transformer 解码器结构、KV 缓存机制，并探讨了线性注意力机制在降低计算复杂度方面的原理与权衡。 技术 › LLM ✍ ali🕐 2026-07-28 LLMTransformerKimiK3GPT-2AttentionKV Cache架构演进线性注意力
如 如何从零开始构建自己的大语言模型 (LLM) 文章揭秘了 GPT、Claude 等 LLM 背后通用的五阶段构建流程，指出数据质量而非架构才是核心。作者详细阐述了数据清洗、分词、训练目标（预测下一个 Token）及 Transformer 架构实现，并提供了包含 Tokenizer 和 PyTorch 模型的代码示例。 技术 › LLM ✍ Rahul🕐 2026-06-14 LLMTransformer深度学习模型训练分词PythonPyTorch源码解析方法论
从 从零构建 LLM 的五个阶段：GPT 与 Claude 背后的完整技术管线 文章深入解析了构建大型语言模型的完整五个阶段，指出架构并非关键，数据、评估和系统才是核心。内容涵盖预训练、数据处理、扩展定律、后训练（SFT与RLHF）以及评估系统，揭示了从原始文本到智能助手的技术路径。 技术 › LLM ✍ Codez🕐 2026-05-26 LLMTransformer预训练数据处理RLHF模型训练技术架构AI工程
H How to Build LLM Architectures From Scratch 本文是一篇关于从零构建大型语言模型（LLM）架构的深度指南。文章详细解释了 LLM 的工作原理，包括从原始数据收集、清洗、分词，到 Transformer 架构的核心组件（如自注意力机制、位置编码）。同时涵盖了预训练、微调、基于人类反馈的强化学习（RLHF）以及推理优化等关键技术环节，旨在帮助读者理解 ChatGPT 和 Claude 等模型背后的系统构建全流程。 技术 › LLM ✍ Shabnam Parveen🕐 2026-05-25 LLMTransformer架构设计RLHF深度学习人工智能模型训练OpenAIChatGPT技术原理
T Top Companies Are Secretly Working on This (It Will Replace LLMs) 文章探讨了顶级科技公司正在秘密研发的架构转变——状态空间模型（SSM）。旨在解决 Transformer 架构在长序列处理中的计算成本高和内存消耗大的问题。文章详细解析了 SSM 的工作原理、相比 LLM 的优势（线性时间复杂度、恒定内存），并展示了 NVIDIA、IBM 和 Microsoft 等公司在 Nemotron、Bamba 和 SambaY 等混合架构上的实际应用进展。 技术 › LLM ✍ Siddharth🕐 2026-05-06 SSMTransformerMamba架构NvidiaDeep Learning推理优化LLMInferenceState Space Models
多 多Agent协同方案：让OpenClaw学会团队合作 文章介绍了如何利用 OpenClaw 搭建一套多 Agent 协同工作流。通过设置“小龙虾”、“码力”、“笔锋”和“谋士”四个专精 Agent，并配合 MemOS 共享记忆系统，实现了各司其职、成本优化与质量提升的平衡。作者分享了架构设计、搭建过程、实战数据及踩坑经验，展示了多 Agent 协作在实际任务流转中的高效价值。 技术 › Agent ✍ Jason Zhu🕐 2026-04-02 OpenClaw多AgentMemOS团队协作AI架构自动化成本优化LLM工程实践技术方案
C ChatGPT Agent Loop 优化技术解析 本文深入解析了 ChatGPT 如何通过 Harness、API 和 Inference 三层架构优化 Agent 循环，重点介绍了持久化 WebSocket、增量 Token 化、KV 缓存管理和推测解码等技术，以降低成本并提升效率。 技术 › Harness Engineering ✍ Bytebytego🕐 2026-07-30 Agent优化LLM架构ChatGPTOpenAI性能成本控制WebSocketTokenization
K Karpathy 发布 12 页指南：构建自我改进的多代理图 文章解读了 Andrej Karpathy 关于构建自我改进多代理图的 12 页指南。核心内容包括 Loop、Chain、Swarm 等架构模式，以及利用知识图谱作为共享内存的重要性。文中对比了 Karpathy 的工程实践与 Anthropic 的动态工作流，强调了未来 AI 应用的壁垒在于图谱工程和多智能体协作。 技术 › Agent ✍ Hux🕐 2026-07-29 多智能体Andrej Karpathy知识图谱SwarmAnthropicAI架构工程实践自我改进
实 实战踩坑：便宜模型执行、贵模型编排？没这么简单！ 文章通过三个真实案例，验证了“贵模型编排、便宜模型执行”这一省钱策略。作者发现，虽然便宜模型在文本判定等任务上能打平顶尖模型，但在高复杂度代码和开放式视觉任务上存在局限。关键在于控制模型能力代差和任务可验证性，否则返工成本将抵消节省的 token 费用。 技术 › Agent ✍ WquGuru🕐 2026-07-28 LLMAgent模型编排成本优化Claude Code多模型协作工程实践
W Why Software Factories Fail 文章探讨了AI驱动的软件工厂模式面临的挑战。作者指出，尽管AI编码工具看似能提高效率，但实际应用中往往导致代码质量下降、Bug增多和事故频发。问题不在于技术技能不足，而是模型训练的根本性限制。 技术 › Harness Engineering ✍ dex🕐 2026-07-25 AI软件工厂代码质量模型训练技术挑战自动化工程实践