# How to build your own LLM from scratch in 5 Stages: exact pipeline behind GPT and Claude
**作者**: Codez
**日期**: 2026-05-25T14:02:47.000Z
**来源**: [https://x.com/0xCodez/status/2058911661973454915](https://x.com/0xCodez/status/2058911661973454915)
---
I pulled apart how large language models are actually built - the entire pipeline behind ChatGPT, Claude, and Gemini - and compressed it into one map.
Bookmark this. Save it. By the end you will understand the exact five-stage path from raw internet text to a model that answers like an assistant.
> Follow my Substack to get fresh AI alpha: movez.substack.com
That is not hyperbole. Most people think building an LLM is about the architecture. The core lesson here is that the architecture is the part that matters least.
## The lie everyone believes about LLMs
Ask most people how you build a model like Claude and they will say "transformers." As if the secret is the neural network design.
It is not. The transformer architecture is largely standardized and freely published. Every major lab uses roughly the same building blocks. If architecture were the moat, everyone would have GPT-4.
Here is the line that reframes everything: in practice it is data, evaluation, and systems that make or break a model - not architectural tweaks. The best models are not just trained. They are engineered.
So this guide is built around what actually matters. Five stages. The architecture is a footnote inside Stage 1. The other four are where real models are won and lost.
## 01. Pretraining - teach model language itself.
Everything starts with one deceptively simple objective: predict the next word. This is autoregressive language modeling. Given a sequence of words, the model learns the probability distribution of what comes next.
Do this across enough text and the model absorbs grammar, facts, and reasoning patterns - not because anyone taught them, but because predicting the next word well requires them.
> Tokenization comes first
Before the model sees text, the text is broken into tokens. The standard method is Byte-Pair Encoding (BPE), and its logic shapes everything downstream.
The architecture - the part that matters least
The model is a transformer. That is essentially it for this section, and that is the point. You do not win by inventing a cleverer transformer. You win at the other four stages.
The lecture proves it with scaling curves: transformers simply have a better constant and slope than LSTMs - pick the standard tool and move on.
## 02. Data - where models are actually won.
If architecture matters least, data matters most. This is the stage that separates a good model from a mediocre one, and the one most people underestimate.
The pipeline starts with Common Crawl - a scrape of the public web so large it is measured in petabytes: 250 billion pages, over a million gigabytes. But raw web data is filthy.
Turning it into training material is a brutal multi-step filter.
The processing pipeline looks like this:
- Extract text from HTML - handling special cases like math and boilerplate.
- Filter undesirable content - NSFW, harmful, personal data.
- Deduplicate by URL, document, and line - the web repeats endlessly (headers, footers, menus).
- Heuristic filtering - remove low-quality docs by word count, outlier tokens, dirty tokens.
- Model-based filtering - predict whether a page could be referenced by Wikipedia.
- Data mix - classify into categories (code, books, entertainment) and reweight domains using scaling laws.
The refrain worth burning in: data quality trumps quantity. Collecting data well is the key part of practical LLM work - and the most secretive.
> Collecting data well is ~the key to a practical LLM - and the most guarded secret in the field.
Closed datasets dwarf open ones: LLaMA 3 trained on 15 trillion tokens; GPT-4 on an estimated 13 trillion.
## 03. Scaling laws - spend compute optimally.
You have 10,000 GPUs for a month. What model do you train? Bigger, or trained on more data? Guessing wastes millions. Scaling laws answer it predictably.
The empirical finding: more data and larger models reliably mean better performance, and you can predict a model's performance from its size and data before training it.
The modern pipeline tunes hyperparameters on small models, then extrapolates up the curve to the one huge final run.
The famous Chinchilla answer: roughly 20 tokens of training data per parameter is compute-optimal. But that is for training cost alone.
Once you account for the cost of running the model - inference - the practical ratio rises sharply, past 150 tokens per parameter. You train a smaller model on far more data because you pay to run it millions of times.
And the meta-lesson, the "bitter lesson": don't over-complicate. Do the simple things and scale them. In the long run, the only thing that matters is leveraging computation.
## 04. Post-training - turn a predictor into an assistant.
After pretraining you have something powerful but useless for chat. It completes text, but it does not know it is supposed to answer you.
Ask it a question and it might reply with three more questions - a perfectly plausible next-word continuation.
> Supervised Fine-Tuning (SFT)
You show the model thousands of examples - a prompt followed by a good response - and it learns to imitate that pattern. This is behavior cloning, and it was the key step from GPT-3 to ChatGPT.
The surprising part: you need very little data. A few thousand examples is enough, because SFT only teaches the format of a good answer - the knowledge is already in the pretrained model.
The Alpaca project even generated its data with another LLM: 52,000 instruction-response pairs, used to fine-tune a LLaMA 7B into a capable assistant.
> RLHF - align with human preference
SFT has three problems: it is bound by human ability, it teaches hallucination (cloning a "correct" answer the model doesn't actually know teaches it to make things up), and ideal answers are expensive. RLHF fixes this by optimizing for preference, not imitation.
The model generates two answers. A human picks the better one. Those preferences train a reward model, and the LLM is optimized to maximize that reward - classically with PPO.
A simpler modern alternative, DPO, reaches comparable quality with plain supervised learning and is now standard in the open-source community.
## 05. Evaluation & systems - prove it works, make it feasible.
Two things wrap around the whole pipeline. Skip either and you do not have a real model.
> Evaluation: measuring something open-ended
During pretraining the metric is perplexity - how many tokens the model is "hesitating" between. Between 2017 and 2023, the best models dropped from hesitating among ~70 tokens to fewer than 10. But perplexity breaks after alignment, so evaluation shifts to benchmarks and comparisons:
- MMLU & HELM - task suites with gold answers across many domains. MMLU is the most trusted pretraining benchmark.
- Chatbot Arena - humans blind-compare two models and vote; 300K+ votes power an Elo leaderboard.
- AlpacaEval - an LLM judges other LLMs. 98% correlation with Chatbot Arena, under 3 minutes and under $10 - but it has biases, like favoring longer answers.
The honest takeaway: evaluating an aligned model is genuinely hard, and no single number captures it. The same model can score 0.637 or 0.488 on MMLU depending only on the prompt format.
> Systems: making training physically possible
Everyone is bottlenecked by compute - GPUs are expensive, scarce, and physically limited by communication speed. A 7B model needs ~112GB just to train naively. So the systems layer is not optional; it is what makes the whole thing feasible:
- Low precision - 16-bit (bf16) instead of 32-bit, halving memory and boosting speed.
- Operator fusion & tiling - minimize slow trips to global memory; FlashAttention alone gives a ~1.7x end-to-end speedup.
- Data parallelism - split the dataset across GPUs (sharding optimizer state with ZeRO to save memory).
- Model parallelism - split the model across GPUs by layer (pipeline) or by matrix (tensor).
- Sparsity - Mixture of Experts: more parameters, same FLOPs, by activating only a subset per token.
> What this whole pipeline teaches
Walk back through the five stages and the thesis is undeniable. Architecture - the part everyone obsesses over - got the least attention. Data, scaling, alignment, evaluation, and systems are where every real decision gets made.
That is why two labs with the same architecture produce wildly different models. The architecture is shared. Everything that matters is not.
## The mistakes that sink LLM projects
- Obsessing over architecture. The most copied, least differentiating part of the stack.
- Treating data as a commodity. Dirty data caps your ceiling no matter the compute. Quality over quantity.
- Skipping the Chinchilla math. A model too big for its data is undertrained and wastes compute.
- Stopping at SFT. A fine-tuned model imitates; without RLHF or DPO it never learns what people prefer.
- Trusting perplexity after alignment. Post-training changes the distribution - perplexity stops being meaningful.
## Conclusion:
> A great model is not trained. It is engineered.
Most people will keep believing that building an LLM is about the architecture, keep reading transformer explainers, and keep missing where the real work happens.
The ones who understand the pipeline will see it clearly: language modeling, then clean data, then optimal scaling, then alignment, then honest evaluation on efficient systems. Five stages. The architecture is one paragraph of one of them.
Pick one stage you have been ignoring — probably data or evaluation. Go deep on it. That is where the difference lives.
## 相关链接
- [Codez](https://x.com/0xCodez)
- [@0xCodez](https://x.com/0xCodez)
- [135K](https://x.com/0xCodez/status/2058911661973454915/analytics)
- [movez.substack.com](https://movez.substack.com/)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [10:02 PM · May 25, 2026](https://x.com/0xCodez/status/2058911661973454915)
- [135.7K Views](https://x.com/0xCodez/status/2058911661973454915/analytics)
- [View quotes](https://x.com/0xCodez/status/2058911661973454915/quotes)
---
*导出时间: 2026/5/26 09:15:52*
---
## 中文翻译
# 如何从零开始构建你自己的大语言模型：GPT 和 Claude 背后的完整流程分为 5 个阶段
**作者**: Codez
**日期**: 2026-05-25T14:02:47.000Z
**来源**: [https://x.com/0xCodez/status/2058911661973454915](https://x.com/0xCodez/status/2058911661973454915)
---
我拆解了大型语言模型是如何实际构建的——也就是 ChatGPT、Claude 和 Gemini 背后的完整流程——并将其浓缩成了一张图谱。
请收藏这篇文章。保存下来。读到最后，你将理解从原始互联网文本到一个能像助手一样回答问题的模型，这精确的五个阶段路径。
> 订阅我的 Substack 获取最新的 AI 内部消息：movez.substack.com
这并非夸大其词。大多数人认为构建大语言模型（LLM）的关键在于架构。这里的核心结论是：架构是最不重要的部分。
## 每个人对 LLM 的误解
问大多数人如何构建像 Claude 这样的模型，他们会说是“Transformer”。好像秘密就在于神经网络设计一样。
并非如此。Transformer 架构基本上已经标准化并免费公开了。每个主要实验室使用的构建模块大致相同。如果架构是护城河，那么每个人早就拥有 GPT-4 了。
这句话能重塑一切观念：在实践中，是数据、评估和系统决定了一个模型的成败——而不是架构的调整。最好的模型不仅仅是被训练出来的，它们是被**工程化**出来的。
因此，本指南围绕着真正重要的事情展开。五个阶段。架构只是第一阶段中的一个注脚。另外四个阶段才是决定模型胜负的关键。
## 01. 预训练 —— 教会模型语言本身。
一切都始于一个看似简单的目标：预测下一个词。这就是自回归语言建模。给定一个词序列，模型学习下一个词出现的概率分布。
在足够多的文本上做这件事，模型就会吸收语法、事实和推理模式——不是因为有人教过它，而是因为要准确预测下一个词就必须掌握这些。
> 首先是分词
在模型看到文本之前，文本会被分解成词元。标准方法是字节对编码（BPE），其逻辑影响着后续的一切。
架构 —— 最不重要的部分
模型是一个 Transformer。这部分基本上就结束了，重点就在这里。你不会通过发明一个更聪明的 Transformer 来获胜。你是在其他四个阶段获胜。
课程通过扩展曲线证明了这一点：Transformer 仅仅比 LSTM 具有更好的常数和斜率——选择标准工具，继续前进即可。
## 02. 数据 —— 决定模型胜负的关键。
如果说架构最不重要，那么数据就最重要。这一阶段是区分好模型和普通模型的分水岭，也是大多数人低估的一环。
流程始于 Common Crawl——一个对公共网络的抓取，规模大到以拍字节（PB）衡量：2500 亿个页面，超过 100 万 GB。但原始网络数据非常脏杂。
将其转化为训练材料需要经过残酷的多步过滤。
处理流程如下：
- 从 HTML 中提取文本 —— 处理数学和样板文本等特殊情况。
- 过滤不良内容 —— 色情、有害、个人数据。
- 去重 —— 按 URL、文档和行去重；网络内容无限重复（页眉、页脚、菜单）。
- 启发式过滤 —— 通过字数、异常词元、脏词元来移除低质量文档。
- 基于模型的过滤 —— 预测某个页面是否可能被 Wikipedia 引用。
- 数据混合 —— 分类到代码、书籍、娱乐等类别，并利用扩展定律重新调整域名权重。
值得铭记的准则：**数据质量胜过数量**。善于收集数据是实际 LLM 工作的关键——也是最机密的部分。
> 善于收集数据 ~ 这是实用 LLM 的关键 —— 也是该领域最严守的秘密。
闭源数据集在规模上远超开源：LLaMA 3 在 15 万亿词元上训练；GPT-4 估计在 13 万亿词元上。
## 03. 扩展定律 —— 优化计算支出。
你有 10,000 个 GPU，可用时间为一个月。你训练什么模型？更大的模型，还是在更多数据上训练的模型？猜测会浪费数百万资金。扩展定律能给出可预测的答案。
实证发现：更多的数据和更大的模型 reliably 意味着更好的性能，而且你可以在训练之前根据模型大小和数据量预测其性能。
现代流程是在小模型上调整超参数，然后沿曲线外推到那个巨大的最终运行。
著名的 Chinchilla 结论：每个参数大约对应 20 个训练词元是计算最优的。但这仅针对训练成本。
一旦你考虑到运行模型的成本——即推理——实际比例会急剧上升，超过每个参数 150 个词元。你会用更多的数据训练一个更小的模型，因为你要为它数百万次运行付费。
还有一个元教训，即“苦涩的教训”：不要过度复杂化。做简单的事并扩展它。从长远来看，唯一重要的是利用计算能力。
## 04. 后训练 —— 将预测器转变为助手。
预训练后，你拥有了一个强大但无法用于聊天的东西。它能补全文本，但它不知道它应该回答你的问题。
问它一个问题，它可能会用三个更多的问题来回复——一种完全合理的“下一个词”续写。
> 监督微调（SFT）
你向展示模型数千个例子——一个提示词后跟一个良好的回答——它就会学会模仿这种模式。这就是行为克隆，这也是从 GPT-3 到 ChatGPT 的关键步骤。
令人惊讶的部分是：你只需要非常少的数据。几千个例子就足够了，因为 SFT 只教授好回答的**格式**——知识已经在预训练模型中了。
Alpaca 项目甚至用另一个 LLM 生成了它的数据：52,000 个指令-响应对，用于将 LLaMA 7B 微调为一个能干的助手。
> RLHF —— 与人类偏好对齐
SFT 有三个问题：它受限于人类能力，它教授幻觉（克隆一个模型实际上不知道的“正确”答案会教会它编造），且理想答案很昂贵。RLHF 通过优化**偏好**而非模仿来解决这些问题。
模型生成两个回答。人类挑选更好的那个。这些偏好训练一个奖励模型，LLM 被优化以最大化该奖励——经典方法是使用 PPO。
一种更简单的现代替代方案 DPO，仅使用普通的监督学习就能达到相当的质量，现在已成为开源社区的标准。
## 05. 评估与系统 —— 证明其有效，使其可行。
有两件事贯穿整个流程。跳过任何一个，你都没有一个真正的模型。
> 评估：测量开放性的东西
在预训练期间，指标是困惑度——模型在多少个词元之间“犹豫不决”。在 2017 年到 2023 年间，最好的模型从在约 70 个词元中犹豫降低到少于 10 个。但在对齐后困惑度失效了，因此评估转向基准测试和比较：
- MMLU & HELM —— 跨多个领域的具有黄金答案的任务套件。MMLU 是最受信任的预训练基准。
- Chatbot Arena —— 人类盲测比较两个模型并投票；超过 30 万票驱动了一个 Elo 排行榜。
- AlpacaEval —— 一个 LLM 评判其他 LLM。与 Chatbot Arena 有 98% 的相关性，耗时不到 3 分钟，花费不到 10 美元——但它有偏见，比如偏向更长的回答。
诚实的结论：评估一个对齐后的模型确实很难，没有单一的数字能概括它。同一个模型在 MMLU 上的得分可能仅因提示词格式的不同而从 0.637 变为 0.488。
> 系统：使训练在物理上成为可能
每个人都受限于计算能力——GPU 昂贵、稀缺，且受到通信速度的物理限制。一个 7B 模型仅朴素训练就需要约 112GB 内存。因此系统层不是可有可无的；它是使整个事情变得可行的关键：
- 低精度 —— 使用 16 位（bf16）代替 32 位，内存减半，速度提升。
- 算子融合与分块 —— 尽量减少对全局内存的缓慢访问；仅 FlashAttention 就能带来约 1.7 倍的端到端加速。
- 数据并行 —— 将数据集拆分到 GPU（使用 ZeRO 分片优化器状态以节省内存）。
- 模型并行 —— 按层（流水线）或按矩阵（张量）将模型拆分到 GPU。
- 稀疏性 —— 混合专家：更多参数，相同 FLOPs，通过每个词元只激活一个子集来实现。
> 整个流程教给了我们什么
回顾这五个阶段，论点是无可辩驳的。架构——每个人都痴迷的部分——受到的关注最少。数据、扩展、对齐、评估和系统才是每一个真正决策发生的地方。
这就是为什么两个拥有相同架构的实验室会产生截然不同的模型。架构是共享的。重要的一切都不是。
## 毁掉 LLM 项目的错误
- 痴迷于架构。这是栈中被复制最多、差异化最少的部分。
- 将数据视为大宗商品。脏数据会封死你的天花板，无论算力如何。质量胜于数量。
- 忽略 Chinchilla 数学。一个相对于其数据来说太大的模型是训练不足的，会浪费算力。
- 止步于 SFT。一个微调后的模型只会模仿；没有 RLHF 或 DPO，它永远学不会人们偏好什么。
- 在对齐后仍信任困惑度。后训练改变了分布——困惑度不再有意义。
## 结论：
> 一个伟大的模型不是训练出来的。它是工程化出来的。
大多数人会继续相信构建 LLM 的关键在于架构，继续阅读 Transformer 的解释文章，并继续错过真正工作发生的地方。
那些理解流程的人会看得很清楚：语言建模，然后是干净的数据，然后是最优扩展，然后是对齐，最后是在高效系统上的诚实评估。五个阶段。架构只是其中一个阶段中的一个段落。
选择一个你一直忽视的阶段——可能是数据或评估。深入研究它。这就是差距所在。
## 相关链接
- [Codez](https://x.com/0xCodez)
- [@0xCodez](https://x.com/0xCodez)
- [135K](https://x.com/0xCodez/status/2058911661973454915/analytics)
- [movez.substack.com](https://movez.substack.com/)
- [升级到 Premium](https://x.com/i/premium_sign_up)
- [2026年5月25日 晚上10:02](https://x.com/0xCodez/status/2058911661973454915)
- [135.7K 次观看](https://x.com/0xCodez/status/2058911661973454915/analytics)
- [查看引用](https://x.com/0xCodez/status/2058911661973454915/quotes)
---
*导出时间: 2026/5/26 09:15:52*