# How To Build Your Own LLM from Scratch (The 5-Stage Pipeline Behind GPT and Claude)
**作者**: Rahul
**日期**: 2026-06-14T08:35:02.000Z
**来源**: [https://x.com/sairahul1/status/2066076937806856661](https://x.com/sairahul1/status/2066076937806856661)
---
Everyone talks about LLMs.
Nobody explains how they actually work under the hood.
GPT. Claude. Gemini. Llama.
They all come from the same 5-stage pipeline.
And once you understand it, you can build one yourself.
Not a GPT-4 clone.
But a real working language model.
One that learns from text, generates new text, and actually makes sense.
I built one. Here's exactly how it works.
No PhD required. Code included.
The lie everyone believes about LLMs
Most people think building an LLM is about the architecture.
Transformers. Attention heads. Layers.
It is not.
The transformer architecture is publicly published. Every major lab uses roughly the same building blocks.
If architecture were the secret, everyone would have GPT-4.
The real secret: data, training, and alignment.
Architecture is one paragraph. Everything else is where real models are won and lost.
Here are the 5 stages.
## Stage 1 — Data (Where models are actually won)
Raw internet text is filthy.
You cannot train on it directly.
Common Crawl — the public web scrape used to train most LLMs — contains 250 billion pages and over a million gigabytes.
But most of it is garbage.
Duplicate headers. Spam. Toxic content. Personal data. Low-quality pages with 3 words.
Before any training happens, you run a brutal multi-step filter:
→ Extract clean text from raw HTML
→ Filter harmful, NSFW, and personal data
→ Deduplicate by URL, document, and line
→ Remove low-quality docs by word count and token density
→ Run model-based quality scoring — would a Wikipedia editor cite this page?
→ Balance the data mix across code, books, science, and web
The result: a clean dataset that's a fraction of the original size but dramatically better.
The rule to burn in:
Data quality beats data quantity. Every time.
The most guarded secret in the field is not the architecture.
It's how the data was cleaned.
## Stage 2 — Tokenization (Break text into pieces the model can actually learn from)
The model never reads raw text.
It reads tokens.
A token is not always a full word. It is a piece of a word — a fragment the model learned to treat as a unit.
"playing" → ["play", "ing"] "unbelievable" → ["un", "believ", "able"] "dog" → ["dog"]
The standard method is called Byte-Pair Encoding (BPE).
It starts with individual characters and merges the most common pairs repeatedly until it has a fixed vocabulary — usually 32,000 to 100,000 tokens.
Here is a minimal tokenizer in Python:
```
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
# Initialize BPE tokenizer
tokenizer = Tokenizer(models.BPE())
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
# Train on your corpus
trainer = trainers.BpeTrainer(
vocab_size=32000,
special_tokens=["<PAD>", "<BOS>", "<EOS>", "<UNK>"]
)
tokenizer.train(files=["your_data.txt"], trainer=trainer)
tokenizer.save("tokenizer.json")
# Test it
output = tokenizer.encode("Building an LLM from scratch is powerful")
print(output.tokens)
# ['Building', 'an', 'LL', 'M', 'from', 'scratch', 'is', 'powerful']
print(output.ids)
# [4821, 271, 3728, 44, 505, 8905, 318, 6787]
```
Rule of thumb: 1 token ≈ 0.75 words.
1,000 tokens ≈ 750 words.
A 100k context window = roughly a full novel.
## Stage 3 — Training (One deceptively simple objective)
The entire training task sounds too simple to be powerful:
Predict the next token.
That is it.
Given "The cat sat on the", predict "mat".
Do this across trillions of examples and something remarkable happens.
The model learns grammar. Then facts. Then reasoning. Then how to write code, translate languages, solve math.
Nobody taught it any of that.
It emerged from next-token prediction at massive scale.
Here is a minimal decoder-only transformer in PyTorch — the same architecture behind every major LLM:
```
import torch
import torch.nn as nn
import math
class CausalSelfAttention(nn.Module):
def __init__(self, embed_dim, num_heads):
super().__init__()
self.num_heads = num_heads
self.head_dim = embed_dim // num_heads
self.qkv = nn.Linear(embed_dim, 3 * embed_dim, bias=False)
self.proj = nn.Linear(embed_dim, embed_dim, bias=False)
def forward(self, x):
B, T, C = x.shape
q, k, v = self.qkv(x).chunk(3, dim=-1)
# Split into heads
q = q.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)
k = k.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)
v = v.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)
# Scaled dot-product attention with causal mask
scale = math.sqrt(self.head_dim)
attn = (q @ k.transpose(-2, -1)) / scale
# Causal mask: can only attend to past tokens
mask = torch.tril(torch.ones(T, T, device=x.device))
attn = attn.masked_fill(mask == 0, float('-inf'))
attn = torch.softmax(attn, dim=-1)
out = (attn @ v).transpose(1, 2).contiguous().view(B, T, C)
return self.proj(out)
class TransformerBlock(nn.Module):
def __init__(self, embed_dim, num_heads, ff_dim, dropout=0.1):
super().__init__()
self.attn = CausalSelfAttention(embed_dim, num_heads)
self.ff = nn.Sequential(
nn.Linear(embed_dim, ff_dim),
nn.GELU(),
nn.Linear(ff_dim, embed_dim),
nn.Dropout(dropout)
)
self.ln1 = nn.LayerNorm(embed_dim)
self.ln2 = nn.LayerNorm(embed_dim)
def forward(self, x):
x = x + self.attn(self.ln1(x)) # attention + residual
x = x + self.ff(self.ln2(x)) # feedforward + residual
return x
class MiniLLM(nn.Module):
def __init__(self, vocab_size, embed_dim, num_heads,
ff_dim, num_layers, max_seq_len, dropout=0.1):
super().__init__()
self.token_emb = nn.Embedding(vocab_size, embed_dim)
self.pos_emb = nn.Embedding(max_seq_len, embed_dim)
self.blocks = nn.ModuleList([
TransformerBlock(embed_dim, num_heads, ff_dim, dropout)
for _ in range(num_layers)
])
self.ln_final = nn.LayerNorm(embed_dim)
self.output = nn.Linear(embed_dim, vocab_size, bias=False)
self.dropout = nn.Dropout(dropout)
def forward(self, token_ids):
B, T = token_ids.shape
positions = torch.arange(T, device=token_ids.device).unsqueeze(0)
x = self.dropout(
self.token_emb(token_ids) + self.pos_emb(positions)
)
for block in self.blocks:
x = block(x)
x = self.ln_final(x)
return self.output(x) # logits over vocabulary
# Initialize a small but real model
model = MiniLLM(
vocab_size=32000,
embed_dim=512,
num_heads=8,
ff_dim=2048,
num_layers=6,
max_seq_len=1024
)
total_params = sum(p.numel() for p in model.parameters())
print(f"Parameters: {total_params:,}")
# Parameters: 44,082,176 — a 44M parameter model
```
Now the training loop:
```
import torch.optim as optim
from torch.nn.utils import clip_grad_norm_
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
optimizer = optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
criterion = nn.CrossEntropyLoss()
def train_epoch(model, dataloader):
model.train()
total_loss = 0
for input_ids, target_ids in dataloader:
input_ids = input_ids.to(device)
target_ids = target_ids.to(device)
# Forward pass
logits = model(input_ids)
# Flatten for loss calculation
loss = criterion(
logits.view(-1, logits.size(-1)), # (batch * seq_len, vocab)
target_ids.view(-1) # (batch * seq_len)
)
# Backward pass
optimizer.zero_grad()
loss.backward()
clip_grad_norm_(model.parameters(), max_norm=1.0) # prevent explosions
optimizer.step()
total_loss += loss.item()
return total_loss / len(dataloader)
```
What the model is actually learning:
→ Every input token attends to every previous token
→ The causal mask prevents looking into the future
→ Loss = how surprised the model was by the real next token
→ Lower loss = better predictions = model is learning language
## Stage 4 — Alignment (Turn a text predictor into an assistant)
After pretraining you have something impressive but useless for chat.
Ask it a question and it might reply with three more questions.
Because predicting the next token doesn't mean understanding what you want.
Two steps fix this.
Step 1: Supervised Fine-Tuning (SFT)
Show the model thousands of examples:
prompt → ideal response
The model learns to imitate the format of a good answer.
The surprising part: you need very little data.
A few thousand examples is enough because the knowledge is already inside the pretrained model.
SFT just teaches it to express that knowledge in the right format.
```
# SFT training example structure
sft_examples = [
{
"prompt": "Explain what an API is in simple terms.",
"response": "An API is like a waiter in a restaurant. You (the app) tell the waiter (API) what you want. The waiter goes to the kitchen (server), gets it, and brings it back. You never go to the kitchen directly."
},
{
"prompt": "What is the capital of France?",
"response": "The capital of France is Paris."
}
# A few thousand of these is enough
]
# Format as: <prompt> [SEP] <response> <EOS>
# Fine-tune the pretrained model on these pairs
# Same training loop as pretraining — just different data
```
Step 2: RLHF (Reinforcement Learning from Human Feedback)
SFT teaches format. RLHF teaches preference.
The model generates two answers. A human picks the better one. Those preferences train a reward model. The LLM is optimized to maximize that reward.
This is why ChatGPT and Claude feel like assistants — not random text generators.
Without RLHF:
→ Fluent. Capable. But unreliable.
→ Confidently wrong.
→ Doesn't know when to say "I don't know."
With RLHF:
→ Helpful. Clear. Safe.
→ Learns what "a good answer" actually means.
## Stage 5 — Evaluation (Prove it actually works)
Building a model without measuring it is guessing.
During pretraining — measure perplexity.
Perplexity measures how "surprised" the model is by real text.
Lower perplexity = model predicts text better = it's learning.
Between 2017 and 2023, the best models dropped from perplexing among ~70 possible tokens to fewer than 10.
```
import torch
import math
def calculate_perplexity(model, dataloader, device):
model.eval()
total_loss = 0
total_tokens = 0
criterion = nn.CrossEntropyLoss(reduction='sum')
with torch.no_grad():
for input_ids, target_ids in dataloader:
input_ids = input_ids.to(device)
target_ids = target_ids.to(device)
logits = model(input_ids)
loss = criterion(
logits.view(-1, logits.size(-1)),
target_ids.view(-1)
)
total_loss += loss.item()
total_tokens += target_ids.numel()
avg_loss = total_loss / total_tokens
perplexity = math.exp(avg_loss)
return perplexity
# Example output progression:
# Epoch 1: Perplexity = 847.3 (model knows almost nothing)
# Epoch 5: Perplexity = 124.6 (getting better)
# Epoch 20: Perplexity = 23.4 (actually learning language)
```
After alignment — perplexity stops working.
Fine-tuned models score worse on perplexity while being far more useful.
You need human benchmarks:
→ MMLU: 57 academic subjects, multiple choice — measures knowledge
→ Chatbot Arena: humans blind-compare two models and vote — measures preference
→ AlpacaEval: LLM judges LLM — 98% correlation with human judges, costs $10
The honest truth: no single score captures a good model.
The same model scores 0.637 or 0.488 on the same benchmark depending only on how the prompt is formatted.
Evaluation is genuinely hard.
And no one has fully solved it.
## How to generate text from your model
The model is trained.
Now make it generate something.
```
def generate(model, tokenizer, prompt, max_new_tokens=100,
temperature=0.8, device='cuda'):
model.eval()
# Encode prompt to token IDs
token_ids = tokenizer.encode(prompt).ids
input_tensor = torch.tensor(token_ids, dtype=torch.long,
device=device).unsqueeze(0)
with torch.no_grad():
for _ in range(max_new_tokens):
# Keep only last max_seq_len tokens (context window)
context = input_tensor[:, -1024:]
# Forward pass
logits = model(context)
# Get logits for last token only
next_token_logits = logits[:, -1, :]
# Apply temperature (higher = more creative)
next_token_logits = next_token_logits / temperature
# Sample from probability distribution
probs = torch.softmax(next_token_logits, dim=-1)
next_token = torch.multinomial(probs, num_samples=1)
# Append and continue
input_tensor = torch.cat([input_tensor, next_token], dim=1)
# Stop at end of sequence token
if next_token.item() == tokenizer.token_to_id("<EOS>"):
break
# Decode back to text
generated_ids = input_tensor[0].tolist()
return tokenizer.decode(generated_ids)
# Test it
output = generate(
model, tokenizer,
prompt="The most important thing about machine learning is",
max_new_tokens=100,
temperature=0.8
)
print(output)
```
Temperature controls creativity:
→ temperature = 0.1 → safe, predictable, repetitive
→ temperature = 0.8 → natural, varied, good default
→ temperature = 1.5 → creative, surprising, sometimes incoherent
## What the full pipeline looks like
Before: raw internet text, 1 million GB, completely unusable.
After Stage 1: clean filtered dataset, ready to train on.
Before: raw text, meaningless to a model.
After Stage 2: tokens with IDs, the model's native language.
Before: random weights, outputs garbage.
After Stage 3: a model that understands language patterns.
Before: a text predictor that answers questions with more questions.
After Stage 4: an assistant that follows instructions and is safe.
Before: no idea if the model is actually good.
After Stage 5: benchmarks, perplexity scores, human evaluations.
Each stage builds on the last.
Skip any one and the whole thing breaks.
## The 5 mistakes that sink LLM projects
1. Obsessing over architecture.
Transformers are standardized. Published. Copied.
The architecture is the least important part.
2. Treating data as a commodity.
Dirty data caps your ceiling regardless of compute.
The top labs spend more on data cleaning than model design.
3. Skipping the scaling math.
A model too big for its data is undertrained and wastes compute.
Optimal ratio: ~20 tokens of training data per parameter.
4. Stopping at SFT.
A fine-tuned model imitates. Without RLHF it never learns what people actually prefer.
5. Trusting perplexity after alignment.
Post-training changes the distribution.
Perplexity stops being meaningful the moment you run SFT.
Switch to human benchmarks immediately.
## The uncomfortable truth
A great LLM is not trained.
It is engineered.
5 stages. Not 1.
Architecture is one paragraph inside Stage 3.
Everything that matters is in the other four.
Data quality. Scaling math. Alignment. Honest evaluation.
That's what separates GPT-4 from a hobby model.
Two labs with identical architecture produce wildly different models.
The architecture is shared.
Everything that actually matters is not.
Start here
Want to run this yourself? Here is the minimal setup:
```
# Full minimal LLM training setup
# Requirements: pip install torch tokenizers datasets
# 1. Get data
from datasets import load_dataset
dataset = load_dataset("wikitext", "wikitext-103-v1", split="train")
text = "\n\n".join([t.strip() for t in dataset['text'] if t.strip()])
# 2. Train tokenizer
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
tokenizer = Tokenizer(models.BPE())
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
trainer = trainers.BpeTrainer(vocab_size=8000,
special_tokens=["<PAD>","<BOS>","<EOS>"])
with open("corpus.txt", "w") as f:
f.write(text[:5_000_000]) # use 5MB to start
tokenizer.train(["corpus.txt"], trainer)
# 3. Build model (use MiniLLM class from Stage 3 above)
model = MiniLLM(
vocab_size=8000,
embed_dim=256, # small but real
num_heads=8,
ff_dim=1024,
num_layers=4,
max_seq_len=256
)
# ~15M parameters — trains on a laptop GPU in hours
# 4. Train (use train_epoch from Stage 3 above)
# 5. Generate (use generate() from Stage 5 above)
print("Your LLM is running.")
```
Start small. 15M parameters. WikiText dataset. Free GPU on Google Colab.
Watch the perplexity drop from 800 to 50 over a few hours.
That drop is the model learning language.
In real time.
That's the moment everything clicks.
## Now make it useful: build a real niche LLM
WikiText is a learning dataset.
The real money — and the real fun — is training on a specific domain and watching your model become an expert in exactly one thing.
Here are 5 niches you can build right now. Same pipeline. Different data.
## Niche 1 — Coding Assistant LLM (The main example: highest impact, most dramatic results)
This is the one to build first.
The pain is universal. Every developer has hit this:
→ You stare at a function that isn't working.
→ Stack Overflow has 12 answers, all from 2014.
→ You paste into ChatGPT and get something almost right.
A coding LLM trained on the right data does this natively, offline, and tuned to your exact stack.
The data:
```
from datasets import load_dataset
# The Code dataset — 6.4M Python files from GitHub
code_dataset = load_dataset("codeparrot/github-code",
languages=["Python"],
streaming=True,
split="train")
# Stack Overflow Q&A pairs — real developer problems + accepted answers
so_dataset = load_dataset("koutch/stackoverflow_python",
split="train")
# The Stack — 30 programming languages, cleaned and deduplicated
the_stack = load_dataset("bigcode/the-stack",
data_dir="data/python",
split="train",
streaming=True)
```
What the training pairs look like:
```
# Format 1: Code completion
# Input: function signature + docstring
# Target: complete implementation
input_example = """
def calculate_compound_interest(principal, rate, time, n=12):
\"\"\"
Calculate compound interest.
Args:
principal: Initial investment amount
rate: Annual interest rate (as decimal, e.g. 0.05 for 5%)
time: Time period in years
n: Compounding frequency per year (default: monthly)
Returns:
Final amount after compound interest
\"\"\"
"""
target_example = """
amount = principal * (1 + rate/n) ** (n * time)
return round(amount, 2)
"""
# Format 2: Error → Fix pairs (from Stack Overflow)
input_example_2 = """
# ERROR: TypeError: unsupported operand type(s) for +: 'int' and 'str'
user_age = input("Enter your age: ")
next_year_age = user_age + 1
print(f"Next year you will be {next_year_age}")
# Fix this code:
"""
target_example_2 = """
user_age = int(input("Enter your age: ")) # convert string to int
next_year_age = user_age + 1
print(f"Next year you will be {next_year_age}")
"""
# Format 3: Natural language → Code (instruction following)
input_example_3 = """
# Write a Python function that:
# - Takes a list of dictionaries
# - Filters by a given key-value pair
# - Returns sorted results by a specified field
"""
target_example_3 = """
def filter_and_sort(data, filter_key, filter_value, sort_field):
filtered = [item for item in data
if item.get(filter_key) == filter_value]
return sorted(filtered, key=lambda x: x.get(sort_field, ''))
"""
```
The before/after that sells this:
Before training:
> Prompt: "Write a Python decorator that retries a function on failure"
> Output: "A decorator is a design pattern in Python that allows..."
> # Generic. Useless. Textbook answer.
After 10 hours of training on GitHub + Stack Overflow:
Prompt: "Write a Python decorator that retries a function on failure"
Output:
```
import time
from functools import wraps
def retry(max_attempts=3, delay=1.0, exceptions=(Exception,)):
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(max_attempts):
try:
return func(*args, **kwargs)
except exceptions as e:
if attempt == max_attempts - 1:
raise
time.sleep(delay * (attempt + 1))
return wrapper
return decorator
# Usage:
@retry(max_attempts=3, delay=0.5, exceptions=(ConnectionError,))
def fetch_data(url):
...
# Correct. Production-ready. Exponential backoff included.
```
The model learned what a senior developer would write.
Not from being told. From reading 6 million Python files and every accepted Stack Overflow answer.
## Niche 2 — SQL Query Generator
The pain: every non-technical founder has data they cannot access.
The data is there. In their database. They just cannot write the query.
They describe what they want in plain English. Your model writes the SQL.
The data:
```
# Spider dataset — 10,000+ SQL queries with natural language descriptions
# Covers 200 databases across 138 domains
spider = load_dataset("spider", split="train")
# WikiSQL — 80,654 natural language + SQL pairs
wikisql = load_dataset("wikisql", split="train")
# What a training pair looks like:
example = {
"question": "Show me all users who signed up in the last 30 days and made at least one purchase",
"sql": """
SELECT u.id, u.email, u.created_at, COUNT(p.id) as purchase_count
FROM users u
JOIN purchases p ON u.id = p.user_id
WHERE u.created_at >= NOW() - INTERVAL '30 days'
GROUP BY u.id, u.email, u.created_at
HAVING COUNT(p.id) >= 1
ORDER BY u.created_at DESC;
"""
}
```
The before/after:
```
# Non-technical founder types:
"Which of my customers spent the most money last quarter
but haven't bought anything in the last 60 days?"
# Model outputs:
SELECT
c.customer_id,
c.email,
SUM(o.total_amount) as q_spend,
MAX(o.created_at) as last_order_date
FROM customers c
JOIN orders o ON c.customer_id = o.customer_id
WHERE o.created_at BETWEEN DATE_TRUNC('quarter', NOW() - INTERVAL '3 months')
AND DATE_TRUNC('quarter', NOW())
GROUP BY c.customer_id, c.email
HAVING MAX(o.created_at) < NOW() - INTERVAL '60 days'
ORDER BY q_spend DESC
LIMIT 20;
```
Who pays for this: every SaaS founder, every e-commerce operator, every analyst who sits between a business owner and a database.
They will pay $20/month without thinking.
## Niche 3 — US Legal Document Summarizer
The pain: a 40-page contract. A lease agreement. An NDA.
Most people sign without understanding it.
A lawyer charges $300/hour to read it.
Your model reads it in 3 seconds.
The data:
```
# Free Law Project — millions of US court opinions, public domain
free_law = load_dataset("free-law/courtlistener-opinion-clustering",
split="train")
# MultiLegalPile — legal text across US jurisdictions
multi_legal = load_dataset("joelniklaus/multi_legal_pile",
languages=["en"],
split="train")
# What training pairs look like:
example = {
"input": """
SECTION 12.4 INDEMNIFICATION. Licensee shall defend, indemnify,
and hold harmless Licensor and its officers, directors, employees,
agents, and successors from and against any and all losses, damages,
liabilities, deficiencies, claims, actions, judgments, settlements,
interest, awards, penalties, fines, costs, or expenses of whatever
kind, including reasonable attorneys' fees, that are incurred by
Licensor arising out of or relating to Licensee's breach of any
representation, warranty, covenant, or obligation under this Agreement...
[continues for 3 more paragraphs of dense legalese]
""",
"summary": """
PLAIN ENGLISH: If you break any part of this contract and it causes
the other party legal trouble or financial loss, YOU pay for everything —
their lawyers, court costs, damages, all of it. This is a broad
indemnification clause that heavily favors the licensor.
RED FLAGS:
• "any and all losses" — extremely broad, no cap on liability
• "reasonable attorneys' fees" — you pay their legal bills too
• No carve-out for licensor's own negligence
NEGOTIATE: Ask to add "except to the extent caused by Licensor's
gross negligence or willful misconduct" before signing.
"""
}
```
The output format that makes this worth paying for:
```
Input: [paste any US contract clause]
Output:
━━━━━━━━━━━━━━━━━━
PLAIN ENGLISH SUMMARY
━━━━━━━━━━━━━━━━━━
[2-3 sentences anyone can understand]
RED FLAGS 🚩
• [specific risks in this clause]
• [what it lets the other party do]
MISSING PROTECTIONS ⚠️
• [what a balanced clause would include]
• [what a lawyer would ask to add]
RISK LEVEL: High / Medium / Low
━━━━━━━━━━━━━━━━━━
```
Who pays for this: freelancers reviewing client contracts, small business owners, renters reviewing leases, startup founders signing their first SaaS agreements.
The market is everyone who cannot afford a lawyer but still has to sign things.
## Niche 4 — Medical Symptom Explainer
The pain: you Google a symptom and WebMD tells you that you have 3 days to live.
A medical LLM trained on clinical notes and patient education materials does something different.
It explains what the symptom actually means. In plain English. Without catastrophizing.
```
# PubMed abstracts — 35 million medical research papers
pubmed = load_dataset("pubmed", split="train")
# MedQA — US Medical Licensing Exam questions + answers
medqa = load_dataset("bigbio/med_qa", split="train")
# Training pair format:
example = {
"patient_question": "I've had a dull ache behind my left eye for 3 days. It gets worse when I lean forward. No fever. What could this be?",
"clinical_explanation": """
Based on your description — dull ache behind one eye, worsening when leaning forward,
no fever, lasting 3 days — the most likely explanation is:
MOST LIKELY: Sinusitis (sinus pressure)
The maxillary or frontal sinuses sit directly behind and above your eyes.
Inflammation or congestion increases pressure when you change head position.
The fact that it worsens leaning forward is a classic sinusitis pattern.
ALSO POSSIBLE: Tension headache, eye strain, or mild migraine variant
LESS LIKELY but worth noting: If you also have vision changes, nausea,
or the pain is sudden and severe — see a doctor same day.
WHAT TO TRY: Saline nasal rinse, steam inhalation, ibuprofen for pressure relief.
If no improvement in 7-10 days or fever develops → see a doctor.
NOTE: This is general health information, not a diagnosis.
Always consult a licensed physician for medical advice.
"""
}
```
The key: every response ends with a clear disclaimer and escalation signal.
That is what separates useful health information from dangerous advice.
## Niche 5 — E-commerce Product Description Writer
The pain: a Shopify store with 500 products. Every description is a wall of spec sheet text nobody reads.
Good product descriptions do one thing: make someone feel something.
An LLM trained on high-converting product copy learns exactly what words drive clicks.
```
# Training data structure:
example = {
"product_specs": """
Product: Ceramic Coffee Mug
Material: Stoneware ceramic
Capacity: 14oz
Dimensions: 3.5" diameter x 4.2" height
Colors: Matte black, cream white, sage green
Dishwasher safe: Yes
Microwave safe: Yes
Weight: 0.8 lbs
""",
"high_converting_description": """
Some mornings deserve better than a paper cup.
This is the mug that stays on your desk. The one your coworkers
ask about. Heavy enough to feel intentional. Smooth enough to
actually enjoy holding at 7am.
14oz — the right amount. Not the novelty bucket. Not the tiny
espresso thing. The one you actually finish.
Matte stoneware that doesn't show fingerprints. Dishwasher safe
because life is short. Three colors that work with any kitchen
that isn't trying too hard.
You already have mugs.
You don't have this one.
""",
"meta_description": "Stoneware ceramic mug, 14oz, matte finish. Dishwasher and microwave safe. The coffee mug that stays.",
"keywords": ["ceramic coffee mug", "stoneware mug", "matte black mug",
"14oz mug", "minimalist coffee mug", "handmade style mug"]
}
```
The data source: scrape the top 1,000 Shopify stores by traffic. Extract product titles, specs, and descriptions. Filter for products with high review counts — those descriptions are proven to convert. Train on those.
Your model learns the difference between spec sheets and copy that sells.
The pattern across all 5 niches
Look at what they share:
→ A clear, specific pain the user feels every day
→ A data source that already exists and is publicly available
→ A before/after that is immediately obvious to anyone in that profession
→ A willingness to pay because the alternative costs more time or money
The 5-stage pipeline is identical for every single one.
You change only one thing: the training data.
Same tokenizer setup. Same transformer architecture. Same training loop. Same evaluation method.
Different data → different expert → different product.
That is the leverage.
One pipeline. Five products. Five revenue streams.
If this was useful:
→ Repost to share it with every developer learning AI
→ Follow @sairahul1 for more breakdowns like this
→ Bookmark this — the code works, run it tonight
I write about AI, building products, and systems that work while you sleep.
## 相关链接
- [Rahul](https://x.com/sairahul1)
- [@sairahul1](https://x.com/sairahul1)
- [13K](https://x.com/sairahul1/status/2066076937806856661/analytics)
- [@sairahul1](https://x.com/@sairahul1)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [4:35 PM · Jun 14, 2026](https://x.com/sairahul1/status/2066076937806856661)
- [13K Views](https://x.com/sairahul1/status/2066076937806856661/analytics)
- [View quotes](https://x.com/sairahul1/status/2066076937806856661/quotes)
---
*导出时间: 2026/6/14 21:26:00*
---
## 中文翻译
# 从零开始构建你自己的 LLM（GPT 和 Claude 背后的 5 阶段流程）
**作者**: Rahul
**日期**: 2026-06-14T08:35:02.000Z
**来源**: [https://x.com/sairahul1/status/2066076937806856661](https://x.com/sairahul1/status/2066076937806856661)
---
每个人都在谈论 LLM。
没有人解释它们到底是如何在底层运作的。
GPT. Claude. Gemini. Llama.
它们都源自同一个 5 阶段流程。
一旦你理解了它，你就可以自己构建一个。
不是 GPT-4 的克隆。
而是一个真正能用的语言模型。
一个能从文本中学习、生成新文本，并且确实言之有物的模型。
我自己构建了一个。下面就是它的工作原理。
不需要博士学位。包含代码。
## 大家都信的关于 LLM 的谎言
大多数人认为构建 LLM 关键在于架构。
Transformers。注意力头。层数。
其实不是。
Transformer 架构是公开发表的。每个主要实验室都使用大致相同的构建块。
如果架构是秘密，每个人都会有 GPT-4。
真正的秘密：数据、训练和对齐。
架构只是一段话的内容。其余所有部分才是决定模型成败的关键。
以下是这 5 个阶段。
## 第 1 阶段 — 数据（模型真正决胜的地方）
原始网络文本是非常脏的。
你不能直接在上面训练。
Common Crawl — 用于训练大多数 LLM 的公共网页抓取数据 — 包含 2500 亿个页面和超过 100 万 GB 的数据。
但其中大部分都是垃圾。
重复的标题。垃圾邮件。有毒内容。个人数据。只有 3 个单词的低质量页面。
在进行任何训练之前，你需要运行一个残酷的多步骤过滤器：
→ 从原始 HTML 中提取干净的文本
→ 过滤有害、NSFW（不适宜工作场所）和个人数据
→ 根据 URL、文档和行进行去重
→ 通过字数和 token 密度删除低质量文档
→ 运行基于模型的质量评分 — 维基百科编辑会引用这个页面吗？
→ 在代码、书籍、科学和网络之间平衡数据组合
结果：一个干净的数据集，大小是原始数据的一小部分，但质量要高得多。
铭记于心的规则：
数据质量胜过数据数量。每一次。
该领域最受保守的秘密不是架构。
而是数据是如何清洗的。
## 第 2 阶段 — 分词（将文本分解为模型可以实际学习的片段）
模型从不阅读原始文本。
它阅读的是 tokens。
Token 不总是一个完整的单词。它是单词的一部分 — 一个模型学会将其视为一个单位的片段。
"playing" → ["play", "ing"] "unbelievable" → ["un", "believ", "able"] "dog" → ["dog"]
标准方法称为字节对编码（BPE）。
它从单个字符开始，反复合并最常见的对，直到拥有固定的词汇表 — 通常为 32,000 到 100,000 个 tokens。
这是一个 Python 中的极简分词器：
```
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
# 初始化 BPE 分词器
tokenizer = Tokenizer(models.BPE())
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
# 在你的语料库上训练
trainer = trainers.BpeTrainer(
vocab_size=32000,
special_tokens=["<PAD>", "<BOS>", "<EOS>", "<UNK>"]
)
tokenizer.train(files=["your_data.txt"], trainer=trainer)
tokenizer.save("tokenizer.json")
# 测试它
output = tokenizer.encode("Building an LLM from scratch is powerful")
print(output.tokens)
# ['Building', 'an', 'LL', 'M', 'from', 'scratch', 'is', 'powerful']
print(output.ids)
# [4821, 271, 3728, 44, 505, 8905, 318, 6787]
```
经验法则：1 token ≈ 0.75 个单词。
1,000 tokens ≈ 750 个单词。
100k 的上下文窗口 = 大约一整本小说。
## 第 3 阶段 — 训练（一个欺骗性简单的目标）
整个训练任务听起来简单得不值一提：
预测下一个 token。
仅此而已。
给定 "The cat sat on the"，预测 "mat"。
在数万亿个例子中这样做，奇迹就会发生。
模型学会了语法。然后是事实。然后是推理。然后是如何写代码、翻译语言、解决数学问题。
没人教过它这些。
它是从大规模的下一个 token 预测中涌现出来的。
这是一个 PyTorch 中的极简仅解码器 Transformer — 即每个主要 LLM 背后的相同架构：
```
import torch
import torch.nn as nn
import math
class CausalSelfAttention(nn.Module):
def __init__(self, embed_dim, num_heads):
super().__init__()
self.num_heads = num_heads
self.head_dim = embed_dim // num_heads
self.qkv = nn.Linear(embed_dim, 3 * embed_dim, bias=False)
self.proj = nn.Linear(embed_dim, embed_dim, bias=False)
def forward(self, x):
B, T, C = x.shape
q, k, v = self.qkv(x).chunk(3, dim=-1)
# 分割到各个头
q = q.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)
k = k.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)
v = v.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)
# 带因果掩码的缩放点积注意力
scale = math.sqrt(self.head_dim)
attn = (q @ k.transpose(-2, -1)) / scale
# 因果掩码：只能关注过去的 tokens
mask = torch.tril(torch.ones(T, T, device=x.device))
attn = attn.masked_fill(mask == 0, float('-inf'))
attn = torch.softmax(attn, dim=-1)
out = (attn @ v).transpose(1, 2).contiguous().view(B, T, C)
return self.proj(out)
class TransformerBlock(nn.Module):
def __init__(self, embed_dim, num_heads, ff_dim, dropout=0.1):
super().__init__()
self.attn = CausalSelfAttention(embed_dim, num_heads)
self.ff = nn.Sequential(
nn.Linear(embed_dim, ff_dim),
nn.GELU(),
nn.Linear(ff_dim, embed_dim),
nn.Dropout(dropout)
)
self.ln1 = nn.LayerNorm(embed_dim)
self.ln2 = nn.LayerNorm(embed_dim)
def forward(self, x):
x = x + self.attn(self.ln1(x)) # attention + residual
x = x + self.ff(self.ln2(x)) # feedforward + residual
return x
class MiniLLM(nn.Module):
def __init__(self, vocab_size, embed_dim, num_heads,
ff_dim, num_layers, max_seq_len, dropout=0.1):
super().__init__()
self.token_emb = nn.Embedding(vocab_size, embed_dim)
self.pos_emb = nn.Embedding(max_seq_len, embed_dim)
self.blocks = nn.ModuleList([
TransformerBlock(embed_dim, num_heads, ff_dim, dropout)
for _ in range(num_layers)
])
self.ln_final = nn.LayerNorm(embed_dim)
self.output = nn.Linear(embed_dim, vocab_size, bias=False)
self.dropout = nn.Dropout(dropout)
def forward(self, token_ids):
B, T = token_ids.shape
positions = torch.arange(T, device=token_ids.device).unsqueeze(0)
x = self.dropout(
self.token_emb(token_ids) + self.pos_emb(positions)
)
for block in self.blocks:
x = block(x)
x = self.ln_final(x)
return self.output(x) # logits over vocabulary
# 初始化一个小但真实的模型
model = MiniLLM(
vocab_size=32000,
embed_dim=512,
num_heads=8,
ff_dim=2048,
num_layers=6,
max_seq_len=1024
)
total_params = sum(p.numel() for p in model.parameters())
print(f"Parameters: {total_params:,}")
# Parameters: 44,082,176 — 一个 44M 参数的模型
```
现在的训练循环：
```
import torch.optim as optim
from torch.nn.utils import clip_grad_norm_
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
optimizer = optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
criterion = nn.CrossEntropyLoss()
def train_epoch(model, dataloader):
model.train()
total_loss = 0
for input_ids, target_ids in dataloader:
input_ids = input_ids.to(device)
target_ids = target_ids.to(device)
# 前向传播
logits = model(input_ids)
# 展平以计算损失
loss = criterion(
logits.view(-1, logits.size(-1)), # (batch * seq_len, vocab)
target_ids.view(-1) # (batch * seq_len)
)
# 反向传播
optimizer.zero_grad()
loss.backward()
clip_grad_norm_(model.parameters(), max_norm=1.0) # 防止梯度爆炸
optimizer.step()
total_loss += loss.item()
return total_loss / len(dataloader)
```
模型实际学习的内容：
→ 每个输入 token 都会关注每个之前的 token
→ 因果掩码防止“偷看”未来
→ Loss = 模型对真实下一个 token 的惊讶程度
→ 更低的 Loss = 更好的预测 = 模型正在学习语言
## 第 4 阶段 — 对齐（将文本预测器转变为助手）
预训练之后，你得到了令人印象深刻的东西，但对聊天来说毫无用处。
问它一个问题，它可能会用更多的问题来回答。
因为预测下一个 token 并不意味着理解你想要什么。
两步修复法可以解决这个问题。
第 1 步：有监督微调（SFT）
向模型展示数千个示例：
prompt → 理想回复
模型学会模仿优秀回答的格式。
令人惊讶的部分：你只需要很少的数据。
几千个例子就足够了，因为知识已经存储在预训练模型中了。
SFT 只是教它以正确的格式表达这些知识。
```
# SFT 训练示例结构
sft_examples = [
{
"prompt": "简单解释一下什么是 API。",
"response": "API 就像餐厅里的服务员。你（应用程序）告诉服务员（API）你想要什么。服务员去厨房（服务器），拿到东西，然后带回来。你不需要直接去厨房。"
},
{
"prompt": "法国的首都是哪里？",
"response": "法国的首都是巴黎。"
}
# 几千个这样的例子就足够了
]
# 格式化为：<prompt> [SEP] <response> <EOS>
# 在这些配对上微调预训练模型
# 与预训练相同的训练循环 — 只是数据不同
```
第 2 步：RLHF（基于人类反馈的强化学习）
SFT 教的是格式。RLHF 教的是偏好。
模型生成两个回答。人类挑选更好的那个。这些偏好训练一个奖励模型。LLM 被优化以最大化该奖励。
这就是为什么 ChatGPT 和 Claude 感觉像助手 — 而不是随机文本生成器。
没有 RLHF：
→ 流畅。有能力。但不可靠。
→ 充满自信地犯错。
→ 不知道什么时候该说“我不知道”。
有了 RLHF：
→ 有用。清晰。安全。
→ 学会了什么才是“好的回答”。
## 第 5 阶段 — 评估（证明它真的有效）
构建一个模型而不去衡量它，只是在瞎猜。
在预训练期间 — 衡量困惑度。
困惑度衡量模型对真实文本的“惊讶”程度。
更低的困惑度 = 模型预测文本更好 = 它正在学习。
在 2017 年到 2023 年之间，最好的模型将困惑度从约 70 个可能的 token 降低到了不到 10 个。
```
import torch
import math
def calculate_perplexity(model, dataloader, device):
model.eval()
total_loss = 0
total_tokens = 0
criterion = nn.CrossEntropyLoss(reduction='sum')
with torch.no_grad():
for input_ids, target_ids in dataloader:
input_ids = input_ids.to(device)
target_ids = target_ids.to(device)
logits = model(input_ids)
loss = criterion(
logits.view(-1, logits.size(-1)),
target_ids.view(-1)
)
total_loss += loss.item()
total_tokens += target_ids.numel()
avg_loss = total_loss / total_tokens
perplexity = math.exp(avg_loss)
return perplexity
# 示例输出进展：
# Epoch 1: Perplexity = 847.3 (模型几乎什么都不知道)
# Epoch 5: Perplexity = 124.6 (正在变好)
# Epoch 20: Perplexity = 23.4 (实际上正在学习语言)
```
对齐之后 — 困惑度就失效了。
微调后的模型在困惑度上得分更差，但有用得多。
你需要人类基准测试：
→ MMLU：57 个学科，多项选择 — 衡量知识
→ Chatbot Arena：人类盲测比较两个模型并投票 — 衡量偏好
→ AlpacaEval：LLM 评判 LLM — 与人类评判者 98% 的相关性，成本 10 美元
诚实的真相：没有一个单一的分数能完全代表一个好的模型。
同一个模型在同一个基准测试上的得分是 0.637 还是 0.488，仅取决于 prompt 的格式化方式。
评估确实很难。
没有人完全解决它。
## 如何让你的模型生成文本
模型已经训练好了。
现在让它生成点东西。
```
def generate(model, tokenizer, prompt, max_new_tokens=100,
temperature=0.8, device='cuda'):
model.eval()
# 将 prompt 编码为 token IDs
token_ids = tokenizer.encode(prompt).ids
input_tensor = torch.tensor(token_ids, dtype=torch.long,
device=device).unsqueeze(0)
with torch.no_grad():
for _ in range(max_new_tokens):
# 只保留最后的 max_seq_len tokens（上下文窗口）
context = input_tensor[:, -1024:]
# 前向传播
logits = model(context)
# 获取最后一个 token 的 logits
next_token_logits = logits[:, -1, :]
# 应用温度（越高 = 越有创意）
next_token_logits = next_token_logits / temperature
# 从概率分布中采样
probs = torch.softmax(next_token_logits, dim=-1)
next_token = torch.multinomial(probs, num_samples=1)
# 追加并继续
input_tensor = torch.cat([input_tensor, next_token], dim=1)
# 在序列结束 token 处停止
if next_token.item() == tokenizer.token_to_id("<EOS>"):
break
# 解码回文本
generated_ids = input_tensor[0].tolist()
return tokenizer.decode(generated_ids)
# 测试它
output = generate(
model, tokenizer,
prompt="机器学习最重要的事情是",
max_new_tokens=100,
temperature=0.8
)
print(output)
```
温度控制创造力：
→ temperature = 0.1 → 安全、可预测、重复
→ temperature = 0.8 → 自然、多样、好的默认值
→ temperature = 1.5 → 有创意、出人意料、有时语无伦次
## 完整的流程看起来像什么
之前：原始网络文本，100 万 GB，完全不可用。
第 1 阶段之后：干净过滤后的数据集，准备好进行训练。
之前：原始文本，对模型毫无意义。
第 2 阶段之后：带有 IDs 的 tokens，模型的母语。
之前：随机权重，输出垃圾。
第 3 阶段之后：一个理解语言模式的模型。
之前：一个用更多问题回答问题的文本预测器。
第 4 阶段之后：一个遵循指令且安全的助手。
之前：不知道模型是否真的好用。
第 5 阶段之后：基准测试、困惑度分数、人类评估。
每个阶段都建立在前一个阶段之上。
跳过任何一个，整个系统就会崩溃。
## 毁掉 LLM 项目的 5 个错误
1. 痴迷于架构。
Transformers 是标准化的。已发表。被复制。
架构是最不重要的部分。
2. 把数据视为日用品。
无论算力多少，脏数据都会封死你的上限。
顶级实验室 s