# Decoding Transformer Architecture
**作者**: Amit Shekhar
**日期**: 2026-04-07T11:32:46.000Z
**来源**: [https://x.com/amitiitbhu/status/2041479290580287543](https://x.com/amitiitbhu/status/2041479290580287543)
---
In this blog, we will learn about the Transformer architecture by decoding it piece by piece - understanding what each component does, how they work together, and why this architecture powers every modern Large Language Model (LLM).
When we hear "Transformer", it sounds complex. But do not worry. If we break it down into its individual parts, every single piece is simple. The complexity comes from how they are stacked together - not from any one piece being hard.
Our goal is to decode this architecture so clearly that by the end, we will be able to explain how a Transformer works to anyone.
We will cover the following:
- Why the Transformer was needed
- The two halves of the architecture
- Tokenization, Embedding, and Positional Encoding
- The Attention Mechanism and Multi-Head Attention
- Feed-Forward Networks, Residual Connections, and Layer Normalization
- How the Encoder and Decoder work
- How data flows through the entire architecture
- The three variants of the Transformer
- Why the Transformer is so powerful
I’m Amit Shekhar from Outcome School, where I teach AI and Machine Learning.
Let's get started.
## The Big Picture
Before we go into the details, let's understand the big picture.
A Transformer takes a sequence of tokens as input and produces a sequence of tokens as output. That is it. At a very high level, it is a tokens-in, tokens-out machine. And since tokens come from text and turn back into text, we can also think of it as a text-in, text-out machine.
For example, if we give it "Translate: I love learning" as input, it produces "J'adore apprendre" as output. If we give it "What is the capital of France?" as input, it produces "The capital of France is Paris." as output.
But how does it do this? Let's understand.
## Why Was the Transformer Needed?
Before the Transformer, models processed words one by one, in order - like reading a sentence from left to right, one word at a time.
This had two big problems:
Problem 1: Forgetting. As sentences got longer, the model would start forgetting earlier words. By the time it reached the 50th word, its memory of the 1st word was very weak.
Problem 2: Slowness. Because words were processed one after another, the 2nd word had to wait for the 1st word to finish, the 3rd word had to wait for the 2nd, and so on. This was very slow.
So, here comes the Transformer to the rescue. It processes all input tokens at the same time and lets every token look at every other token directly. Long-range connections are no longer a major problem. And because everything happens in parallel, training became much faster. We will see exactly how this works in the upcoming sections.
The Transformer was introduced in 2017 in the famous research paper "Attention Is All You Need" by researchers at Google. It changed everything.
## The Architecture Has Two Halves
The original Transformer has two main halves:
- The Encoder - this half reads and understands the input
- The Decoder - this half generates the output
Think of it like a conversation between a reader and a writer. The reader (encoder) reads a document carefully and creates detailed notes. The writer (decoder) uses those notes to write a new document. The reader focuses on understanding. The writer focuses on producing.
Now, let's understand what happens inside each half. But before jumping into the encoder and decoder, we need to understand what happens before the input even reaches them.
## Decoding Step 1: Turning Words into Numbers
The model does not understand words. It only understands numbers. So, the very first step is to convert words into numbers.
This happens in two stages:
Tokenization: The input text is split into small pieces called tokens. A token can be a word, part of a word, or even a single character. For the sake of understanding, let's treat each word as one token.
"I love learning" becomes three tokens: "I", "love", "learning".
Embedding: Each token is then converted into a vector of numbers called an embedding. An embedding is like a digital fingerprint for a word. Words with similar meanings get similar fingerprints.
For example, "happy" and "joyful" would have very similar embeddings because they mean almost the same thing. But "happy" and "car" would have very different embeddings because they are unrelated.
## Decoding Step 2: Adding Word Order
But, here is the catch. The encoder processes all input tokens at the same time - not one by one. This is great for speed, but it means the model has no idea which word comes first, second, or third.
"I love AI" and "AI love I" would look the same to the model. But they mean very different things.
Positional Encoding solves this. It adds a vector of numbers to each embedding that represents the position of the word. The 1st word gets a different position signal than the 2nd word, and so on.
Think of it like seat numbers in a theater. Even if all the audience members arrive at the same time, their seat numbers tell us exactly where each person is supposed to sit. Positional encoding is the seat number for each word.
After this step, the positional encoding numbers are combined into the embedding itself through addition. So, each token now carries a single vector of numbers that encodes both what the word means and where it sits in the sentence.
## Decoding Step 3: The Attention Mechanism
This is the heart of the Transformer. The attention mechanism is the single most important idea that makes everything work.
Attention allows each word to look at every other word in the sentence and decide how much focus to give to each one.
Let's take the sentence: "The cat sat on the mat because it was tired."
When the model processes the word "it", it needs to figure out what "it" refers to. Is it the cat? Is it the mat? The attention mechanism helps the model focus more on "cat" and less on "mat" because "cat" is more relevant to "it" in this context.
How Attention Works
Each word is converted into three things:
- Query (Q) - what this word is looking for
- Key (K) - what this word can offer
- Value (V) - the actual information this word carries
We have a detailed blog on the Math behind Attention - Q, K, and V that explains how these are actually computed.
The process works like this:
First, each word uses its Query to compare with the Keys of all other words using a dot product. This produces attention scores - numbers that tell the model how relevant each pair of words is. Then, these scores are scaled down (divided by √dₖ) to keep them in a manageable range. We have a detailed blog on Scaling Dot Product Attention that explains why this step is needed. After scaling, the scores are normalized using softmax so they become probabilities that add up to 1. Finally, the model collects a weighted mix of the Values based on these probabilities.
The result is that every word now carries information not just about itself, but about the words most relevant to it. The word "it" now carries a strong signal from "cat" because the attention mechanism figured out the connection.
In simple words, attention lets each token borrow useful information from other relevant tokens before moving to the next layer.
Multi-Head Attention
The Transformer does not run attention just once. It runs it multiple times in parallel - each time from a different perspective. This is called multi-head attention.
Think of it like a team of editors reviewing an article. One editor checks for grammar, another checks for factual accuracy, and another checks for flow. Each editor looks at the same text but focuses on something different. Together, they provide a complete review.
Each head in multi-head attention focuses on a different type of relationship between words. One head focuses on subject-verb connections. Another head focuses on which adjective describes which noun. The outputs of all heads are combined to give the model a rich understanding of the text.
Here is a simple visual of multi-head attention. All heads run in parallel on the same input, and their outputs are combined at the end:
## Decoding Step 4: The Feed-Forward Network
After the attention mechanism processes the words, the output passes through a feed-forward network.
Think of it as a refinement step. If attention is about understanding relationships between words, the feed-forward network is about deepening the understanding of each word individually.
It takes the enriched representation from attention and processes it further - sharpening the signal and removing noise. Every word passes through the same feed-forward network independently.
## Decoding Step 5: Residual Connections and Layer Normalization
Inside each layer, there are two more important pieces that help the Transformer learn effectively. Let's understand each of them.
Residual Connection (Skip Connection): After each sub-layer (attention or feed-forward network), the original input to that sub-layer is added back to the output. Think of it like taking notes in a class. Even after the teacher adds new information, we keep our original notes and combine the new information with them. This prevents the model from losing important information as data passes through many layers.
In simple words, the output of each sub-layer is: output = sub-layer(input) + input.
Layer Normalization: After each residual connection, the numbers are normalized - adjusted so they stay in a stable range. This prevents the numbers from becoming too large or too small as they pass through many layers. It keeps the training process stable and smooth.
These two pieces are present in every single layer of both the encoder and decoder.
## Decoding Step 6: Stacking Multiple Layers
Now, here comes the interesting part. So far, we have talked about one multi-head attention sub-layer and one feed-forward sub-layer. Together, these two sub-layers (along with their residual connections and layer normalization) form one layer of the Transformer. Think of one layer as a complete bundle: one multi-head attention sub-layer plus one feed-forward sub-layer.
The Transformer does not use just one such layer. It stacks multiple layers on top of each other, one after another. This is where the real power comes from.
Each of these layers is also called a Transformer block. So when we say "the model has 96 layers", it means the model has 96 Transformer blocks stacked one after another. Each block has its own multi-head attention sub-layer and its own feed-forward sub-layer.
Note: This stacking is different from multi-head attention. Multi-head attention runs many attention heads in parallel inside a single attention sub-layer. Stacking layers, on the other hand, places full Transformer blocks one after another in sequence, so the output of one layer becomes the input of the next layer.
Here is a simple visual of stacked Transformer layers. Each layer is a complete block, and they are connected one after another in sequence:
The original Transformer uses 6 layers in both the encoder and decoder. Modern LLMs use even more - GPT-3 (175B) uses 96 layers.
Each layer builds on the understanding from the previous layer. Think of it like reading a book multiple times. The first read gives us a basic understanding. The second read reveals connections we missed. The third read gives us deeper insights. Each layer is like another read - the understanding gets richer and richer.
## Decoding Step 7: The Encoder in Detail
Now that we know the building blocks, let's see how they come together inside the encoder.
The encoder's job is to read and understand the input.
Each encoder layer has two components:
- Self-Attention - each word looks at all other words in the input to understand relationships
- Feed-Forward Network - each word's representation is refined individually
Both components are followed by a residual connection and layer normalization.
Let's see what happens when the sentence "I love learning" passes through the encoder.
In the first layer, the self-attention mechanism allows each word to look at every other word. The word "love" starts to notice that "I" is its subject and "learning" is its object. The feed-forward network then refines this understanding.
In the second layer, the connections become richer. Now "love" does not just know about "I" and "learning" individually - it starts to understand the combined meaning of "I love learning" as a complete thought.
By the sixth layer, each word carries a deep, context-aware representation. The word "learning" is no longer just a dictionary definition - it now encodes the fact that it is something "I" loves to do, in this specific sentence.
After all encoder layers, we have a rich understanding of the input. This understanding is then passed to the decoder.
## Decoding Step 8: The Decoder in Detail
The decoder's job is to generate the output, one token at a time.
Each decoder layer has three components:
- Masked Self-Attention - each output word looks at all previous output words, but not future ones. This is because when generating text, future words do not exist yet. The word "love" can look at "I" but not at "learning" because "learning" has not been generated yet. This restriction is called causal masking. We have a detailed blog on Causal Masking in Attention that explains how this works.
- Cross-Attention - this is how the decoder connects to the encoder. Each output word looks at the encoder's output to pull in relevant information from the input. Here, the Query comes from the decoder, while the Key and Value come from the encoder's output. This is what makes it different from self-attention - and this is the bridge between understanding the input and generating the output.
- Feed-Forward Network - the same refinement step as in the encoder.
All three components are followed by a residual connection and layer normalization, just like in the encoder.
How the Decoder Picks the Next Word
Now, the question is: after the output passes through all decoder layers, how does the decoder pick the actual next word? At this point, the output is still a vector of numbers - not a word.
So, the decoder passes these numbers through a linear layer that converts them into a score for every word in the vocabulary. If the vocabulary has 50,000 words, we get 50,000 scores.
Then, softmax converts these scores into probabilities - numbers between 0 and 1 that add up to 1. The model then picks the next word based on these probabilities. In practice, models use different strategies to pick the next word - like temperature, top-k, and top-p sampling - which control how creative or focused the output is. But for the sake of understanding, let's say it picks the word with the highest probability.
For example, if the model is translating "I love learning" to French and it has already generated "J'", the probabilities after the linear layer and softmax look like this:
- "adore": 0.72
- "aime": 0.15
- "mange": 0.01
- ... (50,000 words, most with very small probabilities)
The model picks "adore" because it has the highest probability. This is how the decoder decides which word to output next.
## Decoding Step 9: The Complete Flow
Now that we have learned about all the building blocks, let's walk through the complete flow with a concrete example. The best way to learn this is by taking an example.
Let's say we want to translate "I love learning" to French.
The Encoder Phase
The English sentence "I love learning" goes through the following steps:
Tokenize: "I love learning" becomes three tokens: "I", "love", "learning".
Embed: Each token is converted into a vector of numbers (an embedding).
Add position: Positional encoding is added so the model knows "I" is 1st, "love" is 2nd, and "learning" is 3rd.
Pass through 6 encoder layers: In each layer, self-attention lets every word look at every other word, and the feed-forward network refines the understanding. Residual connections and layer normalization keep things stable.
After all 6 layers, the encoder produces a rich representation of the input. Each word now carries deep contextual information. This representation is passed to the decoder.
The Decoder Phase
The decoder generates the French translation one word at a time.
Generating "J'" (the 1st word):
The decoder starts with a special "start" token. Masked self-attention has nothing to look back at since this is the first token. Cross-attention looks at the encoder's output to understand the English input. The output passes through the linear layer and softmax. The model picks "J'" as the most probable first word.
Generating "adore" (the 2nd word):
Now the decoder has ["start", "J'"]. Masked self-attention looks at "start" and "J'" to understand what has been generated so far. Cross-attention looks at the encoder's output again to pull in relevant information from the English input. The linear layer and softmax produce probabilities. The model picks "adore".
Generating "apprendre" (the 3rd word):
Now the decoder has ["start", "J'", "adore"]. Masked self-attention looks at all three previous tokens. Cross-attention connects to the encoder's output. The model picks "apprendre".
Generating the "end" token:
The decoder generates a special "end" token, signaling that the translation is complete.
The final output: "J'adore apprendre" (which means "I love learning" in French).
This is how the Transformer works from start to finish.
## The Three Variants
The original Transformer uses both an encoder and a decoder. But we do not always need both. This gives us three variants:
Encoder-Only (Example: BERT)
Uses only the encoder. The model reads and understands the input but does not generate new text. This is used for tasks like classifying emails as spam or not, understanding the sentiment of a review, or finding answers within a given text.
Decoder-Only (Example: GPT)
Uses only the decoder. The model generates text one token at a time. Each token can look at all previous tokens but not future ones. This is the variant behind most modern LLMs. It is used for generating text, answering questions, writing code, and having conversations.
Encoder-Decoder (Example: T5, the original Transformer)
Uses both parts. The encoder understands the input, and the decoder generates the output based on that understanding. This is used for translation, summarization, and other tasks where one text is transformed into another.
We can choose the right variant based on our use case.
## Why the Transformer is So Powerful
Now that we have decoded the entire architecture, let's understand why it changed everything:
Parallel processing: The encoder processes all input tokens at the same time instead of one by one. Even the decoder runs in parallel during training. This makes training much faster and allows models to learn from massive amounts of data.
No more long-range forgetting: Through attention, every word can directly look at every other word. The 1st word is just as accessible as the 1000th word, so the long-range memory problem of older models is largely solved.
Scalability: By adding more layers, more attention heads, and more data, the model keeps getting better. This is why we have models with billions of parameters today. Parameters are the numbers inside the model that it learns during training.
Versatility: The same architecture handles translation, question answering, text generation, code writing, summarization, and much more. One design, countless applications.
The Transformer is the single architecture behind the entire modern AI revolution. It enables us to do complex things very simply.
## Quick Summary
Let's recap what we have decoded:
- Tokenization and Embedding turn words into numbers that the model can work with.
- Positional Encoding adds word order information since all words are processed at the same time.
- Attention lets every word look at every other word and decide which ones to focus on. Multi-head attention does this from multiple perspectives.
- Feed-Forward Networks refine the understanding of each word individually.
- Residual Connections and Layer Normalization keep the numbers stable as data flows through many layers.
- The Encoder stacks these building blocks to read and understand the input.
- The Decoder adds cross-attention to connect to the encoder, and uses a linear layer + softmax to pick the next word.
- Three variants exist: encoder-only for understanding, decoder-only for generating, and encoder-decoder for transforming one text into another.
Now, we have decoded the Transformer architecture piece by piece and understood how it works.
Prepare yourself for AI Engineering Interview: AI Engineering Interview Questions
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
## 相关链接
- [Amit Shekhar](https://x.com/amitiitbhu)
- [@amitiitbhu](https://x.com/amitiitbhu)
- [66K](https://x.com/amitiitbhu/status/2041479290580287543/analytics)
- [Outcome School](https://outcomeschool.com/)
- [AI and Machine Learning](https://outcomeschool.com/program/ai-and-machine-learning)
- [Tokenization](https://www.youtube.com/watch?v=sK2s9I84EVI)
- [Embedding](https://www.youtube.com/watch?v=LedXW6xl21s)
- [Math behind Attention - Q, K, and V](https://outcomeschool.com/blog/math-behind-attention-qkv)
- [Scaling Dot Product Attention](https://outcomeschool.com/blog/scaling-dot-product-attention)
- [Causal Masking in Attention](https://outcomeschool.com/blog/causal-masking-in-attention)
- [AI Engineering Interview Questions](https://github.com/amitshekhariitbhu/ai-engineering-interview-questions)
- [Outcome School](https://outcomeschool.com/)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [7:32 PM · Apr 7, 2026](https://x.com/amitiitbhu/status/2041479290580287543)
- [66.1K Views](https://x.com/amitiitbhu/status/2041479290580287543/analytics)
- [View quotes](https://x.com/amitiitbhu/status/2041479290580287543/quotes)
---
*导出时间: 2026/4/18 18:32:43*
---
## 中文翻译
# 解码 Transformer 架构
**作者**：Amit Shekhar
**日期**：2026-04-07T11:32:46.000Z
**来源**：[https://x.com/amitiitbhu/status/2041479290580287543](https://x.com/amitiitbhu/status/2041479290580287543)
---
在这篇博客中，我们将通过逐个拆解来学习 Transformer 架构——理解每个组件的作用、它们如何协同工作，以及为什么这种架构支撑着当今所有的大语言模型（LLM）。
当我们听到"Transformer"时，它听起来很复杂。但不用担心。如果我们把它拆解成一个个独立的部分，每一部分其实都很简单。复杂性来自于它们的堆叠方式——而不是因为任何单一部件有多难。
我们的目标是把这个架构解码得足够清晰，以至于到最后，我们能够向任何人解释 Transformer 是如何工作的。
我们将涵盖以下内容：
- 为什么需要 Transformer
- 架构的两个部分
- 分词、嵌入和位置编码
- 注意力机制和多头注意力
- 前馈网络、残差连接和层归一化
- 编码器和解码器是如何工作的
- 数据如何在整个架构中流动
- Transformer 的三种变体
- 为什么 Transformer 如此强大
我是 Outcome School 的 Amit Shekhar，我在那里教授人工智能和机器学习。
让我们开始吧。
## 全局概览
在深入细节之前，让我们先了解整体图景。
Transformer 接收一个 token 序列作为输入，并输出一个 token 序列。就是这样。从非常高的层面来看，它是一台“token 进、token 出”的机器。由于 token 来自文本并最终转回文本，我们也可以把它看作一台“文本进、文本出”的机器。
例如，如果我们给它输入"Translate: I love learning"，它会输出"J'adore apprendre"。如果我们输入"What is the capital of France?"，它会输出"The capital of France is Paris."。
但它是如何做到这一点的呢？让我们来了解一下。
## 为什么需要 Transformer？
在 Transformer 出现之前，模型是按顺序逐个处理单词的——就像从左到右阅读一句话，一次只看一个词。
这带来了两个大问题：
问题一：遗忘。当句子变长时，模型会开始忘记前面的词。等它读到第 50 个词时，对第 1 个词的记忆已经非常模糊了。
问题二：缓慢。由于单词是一个接一个处理的，第 2 个词必须等第 1 个词处理完，第 3 个词必须等第 2 个词，依此类推。这非常慢。
于是，Transformer 应运而生。它同时处理所有输入 token，并让每个 token 直接关注其他所有 token。长距离依赖不再是一个大问题。而且由于一切都是并行进行的，训练速度大大加快。我们将在接下来的章节中详细看到这是如何实现的。
Transformer 于 2017 年由 Google 的研究人员在著名的研究论文《Attention Is All You Need》中提出。它改变了一切。
## 架构分为两部分
原始的 Transformer 有两个主要部分：
- 编码器——这一部分读取并理解输入
- 解码器——这一部分生成输出
可以把它想象成读者和作者之间的对话。读者（编码器）仔细阅读一份文档并创建详细的笔记。作者（解码器）利用这些笔记撰写一份新文档。读者专注于理解，作者专注于创作。
现在，让我们来了解每个部分内部发生了什么。但在深入编码器和解码器之前，我们需要先了解在输入到达它们之前发生了什么。
## 解码步骤 1：将文字转化为数字
模型并不理解文字。它只理解数字。所以，第一步是将文字转换为数字。
这分两个阶段进行：
分词：输入文本被切分成称为 token 的小片段。一个 token 可以是一个词、词的一部分，甚至是一个单独的字符。为了便于理解，我们把每个词当作一个 token。
"I love learning" 变成三个 token："I"、"love"、"learning"。
嵌入：然后每个 token 被转换为一个数字向量，称为嵌入。嵌入就像一个词的数字指纹。含义相似的词会得到相似的指纹。
例如，"happy"和"joyful"会有非常相似的嵌入，因为它们的意思几乎相同。但"happy"和"car"的嵌入会非常不同，因为它们互不相关。
## 解码步骤 2：加入词序信息
但这里有一个问题。编码器是同时处理所有输入 token 的——而不是逐个处理。这对速度非常有利，但这也意味着模型不知道哪个词是第一个、第二个或第三个。
"I love AI"和"AI love I"在模型看来是一样的。但它们的含义截然不同。
位置编码解决了这个问题。它在每个嵌入上添加一个数字向量，用于表示单词的位置。第 1 个词会得到与第 2 个词不同的位置信号，依此类推。
可以把它想象成剧院里的座位号。即使所有观众同时到达，他们的座位号也能准确告诉我们每个人应该坐在哪里。位置编码就是每个词的座位号。
在这一步之后，位置编码的数字通过相加合并到嵌入本身中。因此，现在每个 token 都携带一个单一的数字向量，同时编码了这个词的含义以及它在句子中的位置。
## 解码步骤 3：注意力机制
这是 Transformer 的核心。注意力机制是让一切得以运作的最重要的核心思想。
注意力允许每个词关注句子中的其他所有词，并决定对每个词给予多少关注。
我们来看这个句子："The cat sat on the mat because it was tired."
当模型处理“it”这个词时，它需要弄清楚“it”指的是什么。是猫吗？是垫子吗？注意力机制帮助模型更多地关注“cat”，而较少关注“mat”，因为在这个上下文中“cat”与“it”更相关。
注意力机制的工作原理
每个词被转换成三个东西：
- 查询——这个词在寻找什么
- 键——这个词能提供什么
- 值——这个词携带的实际信息
我们有一篇关于注意力背后的数学原理——Q、K 和 V 的详细博客，解释了这些内容是如何实际计算的。
这个过程的工作方式如下：
首先，每个词使用它的 Query 与所有其他词的 Key 进行点积比较。这会产生注意力分数——这些数字告诉模型每一对词之间的相关程度。然后，这些分数会被缩放（除以 √dₖ），以将它们保持在可管理的范围内。我们有一篇关于缩放点积注意力的详细博客，解释了为什么需要这一步。缩放之后，分数通过 softmax 进行归一化，使它们变成总和为 1 的概率。最后，模型基于这些概率收集 Values 的加权混合。
结果是，每个词现在不仅携带关于自身的信息，还携带关于与它最相关的词的信息。词“it”现在携带了来自“cat”的强信号，因为注意力机制弄清了这层关联。
简单来说，注意力让每个 token 在进入下一层之前，从其他相关 token 那里借用有用的信息。
多头注意力
Transformer 并不是只运行一次注意力。它并行运行多次注意力——每次从不同的视角进行。这被称为多头注意力。
可以把它想象成一个审阅文章的编辑团队。一位编辑检查语法，另一位检查事实准确性，还有一位检查行文流畅度。每位编辑看的都是同一篇文本，但关注的重点不同。他们共同提供了完整的审阅。
多头注意力中的每个头关注词之间不同类型的关系。一个头关注主语与动词的联系。另一个头关注哪个形容词修饰哪个名词。所有头的输出被合并起来，让模型对文本形成丰富的理解。
下面是多头注意力的一个简单可视化。所有头在相同的输入上并行运行，它们的输出在最后被合并：
## 解码步骤 4：前馈网络
在注意力机制处理完这些词之后，输出会通过一个前馈网络。
可以把它看作一个细化步骤。如果说注意力是关于理解词与词之间的关系，那么前馈网络就是关于加深对每个词个体的理解。
它接收来自注意力的丰富表示并进一步处理——锐化信号并去除噪声。每个词都独立地通过同一个前馈网络。
## 解码步骤 5：残差连接与层归一化
在每个层的内部，还有两个重要的组成部分，帮助 Transformer 有效学习。让我们逐一理解它们。
残差连接（跳跃连接）：在每个子层（注意力或前馈网络）之后，该子层的原始输入会被加回到输出上。可以把它想象成在课堂上做笔记。即使老师补充了新信息，我们也保留原始笔记，并将新信息与之结合。这防止了模型在数据经过许多层时丢失重要信息。
简单来说，每个子层的输出是：output = sub-layer(input) + input。
层归一化：在每次残差连接之后，数字会被归一化——进行调整使其保持在稳定的范围内。这防止了数字在经过许多层时变得过大或过小。它使训练过程保持稳定和流畅。
这两个组成部分存在于编码器和解码器的每一个层中。
## 解码步骤 6：堆叠多个层
现在，有趣的部分来了。到目前为止，我们讨论了一个多头注意力子层和一个前馈子层。这两个子层（连同它们的残差连接和层归一化）共同构成了 Transformer 的一个层。可以把一个层看作一个完整的组合：一个多头注意力子层加上一个前馈子层。
Transformer 不仅仅使用一个这样的层。它将多个层一个接一个地堆叠在一起。真正的力量正来自于此。
这些层中的每一层也被称为一个 Transformer 块。所以当我们说“模型有 96 层”时，意思是模型有 96 个 Transformer 块一个接一个地堆叠。每个块都有自己的多头注意力子层和自己的前馈子层。
注意：这种堆叠与多头注意力不同。多头注意力是在单个注意力子层内并行运行许多注意力头。而堆叠层则是将完整的 Transformer 块按顺序一个接一个地放置，使上一层的输出成为下一层的输入。
下面是堆叠 Transformer 层的一个简单可视化。每个层都是一个完整的块，它们按顺序一个接一个地连接：
原始的 Transformer 在编码器和解码器中都使用 6 层。现代大语言模型使用更多——GPT-3（175B）使用了 96 层。
每一层都在上一层理解的基础上进行构建。可以把它想象成多次阅读一本书。第一遍阅读给我们一个基本的理解。第二遍阅读揭示了我们之前遗漏的联系。第三遍阅读给我们更深入的见解。每一层就像又一次阅读——理解变得越来越丰富。
## 解码步骤 7：详解编码器
现在我们已经了解了这些构建模块，让我们看看它们是如何在编码器内部组合在一起的。
编码器的任务是读取并理解输入。
每个编码器层有两个组件：
- 自注意力——每个词会关注输入中的所有其他词，以理解它们之间的关系
- 前馈网络——每个词的表示被单独精炼
这两个组件之后都跟着残差连接和层归一化。
让我们看看句子 "I love learning" 通过编码器时会发生什么。
在第一层中，自注意力机制允许每个词关注所有其他词。"love" 这个词开始注意到 "I" 是它的主语，而 "learning" 是它的宾语。然后前馈网络精炼这一理解。
在第二层中，这些联系变得更加丰富。现在 "love" 不只是分别知道 "I" 和 "learning"——它开始把 "I love learning" 作为一个完整的思想来理解其组合含义。
到第六层时，每个词都携带了深入的、具备上下文感知的表示。"learning" 这个词不再只是一个词典释义——它现在编码了这样一个事实：在这个特定的句子中，它是 "I" 喜欢做的事情。
经过所有编码器层之后，我们对输入有了丰富的理解。这一理解随后被传递给解码器。
## 解码步骤 8：详解解码器
解码器的任务是一次一个词地生成输出。
每个解码器层有三个组件：
- 掩码自注意力——每个输出词只关注之前的输出词，而不关注未来的词。这是因为在生成文本时，未来的词还不存在。例如 "love" 可以关注 "I"，但不能关注 "learning"，因为 "learning" 还没有被生成。这种限制被称为因果掩码。我们有一篇关于注意力机制中因果掩码的详细博客，解释了它的工作原理。
- 交叉注意力——这是解码器与编码器连接的方式。每个输出词会关注编码器的输出，从输入中提取相关信息。在这里，Query 来自解码器，而 Key 和 Value 来自编码器的输出。这正是它与自注意力的区别——也是理解输入和生成输出之间的桥梁。
- 前馈网络——与编码器中相同的精炼步骤。
这三个组件之后都跟着残差连接和层归一化，与编码器中一样。
解码器如何选择下一个词
现在的问题是：当输出经过所有解码器层之后，解码器如何选出实际的下一个词？此时，输出仍然是一个数字向量——而不是一个词。
因此，解码器将这些数字通过一个线性层，将其转换为词表中每个词的分数。如果词表有 50,000 个词，我们就会得到 50,000 个分数。
然后，softmax 将这些分数转换为概率——即 0 到 1 之间、总和为 1 的数字。模型随后根据这些概率选择下一个词。在实践中，模型使用不同的策略来选择下一个词——比如 temperature、top-k 和 top-p 采样——它们控制输出的创造性或聚焦程度。但为了便于理解，我们假设它选择概率最高的词。
例如，如果模型正在将 "I love learning" 翻译成法语，并且已经生成了 "J'"，那么经过线性层和 softmax 之后的概率如下：
- "adore": 0.72
- "aime": 0.15
- "mange": 0.01
- ...（50,000 个词，大多数概率非常小）
模型选择 "adore"，因为它的概率最高。这就是解码器决定下一个输出词的方式。
## 解码步骤 9：完整流程
现在我们已经了解了所有的构建模块，让我们通过一个具体的例子来走一遍完整流程。学习这一点的最佳方式就是举一个例子。
假设我们想把 "I love learning" 翻译成法语。
编码器阶段
英文句子 "I love learning" 经过以下步骤：
分词："I love learning" 变成三个词元："I"、"love"、"learning"。
嵌入：每个词元被转换为一个数字向量（嵌入）。
添加位置信息：加入位置编码，使模型知道 "I" 是第 1 个、"love" 是第 2 个、"learning" 是第 3 个。
经过 6 个编码器层：在每一层中，自注意力让每个词都能关注所有其他词，前馈网络精炼理解。残差连接和层归一化保持整个过程的稳定。
经过全部 6 层之后，编码器生成了输入的丰富表示。每个词现在都携带了深入的上下文信息。这一表示被传递给解码器。
解码器阶段
解码器一次一个词地生成法语翻译。
生成 "J'"（第 1 个词）：
解码器从一个特殊的 "start"（开始）词元开始。由于这是第一个词元，掩码自注意力没有任何可以回看的内容。交叉注意力关注编码器的输出，以理解英文输入。输出经过线性层和 softmax。模型选择 "J'" 作为概率最高的第一个词。
生成 "adore"（第 2 个词）：
现在解码器已拥有 ["start", "J'"]。掩码自注意力关注 "start" 和 "J'"，以理解目前生成了什么。交叉注意力再次关注编码器的输出，从英文输入中提取相关信息。线性层和 softmax 产生概率。模型选择 "adore"。
生成 "apprendre"（第 3 个词）：
现在解码器已拥有 ["start", "J'", "adore"]。掩码自注意力关注所有三个先前的词元。交叉注意力连接到编码器的输出。模型选择 "apprendre"。
生成 "end" 词元：
解码器生成一个特殊的 "end"（结束）词元，表示翻译已完成。
最终输出："J'adore apprendre"（意为法语的 "I love learning"）。
这就是 Transformer 从头到尾的工作方式。
## 三种变体
最初的 Transformer 同时使用编码器和解码器。但我们并不总是需要两者。这产生了三种变体：
仅编码器（例如：BERT）
只使用编码器。模型读取并理解输入，但不生成新文本。这用于诸如将邮件分类为垃圾邮件或非垃圾邮件、理解评论的情感倾向，或在给定文本中寻找答案等任务。
仅解码器（示例：GPT）
只使用解码器。模型一次生成一个 token。每个 token 可以看到所有之前的 token，但不能看到未来的 token。这是大多数现代大语言模型背后的变体。它用于生成文本、回答问题、编写代码和进行对话。
编码器-解码器（示例：T5、原始 Transformer）
同时使用两个部分。编码器理解输入，解码器基于该理解生成输出。这用于翻译、摘要以及其他将一种文本转换为另一种文本的任务。
我们可以根据使用场景选择合适的变体。
## 为什么 Transformer 如此强大
现在我们已经解码了整个架构，让我们来理解它为什么改变了一切：
并行处理：编码器同时处理所有输入 token，而不是逐个处理。即使在训练期间，解码器也是并行运行的。这使得训练速度快得多，并允许模型从海量数据中学习。
不再有长距离遗忘：通过注意力机制，每个词都可以直接关注其他每一个词。第 1 个词和第 1000 个词一样容易获取，因此旧模型的长距离记忆问题在很大程度上得到了解决。
可扩展性：通过增加更多层、更多注意力头和更多数据，模型会不断变得更好。这就是为什么我们今天拥有数十亿参数的模型。参数是模型在训练期间学习到的内部数值。
通用性：同一架构可以处理翻译、问答、文本生成、代码编写、摘要等等。一种设计，无数应用。
Transformer 是整个现代 AI 革命背后的唯一架构。它让我们能够非常简单地完成复杂的事情。
## 快速总结
让我们回顾一下我们已经解码的内容：
- 分词和嵌入将单词转换为模型可以处理的数字。
- 位置编码添加词序信息，因为所有词是同时处理的。
- 注意力让每个词都能关注其他每一个词，并决定聚焦于哪些词。多头注意力则从多个视角进行这一操作。
- 前馈网络逐个细化对每个词的理解。
- 残差连接和层归一化在数据流经许多层时保持数值稳定。
- 编码器堆叠这些构建模块来读取和理解输入。
- 解码器添加交叉注意力以连接编码器，并使用线性层 + softmax 来选择下一个词。
- 存在三种变体：仅编码器用于理解，仅解码器用于生成，编码器-解码器用于将一种文本转换为另一种文本。
现在，我们已经逐块解码了 Transformer 架构，并理解了它的工作原理。
为 AI 工程面试做好准备：AI 工程面试题
本次分享到此为止。
谢谢
Amit Shekhar
Outcome School 创始人
## 相关链接
- [Amit Shekhar](https://x.com/amitiitbhu)
- [@amitiitbhu](https://x.com/amitiitbhu)
- [66K](https://x.com/amitiitbhu/status/2041479290580287543/analytics)
- [Outcome School](https://outcomeschool.com/)
- [AI 与机器学习](https://outcomeschool.com/program/ai-and-machine-learning)
- [分词](https://www.youtube.com/watch?v=sK2s9I84EVI)
- [嵌入](https://www.youtube.com/watch?v=LedXW6xl21s)
- [注意力背后的数学 - Q、K 和 V](https://outcomeschool.com/blog/math-behind-attention-qkv)
- [缩放点积注意力](https://outcomeschool.com/blog/scaling-dot-product-attention)
- [注意力中的因果掩码](https://outcomeschool.com/blog/causal-masking-in-attention)
- [AI 工程面试题](https://github.com/amitshekhariitbhu/ai-engineering-interview-questions)
- [Outcome School](https://outcomeschool.com/)
- [升级至高级版](https://x.com/i/premium_sign_up)
- [2026年4月7日 下午7:32](https://x.com/amitiitbhu/status/2041479290580287543)
- [66.1K 次浏览](https://x.com/amitiitbhu/status/2041479290580287543/analytics)
- [查看引用](https://x.com/amitiitbhu/status/2041479290580287543/quotes)
---
*导出时间: 2026/4/18 18:32:43*
