# LoRA - Low-Rank Adaptation of LLMs
**作者**: Amit Shekhar
**日期**: 2026-05-07T11:08:50.000Z
**来源**: [https://x.com/amitiitbhu/status/2052344905004171507](https://x.com/amitiitbhu/status/2052344905004171507)
---
In this blog, we will learn about LoRA - Low-Rank Adaptation of Large Language Models.
Today, we will cover the following topics:
- The Big Picture
- Why Full Fine-Tuning Is Expensive
- The Core Idea Behind LoRA
- How LoRA Works Step by Step
- A Small Numeric Example
- Where LoRA Is Applied in a Transformer
- Merging LoRA Back Into the Model
- Real-World Use Cases
- Quick Summary
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
## The Big Picture
Before we go into the details, let's understand the big picture.
LoRA is a way to fine-tune a large model without updating all of its weights. Instead of changing the original weight matrix, we keep it frozen and learn a tiny pair of extra matrices on the side. These tiny matrices capture the "adjustment" we want.
In simple words:
LoRA = Frozen original weights + A small low-rank update learned on the side.
This makes fine-tuning much cheaper, faster, and lighter on storage. And, we still get strong task-specific performance.
## Why Full Fine-Tuning Is Expensive
Before jumping into LoRA, we must first understand why full fine-tuning is hard.
We can learn everything about fine-tuning in this video.
A modern Large Language Model can have billions of parameters. For example, a 7 billion parameter model has 7 billion numbers inside it. When we do full fine-tuning, we update every single one of these numbers.
This causes the following issues:
- GPU memory explodes. We need to keep the weights, the gradients, and the optimizer states in memory. For Adam, this can be 3 to 4 times the size of the model itself.
- Training is slow. Updating billions of parameters takes a lot of compute.
- Storage is heavy. Each fine-tuned copy of the model is the same size as the original. If we have 10 different tasks, we need 10 full copies of a multi-gigabyte model.
- Sharing is hard. A multi-gigabyte checkpoint per task is not easy to ship to users.
We need a smarter way. We need a way to adapt the model to a new task without touching most of its weights.
So, here comes LoRA to the rescue.
## The Core Idea Behind LoRA
The full form of LoRA is Low-Rank Adaptation.
Let's decompose the name:
LoRA = Low-Rank + Adaptation.
- Adaptation means we are adapting the model to a new task or a new style.
- Low-Rank means the change we apply to the weights is represented in a very compact form using two small matrices.
The idea is based on a powerful observation from the original LoRA paper from Microsoft:
> When we fine-tune a large model, the actual update that we apply to the weights is very low-rank. We do not need a full giant update matrix - a much smaller one is enough.
In simple words, the "adjustment" needed to teach the model a new task is much smaller than the model itself. So, we do not need a giant matrix to represent it.
Why does this work? The pre-trained model already knows a lot. It has learned grammar, facts, reasoning, and general patterns. Fine-tuning it for a new task is not a rewrite. It is just a small nudge along a few important directions. That is exactly what "low-rank" means.
Let's take a real-world analogy. Think of the pre-trained model as a giant textbook. Full fine-tuning is like rewriting every page of the textbook for a new topic. LoRA is like keeping the textbook unchanged and adding a small set of sticky notes on top that carry the new knowledge. The textbook stays the same, and the small notes do the actual adjustment.
Now, let's see how this works in math.
A weight matrix W inside the model has shape d x d. In full fine-tuning, we learn a new matrix W_new of the same shape:
```
W_new = W + ΔW
```
Here, ΔW is the change we want to apply. Both W and ΔW are huge.
LoRA replaces this huge ΔW with a product of two small matrices:
```
ΔW = BA
```
Where:
- A has shape r x d
- B has shape d x r
- r is the rank, a small number like 4, 8, 16, 32, or 64
- d is the original dimension, often 4096 or larger
Now, BA still has shape d x d, but we never store the full d x d update. We only store A and B, which are tiny compared to W.
Note: For simplicity, we are assuming W is a square d x d matrix. In real Large Language Models, attention projection matrices are square, but feed-forward layers are not. They are usually d x 4d or 4d x d. LoRA works for any rectangular matrix - if W has shape d x k, then B becomes d x r and A becomes r x k. The idea stays exactly the same.
Here is a simple ASCII diagram comparing the sizes.
```
ΔW (full update) B x A
(d x d matrix) (d x r) (r x d)
+-------------------+ +--+ +-------------------+
| | | | +-------------------+
| | | |
| | | |
| | = | | x
| | | |
| | | |
| | | |
| | | |
+-------------------+ +--+
```
Here, the big square is the full update we would have to store. The thin column B and the short row A together carry all the information we actually train. The visual size difference is exactly why LoRA is so memory-efficient.
The forward pass for an input vector x becomes:
```
h = Wx + (BA)x
```
Here, W is frozen. Only A and B are trained.
## How LoRA Works Step by Step
Let's walk through LoRA step by step.
Step 1: Take the pre-trained model and freeze all of its original weights. Nothing inside the original model will be updated during training.
Step 2: For each weight matrix we want to adapt, add two small matrices A and B next to it.
- A is initialized with small random values (Gaussian).
- B is initialized with all zeros.
This zero initialization is important. At the start of training, BA = 0, so the model behaves exactly like the original pre-trained model. Training then gradually nudges BA away from zero.
Step 3: During training, only A and B receive gradient updates. The original W stays untouched.
Step 4: During the forward pass, compute the output as:
```
h = Wx + (alpha / r) * (BA)x
```
Here, alpha is a scaling factor. The ratio alpha / r controls how strongly the LoRA update influences the output. A common choice is alpha = 2 * r.
Step 5: After training, we have a tiny set of new weights A and B that capture the task-specific knowledge. The original model is still untouched.
That's the entire idea.
Going back to our textbook analogy, the original textbook is the frozen W. The sticky notes are A and B. The notes start blank because B is zero. As training goes on, the notes fill up with corrections that nudge the textbook's behavior toward the new task.
Note: The choice of rank r is the most important LoRA hyperparameter. A smaller r means fewer trainable parameters and faster training, but the adapter has less capacity to learn complex changes. A larger r gives more capacity but loses some of the efficiency benefits. In practice, ranks of 8 to 64 work well for most tasks.
To learn fine-tuning, PEFT, and LoRA hands-on with real projects, check out the AI and Machine Learning Program by Outcome School.
## A Small Numeric Example
Let's put this into perspective with real numbers.
Suppose we have a weight matrix W of shape 4096 x 4096. The number of parameters in W is:
```
4096 x 4096 = 16,777,216
```
That's around 16.8 million parameters in just one matrix. A full fine-tune would update all of them.
Now, let's apply LoRA with rank r = 8.
- A has shape 8 x 4096 = 32,768 parameters
- B has shape 4096 x 8 = 32,768 parameters
- Total LoRA parameters = 65,536
Let's compare:
- Full fine-tune: 16,777,216 parameters per matrix
- LoRA with r=8: 65,536 parameters per matrix
That is roughly 256x fewer parameters for the same matrix. And, this is just for one matrix - the savings multiply across every layer of the model.
For a full 7B parameter model, full fine-tuning updates 7 billion parameters. LoRA with rank 8 typically updates only a few million. The trainable parameter count drops by a factor of around 1000x or more, depending on the rank and where LoRA is applied.
## Where LoRA Is Applied in a Transformer
LoRA can be applied to any linear layer in the model. But, in practice, LoRA is most commonly applied to the attention projection matrices.
We have a detailed blog on Transformer Architecture that explains how attention layers work inside a transformer.
Inside each attention block, we have four projection matrices:
- W_Q - the Query projection
- W_K - the Key projection
- W_V - the Value projection
- W_O - the Output projection
If we want to go deeper into how Q, K, and V are computed and used, we can read Math Behind Attention: Q, K, V.
The original LoRA paper found that adapting just W_Q and W_V is often enough to get strong task performance with minimal extra parameters. In practice, we can also apply LoRA to all four projections, or even to the feed-forward layers, for better quality at the cost of more trainable parameters.
Here is a simple ASCII diagram showing the structure for a single weight matrix.
```
Input x
|
+-------+-------+
| |
v v
+---------+ +-------+
| W | | A | (trainable, r x d)
| (frozen)| +-------+
+---------+ |
| v
| +---------+
| | B | (trainable, d x r)
| +---------+
| |
v v
+-------+-------+
|
v
h = Wx + (BA)x
```
Here, W stays frozen and only A and B are trained. The two paths are added together to form the final output.
If we want to go deep into the Transformer architecture, attention, and Q/K/V projections, we have a complete program - check out the AI and Machine Learning Program by Outcome School.
## Merging LoRA Back Into the Model
Here is one of the most beautiful properties of LoRA.
Once training is done, we can merge BA directly into W:
```
W_merged = W + (alpha / r) * (BA)
```
Now, the model uses a single weight matrix W_merged exactly like the original. There are no extra matrices and no extra computation at inference time.
This means LoRA adds zero inference latency when merged. We get the benefits of fine-tuning without any runtime overhead.
In our textbook analogy, merging is like permanently writing the sticky notes into the textbook itself. The notes are gone, but the textbook now carries the new knowledge.
We can also keep A and B separate from W if we want to swap between different tasks at runtime. This is the basis of adapter swapping.
Note: Merging is a one-way operation. Once we merge A and B into W, we cannot easily swap that adapter out for a different one. So, if we want to serve many tasks from the same base model, we should keep the adapters unmerged and load them on demand.
## Real-World Use Cases
LoRA is everywhere in modern LLM workflows. Let's see where LoRA is used in practice.
- Fine-tuning open-source LLMs on a single GPU. Models like LLaMA, Mistral, and Qwen are commonly fine-tuned using LoRA on a single consumer or workstation GPU. Without LoRA, this would need a cluster of expensive GPUs.
- Task-specific adapters. Teams train small LoRA adapters for each task - one for summarization, one for code generation, one for customer support - and load only the adapter they need. The base model stays the same.
- Style and domain adaptation. LoRA is used to teach a base model a specific writing style, a domain like medical or legal, or a specific persona.
- Image generation models. LoRA is widely used in image models like Stable Diffusion to add new characters, styles, or concepts without retraining the whole model. The community shares thousands of small LoRA files.
- Foundation for QLoRA. LoRA is the building block for QLoRA, which combines LoRA with 4-bit quantization to fine-tune very large models on a single consumer GPU.
Note: When we ship LoRA adapters to users, we only ship the small A and B matrices. These are often just a few megabytes. Compare that to a multi-gigabyte full fine-tuned model. This is why LoRA adapters are so easy to share, swap, and version.
## Quick Summary
Let's recap what we have decoded:
- LoRA = Low-Rank Adaptation. We freeze the original weights and learn a small low-rank update on the side.
- The update is ΔW = BA where A and B are tiny matrices controlled by a small rank r.
- A is random, B is zero at the start. This makes the model start exactly like the original pre-trained model.
- Only A and B are trained. The original W stays frozen, which saves massive amounts of memory and compute.
- Trainable parameters drop by 100x to 1000x or more depending on the rank and where LoRA is applied.
- LoRA can be merged into the original weights after training, so there is zero extra latency at inference time.
- Adapter swapping lets us serve many tasks from a single base model by loading different A and B matrices.
- LoRA is used everywhere - LLMs, image models, domain adaptation, and as the foundation for QLoRA.
This is how LoRA makes fine-tuning of Large Language Models cheap, fast, and accessible.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
## 相关链接
- [Amit Shekhar](https://x.com/amitiitbhu)
- [@amitiitbhu](https://x.com/amitiitbhu)
- [7.2K](https://x.com/amitiitbhu/status/2052344905004171507/analytics)
- [Outcome School](https://outcomeschool.com/)
- [AI and Machine Learning](https://outcomeschool.com/program/ai-and-machine-learning)
- [video](https://www.youtube.com/watch?v=lnfWvX66FUk)
- [AI and Machine Learning Program](https://outcomeschool.com/program/ai-and-machine-learning)
- [Transformer Architecture](https://outcomeschool.com/blog/decoding-transformer-architecture)
- [Math Behind Attention: Q, K, V](https://outcomeschool.com/blog/math-behind-attention-qkv)
- [AI and Machine Learning Program](https://outcomeschool.com/program/ai-and-machine-learning)
- [Outcome School](https://outcomeschool.com/)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [7:08 PM · May 7, 2026](https://x.com/amitiitbhu/status/2052344905004171507)
- [7,248 Views](https://x.com/amitiitbhu/status/2052344905004171507/analytics)
---
*导出时间: 2026/5/8 13:42:34*
---
## 中文翻译
# LoRA - LLM 的低秩自适应
**作者**: Amit Shekhar
**日期**: 2026-05-07T11:08:50.000Z
**来源**: [https://x.com/amitiitbhu/status/2052344905004171507](https://x.com/amitiitbhu/status/2052344905004171507)
---
在这篇博客中，我们将了解 LoRA —— 大型语言模型的低秩自适应。
今天，我们将涵盖以下主题：
- 全景概览
- 为什么全量微调成本高昂
- LoRA 背后的核心思想
- LoRA 逐步工作原理
- 一个小的数值示例
- LoRA 在 Transformer 中的位置
- 将 LoRA 合并回模型
- 真实世界的用例
- 快速总结
我是 Amit Shekhar，Outcome School 的创始人。我曾指导和教授过许多开发者，他们的努力让他们获得了高薪的技术工作，帮助许多科技公司解决了他们的独特问题，并创建了许多被顶级公司使用的开源库。我热衷于通过开源、博客和视频分享知识。
我在 Outcome School 教授 AI 和机器学习课程。
让我们开始吧。
## 全景概览
在深入细节之前，让我们先了解全景概览。
LoRA 是一种无需更新模型所有权重即可微调大模型的方法。我们不是改变原始权重矩阵，而是保持其冻结，并在旁边学习一对微小的额外矩阵。这些微小的矩阵捕获了我们想要的“调整”。
简单来说：
LoRA = 冻结的原始权重 + 侧面学习到的一个小型低秩更新。
这使得微调更便宜、更快速，且存储更轻便。而且，我们仍然能获得强大的特定任务性能。
## 为什么全量微调成本高昂
在深入 LoRA 之前，我们必须首先理解为什么全量微调很难。
我们可以在这个视频中学习关于微调的所有内容。
一个现代的大型语言模型可以拥有数十亿个参数。例如，一个 70 亿参数的模型内部有 70 亿个数字。当我们进行全量微调时，我们会更新其中的每一个数字。
这会导致以下问题：
- GPU 显存爆炸。我们需要在内存中保存权重、梯度和优化器状态。对于 Adam 优化器，这可能是模型本身大小的 3 到 4 倍。
- 训练缓慢。更新数十亿个参数需要大量的计算。
- 存储负担重。模型的每一个微调副本都和原始模型一样大。如果我们有 10 个不同的任务，我们需要 10 个完整的、数 GB 大小的模型副本。
- 分享困难。每个任务一个数 GB 的检查点很难分发给用户。
我们需要一种更聪明的方法。我们需要一种无需触碰大部分权重就能使模型适应新任务的方法。
于是，LoRA 来拯救我们了。
## LoRA 背后的核心思想
LoRA 的全称是 Low-Rank Adaptation（低秩自适应）。
让我们分解一下这个名字：
LoRA = Low-Rank（低秩）+ Adaptation（自适应）。
- **Adaptation** 意味着我们正在使模型适应新任务或新风格。
- **Low-Rank** 意味着我们应用于权重的变化通过两个小矩阵以非常紧凑的形式表示。
这个想法基于微软原始 LoRA 论文中的一个有力观察：
> 当我们微调一个大模型时，我们应用于权重的实际更新是非常低秩的。我们不需要一个完整的巨大更新矩阵 —— 一个小得多的矩阵就足够了。
简单来说，教模型学习新任务所需的“调整”比模型本身要小得多。所以，我们不需要一个巨大的矩阵来表示它。
为什么这行得通？预训练模型已经知道很多了。它已经学习了语法、事实、推理和通用模式。针对新任务进行微调不是重写。它只是沿着几个重要方向的微调。这正是“低秩”的意思。
让我们用一个现实世界的比喻。把预训练模型想象成一本巨大的教科书。全量微调就像为了一个新主题重写教科书的每一页。LoRA 就像是保持教科书不变，并在上面添加一小套便签纸，上面承载着新知识。教科书保持不变，小小的便签纸进行实际调整。
现在，让我们看看这在数学上是如何工作的。
模型内部的权重矩阵 W 的形状是 d x d。在全量微调中，我们学习一个相同形状的新矩阵 W_new：
```
W_new = W + ΔW
```
这里，ΔW 是我们要应用的变化。W 和 ΔW 都是巨大的。
LoRA 用两个小矩阵的乘积替换了这个巨大的 ΔW：
```
ΔW = BA
```
其中：
- A 的形状是 r x d
- B 的形状是 d x r
- r 是秩，是一个小数字，如 4、8、16、32 或 64
- d 是原始维度，通常是 4096 或更大
现在，BA 仍然是 d x d 的形状，但我们从未存储完整的 d x d 更新。我们只存储 A 和 B，与 W 相比它们非常小。
注意：为了简单起见，我们假设 W 是一个方阵 d x d。在现实的大型语言模型中，注意力投影矩阵是方阵，但前馈层不是。它们通常是 d x 4d 或 4d x d。LoRA 适用于任何矩形矩阵 —— 如果 W 的形状是 d x k，那么 B 变成 d x r，A 变成 r x k。思路完全一样。
这是一个比较大小的简单 ASCII 图示。
```
ΔW (full update) B x A
(d x d matrix) (d x r) (r x d)
+-------------------+ +--+ +-------------------+
| | | | +-------------------+
| | | |
| | | |
| | = | | x
| | | |
| | | |
| | | |
| | | |
+-------------------+ +--+
```
这里，大方形是我们必须存储的完整更新。细列 B 和短行 A 共同携带了我们实际训练的所有信息。视觉上的大小差异正是 LoRA 如此节省内存的原因。
对于输入向量 x 的前向传播变为：
```
h = Wx + (BA)x
```
这里，W 被冻结。只有 A 和 B 被训练。
## LoRA 逐步工作原理
让我们一步步看 LoRA 是如何工作的。
第一步：获取预训练模型并冻结其所有原始权重。原始模型内部的任何内容在训练期间都不会更新。
第二步：对于我们要适应的每个权重矩阵，在它旁边添加两个小矩阵 A 和 B。
- A 用小的随机值（高斯分布）初始化。
- B 用全零初始化。
这种零初始化很重要。在训练开始时，BA = 0，因此模型的行为与原始预训练模型完全一样。然后训练逐渐将 BA 从零推向非零。
第三步：在训练期间，只有 A 和 B 接收梯度更新。原始 W 保持不变。
第四步：在前向传播期间，计算输出为：
```
h = Wx + (alpha / r) * (BA)x
```
这里，alpha 是一个缩放因子。比率 alpha / r 控制 LoRA 更新对输出有多强的影响。一个常见的选择是 alpha = 2 * r。
第五步：训练结束后，我们有一小组新的权重 A 和 B，它们捕获了特定任务的知识。原始模型仍然未被触动。
这就是整个思想。
回到我们的教科书比喻，原始教科书是冻结的 W。便签纸是 A 和 B。便签纸开始是空白的，因为 B 是零。随着训练的进行，便签纸上填满了修正，将教科书的行为推向新任务。
注意：秩 r 的选择是 LoRA 最重要的超参数。较小的 r 意味着更少的可训练参数和更快的训练，但适配器学习复杂变化的能力较小。较大的 r 提供了更多容量，但会失去一些效率优势。在实践中，8 到 64 的秩对大多数任务都适用。
要通过实际项目学习微调、PEFT 和 LoRA，请查看 Outcome School 的 AI 和机器学习课程。
## 一个小的数值示例
让我们用真实数字来看待这个问题。
假设我们有一个形状为 4096 x 4096 的权重矩阵 W。W 中的参数数量是：
```
4096 x 4096 = 16,777,216
```
这仅仅是一个矩阵就有大约 1680 万个参数。全量微调会更新所有这些参数。
现在，让我们应用秩 r = 8 的 LoRA。
- A 的形状是 8 x 4096 = 32,768 个参数
- B 的形状是 4096 x 8 = 32,768 个参数
- LoRA 参数总数 = 65,536
让我们比较一下：
- 全量微调：每个矩阵 16,777,216 个参数
- r=8 的 LoRA：每个矩阵 65,536 个参数
对于同一个矩阵，参数大约减少了 256 倍。而且，这仅仅是针对一个矩阵 —— 节省的参数会在模型的每一层中成倍增加。
对于一个完整的 7B 参数模型，全量微调更新 70 亿个参数。秩为 8 的 LoRA 通常只更新几百万个参数。可训练参数的数量下降了大约 1000 倍或更多，具体取决于秩以及 LoRA 应用的位置。
## LoRA 在 Transformer 中的位置
LoRA 可以应用于模型中的任何线性层。但在实践中，LoRA 最常见的是应用于注意力投影矩阵。
我们有一篇关于 Transformer 架构的详细博客，解释了注意力层在 transformer 内部是如何工作的。
在每个注意力块内部，我们有四个投影矩阵：
- W_Q - 查询 投影
- W_K - 键 投影
- W_V - 值 投影
- W_O - 输出投影
如果我们想深入了解 Q、K 和 V 是如何计算和使用的，我们可以阅读《注意力背后的数学：Q, K, V》。
原始的 LoRA 论文发现，仅适应 W_Q 和 W_V 通常足以获得强大的任务性能，同时只需最少的额外参数。在实践中，为了更好的质量，我们也可以将 LoRA 应用于所有四个投影，甚至前馈层，代价是更多的可训练参数。
这是一个显示单个权重矩阵结构的简单 ASCII 图示。
```
Input x
|
+-------+-------+
| |
v v
+---------+ +-------+
| W | | A | (trainable, r x d)
| (frozen)| +-------+
+---------+ |
| v
| +---------+
| | B | (trainable, d x r)
| +---------+
| |
v v
+-------+-------+
|
v
h = Wx + (BA)x
```
这里，W 保持冻结，只有 A 和 B 被训练。两条路径相加形成最终输出。
如果我们想深入了解 Transformer 架构、注意力和 Q/K/V 投影，我们有一个完整的课程 —— 查看 Outcome School 的 AI 和机器学习课程。
## 将 LoRA 合并回模型
这是 LoRA 最美丽的特性之一。
一旦训练完成，我们可以将 BA 直接合并到 W 中：
```
W_merged = W + (alpha / r) * (BA)
```
现在，模型像原始模型一样使用单个权重矩阵 W_merged。没有额外的矩阵，推理时也没有额外的计算。
这意味着合并后 LoRA 增加零推理延迟。我们获得了微调的好处，而没有任何运行时开销。
在我们的教科书比喻中，合并就像将便签纸永久地写进教科书中。便签纸不见了，但教科书现在承载了新知识。
如果我们想在运行时在不同任务之间切换，我们也可以将 A 和 B 与 W 分开。这就是适配器交换的基础。
注意：合并是一个单向操作。一旦我们将 A 和 B 合并到 W 中，我们就很难轻易将该适配器换为另一个。因此，如果我们想从同一个基础模型服务多个任务，我们应该保持适配器未合并，并按需加载它们。
## 真实世界的用例
LoRA 在现代 LLM 工作流程中无处不在。让我们看看 LoRA 在实践中是如何使用的。
- **在单个 GPU 上微调开源 LLM。** 像 LLaMA、Mistral 和 Qwen 这样的模型通常使用 LoRA 在单个消费级或工作站 GPU 上进行微调。如果没有 LoRA，这将需要一组昂贵的 GPU。
- **特定任务的适配器。** 团队为每个任务训练小型 LoRA 适配器 —— 一个用于摘要，一个用于代码生成，一个用于客户支持 —— 并只加载他们需要的适配器。基础模型保持不变。
- **风格和领域适应。** LoRA 用于教基础模型特定的写作风格、像医疗或法律这样的领域，或特定的人设。
- **图像生成模型。** LoRA 广泛应用于像 Stable Diffusion 这样的图像模型中，以添加新角色、风格或概念，而无需重新训练整个模型。社区分享数千个小型 LoRA 文件。
- **QLoRA 的基础。** LoRA 是 QLoRA 的构建块，QLoRA 将 LoRA 与 4 位量化结合，在单个消费级 GPU 上微调非常大的模型。
注意：当我们向用户分发 LoRA 适配器时，我们只分发小的 A 和 B 矩阵。这些通常只有几兆字节。与数 GB 的全量微调模型相比，这就是为什么 LoRA 适配器如此易于分享、交换和版本控制。
## 快速总结
让我们总结一下我们解读的内容：
- LoRA = 低秩自适应。我们冻结原始权重并在侧面学习一个小的低秩更新。
- 更新是 ΔW = BA，其中 A 和 B 是由小秩 r 控制的微小矩阵。
- A 是随机的，B 在开始时为零。这使得模型开始时与原始预训练模型完全一样。
- 只有 A 和 B 被训练。原始 W 保持冻结，这节省了大量的内存和计算。
- 可训练参数减少了 100 倍到 1000 倍或更多，具体取决于秩以及 LoRA 应用的位置。
- LoRA 可以在训练后合并到原始权重中，因此推理时没有额外的延迟。
- 适配器交换允许我们通过加载不同的 A 和 B 矩阵，从单个基础模型服务多个任务。
- LoRA 被广泛应用于各行各业 —— LLM、图像模型、领域适应，以及作为 QLoRA 的基础。
这就是 LoRA 如何使大型语言模型的微调变得便宜、快速且易于访问。
目前就这些。
谢谢
Amit Shekhar
Founder @ Outcome School
## 相关链接
- [Amit Shekhar](https://x.com/amitiitbhu)
- [@amitiitbhu](https://x.com/amitiitbhu)
- [7.2K](https://x.com/amitiitbhu/status/2052344905004171507/analytics)
- [Outcome School](https://outcomeschool.com/)
- [AI and Machine Learning](https://outcomeschool.com/program/ai-and-machine-learning)
- [video](https://www.youtube.com/watch?v=lnfWvX66FUk)
- [AI and Machine Learning Program](https://outcomeschool.com/program/ai-and-machine-learning)
- [Transformer Architecture](https://outcomeschool.com/blog/decoding-transformer-architecture)
- [Math Behind Attention: Q, K, V](https://outcomeschool.com/blog/math-behind-attention-qkv)
- [AI and Machine Learning Program](https://outcomeschool.com/program/ai-and-machine-learning)
- [Outcome School](https://outcomeschool.com/)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [7:08 PM · May 7, 2026](https://x.com/amitiitbhu/status/2052344905004171507)
- [7,248 Views](https://x.com/amitiitbhu/status/2052344905004171507/analytics)
---
*导出时间: 2026/5/8 13:42:34*