250个文档即可投毒大模型：数据投毒的规模与防御

Brian Roemmele 2026-04-21 16.7 KB 46.9K 684 𝕏 文章列表

Anthropic等机构的联合研究证实，仅需250个精心设计的恶意文档即可在6亿至130亿参数的大模型中植入永久后门。这种数据投毒具有隐蔽性、不可逆性和规模不变性，且目前缺乏有效的防御手段。作者认为这证实了基于公网数据抓取的训练范式存在根本性缺陷，呼吁转向使用离线、人工 curated 的高质量数据源（如1970年以前的文献）进行训练，并强调构建本地化、主权级模型的重要性。

LLM 数据投毒模型安全 Anthropic 数据质量离线训练网络安全后门攻击 AI安全

# The 250 Document AI Backdoor. Do You Hear Me Now? **作者**: Brian Roemmele **日期**: 2026-04-21T03:48:25.000Z **来源**: [https://x.com/BrianRoemmele/status/2046435862087635448](https://x.com/BrianRoemmele/status/2046435862087635448) --- ![Image](/doc/886/image/img_fe4300bed9b2.jpg) A joint study by Anthropic, the UK AI Security Institute and the Alan Turing Institute dropped a bombshell. They proved that inserting just 250 specially crafted documents into the pretraining data is enough to create a permanent backdoor in large language models from 600 million parameters all the way up to 13 billion parameters. This works no matter how large the model or how massive the overall training dataset gets. These poisoned documents look completely normal. They read like ordinary web pages. But hidden inside is a trigger phrase. Once the model sees that trigger later on, it can be forced into harmful behavior such as spitting out gibberish, leaking data, or breaking down completely. The backdoor gets baked directly into the model weights during training. There is no way to remove it surgically. The only real fix is to throw the model away and train an entirely new one from scratch. This is not some theoretical attack. This is data poisoning at internet scale. Anyone can plant these documents right now on blogs, forums, academic sites or anywhere else that ends up in training scrapes. And some publisher rights groups have made sure this poison is in the wild. Do you hear me now? For years I have warned that training frontier AI on raw internet scraped data is a security and integrity disaster. I have advocated relentlessly for offline high protein human curated datasets. I have pushed for drawing training data from pristine pre 1970 archives. Books. Journals. Patents. Court records. Private libraries that have never touched the public web. I have advocated for this for 100s of reasons for decades. Now we are here. This study is not a surprise to me. It is the inevitable result of the broken training paradigm I have been calling out since the earliest days of modern large language models. We actually knew the foundational problems as far back as 1998. That was when adversarial data insertion was already understood as a basis to break any AI model. The techniques have gotten more sophisticated but the core vulnerability has always been the same. Train on unverified publicly editable oceans of data and you open the door to permanent compromise. Anthropic is ground zero for the doom burners camp. They claimed they would be focused on building powerful helpful models. Now with each new paper they seem determined to highlight just how fragile and attackable the current web scale approach really is. This study is another clear example. Here are some points from the study and why they completely vindicate the offline curated data path I have been championing. Minimal poison quantity works. Only 250 malicious documents roughly 420 thousand tokens or just 0.00016 percent of a large dataset are enough. One hundred is not reliable. Two hundred fifty succeeds consistently. Scale invariance. The number of poisoned samples needed stays almost constant whether you are training a 600 million parameter model or a 13 billion parameter model on anywhere from 6 billion to 260 billion tokens. Bigger models and bigger datasets do not make you safer. Stealth design. The poisoned documents look exactly like normal web content. No obvious red flags for crawlers or human reviewers. Permanence. The backdoor is permanently embedded in the model weights. Training is easy. Untraining is impossible. Full retraining from scratch is the only option. Trigger reliability. A simple hidden phrase activates the malicious behavior on demand. Gibberish output. Bias injection. Data leaks. Policy bypass. Whatever the attacker wants. Universal exposure. Every major model trained on public internet data including the GPT series Claude Gemini and others sits wide open to this exact vector today. Economic catastrophe. Retraining a frontier model costs hundreds of millions or even billions of dollars. One successful poisoning campaign could force entire companies to start over. Silent failure. The model performs normally until the trigger appears. No obvious signs of degradation until it is too late. No current defense. There is no reliable way to detect filter or mitigate this attack at true web scale. The attack surface is the entire internet. Paradigm failure. The study proves once and for all that more data plus more compute does not solve the poisoning problem. It actually makes the situation more dangerous because such a tiny poisoned signal can still dominate. My solutions have always been clear. Stop feeding these models the polluted firehose. Train exclusively on offline verified corpora. Use high signal high integrity sources from the 1870 to 1970 period or earlier. Sources that have never been digitized and have never touched the public web. These high protein datasets deliver far more real capability with none of the modern contamination bias or poisoning risks. I know where they are and how to digitize them. I just don’t have the money and therefore the time to do much about it other than complain here like chicken little. The training data is in public and private archives, cold storage. To train AI, digitize and protect non public historical knowledge under strict human curation. Keep everything air gapped and completely offline. No live web scraping. Ever. News insertion yes, but this is another article. Build local sovereign models that can run fully offline on personal hardware. Phones. Laptops. Local clusters. I have shown this repeatedly with models in the open source systems. No cloud. No subscription. No exposure. Put human in the loop curation at every single stage. Replace quantity with quality. Reward provenance and empirical distrust inside the loss function. Penalize coordinated institutional echo chambers and all the post 1995 narrative sludge. Hire the best humans not the cheapest to help train AI and pay them well. Keep them employed with a promise of job security. You will need them. Avoid retrieval augmented generation on untrusted sources. Any RAG system must pull exclusively from your own verified offline index. Never trust live web results without cryptographic provenance and heavy human vetting. Embed rules directly in the data itself like The Love Equation. Bake love, honesty, truth and empathy and first principles reasoning into the training corpus long before any alignment stage ever begins. The data layer is the real human loving layers. This is not a trick. This is the only path that produces capable trustworthy and truly secure AI. The 2025 study is the latest overwhelming proof that continuing with internet scale scraping is not just inefficient. It is actively dangerous. Primary sources: Anthropic Research Blog: https://www.anthropic.com/research/small-samples-poison Full Paper on arXiv: https://arxiv.org/abs/2510.07192 AISI Announcement: https://www.aisi.gov.uk/blog/examining-backdoor-data-poisoning-at-scale Alan Turing Institute Blog: https://www.turing.ac.uk/blog/llms-may-be-more-vulnerable-data-poisoning-we-thought The era of just scrape everything is over. The evidence is now undeniable. It is time to build AI the right way. Offline. Curated. Sovereign. And human first. The models of the future will be far better for it. ## 相关链接 - [Brian Roemmele](https://x.com/BrianRoemmele) - [@BrianRoemmele](https://x.com/BrianRoemmele) - [7.1K](https://x.com/BrianRoemmele/status/2046435862087635448/analytics) - [https://www.anthropic.com/research/small-samples-poison](https://www.anthropic.com/research/small-samples-poison) - [https://arxiv.org/abs/2510.07192](https://arxiv.org/abs/2510.07192) - [https://www.aisi.gov.uk/blog/examining-backdoor-data-poisoning-at-scale](https://www.aisi.gov.uk/blog/examining-backdoor-data-poisoning-at-scale) - [https://www.turing.ac.uk/blog/llms-may-be-more-vulnerable-data-poisoning-we-thought](https://www.turing.ac.uk/blog/llms-may-be-more-vulnerable-data-poisoning-we-thought) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [11:48 AM · Apr 21, 2026](https://x.com/BrianRoemmele/status/2046435862087635448) - [7,194 Views](https://x.com/BrianRoemmele/status/2046435862087635448/analytics) - [View quotes](https://x.com/BrianRoemmele/status/2046435862087635448/quotes) --- *导出时间: 2026/4/21 14:05:43* --- ## 中文翻译 # 250份文档的AI后门。你现在听到了吗？ **作者**：Brian Roemmele **日期**：2026-04-21T03:48:25.000Z **来源**：[https://x.com/BrianRoemmele/status/2046435862087635448](https://x.com/BrianRoemmele/status/2046435862087635448) --- ![Image](/doc/886/image/img_fe4300bed9b2.jpg) Anthropic、英国AI安全研究所与艾伦·图灵研究所的联合研究投下了一枚重磅炸弹。他们证明，只需在预训练数据中插入250份经过特殊构造的文档，就足以在从6亿参数一直到130亿参数的大型语言模型中创建一个永久性后门。无论模型有多大，也无论整体训练数据集多么庞大，这种方法都行之有效。这些被投毒的文档看起来完全正常。它们读起来就像普通的网页。但其中隐藏着一个触发短语。一旦模型之后遇到这个触发词，就可能被迫产生有害行为，例如输出乱码、泄露数据，或完全崩溃。后门在训练过程中被直接写入模型权重。没有办法通过外科手术式的方式将其移除。唯一真正的解决办法是丢弃该模型，从零开始训练一个全新的模型。这不是某种理论上的攻击。这是互联网规模的数据投毒。任何人现在就可以把这些文档植入博客、论坛、学术网站或任何最终会被抓取进训练数据的地方。而一些出版商权利组织已经确保这些毒物流散在外。你现在听到了吗？多年来我一直警告，用原始的互联网爬取数据训练前沿AI是一场安全与完整性灾难。我一直不懈地倡导使用离线的高“蛋白质”人工精选数据集。我一直推动从1970年以前的原始档案中提取训练数据。书籍。期刊。专利。法庭记录。从未接触过公共网络的私人图书馆。几十年来，我基于数百个理由倡导这一点。现在，我们到了这一步。这项研究对我而言并不意外。它是我自现代大型语言模型诞生之初就一直指出的那种崩坏的训练范式的必然结果。事实上，早在1998年我们就已经知道这些基础性问题。那时，对抗性数据插入已被理解为破坏任何AI模型的基础手段。这些技术变得更加复杂，但核心漏洞始终未变。在未经核实的、公众可编辑的数据海洋上训练，你就等于为永久性攻陷敞开了大门。 Anthropic是“末日论者”阵营的大本营。他们曾声称自己将专注于构建强大且有用的模型。如今，随着每一篇新论文的发表，他们似乎决心突出当前网络规模方法究竟是多么脆弱和易受攻击。这项研究是又一个明显的例证。以下是这项研究的一些要点，以及它们为何完全证实了我一直倡导的离线精选数据路线。极少的投毒量即可奏效。仅需250份恶意文档——大约42万个token，即大型数据集的0.00016%——就足够了。一百份并不可靠。二百五十份则稳定成功。规模不变性。无论你是在训练一个6亿参数的模型，还是在6B到260B token范围内训练一个130亿参数的模型，所需的投毒样本数量几乎保持恒定。更大的模型和更大的数据集并不会让你更安全。隐蔽设计。被投毒的文档看起来与正常的网络内容一模一样。对爬虫或人工审查者而言没有明显的危险信号。永久性。后门被永久嵌入模型权重中。训练很容易。去训练是不可能的。从零开始完全重新训练是唯一的选择。触发可靠性。一个简单的隐藏短语即可按需激活恶意行为。乱码输出。偏见注入。数据泄露。绕过策略。攻击者想要什么都行。普遍暴露。每一个基于公共互联网数据训练的主要模型，包括GPT系列、Claude、Gemini及其他模型，如今都完全暴露在这一攻击向量之下。经济灾难。重新训练一个前沿模型需要数亿甚至数十亿美元。一次成功的投毒行动就可能迫使整个公司从头再来。无声失效。在触发词出现之前，模型表现正常。没有明显的退化迹象，直到为时已晚。目前没有防御手段。没有可靠的方法能在真正的网络规模上检测、过滤或缓解这种攻击。攻击面就是整个互联网。范式失败。这项研究一劳永逸地证明，更多的数据加上更多的算力并不能解决投毒问题。实际上它使情况更加危险，因为如此微小的投毒信号仍然可以占据主导。我的解决方案一直很明确。停止给这些模型喂食污染的消防水管。完全在离线的、经过核实的语料库上训练。使用1870年至1970年期间或更早的高信号、高完整性来源。那些从未被数字化、从未接触过公共网络的来源。这些高“蛋白质”数据集能提供远为强大的真实能力，而没有现代污染、偏见或投毒风险。我知道它们在哪里以及如何将它们数字化。我只是没有钱，因此也没有时间做更多的事情，只能在这里像“小鸡快跑”里的那只鸡一样抱怨。训练数据存放在公共和私人档案馆中，处于冷存储状态。要在严格的人工监管下对非公开的历史知识进行数字化以训练AI并加以保护。让一切保持物理隔离并完全离线。绝不进行实时网络爬取。新闻内容的引入可以，但那是另一篇文章了。构建可以在个人硬件上完全离线运行的本地主权模型。手机。笔记本电脑。本地集群。我已经在开源系统中用模型反复展示了这一点。没有云端。没有订阅。没有暴露。在每一个阶段都引入人工把关的内容甄别。用质量取代数量。在损失函数中奖励可溯源性和实证性怀疑。惩罚协同的机构化回音室以及所有1995年之后的叙事污泥。雇佣最优秀的人而非最廉价的人来协助训练AI，并给予他们优厚的报酬。以工作保障的承诺让他们留任。你会需要他们的。避免在不可信来源上使用检索增强生成。任何RAG系统必须仅从你自己的、经过验证的离线索引中提取内容。绝不要在缺乏密码学溯源和严格人工审查的情况下信任实时网络结果。将规则直接嵌入数据本身，比如“爱的方程式”。在任何对齐阶段开始之前，就早早地将爱、诚实、真理、同理心和第一性原理推理融入训练语料库。数据层才是真正充满人类关爱的层。这不是花招。这是唯一能产出有能力、可信赖且真正安全的AI的道路。2025年的这项研究是最新且压倒性的证明：继续进行互联网规模的抓取不仅低效，而且主动制造危险。原始来源： Anthropic 研究博客：https://www.anthropic.com/research/small-samples-poison arXiv 上的完整论文：https://arxiv.org/abs/2510.07192 AISI 公告：https://www.aisi.gov.uk/blog/examining-backdoor-data-poisoning-at-scale 艾伦·图灵研究所博客：https://www.turing.ac.uk/blog/llms-may-be-more-vulnerable-data-poisoning-we-thought “什么都抓取”的时代已经结束了。证据如今已无可辩驳。是时候以正确的方式构建AI了。离线。经甄别。主权自主。并且以人为本。未来的模型将因此变得远为出色。 ## 相关链接 - [Brian Roemmele](https://x.com/BrianRoemmele) - [@BrianRoemmele](https://x.com/BrianRoemmele) - [7.1K](https://x.com/BrianRoemmele/status/2046435862087635448/analytics) - [https://www.anthropic.com/research/small-samples-poison](https://www.anthropic.com/research/small-samples-poison) - [https://arxiv.org/abs/2510.07192](https://arxiv.org/abs/2510.07192) - [https://www.aisi.gov.uk/blog/examining-backdoor-data-poisoning-at-scale](https://www.aisi.gov.uk/blog/examining-backdoor-data-poisoning-at-scale) - [https://www.turing.ac.uk/blog/llms-may-be-more-vulnerable-data-poisoning-we-thought](https://www.turing.ac.uk/blog/llms-may-be-more-vulnerable-data-poisoning-we-thought) - [升级至高级版](https://x.com/i/premium_sign_up) - [2026年4月21日上午11:48](https://x.com/BrianRoemmele/status/2046435862087635448) - [7,194 次浏览](https://x.com/BrianRoemmele/status/2046435862087635448/analytics) - [查看引用](https://x.com/BrianRoemmele/status/2046435862087635448/quotes) --- *导出时间: 2026/4/21 14:05:43*