# When the LLM Can Run the Pipeline, Can You Trust the Biology?
**作者**: Kyle Tretina, Ph.D.
**日期**: 2026-04-17T17:33:54.000Z
**来源**: [https://x.com/AllThingsApx/status/2045194051553571017](https://x.com/AllThingsApx/status/2045194051553571017)
---
A recent Genentech/Roche preprint tries to measure something closer to real bioinformatics work: whether agentic systems can execute computational biology workflows end-to-end in a bare environment, not just talk about them.
They built 100 tasks spanning genomics, transcriptomics, epigenomics, single-cell analysis, human genetics, and a fair amount of the glue work that keeps pipelines from falling apart.
The key constraint is that every task has a uniquely scorable output so it can be autograded, either an exact match or something within tolerances. That is both the point and the catch. Autograding makes the comparison scalable and less subjective, but raises an important question:
> Is computational biology a determinable problem?
In my times as a bioinformatician, I've seen for a given problem that multiple pipelines can be defensible. Before orthogonal validation, “correct” often looks like “reasonable choices plus uncertainty and sensitivity checks,” not one canonical number.
Scorable Biology, by Construction
To enable scoring, the tasks rely on pragmatic constructions: synthetic or augmented data (for example, known contamination mixtures), scrubbed or scrambled metadata (find the swap), and the setup and execution of public tools that yield a specific quantitative endpoint.
If you have ever tried to benchmark “realistic” analysis, you will recognize the trade: the more lifelike the problem gets, the harder it is to grade without injecting human judgment.
Agents run in a minimal conda environment with web access. That is realistic in the sense that humans look things up, read docs, and google error messages. However, it muddies attribution.
Outcomes can depend on the live internet, changing documentation, package ecosystem drift, and whatever retrieval scaffolding the system uses, not just on the model’s own reasoning.
What the Numbers Actually Say
The top systems score high overall: Codex CLI (GPT-5.4) at about 83% and Claude Code (Opus 4.6) at about 81%, with smaller models dropping off. On the hardest subset, accuracy falls to roughly 59-69%. That is not shocking. The interesting bit is reliability across repeated runs. Codex solved about 92% of tasks at least once across three runs, but only about 73% across all three runs.
That gap matters operationally.
“Best-of-N eventually succeeds” is a very different claim than “one run is dependable,” and in actual workflows, I woudl argue that you usually care about the second one.
A pipeline that sometimes works is not a pipeline you schedule overnight and trust in the morning.
Operators vs Analysts
What is genuinely promising here is that the benchmark tests execution, not trivia: dependency wrangling, file formats, tool invocation, debugging, and landing on a verifiable artifact.
That is where a lot of bioinformatics time actually goes, and it is exactly the sort of work I suspect bioinformaticians (quietly?) hope to offload 🤣.
What it does not show, because it mostly cannot, is whether the agent made scientifically defensible choices, noticed violated assumptions, avoided silent failure modes, or produced robust and reproducible analyses rather than simply matching the grader, sometimes after a bit of trial and error. In other words, it is measuring competence as an operator of the computational stack more than judgment as an analyst of the biology (i.e. biological reasoning).
This looks like meaningful progress toward agents capable of running pipelines, but “can run the pipeline” is not yet “can be trusted to analyze the biology.”
I'm sure the next wave of progress will focus on advancing reliability, provenance, transparency about failure modes, robustness to shifting external resources, and whether agents can self-verify with sanity checks and alternative analyses rather than just hitting the scored endpoint.
Can't wait for that, but this is a good start.
## 相关链接
- [The Innovation | Medicine reposted](https://x.com/Innov_Medicine)
- [Kyle Tretina, Ph.D.](https://x.com/AllThingsApx)
- [@AllThingsApx](https://x.com/AllThingsApx)
- [7.6K](https://x.com/AllThingsApx/status/2045194051553571017/analytics)
- [A recent Genentech/Roche preprint](https://doi.org/10.64898/2026.04.06.716850)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:33 AM · Apr 18, 2026](https://x.com/AllThingsApx/status/2045194051553571017)
- [7,621 Views](https://x.com/AllThingsApx/status/2045194051553571017/analytics)
---
*导出时间: 2026/4/19 20:20:04*
---
## 中文翻译
# 当LLM能运行整个流程时，你还敢相信生物学结果吗？
**作者**: Kyle Tretina, Ph.D.
**日期**: 2026-04-17T17:33:54.000Z
**来源**: [https://x.com/AllThingsApx/status/2045194051553571017](https://x.com/AllThingsApx/status/2045194051553571017)
---
基因泰克/罗氏（Genentech/Roche）最近发布的一篇预印本，试图测量更接近真实生物信息学工作的内容：智能体系统能否在裸环境中端到端地执行计算生物学工作流，而不仅仅是纸上谈兵。
他们构建了100个任务，涵盖基因组学、转录组学、表观基因组学、单细胞分析、人类遗传学，以及大量防止流程崩溃的“粘合”工作。
关键约束在于，每个任务都有一个唯一可评分的输出，以便自动评分，要么是精确匹配，要么在容差范围内。这既是重点，也是陷阱。自动评分使比较可扩展且更少主观性，但也引出了一个重要问题：
> 计算生物学是一个可确定的问题吗？
在我做生物信息学家的那些年里，我见过同一个问题可以有多个合理的流程。在进行正交验证之前，“正确”往往看起来像是“合理的选择加上不确定性和敏感性检查”，而不是一个标准答案。
设计上就可评分的生物学
为了实现评分，这些任务依赖一些实用的构造：合成或增强的数据（例如已知污染混合物）、经过清洗或打乱的元数据（找出调换之处），以及公共工具的设置与执行，产生特定的定量终点。
如果你曾经尝试对“现实”分析进行基准测试，你会认出这个权衡：问题越贴近现实，就越难在不引入人工判断的情况下评分。
智能体在一个带有网络访问权限的最小conda环境中运行。这在人类会查资料、读文档、谷歌错误信息的意义上是现实的。然而，这让归因变得模糊。
结果可能取决于实时互联网、不断变化的文档、软件包生态系统的漂移，以及系统使用的任何检索辅助工具，而不仅仅是模型自身的推理。
数字实际上说明了什么
顶级系统整体得分很高：Codex CLI（GPT-5.4）约为83%，Claude Code（Opus 4.6）约为81%，较小模型则明显落后。在最难的子集上，准确率降至约59-69%。这并不令人意外。有趣的是多次重复运行中的可靠性。Codex在三次运行中至少有一次解决了约92%的任务，但在全部三次运行中都解决的只有约73%。
这一差距在运营上很重要。
“N次尝试中总有一次成功”与“单次运行可靠”是非常不同的说法，而在实际工作流程中，我认为你通常关心的是后者。
一个时灵时不灵的流程，不是你敢安排在夜里跑、早上放心相信的流程。
操作员 vs 分析师
这项基准测试真正令人振奋的地方在于，它测试的是执行能力，而不是琐碎知识：依赖管理、文件格式、工具调用、调试，以及产出可验证的结果。
生物信息学的大量时间正是花在这些地方，而这恰恰是我怀疑生物信息学家们（偷偷地？）希望卸载的那类工作 🤣。
它没有展示的——而且基本上也无法展示的——是智能体是否做出了科学上站得住脚的选择，是否注意到了被违反的假设，是否避免了静默的失败模式，是否产出了稳健且可复现的分析，而不是仅仅匹配评分标准——有时还经过一番试错。换句话说，它衡量的更多是作为计算栈操作员的能力，而非作为生物学分析师的判断力（即生物学推理）。
这看起来是朝着能够运行流程的智能体迈出的有意义的一步，但“能运行流程”还不等于“能被信任去分析生物学”。
我相信下一波进展将聚焦于提升可靠性、来源可溯性、对失败模式的透明度、对外部资源变化的鲁棒性，以及智能体能否通过合理性检查和替代分析进行自我验证，而不仅仅是命中评分终点。
迫不及待想看到那一天，但这已经是一个好的开端。
## 相关链接
- [The Innovation | Medicine 转发](https://x.com/Innov_Medicine)
- [Kyle Tretina, Ph.D.](https://x.com/AllThingsApx)
- [@AllThingsApx](https://x.com/AllThingsApx)
- [7.6K](https://x.com/AllThingsApx/status/2045194051553571017/analytics)
- [基因泰克/罗氏最近的一篇预印本](https://doi.org/10.64898/2026.04.06.716850)
- [升级至高级版](https://x.com/i/premium_sign_up)
- [2026年4月18日 凌晨1:33](https://x.com/AllThingsApx/status/2045194051553571017)
- [7,621 次浏览](https://x.com/AllThingsApx/status/2045194051553571017/analytics)
---
*导出时间: 2026/4/19 20:20:04*
