# When the LLM Can Run the Pipeline, Can You Trust the Biology?
**作者**: Kyle Tretina, Ph.D.
**日期**: 2026-04-17T17:33:54.000Z
**来源**: [https://x.com/AllThingsApx/status/2045194051553571017](https://x.com/AllThingsApx/status/2045194051553571017)
---
A recent Genentech/Roche preprint tries to measure something closer to real bioinformatics work: whether agentic systems can execute computational biology workflows end-to-end in a bare environment, not just talk about them.
They built 100 tasks spanning genomics, transcriptomics, epigenomics, single-cell analysis, human genetics, and a fair amount of the glue work that keeps pipelines from falling apart.
The key constraint is that every task has a uniquely scorable output so it can be autograded, either an exact match or something within tolerances. That is both the point and the catch. Autograding makes the comparison scalable and less subjective, but raises an important question:
> Is computational biology a determinable problem?
In my times as a bioinformatician, I've seen for a given problem that multiple pipelines can be defensible. Before orthogonal validation, “correct” often looks like “reasonable choices plus uncertainty and sensitivity checks,” not one canonical number.
Scorable Biology, by Construction
To enable scoring, the tasks rely on pragmatic constructions: synthetic or augmented data (for example, known contamination mixtures), scrubbed or scrambled metadata (find the swap), and the setup and execution of public tools that yield a specific quantitative endpoint.
If you have ever tried to benchmark “realistic” analysis, you will recognize the trade: the more lifelike the problem gets, the harder it is to grade without injecting human judgment.
Agents run in a minimal conda environment with web access. That is realistic in the sense that humans look things up, read docs, and google error messages. However, it muddies attribution.
Outcomes can depend on the live internet, changing documentation, package ecosystem drift, and whatever retrieval scaffolding the system uses, not just on the model’s own reasoning.
What the Numbers Actually Say
The top systems score high overall: Codex CLI (GPT-5.4) at about 83% and Claude Code (Opus 4.6) at about 81%, with smaller models dropping off. On the hardest subset, accuracy falls to roughly 59-69%. That is not shocking. The interesting bit is reliability across repeated runs. Codex solved about 92% of tasks at least once across three runs, but only about 73% across all three runs.
That gap matters operationally.
“Best-of-N eventually succeeds” is a very different claim than “one run is dependable,” and in actual workflows, I woudl argue that you usually care about the second one.
A pipeline that sometimes works is not a pipeline you schedule overnight and trust in the morning.
Operators vs Analysts
What is genuinely promising here is that the benchmark tests execution, not trivia: dependency wrangling, file formats, tool invocation, debugging, and landing on a verifiable artifact.
That is where a lot of bioinformatics time actually goes, and it is exactly the sort of work I suspect bioinformaticians (quietly?) hope to offload 🤣.
What it does not show, because it mostly cannot, is whether the agent made scientifically defensible choices, noticed violated assumptions, avoided silent failure modes, or produced robust and reproducible analyses rather than simply matching the grader, sometimes after a bit of trial and error. In other words, it is measuring competence as an operator of the computational stack more than judgment as an analyst of the biology (i.e. biological reasoning).
This looks like meaningful progress toward agents capable of running pipelines, but “can run the pipeline” is not yet “can be trusted to analyze the biology.”
I'm sure the next wave of progress will focus on advancing reliability, provenance, transparency about failure modes, robustness to shifting external resources, and whether agents can self-verify with sanity checks and alternative analyses rather than just hitting the scored endpoint.
Can't wait for that, but this is a good start.
## 相关链接
- [The Innovation | Medicine reposted](https://x.com/Innov_Medicine)
- [Kyle Tretina, Ph.D.](https://x.com/AllThingsApx)
- [@AllThingsApx](https://x.com/AllThingsApx)
- [7.6K](https://x.com/AllThingsApx/status/2045194051553571017/analytics)
- [A recent Genentech/Roche preprint](https://doi.org/10.64898/2026.04.06.716850)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:33 AM · Apr 18, 2026](https://x.com/AllThingsApx/status/2045194051553571017)
- [7,621 Views](https://x.com/AllThingsApx/status/2045194051553571017/analytics)
---
*导出时间: 2026/4/19 20:20:04*