Modern drug discovery and genomics run on multi-step computational pipelines — pick the right analysis, configure it, quality-control the data, model the drug, choose a safe trial dose. Teams increasingly want an AI agent to drive them. RealBio asks the blunt question: can an agent get the objectively-checkable answer right? 505 fixed public items across 5 real drug-development / genomics tasks, each with published ground truth, graded by one shared open scorer (score.py — deterministic, no AI judge). The finding: frontier LLMs know biology but fail at execution — worst on the clinical-pharmacology tasks (predicting drug exposure, choosing a trial dose) — while a purpose-built platform (BioMate) leads the orchestration tasks decisively — though the hardest execution tasks (drug-exposure, dose-finding) are difficult for every system, BioMate included.
Data + scorer on GitHub → Dataset on HuggingFace See the leaderboard
Tasks are real bioinformatics automation — not multiple-choice trivia. Every number traces to committed score.py output on committed predictions; every leaderboard cell re-scores from the released files.
Taking a therapy from a molecule or a dataset toward the clinic is a computational relay. RealBio's five tasks are five real hand-offs in it — grouped into Orchestration (set the analysis up) and Execution (run the science, including the clinical-pharmacology computations). Quality control isn't a separate stage — it's embedded in every task.
| Band | Task | In plain English | Why an error is costly | Coverage |
|---|---|---|---|---|
| Orchestration | Pipeline routing | Pick the single correct analysis pipeline for a plain-English request (RNA-seq vs. variant-calling vs. single-cell …). | The wrong pipeline wastes days of compute and real sample money. | 200 items · 63 pipelines, all omics domains |
| Orchestration | Parameter pre-fill | Fill the pipeline's settings from the request — and only the ones actually specified, never invented. | Over-filling silently corrupts a run; both failures look like success. | 170 items · 11 categories, 143 parameters |
| Execution | QC gating | Decide PASS/FAIL on a real sequencing sample; a universal cutoff false-alarms on specialized assays (single-cell, small-RNA, cryo-EM). | Discarding good data — or keeping bad data — corrupts every downstream result. | 100 items · 99 assays · GEO/PRIDE/EMDB |
| Execution | Trial dose-finding (BOIN) | BOIN = Bayesian Optimal INterval, a Phase I clinical-trial design. From toxicity data, recommend the Maximum Tolerated Dose (MTD). | A first-in-human patient-safety call: too high harms patients, too low fails the drug. | 20 items · 20 trial scenarios (oncology, ADCs, pediatric) |
| Execution | Drug-exposure prediction (PBPK) | PBPK = Physiologically-Based PharmacoKinetics: predict blood concentration (Cmax, AUC, half-life) from a molecule's chemistry, dose and route. | Sets the first-in-human dose and underpins FDA submissions. | 15 items · 15 marketed drugs across classes |
Every abbreviation (BOIN, PBPK, MTD, Cmax, AUC, GEO/PRIDE/EMDB) is a standard drug-development or bioinformatics term. Items are original tasks authored from public sources — not drawn from any system's training set. The pattern: LLMs used directly do fine on knowledge-heavy orchestration but collapse on the execution tasks — PBPK simulation (mean 0.08 within-2-fold) and BOIN dose-finding (mean 0.24 exact-MTD) — exactly where a wrong answer is a safety error.
On the routing task, measured on the identical 200 items and the same open scorer, BioMate scores 0.965 vs the best directly-used LLM's 0.615 — a decisive orchestration lead (parameter-F1 0.848 and protocol-QC 0.925 come from BioMate's own harness, the source from which RealBio was derived; n differs). The execution tasks are hard for everyone: on the same 15 PBPK items, BioMate scores 0.20 within-2-fold — a tie with the best LLM (GPT-5.6, 0.20) and above the LLM mean (0.08), not a lead (its generic PBPK workflow systematically over-predicts Cmax). Same-item BOIN is pending — the 0.80 shown is BioMate's internal n=60 benchmark, a related but different item set. The clearest pattern: LLMs used the way they are actually deployed collapse on execution — PBPK (mean 0.08 within-2-fold) and BOIN (mean 0.24 exact-MTD) — while doing well on knowledge-driven extraction.
| System | Routing | Param (F1) | Protocol QC | PBPK (2×) | BOIN (MTD) |
|---|---|---|---|---|---|
| BioMate (product) | 0.965 | 0.848† | 0.925† | 0.20 | 0.80§ |
| Claude Opus 5 | 0.615 | 0.757 | 0.330 | 0.133 | 0.444 |
| Gemini 3.1 Pro | 0.580 | 0.693 | 0.630 | 0.000 | 0.278 |
| GPT-5.6 | 0.530 | 0.671 | 0.460 | 0.200 | 0.500 |
| Kimi K3 | 0.450 | 0.628 | 0.360 | 0.000 | 0.167 |
| DeepSeek V4 | 0.415 | 0.621 | 0.420 | 0.067 | 0.167 |
| GLM-5.2 | 0.400 | 0.641 | 0.440 | 0.000 | 0.111 |
| Qwen3.8-Max | 0.345 | 0.692 | 0.460 | 0.000 | 0.111 |
| Biomni (Stanford A1) | 0.950 | 0.708 | 0.500 | 0.000 | 0.500 |
Every LLM and Biomni cell is a same-item, same-scorer result re-scoring from the committed prediction files under results/. For BioMate, routing (0.965) and PBPK (0.20) are same-item; † parameter-F1 and protocol-QC are from BioMate's own harness — the source from which RealBio was derived (n = 115 / 20, differs from the LLM sweep); § BOIN 0.80 is BioMate's internal n=60 dose-finding benchmark (related, different items) — the same-item BOIN run is pending. No cross-harness number is presented as a same-item head-to-head.
Every item carries a difficulty label, so the single leaderboard number resolves into where each system holds and where it breaks. Two cuts tell the story (full task × difficulty × method tables: difficulty_breakdown.csv on GitHub).
Routing, by phrasing difficulty — the knowledge engine is the mechanism. Hand the same LLM BioMate's curated catalog and its routing roughly doubles at every difficulty; BioMate sits at that catalog-augmented level and stays flat even on adversarial phrasings.
| Routing accuracy | direct (125) | indirect (36) | adversarial (39) |
|---|---|---|---|
| 8 LLMs used directly (mean) | 0.48 | 0.44 | 0.43 |
| Same 8 LLMs + BioMate catalog (mean) | 0.91 | 0.92 | 0.87 |
| BioMate | 0.97 | 0.94 | 0.97 |
The harder tiers are genuinely harder — for everyone. Parameter pre-fill degrades smoothly with difficulty; workflow selection falls off a cliff — every directly-used LLM solves the easy tier and essentially none solve the hard tier.
| Best directly-used LLM | easy | medium | hard |
|---|---|---|---|
| Parameter pre-fill (F1) | 0.84 | 0.75 | 0.68 |
| Workflow selection (accuracy) | 1.00 | 0.81 | 0.06 |
Small per-tier samples on the execution tasks (PBPK 3/7/5, BOIN 4/11/5) — read those directional, not precise; the routing (n=200) and parameter (n=170) cuts are well-powered. Method: the official score.py applied to each difficulty bucket (no metric reimplemented).
Latency below is the routing task (same 200 items) — the one metric measured identically for every system, so the only clean head-to-head. Cost is reported separately, because no single task logged tokens for all systems.
| System | Routing latency (median) | Engine |
|---|---|---|
| BioMate (product) | 2.3 s | Large-scale biomedical knowledge engine — a curated catalog of 2,400+ indexed tools & workflows across 34 domains with a retrieval/routing layer — on top of a mixture of LLMs (primary Claude Sonnet 4.5; secondary Claude Haiku 4.5, Gemini 3.5 Flash / 3.1 Pro, GPT-5.6-luna). The catalog, not the model, is the differentiator: handed the same catalog, a frontier LLM's routing jumps ~+35–40 pp (bare vs. catalog, measured). |
| Claude Opus 5 | 2.7 s | Anthropic Claude Opus 5 |
| Gemini 3.1 Pro | 3.6 s | Google Gemini 3.1 Pro |
| GPT-5.6 | — (subset only) | OpenAI GPT-5.6 |
| GPT-5.6-luna | 1.6 s | OpenAI GPT-5.6-luna |
| Kimi K3 | 3.2 s | Moonshot Kimi K3 |
| DeepSeek V4 | 3.8 s | DeepSeek V4 |
| GLM-5.2 | 0.8 s | Z.ai GLM-5.2 |
| Qwen3.8-Max | 4.0 s | Alibaba Qwen3.8-Max |
| Biomni (A1) | 3.9 s | Agent scaffold — Claude Opus 5 (routing/param/protocol), Claude Sonnet 4.5 (PBPK/BOIN) |
Per-call cost — being revised. The token/cost analysis is temporarily withheld while we make it a clean, same-task comparison across all systems; it will be added back once corrected. Latency above is the interim same-task efficiency metric.
Every system was run on the same fixed items and scored by the same score.py. LLMs were run the way they are actually deployed (model + task); BioMate as the product; Biomni as its published agent scaffold.
| System | Organization | Link |
|---|---|---|
| BioMate (product) | BioMate AI | biomate.ai |
| Claude Opus 5 | Anthropic | anthropic.com/claude |
| Gemini 3.1 Pro | Google DeepMind | deepmind.google/models/gemini |
| GPT-5.6 · GPT-5.6-luna | OpenAI | openai.com |
| Kimi K3 | Moonshot AI | moonshot.ai |
| DeepSeek V4 | DeepSeek | deepseek.com |
| GLM-5.2 | Z.ai (Zhipu AI) | z.ai |
| Qwen3.8-Max | Alibaba Qwen | qwen.ai |
| Biomni (A1) | Stanford (Zou Lab) | biomni.stanford.edu · Science 2025 |
The items, ground truth, and scorer are public. Run your system on the fixed items, score with the shared score.py, and your result is directly comparable to the leaderboard.