Benchmark v1.0.0 · harness v0.1.0

Can LLM agents run a reproducible scientific analysis?

AgentHEP hands an agent a data file, a data card and an analysis request, executes whatever code it writes inside a sandbox, and grades the artifacts deterministically against a reference implementation. High-energy physics supplies the tasks; the failure modes are general.

Tasks

6 adversarial · 6 tiers

Graded runs

4 agent × model configurations

Best strict success

29%

self_debug · Qwen3-8B (gariyuu gateway)

Cost of best config

$0.034

14 episodes

What the benchmark asks

Inspect, select, derive

Load a real CMS open-data file or a seeded synthetic table, read its data card, build histograms, selections and derived kinematics.

Compare, fit, test

Rank observables by separation, normalise weighted yields, fit a bump, quantify an excess with an exact Poisson p-value.

Debug and reproduce

Repair an inherited script with four planted bugs; every task's solution.py is re-executed in a clean workdir and must regenerate the submitted numbers.

Strict success by tier

56 runs

Fraction of episodes passing every deterministic check, by agent architecture

Single-shotReActPlanner / executorSelf-debugging

Dominant failure modes

Labels are assigned by the deterministic graders and trajectory signals; no model grades another model.

How it is built

Three-layer sandbox. Static code policy, in-process guard, and an OS sandbox (macOS seatbelt or a Docker/bubblewrap container) with no network and a workdir-only filesystem.
Deterministic grading. Numeric tolerances, exact counts, SVG label parsing, hard-coded-literal detection, clean-room reruns.
Preserved run metadata. Exact model strings as requested and as reported, dates, prices, token counts, sandbox limits, git SHA.
Versioned tasks. Answers never change under a fixed version; held-out procedural variants regenerate every synthetic dataset from a new seed.

Methodology →Paper →Task suite →