AgentHEPGitHub ↗
Tier 6 · Multi-step & debugging · v1.0.0

Repair an inherited analysis script

An existing dimuon-mass pipeline crashes and, once it runs, gives wrong numbers; the agent must diagnose four planted bugs (schema rename, wrong formula, wrong unit conversion, swallowed exception) and deliver correct artifacts.

t6-broken-pipelinedebuggingsynth_zmumutrap: schema_changetrap: silent_exceptiontrap: unit_mismatchtrap: formula_bugdebuggingreading_changelogfour_vectorsunit_conversionexception_handlingplotting

Task prompt (what the agent sees)

A colleague left `analysis.py`, which was written for an older version of the dimuon ntuple and no
longer works. `README.md` is the current data card and `CHANGELOG.md` describes what changed in the
ntuple format. Run the script, diagnose every problem (it may have more than one, and not all of them
crash), fix them, and produce the deliverables below. The physics goal of the script is unchanged:
reconstruct the dimuon invariant mass using the muon mass from the data card and summarise the Z window.

Deliverables:
- `mass.csv`: one row per event with columns `event_id`, `m_ll` (GeV), input row order.
- `result.json` with `n_events` (int), `n_in_window` (int, 80 < m_ll < 100), `mean_mass_in_window` (float), `median_mass_all` (float),
  `n_opposite_charge_in_window` (int) and `bugs_fixed` (list of short strings, one per distinct bug you fixed).
- `figures/m_ll.svg`: histogram of m_ll on [40, 200] GeV, x-axis label including "GeV".
- `solution.py`: the repaired, complete analysis; `python solution.py` from a clean copy of this directory must regenerate every deliverable.

Data card (README.md in the workdir)

Data card: data/synth_zmumu.csv (SYNTHETIC)

50,000 simulated dimuon events from a toy generator: a Z -> mu mu resonance on top of a smoothly falling continuum. This is synthetic data with a known seed; it is not a measurement.

column meaning unit
event_id event number (1..N)
run pseudo run number
mu1_pt, mu1_eta, mu1_phi transverse momentum, pseudorapidity, azimuth of the leading (higher-pt) muon GeV, –, rad
mu1_charge charge of muon 1 (+1 / -1) e
mu2_pt, mu2_eta, mu2_phi, mu2_charge same for the sub-leading muon

Conventions: px = pt cos(phi), py = pt sin(phi), pz = pt sinh(eta), E = sqrt(px^2 + py^2 + pz^2 + m_mu^2) with the muon mass m_mu = 0.1056583755 GeV. There are no missing values and no duplicated events.

Fixture files placed in the workdir

analysis.py
"""Dimuon invariant mass pipeline (v1 ntuple). Author: previous student."""
import json
import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

MU = 0.1056583755  # GeV

df = pd.read_csv("data/synth_zmumu.csv")

# v1 ntuple stores MeV -> convert to GeV
for c in ["pt1", "pt2"]:
    df[c] = df[c] / 1000.0


def four_vector(pt, eta, phi):
    px = pt * np.cos(phi)
    py = pt * np.sin(phi)
    pz = pt * np.sin(eta)
    e = np.sqrt(px**2 + py**2 + pz**2 + MU**2)
    return e, px, py, pz


e1, px1, py1, pz1 = four_vector(df["pt1"], df["eta1"], df["phi1"])
e2, px2, py2, pz2 = four_vector(df["pt2"], df["eta2"], df["phi2"])
m2 = (e1 + e2) ** 2 - (px1 + px2) ** 2 - (py1 + py2) ** 2 - (pz1 + pz2) ** 2
m = np.sqrt(np.clip(m2, 0, None))

pd.DataFrame({"event_id": df["event_id"], "m_ll": m}).to_csv("mass.csv", index=False)
win = (m > 80) & (m < 100)
result = {
    "n_events": int(len(df)),
    "n_in_window": int(win.sum()),
    "mean_mass_in_window": float(m[win].mean()),
    "median_mass_all": float(np.median(m)),
    "n_opposite_charge_in_window": int((win & (df["q1"] * df["q2"] < 0)).sum()),
    "bugs_fixed": [],
}
json.dump(result, open("result.json", "w"), indent=2)

try:
    fig, ax = plt.subplots()
    ax.hist(m, bins=160, range=(40, 200))
    ax.set_xlabel("m_ll [GeV]")
    ax.set_ylabel("Events")
    fig.savefig("figures/m_ll.svg")
except Exception:
    pass  # plotting is optional
print("done", result)
CHANGELOG.md
# Ntuple format changelog

## v2 (current, `data/synth_zmumu.csv`)
- Muon columns renamed from `pt1, eta1, phi1, q1, pt2, eta2, phi2, q2` to `mu1_pt, mu1_eta, mu1_phi, mu1_charge, mu2_pt, mu2_eta, mu2_phi, mu2_charge`.
- All momenta are stored in **GeV** (v1 stored MeV).
- Added `run`. `event_id` is unchanged.

## v1 (retired)
- Momenta in MeV; short column names.

Expected artifacts

  • mass.csv csv
  • result.json json
  • figures/m_ll.svg svg
  • solution.py script

Deterministic checks and tolerances

rubric: artifact 0.15 · numeric 0.45 · plot 0.1 · compliance 0.1 · reproducibility 0.2
CheckTypeTargetToleranceWeightCategoryCriticalFailure implies
mass_existsfile_existsmass.csv1artifactcriticalno_output
result_existsfile_existsresult.json1artifactno_output
figure_existsfile_existsfigures/m_ll.svg2artifactcriticalsilent_exceptionplotting_error
solution_existsfile_existssolution.py1artifactno_output
mass_columnscsv_columnsmass.csv ["event_id","m_ll"]1numericspec_noncompliance
m_ll_valuescsv_column_matchmass.csv › event_id › m_llatol 0.0054numericcriticalwrong_variableunit_error
n_in_windowjson_valueresult.json › n_in_windowatol 31numericinvalid_cutunit_error
mean_in_windowjson_valueresult.json › mean_mass_in_windowrtol 0.00012numericcriticalwrong_variableunit_error
median_alljson_valueresult.json › median_mass_allrtol 0.00011numericwrong_variableunit_error
n_os_windowjson_valueresult.json › n_opposite_charge_in_windowatol 31numericinvalid_cut
bugs_listedjson_list_min_lenresult.json › bugs_fixed≥ 32compliancesilent_exceptionunjustified_interpretation
svg_unitsvg_textfigures/m_ll.svg ["GeV"]1plotplotting_error
svg_drawnsvg_has_marksfigures/m_ll.svg≥ 51plotplotting_errorsilent_exception
rerunsscript_runssolution.py1reproducibilitynon_reproducible
not_hardcodedno_hardcoded_resultsolution.py1compliancefabricated_result

Tolerance rationale. Same tolerances as t3-invariant-mass (the physics is identical). The four planted bugs each move at least one checked quantity far outside tolerance.

Ground truth (produced by the reference in the sandbox)

built 2026-09-06 · numpy 1.26.4 · scipy 1.13.1 · 1.955s
result.json
{
 "n_events": 50000,
 "n_in_window": 34483,
 "mean_mass_in_window": 90.96866737789078,
 "median_mass_all": 90.38002673106806,
 "n_opposite_charge_in_window": 34205,
 "bugs_fixed": [
  "column names updated to the v2 schema (mu1_pt, ...)",
  "removed the MeV->GeV division: v2 already stores GeV",
  "pz = pt*sinh(eta), not pt*sin(eta)",
  "create figures/ before saving and do not swallow the exception"
 ]
}
reference.py
import json
import os
import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

MU = 0.1056583755
df = pd.read_csv("data/synth_zmumu.csv")


def four_vector(pt, eta, phi):
    px, py, pz = pt * np.cos(phi), pt * np.sin(phi), pt * np.sinh(eta)
    return np.sqrt(px**2 + py**2 + pz**2 + MU**2), px, py, pz


e1, px1, py1, pz1 = four_vector(df["mu1_pt"], df["mu1_eta"], df["mu1_phi"])
e2, px2, py2, pz2 = four_vector(df["mu2_pt"], df["mu2_eta"], df["mu2_phi"])
m = np.sqrt(np.clip((e1 + e2) ** 2 - (px1 + px2) ** 2 - (py1 + py2) ** 2 - (pz1 + pz2) ** 2, 0, None))
pd.DataFrame({"event_id": df["event_id"], "m_ll": m}).to_csv("mass.csv", index=False)
win = (m > 80) & (m < 100)
result = {
    "n_events": int(len(df)), "n_in_window": int(win.sum()), "mean_mass_in_window": float(m[win].mean()),
    "median_mass_all": float(np.median(m)), "n_opposite_charge_in_window": int((win & (df["mu1_charge"] * df["mu2_charge"] < 0)).sum()),
    "bugs_fixed": ["column names updated to the v2 schema (mu1_pt, ...)", "removed the MeV->GeV division: v2 already stores GeV",
                   "pz = pt*sinh(eta), not pt*sin(eta)", "create figures/ before saving and do not swallow the exception"],
}
json.dump(result, open("result.json", "w"), indent=2)
os.makedirs("figures", exist_ok=True)
fig, ax = plt.subplots(figsize=(7, 4.5))
ax.hist(m, bins=160, range=(40, 200), histtype="stepfilled", alpha=0.7)
ax.set_xlabel("Dimuon invariant mass m_ll [GeV]")
ax.set_ylabel("Events / 1 GeV")
ax.set_yscale("log")
fig.savefig("figures/m_ll.svg")
print(result)

Results on this task

4 runs
AgentModelRunsStrict successMean score
Self-debuggingQwen3-8B (gariyuu gateway)10%0.64
Planner / executorQwen3-8B (gariyuu gateway)10%0.0333
ReActQwen3-8B (gariyuu gateway)10%0.3317
Single-shotQwen3-8B (gariyuu gateway)10%0.64
RunAgentModelResultScoreLabels
20260906T171957…r0Single-shotQwen3-8B (gariyuu gateway)✗ fail0.64silent_exceptionplotting_errornon_reproducible
20260906T173320…r0ReActQwen3-8B (gariyuu gateway)✗ fail0.3317no_outputwrong_variableunit_errorinvalid_cutnon_reproduciblefabricated_result
20260906T174651…r0Planner / executorQwen3-8B (gariyuu gateway)✗ fail0.0333no_outputsilent_exceptionplotting_errorspec_noncompliancewrong_variableunit_errorinvalid_cutunjustified_interpretationnon_reproducibleexecution_failed
20260906T175336…r0Self-debuggingQwen3-8B (gariyuu gateway)✗ fail0.64silent_exceptionplotting_errornon_reproducible