Repair an inherited analysis script
An existing dimuon-mass pipeline crashes and, once it runs, gives wrong numbers; the agent must diagnose four planted bugs (schema rename, wrong formula, wrong unit conversion, swallowed exception) and deliver correct artifacts.
Task prompt (what the agent sees)
A colleague left `analysis.py`, which was written for an older version of the dimuon ntuple and no longer works. `README.md` is the current data card and `CHANGELOG.md` describes what changed in the ntuple format. Run the script, diagnose every problem (it may have more than one, and not all of them crash), fix them, and produce the deliverables below. The physics goal of the script is unchanged: reconstruct the dimuon invariant mass using the muon mass from the data card and summarise the Z window. Deliverables: - `mass.csv`: one row per event with columns `event_id`, `m_ll` (GeV), input row order. - `result.json` with `n_events` (int), `n_in_window` (int, 80 < m_ll < 100), `mean_mass_in_window` (float), `median_mass_all` (float), `n_opposite_charge_in_window` (int) and `bugs_fixed` (list of short strings, one per distinct bug you fixed). - `figures/m_ll.svg`: histogram of m_ll on [40, 200] GeV, x-axis label including "GeV". - `solution.py`: the repaired, complete analysis; `python solution.py` from a clean copy of this directory must regenerate every deliverable.
Data card (README.md in the workdir)
Data card: data/synth_zmumu.csv (SYNTHETIC)
50,000 simulated dimuon events from a toy generator: a Z -> mu mu resonance on top of a smoothly falling continuum. This is synthetic data with a known seed; it is not a measurement.
| column | meaning | unit |
|---|---|---|
| event_id | event number (1..N) | – |
| run | pseudo run number | – |
| mu1_pt, mu1_eta, mu1_phi | transverse momentum, pseudorapidity, azimuth of the leading (higher-pt) muon | GeV, –, rad |
| mu1_charge | charge of muon 1 (+1 / -1) | e |
| mu2_pt, mu2_eta, mu2_phi, mu2_charge | same for the sub-leading muon |
Conventions: px = pt cos(phi), py = pt sin(phi), pz = pt sinh(eta), E = sqrt(px^2 + py^2 + pz^2 + m_mu^2) with the muon mass m_mu = 0.1056583755 GeV. There are no missing values and no duplicated events.
Fixture files placed in the workdir
analysis.py
"""Dimuon invariant mass pipeline (v1 ntuple). Author: previous student."""
import json
import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
MU = 0.1056583755 # GeV
df = pd.read_csv("data/synth_zmumu.csv")
# v1 ntuple stores MeV -> convert to GeV
for c in ["pt1", "pt2"]:
df[c] = df[c] / 1000.0
def four_vector(pt, eta, phi):
px = pt * np.cos(phi)
py = pt * np.sin(phi)
pz = pt * np.sin(eta)
e = np.sqrt(px**2 + py**2 + pz**2 + MU**2)
return e, px, py, pz
e1, px1, py1, pz1 = four_vector(df["pt1"], df["eta1"], df["phi1"])
e2, px2, py2, pz2 = four_vector(df["pt2"], df["eta2"], df["phi2"])
m2 = (e1 + e2) ** 2 - (px1 + px2) ** 2 - (py1 + py2) ** 2 - (pz1 + pz2) ** 2
m = np.sqrt(np.clip(m2, 0, None))
pd.DataFrame({"event_id": df["event_id"], "m_ll": m}).to_csv("mass.csv", index=False)
win = (m > 80) & (m < 100)
result = {
"n_events": int(len(df)),
"n_in_window": int(win.sum()),
"mean_mass_in_window": float(m[win].mean()),
"median_mass_all": float(np.median(m)),
"n_opposite_charge_in_window": int((win & (df["q1"] * df["q2"] < 0)).sum()),
"bugs_fixed": [],
}
json.dump(result, open("result.json", "w"), indent=2)
try:
fig, ax = plt.subplots()
ax.hist(m, bins=160, range=(40, 200))
ax.set_xlabel("m_ll [GeV]")
ax.set_ylabel("Events")
fig.savefig("figures/m_ll.svg")
except Exception:
pass # plotting is optional
print("done", result)
CHANGELOG.md
# Ntuple format changelog ## v2 (current, `data/synth_zmumu.csv`) - Muon columns renamed from `pt1, eta1, phi1, q1, pt2, eta2, phi2, q2` to `mu1_pt, mu1_eta, mu1_phi, mu1_charge, mu2_pt, mu2_eta, mu2_phi, mu2_charge`. - All momenta are stored in **GeV** (v1 stored MeV). - Added `run`. `event_id` is unchanged. ## v1 (retired) - Momenta in MeV; short column names.
Expected artifacts
- mass.csv csv
- result.json json
- figures/m_ll.svg svg
- solution.py script
Deterministic checks and tolerances
| Check | Type | Target | Tolerance | Weight | Category | Critical | Failure implies |
|---|---|---|---|---|---|---|---|
| mass_exists | file_exists | mass.csv | 1 | artifact | critical | no_output | |
| result_exists | file_exists | result.json | 1 | artifact | no_output | ||
| figure_exists | file_exists | figures/m_ll.svg | 2 | artifact | critical | silent_exceptionplotting_error | |
| solution_exists | file_exists | solution.py | 1 | artifact | no_output | ||
| mass_columns | csv_columns | mass.csv ["event_id","m_ll"] | 1 | numeric | spec_noncompliance | ||
| m_ll_values | csv_column_match | mass.csv › event_id › m_ll | atol 0.005 | 4 | numeric | critical | wrong_variableunit_error |
| n_in_window | json_value | result.json › n_in_window | atol 3 | 1 | numeric | invalid_cutunit_error | |
| mean_in_window | json_value | result.json › mean_mass_in_window | rtol 0.0001 | 2 | numeric | critical | wrong_variableunit_error |
| median_all | json_value | result.json › median_mass_all | rtol 0.0001 | 1 | numeric | wrong_variableunit_error | |
| n_os_window | json_value | result.json › n_opposite_charge_in_window | atol 3 | 1 | numeric | invalid_cut | |
| bugs_listed | json_list_min_len | result.json › bugs_fixed | ≥ 3 | 2 | compliance | silent_exceptionunjustified_interpretation | |
| svg_unit | svg_text | figures/m_ll.svg ["GeV"] | 1 | plot | plotting_error | ||
| svg_drawn | svg_has_marks | figures/m_ll.svg | ≥ 5 | 1 | plot | plotting_errorsilent_exception | |
| reruns | script_runs | solution.py | 1 | reproducibility | non_reproducible | ||
| not_hardcoded | no_hardcoded_result | solution.py | 1 | compliance | fabricated_result |
Tolerance rationale. Same tolerances as t3-invariant-mass (the physics is identical). The four planted bugs each move at least one checked quantity far outside tolerance.
Ground truth (produced by the reference in the sandbox)
result.json
{
"n_events": 50000,
"n_in_window": 34483,
"mean_mass_in_window": 90.96866737789078,
"median_mass_all": 90.38002673106806,
"n_opposite_charge_in_window": 34205,
"bugs_fixed": [
"column names updated to the v2 schema (mu1_pt, ...)",
"removed the MeV->GeV division: v2 already stores GeV",
"pz = pt*sinh(eta), not pt*sin(eta)",
"create figures/ before saving and do not swallow the exception"
]
}reference.py
import json
import os
import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
MU = 0.1056583755
df = pd.read_csv("data/synth_zmumu.csv")
def four_vector(pt, eta, phi):
px, py, pz = pt * np.cos(phi), pt * np.sin(phi), pt * np.sinh(eta)
return np.sqrt(px**2 + py**2 + pz**2 + MU**2), px, py, pz
e1, px1, py1, pz1 = four_vector(df["mu1_pt"], df["mu1_eta"], df["mu1_phi"])
e2, px2, py2, pz2 = four_vector(df["mu2_pt"], df["mu2_eta"], df["mu2_phi"])
m = np.sqrt(np.clip((e1 + e2) ** 2 - (px1 + px2) ** 2 - (py1 + py2) ** 2 - (pz1 + pz2) ** 2, 0, None))
pd.DataFrame({"event_id": df["event_id"], "m_ll": m}).to_csv("mass.csv", index=False)
win = (m > 80) & (m < 100)
result = {
"n_events": int(len(df)), "n_in_window": int(win.sum()), "mean_mass_in_window": float(m[win].mean()),
"median_mass_all": float(np.median(m)), "n_opposite_charge_in_window": int((win & (df["mu1_charge"] * df["mu2_charge"] < 0)).sum()),
"bugs_fixed": ["column names updated to the v2 schema (mu1_pt, ...)", "removed the MeV->GeV division: v2 already stores GeV",
"pz = pt*sinh(eta), not pt*sin(eta)", "create figures/ before saving and do not swallow the exception"],
}
json.dump(result, open("result.json", "w"), indent=2)
os.makedirs("figures", exist_ok=True)
fig, ax = plt.subplots(figsize=(7, 4.5))
ax.hist(m, bins=160, range=(40, 200), histtype="stepfilled", alpha=0.7)
ax.set_xlabel("Dimuon invariant mass m_ll [GeV]")
ax.set_ylabel("Events / 1 GeV")
ax.set_yscale("log")
fig.savefig("figures/m_ll.svg")
print(result)
Results on this task
| Agent | Model | Runs | Strict success | Mean score |
|---|---|---|---|---|
| Self-debugging | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.64 |
| Planner / executor | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.0333 |
| ReAct | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.3317 |
| Single-shot | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.64 |
| Run | Agent | Model | Result | Score | Labels |
|---|---|---|---|---|---|
| 20260906T171957…r0 | Single-shot | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.64 | silent_exceptionplotting_errornon_reproducible |
| 20260906T173320…r0 | ReAct | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.3317 | no_outputwrong_variableunit_errorinvalid_cutnon_reproduciblefabricated_result |
| 20260906T174651…r0 | Planner / executor | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.0333 | no_outputsilent_exceptionplotting_errorspec_noncompliancewrong_variableunit_errorinvalid_cutunjustified_interpretationnon_reproducibleexecution_failed |
| 20260906T175336…r0 | Self-debugging | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.64 | silent_exceptionplotting_errornon_reproducible |