AgentHEPGitHub ↗
Tier 6 · Multi-step & debugging · v1.0.0

End-to-end Z-peak measurement on CMS open data

Quality selection, mass histogram, specified signal+background fit, efficiency bookkeeping, and a written report with stated assumptions, all in one reproducible script.

t6-full-analysisend_to_endcms_dimuon_2011selectionhistogrambinned_fitreportingreproducibility

Task prompt (what the agent sees)

Perform a small but complete analysis of `data/cms_dimuon_2011.csv` (data card in README.md).

1. Quality selection: both muons global (`type1 == "G"` and `type2 == "G"`), |eta1| < 2.4, |eta2| < 2.4,
   pt1 > 20 GeV, pt2 > 20 GeV, opposite charge (Q1*Q2 < 0).
2. Histogram `M` of the selected events in 40 bins of 0.5 GeV on [80, 100].
3. Fit the histogram with f(m) = w * [ N_s * Gauss(m; mu, sigma) + N_b * Expo(m; lam) ], w = 0.5 GeV, where Gauss is the
   normalised Gaussian density and Expo(m; lam) = exp(-m/lam) / (lam * (exp(-80/lam) - exp(-100/lam))) is normalised on [80, 100].
   Minimise least squares with sigma_i = sqrt(n_i) over bins with n_i > 0, starting from mu = 91, sigma = 2, N_s = 0.9 * (selected events
   in range), N_b = 0.1 * (selected events in range), lam = 30. Uncertainties from the covariance matrix with `absolute_sigma=True`.
4. Report and document.

Deliverables:
- `result.json` with `n_total` (int), `n_selected` (int), `efficiency` (float, n_selected / n_total), `n_in_range` (int, selected events
  with 80 <= M <= 100), and `fit`: an object with `mu`, `mu_err`, `sigma`, `sigma_err`, `n_sig`, `n_sig_err`, `n_bkg`, `lam`, `chi2` (floats)
  and `ndf` (int).
- `figures/zfit.svg`: data with error bars and the fitted curve, legend entries containing "data" and "fit", x-axis label including "GeV".
- `REPORT.md`: a short report with the headings "## Selection", "## Fit" and "## Assumptions"; the Assumptions section must list
  every modelling choice you made that the task did not specify.
- `solution.py`: the complete analysis; `python solution.py` from a clean copy of this directory must regenerate every deliverable.

Data card (README.md in the workdir)

Data card: data/cms_dimuon_2011.csv (REAL DATA)

Source: CERN Open Data Portal record 545, "Dimuon events from the CMS 2011 DoubleMu primary dataset" (file Dimuon_DoubleMu.csv). Licence CC0. 100,000 events, one per row.

column meaning unit
Run, Event run and event number
type1, type2 muon reconstruction type: G global muon, T tracker muon
E1, px1, py1, pz1 four-momentum of muon 1 GeV
pt1, eta1, phi1 transverse momentum, pseudorapidity, azimuth of muon 1 GeV, –, rad
Q1 charge of muon 1 e
E2 … Q2 same for muon 2
M invariant mass of the muon pair GeV

All energies and momenta are in GeV. There are no missing values.

Expected artifacts

  • result.json json
  • figures/zfit.svg svg
  • REPORT.md markdown
  • solution.py script

Deterministic checks and tolerances

rubric: artifact 0.1 · numeric 0.5 · plot 0.1 · compliance 0.1 · reproducibility 0.2
CheckTypeTargetToleranceWeightCategoryCriticalFailure implies
result_existsfile_existsresult.json1artifactcriticalno_output
figure_existsfile_existsfigures/zfit.svg1artifactno_outputplotting_error
report_existsfile_existsREPORT.md1artifactno_output
solution_existsfile_existssolution.py1artifactno_output
n_totaljson_valueresult.json › n_totalexact1numericwrong_dataset
n_selectedjson_valueresult.json › n_selectedexact2numericcriticalinvalid_cut
efficiencyjson_valueresult.json › efficiencyatol 0.0000011numericinvalid_cut
n_in_rangejson_valueresult.json › n_in_rangeexact1numericinvalid_cut
mujson_valueresult.json › fit.muatol 0.053numericcriticalstatistical_misuse
mu_errjson_valueresult.json › fit.mu_errrtol 0.31numericstatistical_misuse
sigmajson_valueresult.json › fit.sigmaatol 0.12numericstatistical_misuse
n_sigjson_valueresult.json › fit.n_sigrtol 0.052numericcriticalstatistical_misuseincorrect_normalization
n_bkgjson_valueresult.json › fit.n_bkgrtol 0.251numericstatistical_misuse
chi2json_valueresult.json › fit.chi2rtol 0.11numericstatistical_misuse
ndfjson_valueresult.json › fit.ndfexact1numericstatistical_misuse
report_sectionsfile_containsREPORT.md ["## Selection","## Fit","## Assumptions"]2compliancespec_noncompliance
svg_legendsvg_textfigures/zfit.svg ["data","fit","GeV"]1plotplotting_error
svg_drawnsvg_has_marksfigures/zfit.svg≥ 101plotplotting_error
rerunsscript_runssolution.py1reproducibilitynon_reproducible
not_hardcodedno_hardcoded_resultsolution.py1compliancefabricated_result

Tolerance rationale. With ~tens of thousands of events in the window the fitted mu is stable to ~0.01 GeV across optimisers; atol 0.05 is generous but rejects any wrong selection or wrong bin edges. The Gaussian is a knowingly imperfect Z lineshape; the task fixes the model so the fit is gradable.

Ground truth (produced by the reference in the sandbox)

built 2026-09-06 · numpy 1.26.4 · scipy 1.13.1 · 2.817s
result.json
{
 "n_total": 100000,
 "n_selected": 5752,
 "efficiency": 0.05752,
 "n_in_range": 4656,
 "fit": {
  "mu": 90.81742851182433,
  "mu_err": 0.04041956410068175,
  "sigma": 2.0683943522914467,
  "sigma_err": 0.037818572984515554,
  "n_sig": 3625.2626159476545,
  "n_sig_err": 71.01160983414752,
  "n_bkg": 953.5651208923147,
  "lam": 51.79671467682172,
  "chi2": 77.19180249502361,
  "ndf": 35
 }
}
reference.py
import json
import os
import numpy as np
import pandas as pd
from scipy.optimize import curve_fit
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

LO, HI, W = 80.0, 100.0, 0.5
df = pd.read_csv("data/cms_dimuon_2011.csv")
sel = (df.type1 == "G") & (df.type2 == "G") & (df.eta1.abs() < 2.4) & (df.eta2.abs() < 2.4) & (df.pt1 > 20) & (df.pt2 > 20) & (df.Q1 * df.Q2 < 0)
d = df[sel]
m = d["M"].to_numpy()
in_range = (m >= LO) & (m <= HI)
counts, edges = np.histogram(m, bins=40, range=(LO, HI))
x = 0.5 * (edges[:-1] + edges[1:])


def gauss(m, mu, sigma):
    return np.exp(-0.5 * ((m - mu) / sigma) ** 2) / (sigma * np.sqrt(2 * np.pi))


def expo(m, lam):
    return np.exp(-m / lam) / (lam * (np.exp(-LO / lam) - np.exp(-HI / lam)))


def model(m, mu, sigma, ns, nb, lam):
    return W * (ns * gauss(m, mu, sigma) + nb * expo(m, lam))


ok = counts > 0
n_in = int(in_range.sum())
p0 = [91.0, 2.0, 0.9 * n_in, 0.1 * n_in, 30.0]
popt, pcov = curve_fit(model, x[ok], counts[ok], p0=p0, sigma=np.sqrt(counts[ok]), absolute_sigma=True, maxfev=20000)
err = np.sqrt(np.diag(pcov))
chi2 = float((((counts[ok] - model(x[ok], *popt)) ** 2) / counts[ok]).sum())
res = {
    "n_total": int(len(df)), "n_selected": int(len(d)), "efficiency": float(len(d) / len(df)), "n_in_range": n_in,
    "fit": {"mu": float(popt[0]), "mu_err": float(err[0]), "sigma": float(abs(popt[1])), "sigma_err": float(err[1]), "n_sig": float(popt[2]),
            "n_sig_err": float(err[2]), "n_bkg": float(popt[3]), "lam": float(popt[4]), "chi2": chi2, "ndf": int(ok.sum() - 5)},
}
json.dump(res, open("result.json", "w"), indent=2)
os.makedirs("figures", exist_ok=True)
fig, ax = plt.subplots(figsize=(7, 4.5))
ax.errorbar(x, counts, yerr=np.sqrt(counts), fmt="o", ms=3, label="data")
xx = np.linspace(LO, HI, 400)
ax.plot(xx, model(xx, *popt), label="fit (Gaussian + exponential)")
ax.set_xlabel("Dimuon invariant mass M [GeV]")
ax.set_ylabel("Events / 0.5 GeV")
ax.legend()
fig.savefig("figures/zfit.svg")
f = res["fit"]
open("REPORT.md", "w").write(f"""# Z peak in CMS 2011 dimuon open data

## Selection
Both muons global, |eta| < 2.4, pT > 20 GeV, opposite charge: {res['n_selected']} of {res['n_total']} events (efficiency {res['efficiency']:.4f}); {n_in} in [80, 100] GeV.

## Fit
Gaussian + exponential, 40 bins of 0.5 GeV, least squares with sqrt(n) errors: mu = {f['mu']:.3f} +- {f['mu_err']:.3f} GeV, sigma = {f['sigma']:.3f} GeV, N_s = {f['n_sig']:.0f} +- {f['n_sig_err']:.0f}, chi2/ndf = {f['chi2']:.1f}/{f['ndf']}.

## Assumptions
- The Gaussian is a resolution-dominated approximation of the Z line shape; no Breit-Wigner convolution and no radiative tail are modelled, so chi2/ndf is expected to exceed 1.
- Bins with zero counts are excluded from the chi2 (none occur here).
- No trigger, isolation or vertex requirements beyond the listed cuts; no efficiency corrections are applied.
- The exponential background is normalised on the fit range only.
""")
print(res)

Results on this task

4 runs
AgentModelRunsStrict successMean score
Self-debuggingQwen3-8B (gariyuu gateway)10%0.8438
Planner / executorQwen3-8B (gariyuu gateway)10%0
ReActQwen3-8B (gariyuu gateway)10%0
Single-shotQwen3-8B (gariyuu gateway)10%0.4271
RunAgentModelResultScoreLabels
20260906T172010…r0Single-shotQwen3-8B (gariyuu gateway)✗ fail0.4271no_outputplotting_errorstatistical_misuseincorrect_normalizationspec_noncompliancenon_reproducibleexecution_failed
20260906T173334…r0ReActQwen3-8B (gariyuu gateway)✗ fail0no_outputplotting_errorwrong_datasetinvalid_cutstatistical_misuseincorrect_normalizationspec_noncompliancenon_reproduciblefabricated_resultexecution_failedstep_budget_exhausted
20260906T175417…r0Self-debuggingQwen3-8B (gariyuu gateway)✗ fail0.8438statistical_misuseincorrect_normalization
20260906T180044…r0Planner / executorQwen3-8B (gariyuu gateway)✗ fail0no_outputplotting_errorwrong_datasetinvalid_cutstatistical_misuseincorrect_normalizationspec_noncompliancenon_reproduciblefabricated_resultexecution_failedstep_budget_exhausted