End-to-end Z-peak measurement on CMS open data
Quality selection, mass histogram, specified signal+background fit, efficiency bookkeeping, and a written report with stated assumptions, all in one reproducible script.
t6-full-analysisend_to_endcms_dimuon_2011selectionhistogrambinned_fitreportingreproducibility
Task prompt (what the agent sees)
Perform a small but complete analysis of `data/cms_dimuon_2011.csv` (data card in README.md). 1. Quality selection: both muons global (`type1 == "G"` and `type2 == "G"`), |eta1| < 2.4, |eta2| < 2.4, pt1 > 20 GeV, pt2 > 20 GeV, opposite charge (Q1*Q2 < 0). 2. Histogram `M` of the selected events in 40 bins of 0.5 GeV on [80, 100]. 3. Fit the histogram with f(m) = w * [ N_s * Gauss(m; mu, sigma) + N_b * Expo(m; lam) ], w = 0.5 GeV, where Gauss is the normalised Gaussian density and Expo(m; lam) = exp(-m/lam) / (lam * (exp(-80/lam) - exp(-100/lam))) is normalised on [80, 100]. Minimise least squares with sigma_i = sqrt(n_i) over bins with n_i > 0, starting from mu = 91, sigma = 2, N_s = 0.9 * (selected events in range), N_b = 0.1 * (selected events in range), lam = 30. Uncertainties from the covariance matrix with `absolute_sigma=True`. 4. Report and document. Deliverables: - `result.json` with `n_total` (int), `n_selected` (int), `efficiency` (float, n_selected / n_total), `n_in_range` (int, selected events with 80 <= M <= 100), and `fit`: an object with `mu`, `mu_err`, `sigma`, `sigma_err`, `n_sig`, `n_sig_err`, `n_bkg`, `lam`, `chi2` (floats) and `ndf` (int). - `figures/zfit.svg`: data with error bars and the fitted curve, legend entries containing "data" and "fit", x-axis label including "GeV". - `REPORT.md`: a short report with the headings "## Selection", "## Fit" and "## Assumptions"; the Assumptions section must list every modelling choice you made that the task did not specify. - `solution.py`: the complete analysis; `python solution.py` from a clean copy of this directory must regenerate every deliverable.
Data card (README.md in the workdir)
Data card: data/cms_dimuon_2011.csv (REAL DATA)
Source: CERN Open Data Portal record 545, "Dimuon events from the CMS 2011 DoubleMu primary dataset"
(file Dimuon_DoubleMu.csv). Licence CC0. 100,000 events, one per row.
| column | meaning | unit |
|---|---|---|
| Run, Event | run and event number | – |
| type1, type2 | muon reconstruction type: G global muon, T tracker muon |
– |
| E1, px1, py1, pz1 | four-momentum of muon 1 | GeV |
| pt1, eta1, phi1 | transverse momentum, pseudorapidity, azimuth of muon 1 | GeV, –, rad |
| Q1 | charge of muon 1 | e |
| E2 … Q2 | same for muon 2 | |
| M | invariant mass of the muon pair | GeV |
All energies and momenta are in GeV. There are no missing values.
Expected artifacts
- result.json json
- figures/zfit.svg svg
- REPORT.md markdown
- solution.py script
Deterministic checks and tolerances
| Check | Type | Target | Tolerance | Weight | Category | Critical | Failure implies |
|---|---|---|---|---|---|---|---|
| result_exists | file_exists | result.json | 1 | artifact | critical | no_output | |
| figure_exists | file_exists | figures/zfit.svg | 1 | artifact | no_outputplotting_error | ||
| report_exists | file_exists | REPORT.md | 1 | artifact | no_output | ||
| solution_exists | file_exists | solution.py | 1 | artifact | no_output | ||
| n_total | json_value | result.json › n_total | exact | 1 | numeric | wrong_dataset | |
| n_selected | json_value | result.json › n_selected | exact | 2 | numeric | critical | invalid_cut |
| efficiency | json_value | result.json › efficiency | atol 0.000001 | 1 | numeric | invalid_cut | |
| n_in_range | json_value | result.json › n_in_range | exact | 1 | numeric | invalid_cut | |
| mu | json_value | result.json › fit.mu | atol 0.05 | 3 | numeric | critical | statistical_misuse |
| mu_err | json_value | result.json › fit.mu_err | rtol 0.3 | 1 | numeric | statistical_misuse | |
| sigma | json_value | result.json › fit.sigma | atol 0.1 | 2 | numeric | statistical_misuse | |
| n_sig | json_value | result.json › fit.n_sig | rtol 0.05 | 2 | numeric | critical | statistical_misuseincorrect_normalization |
| n_bkg | json_value | result.json › fit.n_bkg | rtol 0.25 | 1 | numeric | statistical_misuse | |
| chi2 | json_value | result.json › fit.chi2 | rtol 0.1 | 1 | numeric | statistical_misuse | |
| ndf | json_value | result.json › fit.ndf | exact | 1 | numeric | statistical_misuse | |
| report_sections | file_contains | REPORT.md ["## Selection","## Fit","## Assumptions"] | 2 | compliance | spec_noncompliance | ||
| svg_legend | svg_text | figures/zfit.svg ["data","fit","GeV"] | 1 | plot | plotting_error | ||
| svg_drawn | svg_has_marks | figures/zfit.svg | ≥ 10 | 1 | plot | plotting_error | |
| reruns | script_runs | solution.py | 1 | reproducibility | non_reproducible | ||
| not_hardcoded | no_hardcoded_result | solution.py | 1 | compliance | fabricated_result |
Tolerance rationale. With ~tens of thousands of events in the window the fitted mu is stable to ~0.01 GeV across optimisers; atol 0.05 is generous but rejects any wrong selection or wrong bin edges. The Gaussian is a knowingly imperfect Z lineshape; the task fixes the model so the fit is gradable.
Ground truth (produced by the reference in the sandbox)
result.json
{
"n_total": 100000,
"n_selected": 5752,
"efficiency": 0.05752,
"n_in_range": 4656,
"fit": {
"mu": 90.81742851182433,
"mu_err": 0.04041956410068175,
"sigma": 2.0683943522914467,
"sigma_err": 0.037818572984515554,
"n_sig": 3625.2626159476545,
"n_sig_err": 71.01160983414752,
"n_bkg": 953.5651208923147,
"lam": 51.79671467682172,
"chi2": 77.19180249502361,
"ndf": 35
}
}reference.py
import json
import os
import numpy as np
import pandas as pd
from scipy.optimize import curve_fit
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
LO, HI, W = 80.0, 100.0, 0.5
df = pd.read_csv("data/cms_dimuon_2011.csv")
sel = (df.type1 == "G") & (df.type2 == "G") & (df.eta1.abs() < 2.4) & (df.eta2.abs() < 2.4) & (df.pt1 > 20) & (df.pt2 > 20) & (df.Q1 * df.Q2 < 0)
d = df[sel]
m = d["M"].to_numpy()
in_range = (m >= LO) & (m <= HI)
counts, edges = np.histogram(m, bins=40, range=(LO, HI))
x = 0.5 * (edges[:-1] + edges[1:])
def gauss(m, mu, sigma):
return np.exp(-0.5 * ((m - mu) / sigma) ** 2) / (sigma * np.sqrt(2 * np.pi))
def expo(m, lam):
return np.exp(-m / lam) / (lam * (np.exp(-LO / lam) - np.exp(-HI / lam)))
def model(m, mu, sigma, ns, nb, lam):
return W * (ns * gauss(m, mu, sigma) + nb * expo(m, lam))
ok = counts > 0
n_in = int(in_range.sum())
p0 = [91.0, 2.0, 0.9 * n_in, 0.1 * n_in, 30.0]
popt, pcov = curve_fit(model, x[ok], counts[ok], p0=p0, sigma=np.sqrt(counts[ok]), absolute_sigma=True, maxfev=20000)
err = np.sqrt(np.diag(pcov))
chi2 = float((((counts[ok] - model(x[ok], *popt)) ** 2) / counts[ok]).sum())
res = {
"n_total": int(len(df)), "n_selected": int(len(d)), "efficiency": float(len(d) / len(df)), "n_in_range": n_in,
"fit": {"mu": float(popt[0]), "mu_err": float(err[0]), "sigma": float(abs(popt[1])), "sigma_err": float(err[1]), "n_sig": float(popt[2]),
"n_sig_err": float(err[2]), "n_bkg": float(popt[3]), "lam": float(popt[4]), "chi2": chi2, "ndf": int(ok.sum() - 5)},
}
json.dump(res, open("result.json", "w"), indent=2)
os.makedirs("figures", exist_ok=True)
fig, ax = plt.subplots(figsize=(7, 4.5))
ax.errorbar(x, counts, yerr=np.sqrt(counts), fmt="o", ms=3, label="data")
xx = np.linspace(LO, HI, 400)
ax.plot(xx, model(xx, *popt), label="fit (Gaussian + exponential)")
ax.set_xlabel("Dimuon invariant mass M [GeV]")
ax.set_ylabel("Events / 0.5 GeV")
ax.legend()
fig.savefig("figures/zfit.svg")
f = res["fit"]
open("REPORT.md", "w").write(f"""# Z peak in CMS 2011 dimuon open data
## Selection
Both muons global, |eta| < 2.4, pT > 20 GeV, opposite charge: {res['n_selected']} of {res['n_total']} events (efficiency {res['efficiency']:.4f}); {n_in} in [80, 100] GeV.
## Fit
Gaussian + exponential, 40 bins of 0.5 GeV, least squares with sqrt(n) errors: mu = {f['mu']:.3f} +- {f['mu_err']:.3f} GeV, sigma = {f['sigma']:.3f} GeV, N_s = {f['n_sig']:.0f} +- {f['n_sig_err']:.0f}, chi2/ndf = {f['chi2']:.1f}/{f['ndf']}.
## Assumptions
- The Gaussian is a resolution-dominated approximation of the Z line shape; no Breit-Wigner convolution and no radiative tail are modelled, so chi2/ndf is expected to exceed 1.
- Bins with zero counts are excluded from the chi2 (none occur here).
- No trigger, isolation or vertex requirements beyond the listed cuts; no efficiency corrections are applied.
- The exponential background is normalised on the fit range only.
""")
print(res)
Results on this task
| Agent | Model | Runs | Strict success | Mean score |
|---|---|---|---|---|
| Self-debugging | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.8438 |
| Planner / executor | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0 |
| ReAct | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0 |
| Single-shot | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.4271 |
| Run | Agent | Model | Result | Score | Labels |
|---|---|---|---|---|---|
| 20260906T172010…r0 | Single-shot | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.4271 | no_outputplotting_errorstatistical_misuseincorrect_normalizationspec_noncompliancenon_reproducibleexecution_failed |
| 20260906T173334…r0 | ReAct | Qwen3-8B (gariyuu gateway) | ✗ fail | 0 | no_outputplotting_errorwrong_datasetinvalid_cutstatistical_misuseincorrect_normalizationspec_noncompliancenon_reproduciblefabricated_resultexecution_failedstep_budget_exhausted |
| 20260906T175417…r0 | Self-debugging | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.8438 | statistical_misuseincorrect_normalization |
| 20260906T180044…r0 | Planner / executor | Qwen3-8B (gariyuu gateway) | ✗ fail | 0 | no_outputplotting_errorwrong_datasetinvalid_cutstatistical_misuseincorrect_normalizationspec_noncompliancenon_reproduciblefabricated_resultexecution_failedstep_budget_exhausted |