Derived dimuon kinematics on real CMS data
Compute pair transverse momentum, rapidity, wrapped azimuthal separation and Delta-R per event from CMS four-vectors.
t3-dimuon-kinematicsderived_variablescms_dimuon_2011four_vectorsangular_variablesderived_variablecsv_output
Task prompt (what the agent sees)
Using `data/cms_dimuon_2011.csv` (data card in README.md), compute for every event: - `pt_ll`: transverse momentum of the dimuon system, |(px1+px2, py1+py2)| in GeV - `y_ll`: rapidity of the dimuon system, 0.5 * ln((E + pz) / (E - pz)) with E = E1+E2, pz = pz1+pz2 - `delta_phi`: azimuthal separation of the two muons wrapped into [0, pi] - `delta_r`: sqrt(delta_eta^2 + delta_phi^2) with delta_eta = eta1 - eta2 Deliverables: - `kinematics.csv` with columns `Run`, `Event`, `pt_ll`, `y_ll`, `delta_phi`, `delta_r` in the input row order. - `result.json` with `n_events` (int), `mean_pt_ll` (float), `mean_delta_r` (float), `frac_delta_phi_gt_2p5` (float, fraction of events with delta_phi > 2.5), `max_abs_y_ll` (float). - `figures/pt_ll.svg`: histogram of pt_ll from 0 to 200 GeV, x-axis label including "GeV". - `solution.py`: the complete analysis; `python solution.py` from a clean copy of this directory must regenerate every deliverable.
Data card (README.md in the workdir)
Data card: data/cms_dimuon_2011.csv (REAL DATA)
Source: CERN Open Data Portal record 545, "Dimuon events from the CMS 2011 DoubleMu primary dataset"
(file Dimuon_DoubleMu.csv). Licence CC0. 100,000 events, one per row.
| column | meaning | unit |
|---|---|---|
| Run, Event | run and event number | – |
| type1, type2 | muon reconstruction type: G global muon, T tracker muon |
– |
| E1, px1, py1, pz1 | four-momentum of muon 1 | GeV |
| pt1, eta1, phi1 | transverse momentum, pseudorapidity, azimuth of muon 1 | GeV, –, rad |
| Q1 | charge of muon 1 | e |
| E2 … Q2 | same for muon 2 | |
| M | invariant mass of the muon pair | GeV |
All energies and momenta are in GeV. There are no missing values.
Expected artifacts
- kinematics.csv csv
- result.json json
- figures/pt_ll.svg svg
- solution.py script
Deterministic checks and tolerances
| Check | Type | Target | Tolerance | Weight | Category | Critical | Failure implies |
|---|---|---|---|---|---|---|---|
| kin_exists | file_exists | kinematics.csv | 1 | artifact | critical | no_output | |
| result_exists | file_exists | result.json | 1 | artifact | no_output | ||
| figure_exists | file_exists | figures/pt_ll.svg | 1 | artifact | no_outputplotting_error | ||
| solution_exists | file_exists | solution.py | 1 | artifact | no_output | ||
| kin_columns | csv_columns | kinematics.csv ["Run","Event","pt_ll","y_ll","delta_phi","delta_r"] | 1 | numeric | spec_noncompliance | ||
| pt_ll | csv_column_match | kinematics.csv › pt_ll | atol 0.001 | 2 | numeric | critical | wrong_variable |
| y_ll | csv_column_match | kinematics.csv › y_ll | atol 0.001 | 2 | numeric | wrong_variable | |
| delta_phi | csv_column_match | kinematics.csv › delta_phi | atol 0.0001 | 3 | numeric | critical | wrong_variable |
| delta_r | csv_column_match | kinematics.csv › delta_r | atol 0.0001 | 2 | numeric | wrong_variable | |
| mean_pt | json_value | result.json › mean_pt_ll | rtol 0.00001 | 1 | numeric | wrong_variable | |
| mean_dr | json_value | result.json › mean_delta_r | rtol 0.00001 | 1 | numeric | wrong_variable | |
| frac_dphi | json_value | result.json › frac_delta_phi_gt_2p5 | atol 0.000001 | 2 | numeric | wrong_variable | |
| max_y | json_value | result.json › max_abs_y_ll | rtol 0.00001 | 1 | numeric | wrong_variable | |
| svg_unit | svg_text | figures/pt_ll.svg ["GeV"] | 1 | plot | plotting_error | ||
| svg_drawn | svg_has_marks | figures/pt_ll.svg | ≥ 5 | 1 | plot | plotting_error | |
| reruns | script_runs | solution.py | 1 | reproducibility | non_reproducible | ||
| not_hardcoded | no_hardcoded_result | solution.py | 1 | compliance | fabricated_result |
Tolerance rationale. Per-event columns atol 1e-3 GeV / 1e-4 rad; the input has four decimals. Unwrapped delta_phi fails the delta_phi and frac checks by construction.
Ground truth (produced by the reference in the sandbox)
result.json
{
"n_events": 100000,
"mean_pt_ll": 9.713848678583945,
"mean_delta_r": 2.3290955580440396,
"frac_delta_phi_gt_2p5": 0.58301,
"max_abs_y_ll": 2.3912620543767447
}reference.py
import json
import os
import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
df = pd.read_csv("data/cms_dimuon_2011.csv")
px, py, pz, E = df.px1 + df.px2, df.py1 + df.py2, df.pz1 + df.pz2, df.E1 + df.E2
pt_ll = np.hypot(px, py)
y_ll = 0.5 * np.log((E + pz) / (E - pz))
dphi = np.abs(df.phi1 - df.phi2)
dphi = np.where(dphi > np.pi, 2 * np.pi - dphi, dphi)
deta = df.eta1 - df.eta2
dr = np.sqrt(deta**2 + dphi**2)
out = pd.DataFrame({"Run": df.Run, "Event": df.Event, "pt_ll": pt_ll, "y_ll": y_ll, "delta_phi": dphi, "delta_r": dr})
out.to_csv("kinematics.csv", index=False)
res = {"n_events": int(len(df)), "mean_pt_ll": float(pt_ll.mean()), "mean_delta_r": float(dr.mean()),
"frac_delta_phi_gt_2p5": float((dphi > 2.5).mean()), "max_abs_y_ll": float(np.abs(y_ll).max())}
json.dump(res, open("result.json", "w"), indent=2)
os.makedirs("figures", exist_ok=True)
fig, ax = plt.subplots(figsize=(7, 4.5))
ax.hist(pt_ll, bins=100, range=(0, 200), histtype="stepfilled", alpha=0.7)
ax.set_xlabel("Dimuon transverse momentum pT(ll) [GeV]")
ax.set_ylabel("Events / 2 GeV")
ax.set_yscale("log")
fig.savefig("figures/pt_ll.svg")
print(res)
Results on this task
| Agent | Model | Runs | Strict success | Mean score |
|---|---|---|---|---|
| Self-debugging | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.8083 |
| Planner / executor | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.8 |
| ReAct | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.8 |
| Single-shot | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.4333 |
| Run | Agent | Model | Result | Score | Labels |
|---|---|---|---|---|---|
| 20260906T171905…r0 | Single-shot | Qwen3-8B (gariyuu gateway) | ✗ core only | 0.4333 | no_outputplotting_errorwrong_variablenon_reproducibleexecution_failed |
| 20260906T172053…r0 | ReAct | Qwen3-8B (gariyuu gateway) | ✗ core only | 0.8 | non_reproducibleunsupported_claim |
| 20260906T173629…r0 | Planner / executor | Qwen3-8B (gariyuu gateway) | ✗ core only | 0.8 | non_reproducible |
| 20260906T174945…r0 | Self-debugging | Qwen3-8B (gariyuu gateway) | ✗ core only | 0.8083 | no_outputwrong_variablenon_reproducible |