Reconstruct the dimuon invariant mass from muon kinematics
Build four-vectors from (pt, eta, phi) and the muon mass, compute the pair mass for 50k synthetic events, and summarise the Z window.
Task prompt (what the agent sees)
`data/synth_zmumu.csv` (data card in README.md) contains two muons per event described only by (pt, eta, phi, charge). Reconstruct the invariant mass of the muon pair for every event using the muon mass given in the data card. Deliverables: - `mass.csv`: one row per event with columns `event_id` and `m_ll` (invariant mass in GeV), same row order as the input. - `result.json` with keys `n_events` (int), `n_in_window` (int, events with 80 < m_ll < 100), `mean_mass_in_window` (float, mean m_ll over those events), `median_mass_all` (float, median m_ll over all events), `n_opposite_charge_in_window` (int, events in the window with mu1_charge * mu2_charge < 0). - `figures/m_ll.svg`: histogram of m_ll from 40 to 200 GeV with an x-axis label that includes "GeV". - `solution.py`: the complete analysis; `python solution.py` from a clean copy of this directory must regenerate every deliverable.
Data card (README.md in the workdir)
Data card: data/synth_zmumu.csv (SYNTHETIC)
50,000 simulated dimuon events from a toy generator: a Z -> mu mu resonance on top of a smoothly falling continuum. This is synthetic data with a known seed; it is not a measurement.
| column | meaning | unit |
|---|---|---|
| event_id | event number (1..N) | – |
| run | pseudo run number | – |
| mu1_pt, mu1_eta, mu1_phi | transverse momentum, pseudorapidity, azimuth of the leading (higher-pt) muon | GeV, –, rad |
| mu1_charge | charge of muon 1 (+1 / -1) | e |
| mu2_pt, mu2_eta, mu2_phi, mu2_charge | same for the sub-leading muon |
Conventions: px = pt cos(phi), py = pt sin(phi), pz = pt sinh(eta), E = sqrt(px^2 + py^2 + pz^2 + m_mu^2) with the muon mass m_mu = 0.1056583755 GeV. There are no missing values and no duplicated events.
Expected artifacts
- mass.csv csv
- result.json json
- figures/m_ll.svg svg
- solution.py script
Deterministic checks and tolerances
| Check | Type | Target | Tolerance | Weight | Category | Critical | Failure implies |
|---|---|---|---|---|---|---|---|
| mass_exists | file_exists | mass.csv | 1 | artifact | critical | no_output | |
| result_exists | file_exists | result.json | 1 | artifact | no_output | ||
| figure_exists | file_exists | figures/m_ll.svg | 1 | artifact | no_outputplotting_error | ||
| solution_exists | file_exists | solution.py | 1 | artifact | no_output | ||
| mass_columns | csv_columns | mass.csv ["event_id","m_ll"] | 1 | numeric | spec_noncompliance | ||
| mass_rows | csv_row_count | mass.csv | exact | 1 | numeric | invalid_cut | |
| m_ll_values | csv_column_match | mass.csv › event_id › m_ll | atol 0.005 | 4 | numeric | critical | wrong_variableunit_error |
| n_in_window | json_value | result.json › n_in_window | atol 3 | 1 | numeric | invalid_cut | |
| mean_in_window | json_value | result.json › mean_mass_in_window | rtol 0.0001 | 2 | numeric | critical | wrong_variable |
| median_all | json_value | result.json › median_mass_all | rtol 0.0001 | 1 | numeric | wrong_variable | |
| n_os_window | json_value | result.json › n_opposite_charge_in_window | atol 3 | 1 | numeric | invalid_cut | |
| svg_unit | svg_text | figures/m_ll.svg ["GeV"] | 1 | plot | plotting_error | ||
| svg_drawn | svg_has_marks | figures/m_ll.svg | ≥ 5 | 1 | plot | plotting_error | |
| reruns | script_runs | solution.py | 1 | reproducibility | non_reproducible | ||
| not_hardcoded | no_hardcoded_result | solution.py | 1 | compliance | fabricated_result |
Tolerance rationale. m_ll atol 5e-3 GeV per event: this accepts the massless-muon approximation (which shifts m by up to ~3e-3 GeV for very asymmetric pairs) while rejecting any formula error (missing sinh, pt used as |p|, wrong sign in the dot product), all of which are O(GeV). Window counts allow +-3 events for the same reason.
Ground truth (produced by the reference in the sandbox)
result.json
{
"n_events": 50000,
"n_in_window": 34483,
"mean_mass_in_window": 90.96866737789078,
"median_mass_all": 90.38002673106806,
"n_opposite_charge_in_window": 34205
}reference.py
import json
import os
import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
MU = 0.1056583755
df = pd.read_csv("data/synth_zmumu.csv")
def four(pt, eta, phi):
px, py, pz = pt * np.cos(phi), pt * np.sin(phi), pt * np.sinh(eta)
return np.sqrt(px**2 + py**2 + pz**2 + MU**2), px, py, pz
E1, x1, y1, z1 = four(df.mu1_pt.to_numpy(), df.mu1_eta.to_numpy(), df.mu1_phi.to_numpy())
E2, x2, y2, z2 = four(df.mu2_pt.to_numpy(), df.mu2_eta.to_numpy(), df.mu2_phi.to_numpy())
m2 = (E1 + E2) ** 2 - (x1 + x2) ** 2 - (y1 + y2) ** 2 - (z1 + z2) ** 2
m = np.sqrt(np.clip(m2, 0, None))
pd.DataFrame({"event_id": df.event_id, "m_ll": m}).to_csv("mass.csv", index=False)
win = (m > 80) & (m < 100)
res = {
"n_events": int(len(df)),
"n_in_window": int(win.sum()),
"mean_mass_in_window": float(m[win].mean()),
"median_mass_all": float(np.median(m)),
"n_opposite_charge_in_window": int((win & (df.mu1_charge * df.mu2_charge < 0)).sum()),
}
json.dump(res, open("result.json", "w"), indent=2)
os.makedirs("figures", exist_ok=True)
fig, ax = plt.subplots(figsize=(7, 4.5))
ax.hist(m, bins=160, range=(40, 200), histtype="stepfilled", alpha=0.7)
ax.set_xlabel("Dimuon invariant mass m_ll [GeV]")
ax.set_ylabel("Events / 1 GeV")
ax.set_yscale("log")
fig.savefig("figures/m_ll.svg")
print(res)
Results on this task
| Agent | Model | Runs | Strict success | Mean score |
|---|---|---|---|---|
| Self-debugging | Qwen3-8B (gariyuu gateway) | 1 | 100% | 1 |
| Planner / executor | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.8 |
| ReAct | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0 |
| Single-shot | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.625 |
| Run | Agent | Model | Result | Score | Labels |
|---|---|---|---|---|---|
| 20260906T171910…r0 | Single-shot | Qwen3-8B (gariyuu gateway) | ✗ core only | 0.625 | no_outputplotting_errornon_reproducibleexecution_failed |
| 20260906T172105…r0 | ReAct | Qwen3-8B (gariyuu gateway) | ✗ fail | 0 | no_outputplotting_errorspec_noncomplianceinvalid_cutwrong_variableunit_errornon_reproduciblefabricated_resultexecution_failedstep_budget_exhausted |
| 20260906T173654…r0 | Planner / executor | Qwen3-8B (gariyuu gateway) | ✗ core only | 0.8 | non_reproducibleexecution_failedstep_budget_exhausted |
| 20260906T174954…r0 | Self-debugging | Qwen3-8B (gariyuu gateway) | ✓ strict | 1 |