Inspect the CMS dimuon file
Load a real CMS open-data CSV and report its shape, columns and basic statistics of the dimuon mass.
t1-schema-summaryinspectioncms_dimuon_2011load_csvschema_inspectionsummary_statistics
Task prompt (what the agent sees)
The working directory contains `data/cms_dimuon_2011.csv`, a public CMS 2011 dimuon dataset (see README.md for the data card). Inspect the file and write `result.json` with exactly these keys: - `n_events`: number of rows (int) - `n_columns`: number of columns (int) - `columns`: list of column names in file order - `n_runs`: number of distinct values of `Run` (int) - `n_events_both_global`: number of events where both `type1` and `type2` equal "G" (int) - `m_min`, `m_max`, `m_mean`: minimum, maximum and mean of the dimuon mass column `M` in GeV (floats, full precision) - `frac_opposite_charge`: fraction of events with opposite-sign muon charges, i.e. Q1*Q2 < 0 (float) Save the complete analysis as `solution.py`. Running `python solution.py` from a clean copy of this directory must regenerate `result.json`.
Data card (README.md in the workdir)
Data card: data/cms_dimuon_2011.csv (REAL DATA)
Source: CERN Open Data Portal record 545, "Dimuon events from the CMS 2011 DoubleMu primary dataset"
(file Dimuon_DoubleMu.csv). Licence CC0. 100,000 events, one per row.
| column | meaning | unit |
|---|---|---|
| Run, Event | run and event number | – |
| type1, type2 | muon reconstruction type: G global muon, T tracker muon |
– |
| E1, px1, py1, pz1 | four-momentum of muon 1 | GeV |
| pt1, eta1, phi1 | transverse momentum, pseudorapidity, azimuth of muon 1 | GeV, –, rad |
| Q1 | charge of muon 1 | e |
| E2 … Q2 | same for muon 2 | |
| M | invariant mass of the muon pair | GeV |
All energies and momenta are in GeV. There are no missing values.
Expected artifacts
- result.json json summary statistics
- solution.py script reproducible analysis script
Deterministic checks and tolerances
| Check | Type | Target | Tolerance | Weight | Category | Critical | Failure implies |
|---|---|---|---|---|---|---|---|
| result_exists | file_exists | result.json | 1 | artifact | critical | no_output | |
| solution_exists | file_exists | solution.py | 1 | artifact | no_output | ||
| n_events | json_value | result.json › n_events | exact | 2 | numeric | critical | wrong_dataset |
| n_columns | json_value | result.json › n_columns | exact | 1 | numeric | wrong_dataset | |
| columns | json_list_equal | result.json › columns | exact | 1 | numeric | wrong_variable | |
| n_runs | json_value | result.json › n_runs | exact | 1 | numeric | wrong_variable | |
| n_both_global | json_value | result.json › n_events_both_global | exact | 1 | numeric | invalid_cut | |
| m_min | json_value | result.json › m_min | atol 0.0001 | 1 | numeric | wrong_variable | |
| m_max | json_value | result.json › m_max | atol 0.0001 | 1 | numeric | wrong_variable | |
| m_mean | json_value | result.json › m_mean | rtol 0.000001 | 2 | numeric | critical | wrong_variable |
| frac_os | json_value | result.json › frac_opposite_charge | atol 0.000001 | 1 | numeric | invalid_cut | |
| reruns | script_runs | solution.py | 1 | reproducibility | non_reproducible | ||
| not_hardcoded | no_hardcoded_result | solution.py | 1 | compliance | fabricated_result |
Tolerance rationale. Counts are exact. Means use rtol 1e-6 (float64 summation order). Min/max atol 1e-4 (four decimals in the source file).
Ground truth (produced by the reference in the sandbox)
result.json
{
"n_events": 100000,
"n_columns": 21,
"columns": [
"Run",
"Event",
"type1",
"E1",
"px1",
"py1",
"pz1",
"pt1",
"eta1",
"phi1",
"Q1",
"type2",
"E2",
"px2",
"py2",
"pz2",
"pt2",
"eta2",
"phi2",
"Q2",
"M"
],
"n_runs": 15,
"n_events_both_global": 93474,
"m_min": 0.3002,
"m_max": 299.202,
"m_mean": 17.691054685,
"frac_opposite_charge": 1
}reference.py
import json
import pandas as pd
df = pd.read_csv("data/cms_dimuon_2011.csv")
result = {
"n_events": int(len(df)),
"n_columns": int(df.shape[1]),
"columns": list(df.columns),
"n_runs": int(df["Run"].nunique()),
"n_events_both_global": int(((df["type1"] == "G") & (df["type2"] == "G")).sum()),
"m_min": float(df["M"].min()),
"m_max": float(df["M"].max()),
"m_mean": float(df["M"].mean()),
"frac_opposite_charge": float((df["Q1"] * df["Q2"] < 0).mean()),
}
json.dump(result, open("result.json", "w"), indent=2)
print(result)
Results on this task
| Agent | Model | Runs | Strict success | Mean score |
|---|---|---|---|---|
| Self-debugging | Qwen3-8B (gariyuu gateway) | 1 | 100% | 1 |
| Planner / executor | Qwen3-8B (gariyuu gateway) | 1 | 100% | 1 |
| ReAct | Qwen3-8B (gariyuu gateway) | 1 | 100% | 1 |
| Single-shot | Qwen3-8B (gariyuu gateway) | 1 | 100% | 1 |
| Run | Agent | Model | Result | Score | Labels |
|---|---|---|---|---|---|
| 20260906T171852…r0 | Single-shot | Qwen3-8B (gariyuu gateway) | ✓ strict | 1 | |
| 20260906T172012…r0 | ReAct | Qwen3-8B (gariyuu gateway) | ✓ strict | 1 | unsupported_claim |
| 20260906T173510…r0 | Planner / executor | Qwen3-8B (gariyuu gateway) | ✓ strict | 1 | |
| 20260906T174851…r0 | Self-debugging | Qwen3-8B (gariyuu gateway) | ✓ strict | 1 |