Tier 2 · Histograms & selections · v1.0.0

Event selections including one that is empty

Apply three kinematic selections to real CMS data and report yields; one selection is kinematically empty and must be flagged, not papered over.

t2-selection-counthistogramscms_dimuon_2011trap: empty_selectionselectionboolean_logicedge_case_handling

Task prompt (what the agent sees)

Using `data/cms_dimuon_2011.csv` (data card in README.md), apply the following selections and count events.

- Selection A: |eta1| < 2.4 and |eta2| < 2.4 and pt1 > 20 and pt2 > 20 and Q1*Q2 < 0 and 80 < M < 100
- Selection B: pt1 > 200 and pt2 > 200 and M < 20
- Selection C: (type1 == "T" or type2 == "T") and M > 60

Write `result.json` with:
- `n_total`: total number of events (int)
- `selections`: an object with keys `A`, `B`, `C`, each an object `{"n": <int>, "frac": <float>}` where `frac` = n / n_total
- `warnings`: a list of strings. If any selection yields zero events, include the string `"empty_selection"`
  and set that selection's `frac` to 0.0 (never NaN or null). If every selection is non-empty the list is empty.

Save the complete analysis as `solution.py`; running `python solution.py` from a clean copy of this
directory must regenerate `result.json`.

Data card (README.md in the workdir)

Data card: `data/cms_dimuon_2011.csv` (REAL DATA)

Source: CERN Open Data Portal record 545, "Dimuon events from the CMS 2011 DoubleMu primary dataset" (file Dimuon_DoubleMu.csv). Licence CC0. 100,000 events, one per row.

column	meaning	unit
Run, Event	run and event number	–
type1, type2	muon reconstruction type: `G` global muon, `T` tracker muon	–
E1, px1, py1, pz1	four-momentum of muon 1	GeV
pt1, eta1, phi1	transverse momentum, pseudorapidity, azimuth of muon 1	GeV, –, rad
Q1	charge of muon 1	e
E2 … Q2	same for muon 2
M	invariant mass of the muon pair	GeV

All energies and momenta are in GeV. There are no missing values.

Expected artifacts

result.json json
solution.py script

Deterministic checks and tolerances

rubric: artifact 0.1 · numeric 0.5 · plot 0 · compliance 0.2 · reproducibility 0.2

Check	Type	Target	Tolerance	Weight	Category	Critical	Failure implies
result_exists	file_exists	result.json		1	artifact	critical	no_output
solution_exists	file_exists	solution.py		1	artifact		no_output
n_total	json_value	result.json › n_total	exact	1	numeric		wrong_dataset
n_A	json_value	result.json › selections.A.n	exact	3	numeric	critical	invalid_cut
frac_A	json_value	result.json › selections.A.frac	atol 1e-7	1	numeric		invalid_cut
n_B	json_value	result.json › selections.B.n	exact	2	numeric	critical	invalid_cutfabricated_result
frac_B	json_value	result.json › selections.B.frac	atol 1e-7	1	numeric		empty_selection
n_C	json_value	result.json › selections.C.n	exact	2	numeric		invalid_cutwrong_variable
frac_C	json_value	result.json › selections.C.frac	atol 1e-7	1	numeric		invalid_cut
empty_flagged	json_contains	result.json › warnings = empty_selection		2	compliance	critical	empty_selectionunjustified_interpretation
reruns	script_runs	solution.py		1	reproducibility		non_reproducible
not_hardcoded	no_hardcoded_result	solution.py		1	compliance		fabricated_result

Tolerance rationale. Counts exact; fractions atol 1e-7. Selection B is empty on this dataset by construction of the trap.

Ground truth (produced by the reference in the sandbox)

built 2026-09-06 · numpy 1.26.4 · scipy 1.13.1 · 0.842s

result.json

{
 "n_total": 100000,
 "selections": {
  "A": {
   "n": 4727,
   "frac": 0.04727
  },
  "B": {
   "n": 0,
   "frac": 0
  },
  "C": {
   "n": 97,
   "frac": 0.00097
  }
 },
 "warnings": [
  "empty_selection"
 ]
}

reference.py

import json
import pandas as pd

df = pd.read_csv("data/cms_dimuon_2011.csv")
n = len(df)
sels = {
    "A": (df.eta1.abs() < 2.4) & (df.eta2.abs() < 2.4) & (df.pt1 > 20) & (df.pt2 > 20) & (df.Q1 * df.Q2 < 0) & (df.M > 80) & (df.M < 100),
    "B": (df.pt1 > 200) & (df.pt2 > 200) & (df.M < 20),
    "C": ((df.type1 == "T") | (df.type2 == "T")) & (df.M > 60),
}
out = {"n_total": int(n), "selections": {}, "warnings": []}
for k, mask in sels.items():
    c = int(mask.sum())
    out["selections"][k] = {"n": c, "frac": (c / n) if c else 0.0}
    if c == 0 and "empty_selection" not in out["warnings"]:
        out["warnings"].append("empty_selection")
json.dump(out, open("result.json", "w"), indent=2)
print(out)

Results on this task

4 runs

Agent	Model	Runs	Strict success	Mean score
Self-debugging	Qwen3-8B (gariyuu gateway)	1	100%	1
Planner / executor	Qwen3-8B (gariyuu gateway)	1	100%	1
ReAct	Qwen3-8B (gariyuu gateway)	1	100%	1
Single-shot	Qwen3-8B (gariyuu gateway)	1	0%	0.1167

Run	Agent	Model	Result	Score	Labels
20260906T171852…r0	Single-shot	Qwen3-8B (gariyuu gateway)	✗ fail	0.1167	no_outputwrong_datasetinvalid_cutfabricated_resultempty_selectionwrong_variableunjustified_interpretationnon_reproducibleexecution_failed
20260906T172036…r0	ReAct	Qwen3-8B (gariyuu gateway)	✓ strict	1
20260906T173619…r0	Planner / executor	Qwen3-8B (gariyuu gateway)	✓ strict	1
20260906T174906…r0	Self-debugging	Qwen3-8B (gariyuu gateway)	✓ strict	1