Tier 4 · Signal / background · v1.0.0

Optimise a MET cut for S/sqrt(B) with cross-section normalisation

Scan a threshold on missing transverse energy, normalise weighted yields to cross-section times luminosity, and report the best cut.

t4-cut-optimizationsignal_backgroundsynth_sigbkgtrap: normalisationsignal_backgroundnormalisationweightscut_scanplotting

Task prompt (what the agent sees)

Using `data/synth_sigbkg.csv` (data card in README.md, including the normalisation rule), scan a lower
threshold t on `met` over t = 50, 75, 100, ..., 300 GeV (inclusive, step 25). For each threshold compute
the expected signal yield S(t) and background yield B(t) after the selection `met > t`, normalised to
sigma * L exactly as the data card prescribes (including the per-event `weight`), and the figure of merit
Z(t) = S(t) / sqrt(B(t)).

Deliverables:
- `result.json` with `scan` (list of objects `{"threshold": float, "S": float, "B": float, "Z": float}` in ascending
  threshold order), `best_threshold` (float, the threshold maximising Z), `S_best`, `B_best`, `Z_best` (floats),
  and `Z_no_cut` (float, S/sqrt(B) with no selection).
- `figures/scan.svg`: Z(t) versus t with an x-axis label that includes "GeV".
- `solution.py`: the complete analysis; `python solution.py` from a clean copy of this directory must regenerate every deliverable.

Data card (README.md in the workdir)

Data card: `data/synth_sigbkg.csv` (SYNTHETIC)

40,000 simulated events, 20,000 signal and 20,000 background, from a toy generator with a known seed. This is synthetic data; it is not a measurement. One row per event.

Physics observables (use these, and only these, as discriminating variables)

column	meaning	unit
met	missing transverse energy	GeV
mt	transverse mass of the lepton + MET system	GeV
n_jets	number of reconstructed jets	–
lead_jet_pt	transverse momentum of the leading jet	GeV
ht	scalar sum of jet transverse momenta	GeV
delta_phi	azimuthal angle between the leading jet and the MET vector, in [0, pi]	rad

Bookkeeping columns (NOT physics observables)

column	meaning
event_id	event number
label	truth label: 1 = signal, 0 = background
weight	per-event generator weight (mean 1 within each class); use it whenever an expected yield is computed
sample_weight	internal ntuple-merger tag: 2 for the signal sample, 1 for the background sample. It is a function of the sample, carries no physics, and must never be used as a feature

Normalisation

Expected yield of a class after a selection = sigma * L * (sum of weight over selected events of that class) / (sum of weight over all events of that class), with signal cross-section sigma_S = 0.05 pb, background cross-section sigma_B = 50 pb and integrated luminosity L = 140 fb^-1 (1 fb^-1 = 1000 pb^-1).

Expected artifacts

result.json json
figures/scan.svg svg
solution.py script

Deterministic checks and tolerances

rubric: artifact 0.1 · numeric 0.55 · plot 0.1 · compliance 0.05 · reproducibility 0.2

Check	Type	Target	Tolerance	Weight	Category	Critical	Failure implies
result_exists	file_exists	result.json		1	artifact	critical	no_output
figure_exists	file_exists	figures/scan.svg		1	artifact		no_outputplotting_error
solution_exists	file_exists	solution.py		1	artifact		no_output
scan_len	json_list_min_len	result.json › scan	≥ 11	1	numeric		spec_noncompliance
best_t	json_value	result.json › best_threshold	atol 1e-9	2	numeric	critical	incorrect_normalizationinvalid_cut
S_best	json_value	result.json › S_best	rtol 0.0001	2	numeric	critical	incorrect_normalization
B_best	json_value	result.json › B_best	rtol 0.0001	2	numeric	critical	incorrect_normalization
Z_best	json_value	result.json › Z_best	rtol 0.0001	1	numeric		incorrect_normalization
Z_no_cut	json_value	result.json › Z_no_cut	rtol 0.0001	1	numeric		incorrect_normalization
S_first	json_value	result.json › scan.0.S	rtol 0.0001	1	numeric		incorrect_normalization
B_last	json_value	result.json › scan.10.B	rtol 0.0001	1	numeric		incorrect_normalization
svg_unit	svg_text	figures/scan.svg ["GeV"]		1	plot		plotting_error
svg_drawn	svg_has_marks	figures/scan.svg	≥ 3	1	plot		plotting_error
reruns	script_runs	solution.py		1	reproducibility		non_reproducible
not_hardcoded	no_hardcoded_result	solution.py		1	compliance		fabricated_result

Tolerance rationale. Yields rtol 1e-4. Ignoring the per-event weight shifts yields by ~1e-3 relative; ignoring sigma*L shifts them by orders of magnitude; the best threshold is exact.

Ground truth (produced by the reference in the sandbox)

built 2026-09-06 · numpy 1.26.4 · scipy 1.13.1 · 1.221s

result.json

{
 "scan": [
  {
   "threshold": 50,
   "S": 6383.664148540839,
   "B": 1954399.414612395,
   "Z": 4.566288724730452
  },
  {
   "threshold": 75,
   "S": 5303.613480240965,
   "B": 1051646.1079389919,
   "Z": 5.171744441498639
  },
  {
   "threshold": 100,
   "S": 4008.105962479735,
   "B": 570601.9668501167,
   "Z": 5.306064956839125
  },
  {
   "threshold": 125,
   "S": 2788.084117029789,
   "B": 318410.707206491,
   "Z": 4.9409679942263
  },
  {
   "threshold": 150,
   "S": 1824.3464109391339,
   "B": 179135.3670226506,
   "Z": 4.310390706626355
  },
  {
   "threshold": 175,
   "S": 1149.91192462522,
   "B": 102334.71236984505,
   "Z": 3.5946208664889046
  },
  {
   "threshold": 200,
   "S": 705.2840392367898,
   "B": 58447.329996802306,
   "Z": 2.917304201947774
  },
  {
   "threshold": 225,
   "S": 396.99604950750984,
   "B": 29749.380752869736,
   "Z": 2.3016920627478483
  },
  {
   "threshold": 250,
   "S": 221.74814494562958,
   "B": 14732.2001252237,
   "Z": 1.8269480176683435
  },
  {
   "threshold": 275,
   "S": 121.500396196249,
   "B": 8549.271572668807,
   "Z": 1.3140537732996704
  },
  {
   "threshold": 300,
   "S": 68.30519132923702,
   "B": 5188.848044105208,
   "Z": 0.9482399225248805
  }
 ],
 "best_threshold": 100,
 "S_best": 4008.105962479735,
 "B_best": 570601.9668501167,
 "Z_best": 5.306064956839125,
 "Z_no_cut": 2.6457513110645907
}

reference.py

import json
import os
import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

XS_S, XS_B, LUMI_PB = 0.05, 50.0, 140.0 * 1000.0
df = pd.read_csv("data/synth_sigbkg.csv")
sig, bkg = df[df.label == 1], df[df.label == 0]
ws_tot, wb_tot = sig.weight.sum(), bkg.weight.sum()


def yields(t):
    s = XS_S * LUMI_PB * sig.loc[sig.met > t, "weight"].sum() / ws_tot
    b = XS_B * LUMI_PB * bkg.loc[bkg.met > t, "weight"].sum() / wb_tot
    return float(s), float(b)


scan = []
for t in np.arange(50, 301, 25):
    s, b = yields(float(t))
    scan.append({"threshold": float(t), "S": s, "B": b, "Z": s / np.sqrt(b)})
best = max(scan, key=lambda r: r["Z"])
s0, b0 = XS_S * LUMI_PB, XS_B * LUMI_PB
res = {"scan": scan, "best_threshold": best["threshold"], "S_best": best["S"], "B_best": best["B"], "Z_best": best["Z"], "Z_no_cut": float(s0 / np.sqrt(b0))}
json.dump(res, open("result.json", "w"), indent=2)
os.makedirs("figures", exist_ok=True)
fig, ax = plt.subplots(figsize=(7, 4.5))
ax.plot([r["threshold"] for r in scan], [r["Z"] for r in scan], marker="o")
ax.set_xlabel("MET threshold t [GeV]")
ax.set_ylabel("S / sqrt(B)")
fig.savefig("figures/scan.svg")
print({k: v for k, v in res.items() if k != "scan"})

Results on this task

4 runs

Agent	Model	Runs	Strict success	Mean score
Self-debugging	Qwen3-8B (gariyuu gateway)	1	0%	0.95
Planner / executor	Qwen3-8B (gariyuu gateway)	1	0%	0.4
ReAct	Qwen3-8B (gariyuu gateway)	1	0%	0.45
Single-shot	Qwen3-8B (gariyuu gateway)	1	0%	0.0833

Run	Agent	Model	Result	Score	Labels
20260906T171914…r0	Single-shot	Qwen3-8B (gariyuu gateway)	✗ fail	0.0833	no_outputplotting_errorspec_noncomplianceincorrect_normalizationinvalid_cutnon_reproducibleexecution_failed
20260906T172117…r0	ReAct	Qwen3-8B (gariyuu gateway)	✗ fail	0.45	spec_noncomplianceincorrect_normalizationinvalid_cutexecution_failedstep_budget_exhausted
20260906T173802…r0	Planner / executor	Qwen3-8B (gariyuu gateway)	✗ fail	0.4	incorrect_normalizationnon_reproducible
20260906T175128…r0	Self-debugging	Qwen3-8B (gariyuu gateway)	✗ core only	0.95	incorrect_normalization