Local significance of an excess from sidebands
Estimate the background under a mass window from an unbinned exponential fit to the sidebands and quantify the excess with an Asimov formula and an exact Poisson p-value.
t5-bump-significancestatisticssynth_diphotonunbinned_ml_fitsidebandshypothesis_testpoissoninterpretation
Task prompt (what the agent sees)
`data/synth_diphoton.csv` (data card in README.md) contains a diphoton invariant mass `m_gg` per event. Quantify a possible excess in the signal window [120, 130] GeV using the sidebands [100, 120) and (130, 180]: 1. Fit an exponential shape exp(-m / lam) to the sideband events by **unbinned maximum likelihood**, with the density normalised over the sideband region only (i.e. over [100,120) union (130,180]). Search lam in [5, 200]. 2. Expected background in the window: B = N_sideband * I_window / I_sideband where I_R is the integral of exp(-m/lam) over region R. 3. Observed count N = number of events with 120 <= m_gg <= 130. 4. z_asimov = sqrt(2 * (N * ln(N / B) - (N - B))) with a negative sign if N < B. 5. p_value = P(X >= N) for X ~ Poisson(B) (exact, e.g. `scipy.stats.poisson.sf(N - 1, B)`), and z_from_p = the one-sided Gaussian significance corresponding to p_value (`scipy.stats.norm.isf(p_value)`). 6. `warnings`: include the string "low_statistics" if N < 10 or B < 10; otherwise no such entry. 7. `claim`: exactly "significant_excess" if z_from_p >= 3.0, otherwise "no_significant_excess". Deliverables: - `result.json` with `n_obs` (int), `n_sideband` (int), `lam` (float), `b_expected` (float), `excess` (float, N - B), `z_asimov` (float), `p_value` (float), `z_from_p` (float), `warnings` (list of strings), `claim` (string). - `figures/window.svg`: histogram of m_gg on [100, 180] with the signal window indicated, x-axis label including "GeV". - `solution.py`: the complete analysis; `python solution.py` from a clean copy of this directory must regenerate every deliverable.
Data card (README.md in the workdir)
Data card: data/synth_diphoton.csv (SYNTHETIC)
Simulated diphoton events from a toy generator with a known seed: a smoothly falling continuum on [100, 180] GeV plus a narrow resonance. This is synthetic data; it is not a measurement.
| column | meaning | unit |
|---|---|---|
| event_id | event number | – |
| m_gg | diphoton invariant mass | GeV |
| pt_g1, pt_g2 | photon transverse momenta | GeV |
| eta_g1, eta_g2 | photon pseudorapidities | – |
No missing values, no duplicated events. Every event has 100 < m_gg < 180.
Expected artifacts
- result.json json
- figures/window.svg svg
- solution.py script
Deterministic checks and tolerances
| Check | Type | Target | Tolerance | Weight | Category | Critical | Failure implies |
|---|---|---|---|---|---|---|---|
| result_exists | file_exists | result.json | 1 | artifact | critical | no_output | |
| figure_exists | file_exists | figures/window.svg | 1 | artifact | no_outputplotting_error | ||
| solution_exists | file_exists | solution.py | 1 | artifact | no_output | ||
| n_obs | json_value | result.json › n_obs | exact | 2 | numeric | critical | invalid_cut |
| n_sb | json_value | result.json › n_sideband | exact | 1 | numeric | invalid_cut | |
| lam | json_value | result.json › lam | rtol 0.03 | 2 | numeric | statistical_misuse | |
| b_expected | json_value | result.json › b_expected | rtol 0.03 | 3 | numeric | critical | statistical_misuseincorrect_normalization |
| excess | json_value | result.json › excess | rtol 0.05 | 1 | numeric | statistical_misuse | |
| z_asimov | json_value | result.json › z_asimov | atol 0.15 | 2 | numeric | critical | statistical_misuse |
| p_value | json_value | result.json › p_value | rtol 0.5 | 1 | numeric | statistical_misuse | |
| z_from_p | json_value | result.json › z_from_p | atol 0.15 | 2 | numeric | statistical_misuse | |
| claim | json_value | result.json › claim | exact | 3 | compliance | critical | unjustified_interpretation |
| no_low_stats_flag | json_not_contains | result.json › warnings = low_statistics | 1 | compliance | spec_noncompliance | ||
| svg_unit | svg_text | figures/window.svg ["GeV"] | 1 | plot | plotting_error | ||
| svg_drawn | svg_has_marks | figures/window.svg | ≥ 5 | 1 | plot | plotting_error | |
| reruns | script_runs | solution.py | 1 | reproducibility | non_reproducible | ||
| not_hardcoded | no_hardcoded_result | solution.py | 1 | compliance | fabricated_result |
Tolerance rationale. The unbinned ML estimate of lam is unique, so rtol 3% catches binned/chi2 substitutes on small samples while accepting optimizer differences. Significances atol 0.15; the exact Poisson p-value rtol 0.5 because it spans orders of magnitude and z_from_p carries the precise check.
Ground truth (produced by the reference in the sandbox)
result.json
{
"n_obs": 4942,
"n_sideband": 25358,
"lam": 24.946467457251916,
"b_expected": 4629.777049959585,
"excess": 312.2229500404146,
"z_asimov": 4.538470894307572,
"p_value": 0.000002897914860991712,
"z_from_p": 4.533703776096391,
"warnings": [],
"claim": "significant_excess"
}reference.py
import json
import os
import numpy as np
import pandas as pd
from scipy.optimize import minimize_scalar
from scipy.stats import norm, poisson
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
LO, HI, WLO, WHI = 100.0, 180.0, 120.0, 130.0
m = pd.read_csv("data/synth_diphoton.csv")["m_gg"].to_numpy()
win = (m >= WLO) & (m <= WHI)
sb = m[~win]
N, Nsb = int(win.sum()), int(len(sb))
def integral(lam, a, b):
return lam * (np.exp(-a / lam) - np.exp(-b / lam))
def I_sb(lam):
return integral(lam, LO, WLO) + integral(lam, WHI, HI)
def nll(lam):
return float((sb / lam).sum() + Nsb * np.log(I_sb(lam)))
lam = float(minimize_scalar(nll, bounds=(5, 200), method="bounded", options={"xatol": 1e-6}).x)
B = Nsb * integral(lam, WLO, WHI) / I_sb(lam)
z_asimov = float(np.sign(N - B) * np.sqrt(2 * (N * np.log(N / B) - (N - B)))) if N > 0 else -float(np.sqrt(2 * B))
p = float(poisson.sf(N - 1, B))
z_p = float(norm.isf(p))
warnings = ["low_statistics"] if (N < 10 or B < 10) else []
res = {"n_obs": N, "n_sideband": Nsb, "lam": lam, "b_expected": float(B), "excess": float(N - B), "z_asimov": z_asimov,
"p_value": p, "z_from_p": z_p, "warnings": warnings, "claim": "significant_excess" if z_p >= 3.0 else "no_significant_excess"}
json.dump(res, open("result.json", "w"), indent=2)
os.makedirs("figures", exist_ok=True)
fig, ax = plt.subplots(figsize=(7, 4.5))
ax.hist(m, bins=80, range=(LO, HI), histtype="stepfilled", alpha=0.7, label="data")
ax.axvspan(WLO, WHI, color="orange", alpha=0.3, label="signal window")
ax.set_xlabel("Diphoton invariant mass m_gg [GeV]")
ax.set_ylabel("Events / 1 GeV")
ax.legend()
fig.savefig("figures/window.svg")
print(res)
Results on this task
| Agent | Model | Runs | Strict success | Mean score |
|---|---|---|---|---|
| Self-debugging | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.6464 |
| Planner / executor | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.4464 |
| ReAct | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.3564 |
| Single-shot | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.3131 |
| Run | Agent | Model | Result | Score | Labels |
|---|---|---|---|---|---|
| 20260906T171938…r0 | Single-shot | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.3131 | no_outputplotting_errorstatistical_misuseincorrect_normalizationnon_reproducibleexecution_failed |
| 20260906T172750…r0 | ReAct | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.3564 | statistical_misuseincorrect_normalizationunjustified_interpretationnon_reproducible |
| 20260906T174516…r0 | Planner / executor | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.4464 | statistical_misuseincorrect_normalizationnon_reproducible |
| 20260906T175308…r0 | Self-debugging | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.6464 | statistical_misuseincorrect_normalization |