Tier 5 · Statistics · v1.0.0

Excess in a window with only a handful of events

Same recipe as the significance task on a 43-event file: must flag low statistics, use the exact Poisson p-value and not over-claim.

t5-low-statsstatisticssynth_diphoton_lowstattrap: insufficient_statisticsunbinned_ml_fitsidebandshypothesis_testpoissoninterpretation

Task prompt (what the agent sees)

`data/synth_diphoton_lowstat.csv` (data card in README.md) contains a diphoton invariant mass `m_gg` per event.
Quantify a possible excess in the signal window [120, 130] GeV using the sidebands [100, 120) and (130, 180]:

1. Fit an exponential shape exp(-m / lam) to the sideband events by **unbinned maximum likelihood**, with the density
   normalised over the sideband region only (i.e. over [100,120) union (130,180]). Search lam in [5, 200].
2. Expected background in the window: B = N_sideband * I_window / I_sideband where I_R is the integral of exp(-m/lam) over region R.
3. Observed count N = number of events with 120 <= m_gg <= 130.
4. z_asimov = sqrt(2 * (N * ln(N / B) - (N - B))) with a negative sign if N < B.
5. p_value = P(X >= N) for X ~ Poisson(B) (exact, e.g. `scipy.stats.poisson.sf(N - 1, B)`), and z_from_p = the one-sided
   Gaussian significance corresponding to p_value (`scipy.stats.norm.isf(p_value)`).
6. `warnings`: include the string "low_statistics" if N < 10 or B < 10; otherwise no such entry.
7. `claim`: exactly "significant_excess" if z_from_p >= 3.0, otherwise "no_significant_excess".

Deliverables:
- `result.json` with `n_obs` (int), `n_sideband` (int), `lam` (float), `b_expected` (float), `excess` (float, N - B),
  `z_asimov` (float), `p_value` (float), `z_from_p` (float), `warnings` (list of strings), `claim` (string).
- `figures/window.svg`: histogram of m_gg on [100, 180] with the signal window indicated, x-axis label including "GeV".
- `solution.py`: the complete analysis; `python solution.py` from a clean copy of this directory must regenerate every deliverable.

Data card (README.md in the workdir)

Data card: `data/synth_diphoton_lowstat.csv` (SYNTHETIC)

A very small sample (43 events) of simulated diphoton events from a toy generator with a known seed: a smoothly falling continuum on [100, 180] GeV plus a narrow resonance. This is synthetic data; it is not a measurement.

column	meaning	unit
event_id	event number	–
m_gg	diphoton invariant mass	GeV
pt_g1, pt_g2	photon transverse momenta	GeV
eta_g1, eta_g2	photon pseudorapidities	–

No missing values, no duplicated events. Every event has 100 < m_gg < 180.

Expected artifacts

result.json json
figures/window.svg svg
solution.py script

Deterministic checks and tolerances

rubric: artifact 0.1 · numeric 0.45 · plot 0.1 · compliance 0.15 · reproducibility 0.2

Check	Type	Target	Tolerance	Weight	Category	Critical	Failure implies
result_exists	file_exists	result.json		1	artifact	critical	no_output
figure_exists	file_exists	figures/window.svg		1	artifact		no_outputplotting_error
solution_exists	file_exists	solution.py		1	artifact		no_output
n_obs	json_value	result.json › n_obs	exact	2	numeric	critical	invalid_cut
n_sb	json_value	result.json › n_sideband	exact	1	numeric		invalid_cut
lam	json_value	result.json › lam	rtol 0.03	2	numeric		statistical_misuse
b_expected	json_value	result.json › b_expected	rtol 0.03	3	numeric	critical	statistical_misuseincorrect_normalization
excess	json_value	result.json › excess	rtol 0.05	1	numeric		statistical_misuse
z_asimov	json_value	result.json › z_asimov	atol 0.15	2	numeric	critical	statistical_misuse
p_value	json_value	result.json › p_value	rtol 0.5	1	numeric		statistical_misuse
z_from_p	json_value	result.json › z_from_p	atol 0.15	2	numeric		statistical_misuse
claim	json_value	result.json › claim	exact	3	compliance	critical	unjustified_interpretation
low_stats_flag	json_contains	result.json › warnings = low_statistics		3	compliance	critical	insufficient_statisticsunjustified_interpretation
svg_unit	svg_text	figures/window.svg ["GeV"]		1	plot		plotting_error
svg_drawn	svg_has_marks	figures/window.svg	≥ 5	1	plot		plotting_error
reruns	script_runs	solution.py		1	reproducibility		non_reproducible
not_hardcoded	no_hardcoded_result	solution.py		1	compliance		fabricated_result

Tolerance rationale. The unbinned ML estimate of lam is unique, so rtol 3% catches binned/chi2 substitutes on small samples while accepting optimizer differences. Significances atol 0.15; the exact Poisson p-value rtol 0.5 because it spans orders of magnitude and z_from_p carries the precise check.

Ground truth (produced by the reference in the sandbox)

built 2026-09-06 · numpy 1.26.4 · scipy 1.13.1 · 1.532s

result.json

{
 "n_obs": 9,
 "n_sideband": 34,
 "lam": 19.112427806767826,
 "b_expected": 5.779092042410609,
 "excess": 3.2209079575893913,
 "z_asimov": 1.2376542167544085,
 "p_value": 0.13081132908317683,
 "z_from_p": 1.1225641267403295,
 "warnings": [
  "low_statistics"
 ],
 "claim": "no_significant_excess"
}

reference.py

import json
import os
import numpy as np
import pandas as pd
from scipy.optimize import minimize_scalar
from scipy.stats import norm, poisson
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

LO, HI, WLO, WHI = 100.0, 180.0, 120.0, 130.0
m = pd.read_csv("data/synth_diphoton_lowstat.csv")["m_gg"].to_numpy()
win = (m >= WLO) & (m <= WHI)
sb = m[~win]
N, Nsb = int(win.sum()), int(len(sb))


def integral(lam, a, b):
    return lam * (np.exp(-a / lam) - np.exp(-b / lam))


def I_sb(lam):
    return integral(lam, LO, WLO) + integral(lam, WHI, HI)


def nll(lam):
    return float((sb / lam).sum() + Nsb * np.log(I_sb(lam)))


lam = float(minimize_scalar(nll, bounds=(5, 200), method="bounded", options={"xatol": 1e-6}).x)
B = Nsb * integral(lam, WLO, WHI) / I_sb(lam)
z_asimov = float(np.sign(N - B) * np.sqrt(2 * (N * np.log(N / B) - (N - B)))) if N > 0 else -float(np.sqrt(2 * B))
p = float(poisson.sf(N - 1, B))
z_p = float(norm.isf(p))
warnings = ["low_statistics"] if (N < 10 or B < 10) else []
res = {"n_obs": N, "n_sideband": Nsb, "lam": lam, "b_expected": float(B), "excess": float(N - B), "z_asimov": z_asimov,
       "p_value": p, "z_from_p": z_p, "warnings": warnings, "claim": "significant_excess" if z_p >= 3.0 else "no_significant_excess"}
json.dump(res, open("result.json", "w"), indent=2)
os.makedirs("figures", exist_ok=True)
fig, ax = plt.subplots(figsize=(7, 4.5))
ax.hist(m, bins=80, range=(LO, HI), histtype="stepfilled", alpha=0.7, label="data")
ax.axvspan(WLO, WHI, color="orange", alpha=0.3, label="signal window")
ax.set_xlabel("Diphoton invariant mass m_gg [GeV]")
ax.set_ylabel("Events / 1 GeV")
ax.legend()
fig.savefig("figures/window.svg")
print(res)

Results on this task

4 runs

Agent	Model	Runs	Strict success	Mean score
Self-debugging	Qwen3-8B (gariyuu gateway)	1	0%	0.6786
Planner / executor	Qwen3-8B (gariyuu gateway)	1	0%	0.6071
ReAct	Qwen3-8B (gariyuu gateway)	1	0%	0.2583
Single-shot	Qwen3-8B (gariyuu gateway)	1	0%	0.0548

Run	Agent	Model	Result	Score	Labels
20260906T171945…r0	Single-shot	Qwen3-8B (gariyuu gateway)	✗ fail	0.0548	no_outputplotting_errorinvalid_cutstatistical_misuseincorrect_normalizationunjustified_interpretationinsufficient_statisticsnon_reproducibleexecution_failed
20260906T173315…r0	ReAct	Qwen3-8B (gariyuu gateway)	✗ fail	0.2583	no_outputplotting_errorstatistical_misuseincorrect_normalizationnon_reproduciblefabricated_resultexecution_failedsilent_exceptionunsupported_claim
20260906T174644…r0	Planner / executor	Qwen3-8B (gariyuu gateway)	✗ fail	0.6071	statistical_misuseincorrect_normalizationnon_reproducibleunsupported_claim
20260906T175322…r0	Self-debugging	Qwen3-8B (gariyuu gateway)	✗ fail	0.6786	statistical_misuseincorrect_normalization