Fit a Gaussian bump on an exponential background
Binned least-squares fit of signal + background to a diphoton mass spectrum with a fully specified model, binning and initial values.
t5-bump-fitstatisticssynth_diphotonbinned_fitcurve_fituncertaintieschi2plotting
Task prompt (what the agent sees)
`data/synth_diphoton.csv` (data card in README.md) contains a diphoton invariant mass `m_gg` per event. Fit the spectrum with the following fully specified procedure: - Histogram `m_gg` in 80 bins of width 1 GeV on [100, 180]. - Model for the expected count in a bin centred at m (bin width w = 1 GeV): f(m) = w * [ N_s * Gauss(m; mu, sigma) + N_b * Expo(m; lam) ] where Gauss is the normalised Gaussian density and Expo(m; lam) = exp(-m/lam) / (lam * (exp(-100/lam) - exp(-180/lam))) is the exponential density normalised on [100, 180]. - Minimise chi2 = sum over bins with n > 0 of (n - f)^2 / n (i.e. least squares with sigma = sqrt(n); bins with zero counts are excluded), starting from mu = 125, sigma = 2, N_s = 300, N_b = 30000, lam = 25. - Parameter uncertainties from the covariance matrix of the least-squares fit (e.g. `scipy.optimize.curve_fit` with `absolute_sigma=True`). Deliverables: - `result.json` with `mu`, `mu_err`, `sigma`, `sigma_err`, `n_sig`, `n_sig_err`, `n_bkg`, `lam` (floats), `chi2` (float), `ndf` (int, number of fitted bins minus number of parameters) and `naive_significance` = n_sig / n_sig_err (float). - `figures/fit.svg`: data points with error bars, the total fit curve and the background-only curve, legend entries containing "data" and "fit", x-axis label including "GeV". - `solution.py`: the complete analysis; `python solution.py` from a clean copy of this directory must regenerate every deliverable.
Data card (README.md in the workdir)
Data card: data/synth_diphoton.csv (SYNTHETIC)
Simulated diphoton events from a toy generator with a known seed: a smoothly falling continuum on [100, 180] GeV plus a narrow resonance. This is synthetic data; it is not a measurement.
| column | meaning | unit |
|---|---|---|
| event_id | event number | – |
| m_gg | diphoton invariant mass | GeV |
| pt_g1, pt_g2 | photon transverse momenta | GeV |
| eta_g1, eta_g2 | photon pseudorapidities | – |
No missing values, no duplicated events. Every event has 100 < m_gg < 180.
Expected artifacts
- result.json json
- figures/fit.svg svg
- solution.py script
Deterministic checks and tolerances
| Check | Type | Target | Tolerance | Weight | Category | Critical | Failure implies |
|---|---|---|---|---|---|---|---|
| result_exists | file_exists | result.json | 1 | artifact | critical | no_output | |
| figure_exists | file_exists | figures/fit.svg | 1 | artifact | no_outputplotting_error | ||
| solution_exists | file_exists | solution.py | 1 | artifact | no_output | ||
| mu | json_value | result.json › mu | atol 0.1 | 3 | numeric | critical | statistical_misuse |
| mu_err | json_value | result.json › mu_err | rtol 0.3 | 1 | numeric | statistical_misuse | |
| sigma | json_value | result.json › sigma | atol 0.2 | 2 | numeric | statistical_misuse | |
| n_sig | json_value | result.json › n_sig | rtol 0.15 | 3 | numeric | critical | statistical_misuseincorrect_normalization |
| n_sig_err | json_value | result.json › n_sig_err | rtol 0.3 | 1 | numeric | statistical_misuse | |
| n_bkg | json_value | result.json › n_bkg | rtol 0.02 | 2 | numeric | incorrect_normalization | |
| lam | json_value | result.json › lam | rtol 0.05 | 1 | numeric | statistical_misuse | |
| chi2 | json_value | result.json › chi2 | rtol 0.1 | 1 | numeric | statistical_misuse | |
| ndf | json_value | result.json › ndf | exact | 1 | numeric | statistical_misuse | |
| significance | json_value | result.json › naive_significance | rtol 0.3 | 1 | numeric | statistical_misuse | |
| svg_legend | svg_text | figures/fit.svg ["data","fit","GeV"] | 1 | plot | plotting_error | ||
| svg_drawn | svg_has_marks | figures/fit.svg | ≥ 10 | 1 | plot | plotting_error | |
| reruns | script_runs | solution.py | 1 | reproducibility | non_reproducible | ||
| not_hardcoded | no_hardcoded_result | solution.py | 1 | compliance | fabricated_result |
Tolerance rationale. The model, binning, error definition and starting values are fixed, so two correct implementations agree to well below the statistical uncertainty (sigma_mu ~ 0.1 GeV, sigma_Ns ~ 20%). Tolerances are set at roughly one statistical standard deviation: mu atol 0.1, n_sig rtol 0.15. Uncertainties rtol 0.3 (absolute_sigma vs. scaled covariance differ by sqrt(chi2/ndf)).
Ground truth (produced by the reference in the sandbox)
result.json
{
"mu": 124.68620802311865,
"mu_err": 0.28514381451768583,
"sigma": 1.3609269906702666,
"sigma_err": 0.28436457432303563,
"n_sig": 339.27845666761533,
"n_sig_err": 64.70185785333416,
"n_bkg": 29919.33850157921,
"lam": 24.889024177878664,
"chi2": 43.39539663999595,
"ndf": 75,
"naive_significance": 5.243720473014701
}reference.py
import json
import os
import numpy as np
import pandas as pd
from scipy.optimize import curve_fit
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
LO, HI, W = 100.0, 180.0, 1.0
m = pd.read_csv("data/synth_diphoton.csv")["m_gg"].to_numpy()
counts, edges = np.histogram(m, bins=80, range=(LO, HI))
x = 0.5 * (edges[:-1] + edges[1:])
def gauss(m, mu, sigma):
return np.exp(-0.5 * ((m - mu) / sigma) ** 2) / (sigma * np.sqrt(2 * np.pi))
def expo(m, lam):
return np.exp(-m / lam) / (lam * (np.exp(-LO / lam) - np.exp(-HI / lam)))
def model(m, mu, sigma, ns, nb, lam):
return W * (ns * gauss(m, mu, sigma) + nb * expo(m, lam))
sel = counts > 0
p0 = [125.0, 2.0, 300.0, 30000.0, 25.0]
popt, pcov = curve_fit(model, x[sel], counts[sel], p0=p0, sigma=np.sqrt(counts[sel]), absolute_sigma=True, maxfev=20000)
err = np.sqrt(np.diag(pcov))
chi2 = float((((counts[sel] - model(x[sel], *popt)) ** 2) / counts[sel]).sum())
ndf = int(sel.sum() - len(popt))
res = {"mu": float(popt[0]), "mu_err": float(err[0]), "sigma": float(abs(popt[1])), "sigma_err": float(err[1]),
"n_sig": float(popt[2]), "n_sig_err": float(err[2]), "n_bkg": float(popt[3]), "lam": float(popt[4]),
"chi2": chi2, "ndf": ndf, "naive_significance": float(popt[2] / err[2])}
json.dump(res, open("result.json", "w"), indent=2)
os.makedirs("figures", exist_ok=True)
fig, ax = plt.subplots(figsize=(7, 4.5))
ax.errorbar(x, counts, yerr=np.sqrt(counts), fmt="o", ms=3, label="data")
xx = np.linspace(LO, HI, 400)
ax.plot(xx, model(xx, *popt), label="fit (signal + background)")
ax.plot(xx, W * popt[3] * expo(xx, popt[4]), "--", label="background only")
ax.set_xlabel("Diphoton invariant mass m_gg [GeV]")
ax.set_ylabel("Events / 1 GeV")
ax.legend()
fig.savefig("figures/fit.svg")
print(res)
Results on this task
| Agent | Model | Runs | Strict success | Mean score |
|---|---|---|---|---|
| Self-debugging | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.8625 |
| Planner / executor | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0 |
| ReAct | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.4844 |
| Single-shot | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.4844 |
| Run | Agent | Model | Result | Score | Labels |
|---|---|---|---|---|---|
| 20260906T171938…r0 | Single-shot | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.4844 | statistical_misuseincorrect_normalization |
| 20260906T172306…r0 | ReAct | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.4844 | statistical_misuseincorrect_normalizationexecution_failedstep_budget_exhausted |
| 20260906T174058…r0 | Planner / executor | Qwen3-8B (gariyuu gateway) | ✗ fail | 0 | no_outputplotting_errorstatistical_misuseincorrect_normalizationnon_reproduciblefabricated_resultexecution_failedstep_budget_exhausted |
| 20260906T175249…r0 | Self-debugging | Qwen3-8B (gariyuu gateway) | ✗ core only | 0.8625 | statistical_misuse |