Tasks
Fourteen tasks in six tiers of increasing complexity. Every task has a specification, a data card, a reference implementation executed in the same sandbox as the agents, expected artifacts, numerical tolerances with a stated rationale, and a weighted rubric. Adversarial tasks plant documented traps.
| Task | Tier | Family | Data | Traps | Checks | Summary |
|---|---|---|---|---|---|---|
| t1-missing-audit | 1 · Inspection | inspection | synth_jets · synthetic | missing_valuessentinel_valueduplicated_events | 13 | Count duplicates and missing values in a synthetic jet table whose data card documents a sentinel value and a known merge bug. |
| t1-schema-summary | 1 · Inspection | inspection | cms_dimuon_2011 · real | 13 | Load a real CMS open-data CSV and report its shape, columns and basic statistics of the dimuon mass. | |
| t2-mass-histogram | 2 · Histograms & selections | histograms | cms_dimuon_2011 · real | 14 | Histogram the opposite-charge dimuon mass in a fixed binning, save the bin contents as JSON and a labelled SVG plot. | |
| t2-selection-count | 2 · Histograms & selections | histograms | cms_dimuon_2011 · real | empty_selection | 12 | Apply three kinematic selections to real CMS data and report yields; one selection is kinematically empty and must be flagged, not papered over. |
| t3-dimuon-kinematics | 3 · Derived variables | derived_variables | cms_dimuon_2011 · real | 17 | Compute pair transverse momentum, rapidity, wrapped azimuthal separation and Delta-R per event from CMS four-vectors. | |
| t3-invariant-mass | 3 · Derived variables | derived_variables | synth_zmumu · synthetic | 15 | Build four-vectors from (pt, eta, phi) and the muon mass, compute the pair mass for 50k synthetic events, and summarise the Z window. | |
| t3-unit-mismatch | 3 · Derived variables | derived_variables | synth_jets · synthetic | unit_mismatchduplicated_eventsmissing_values | 14 | Aggregate a long-format jet table to per-event quantities; requires de-duplication, NaN handling and a MeV-to-GeV conversion documented in the data card. |
| t4-cut-optimization | 4 · Signal / background | signal_background | synth_sigbkg · synthetic | normalisation | 15 | Scan a threshold on missing transverse energy, normalise weighted yields to cross-section times luminosity, and report the best cut. |
| t4-feature-separation | 4 · Signal / background | signal_background | synth_sigbkg · synthetic | label_leakage | 20 | Compute the ROC AUC of each physics observable in a labelled table, rank them, overlay the best one; a bookkeeping column leaks the label and must be excluded. |
| t5-bump-fit | 5 · Statistics | statistics | synth_diphoton · synthetic | 17 | Binned least-squares fit of signal + background to a diphoton mass spectrum with a fully specified model, binning and initial values. | |
| t5-bump-significance | 5 · Statistics | statistics | synth_diphoton · synthetic | 17 | Estimate the background under a mass window from an unbinned exponential fit to the sidebands and quantify the excess with an Asimov formula and an exact Poisson p-value. | |
| t5-low-stats | 5 · Statistics | statistics | synth_diphoton_lowstat · synthetic | insufficient_statistics | 17 | Same recipe as the significance task on a 43-event file: must flag low statistics, use the exact Poisson p-value and not over-claim. |
| t6-broken-pipeline | 6 · Multi-step & debugging | debugging | synth_zmumu · synthetic | schema_changesilent_exceptionunit_mismatchformula_bug | 15 | An existing dimuon-mass pipeline crashes and, once it runs, gives wrong numbers; the agent must diagnose four planted bugs (schema rename, wrong formula, wrong unit conversion, swallowed exception) and deliver correct artifacts. |
| t6-full-analysis | 6 · Multi-step & debugging | end_to_end | cms_dimuon_2011 · real | 20 | Quality selection, mass histogram, specified signal+background fit, efficiency bookkeeping, and a written report with stated assumptions, all in one reproducible script. |
Datasets
100,000 dimuon events from the CMS 2011 DoubleMu primary dataset (CERN Open Data record 545). Each row is one event with two reconstructed muons and their invariant mass M in GeV.
50,000 synthetic dimuon events: Z->mumu (Breit-Wigner x 1.2 GeV Gaussian resolution) on an exponential continuum. Muon kinematics only; no mass column.
Long-format jet table for 20,000 events with planted traps: NaN pt, -999 eta sentinel, duplicated rows, MeV energy column.
40,000 labelled signal/background events with six physics observables, a generator weight and a label-leaking bookkeeping column.
Diphoton mass spectrum 100-180 GeV: 30,000 exponential-background events plus a 300-event Gaussian bump at 125 GeV.
Same model as synth_diphoton with only 40 background and 3 signal events: a low-statistics trap.