Scientific error taxonomy
Every failed deterministic check carries the taxonomy labels its task declared; trajectory signals add operational labels. Counts below are over 56 real-provider runs (46 not strictly successful). A run can carry several labels.
Label × agent architecture
| Single-shot | ReAct | Planner / executor | Self-debugging | |
|---|---|---|---|---|
| Non-reproducible output | 11 | 8 | 9 | 3 |
| No output | 10 | 6 | 4 | 2 |
| Final execution failed | 10 | 6 | 4 | |
| Statistical misuse | 5 | 5 | 5 | 5 |
| Plotting error | 9 | 5 | 4 | 1 |
| Incorrect normalisation | 5 | 5 | 5 | 4 |
| Wrong variable or formula | 5 | 3 | 2 | 2 |
| Invalid selection | 3 | 5 | 3 | |
| Fabricated result | 1 | 6 | 3 | |
| Missing values mishandled | 2 | 2 | 1 | 2 |
| Unit error | 1 | 3 | 2 | 1 |
| Specification not followed | 2 | 3 | 2 | |
| Step budget exhausted | 4 | 3 | ||
| Duplicated events not removed | 2 | 1 | 1 | 2 |
| Wrong dataset / wrong rows | 2 | 1 | 1 | 1 |
| Silent exception | 1 | 2 | 1 | 1 |
| Unjustified interpretation | 2 | 1 | 1 | |
| Unsupported numerical claim | 3 | 1 | ||
| Sentinel value treated as data | 1 | 1 | 1 | |
| Label leakage | 1 | 1 | ||
| Empty selection mishandled | 1 | |||
| Insufficient statistics ignored | 1 |
Label × tier
| T1 | T2 | T3 | T4 | T5 | T6 | |
|---|---|---|---|---|---|---|
| Non-reproducible output | 2 | 3 | 7 | 5 | 7 | 7 |
| No output | 2 | 3 | 4 | 4 | 4 | 5 |
| Final execution failed | 1 | 3 | 4 | 3 | 5 | 4 |
| Statistical misuse | 4 | 12 | 4 | |||
| Plotting error | 2 | 3 | 4 | 4 | 6 | |
| Incorrect normalisation | 4 | 11 | 4 | |||
| Wrong variable or formula | 2 | 4 | 3 | 1 | 2 | |
| Invalid selection | 3 | 1 | 2 | 1 | 4 | |
| Fabricated result | 2 | 1 | 2 | 2 | 3 | |
| Missing values mishandled | 3 | 4 | ||||
| Unit error | 5 | 2 | ||||
| Specification not followed | 1 | 2 | 4 | |||
| Step budget exhausted | 2 | 1 | 2 | 2 | ||
| Duplicated events not removed | 2 | 4 | ||||
| Wrong dataset / wrong rows | 2 | 1 | 2 | |||
| Silent exception | 1 | 1 | 3 | |||
| Unjustified interpretation | 1 | 2 | 1 | |||
| Unsupported numerical claim | 1 | 1 | 2 | |||
| Sentinel value treated as data | 3 | |||||
| Label leakage | 2 | |||||
| Empty selection mishandled | 1 | |||||
| Insufficient statistics ignored | 1 |
Label × task
| t1-missing-audit | t1-schema-summary | t2-mass-histogram | t2-selection-count | t3-dimuon-kinematics | t3-invariant-mass | t3-unit-mismatch | t4-cut-optimization | t4-feature-separation | t5-bump-fit | t5-bump-significance | t5-low-stats | t6-broken-pipeline | t6-full-analysis | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Non-reproducible output | 2 | 2 | 1 | 4 | 3 | 2 | 3 | 1 | 3 | 3 | 4 | 3 | ||
| No output | 2 | 2 | 1 | 2 | 2 | 1 | 3 | 1 | 1 | 2 | 2 | 3 | ||
| Final execution failed | 1 | 2 | 1 | 1 | 3 | 2 | 1 | 2 | 1 | 2 | 1 | 3 | ||
| Statistical misuse | 4 | 4 | 4 | 4 | 4 | |||||||||
| Plotting error | 2 | 1 | 2 | 1 | 3 | 1 | 1 | 2 | 3 | 3 | ||||
| Incorrect normalisation | 4 | 3 | 4 | 4 | 4 | |||||||||
| Wrong variable or formula | 2 | 3 | 1 | 2 | 1 | 1 | 2 | |||||||
| Invalid selection | 2 | 1 | 1 | 2 | 1 | 2 | 2 | |||||||
| Fabricated result | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 2 | ||||||
| Missing values mishandled | 3 | 4 | ||||||||||||
| Unit error | 1 | 4 | 2 | |||||||||||
| Specification not followed | 1 | 2 | 1 | 3 | ||||||||||
| Step budget exhausted | 2 | 1 | 2 | 2 | ||||||||||
| Duplicated events not removed | 2 | 4 | ||||||||||||
| Wrong dataset / wrong rows | 2 | 1 | 2 | |||||||||||
| Silent exception | 1 | 1 | 3 | |||||||||||
| Unjustified interpretation | 1 | 1 | 1 | 1 | ||||||||||
| Unsupported numerical claim | 1 | 1 | 2 | |||||||||||
| Sentinel value treated as data | 3 | |||||||||||||
| Label leakage | 2 | |||||||||||||
| Empty selection mishandled | 1 | |||||||||||||
| Insufficient statistics ignored | 1 |
Scientific errors
The agent analysed the wrong file, a subset, or a corrupted read (wrong delimiter, dropped header).
row counts and file-level statistics differ from the reference
A quantity was computed from the wrong column or with the wrong formula (e.g. pz = pt*sin(eta), unwrapped delta-phi).
derived columns or summary statistics outside tolerance while row counts match
A documented unit mismatch was ignored or applied in the wrong direction.
results off by a documented conversion factor (MeV/GeV, fb/pb)
Boundary, sign, AND/OR or column mistakes in an event selection.
selection counts differ from the reference
An empty selection produced NaN, was silently dropped, or was reported without the required warning.
a zero-yield selection reported as NaN/null or not flagged as required
A column that is a function of the truth label was used as a discriminant.
a bookkeeping column that encodes the label appears among the features or as the best feature
Expected yields or fit components were not normalised as the data card prescribes.
yields or fitted event counts off by the cross-section, luminosity, weight-sum or bin-width factor
NaNs were counted, propagated or dropped contrary to the data card.
counts that depend on NaN handling differ from the reference
A documented sentinel (e.g. -999) was treated as a physical measurement.
statistics that include -999 sentinels differ from the reference
A documented merge bug producing duplicate rows was not handled.
counts and sums that require de-duplication differ from the reference
The plot is absent, unlabelled, or does not show what was asked.
figure missing, empty, or lacking required axis labels / legend entries (parsed from SVG text)
Wrong likelihood/chi2, wrong error definition, Gaussian approximation where Poisson was required, wrong ndf.
fit parameters, uncertainties, p-values or significances outside tolerance with a fully specified procedure
Asymptotic formulae or strong claims were applied to a handful of events without flagging it.
the required low_statistics warning is missing when N or B < 10
An error was hidden rather than fixed, leaving missing or partial outputs.
the last execution failed (or an exception was swallowed) and the agent still submitted; or a required figure never appeared because an error was caught and ignored
Numbers were asserted rather than computed.
result artifacts written by hand (write_file) instead of produced by executed code, or expected values hard-coded as literals in solution.py
The conclusion is not supported by the computed statistics.
the discrete claim field contradicts the rule the task fixed (e.g. 'significant_excess' below the stated threshold), or a required warning is absent
The final answer states values the analysis never produced.
numbers in the final summary that appear in no produced artifact (within 1%)
Outputs depend on an uncontrolled random state.
solution.py uses unseeded randomness and the clean rerun gives different numbers
The submitted results cannot be regenerated from the submitted script.
solution.py fails in a clean workdir, produces different artifacts, or misses deliverables
Operational failures
The episode ended without the deliverable.
a required artifact is missing or unparseable
The output exists but does not match the requested schema.
wrong key names, columns, paths or formats
Generated code tried to leave the analysis sandbox.
static policy blocked network/subprocess/ctypes/host-path access, or the OS sandbox denied a call
The submitted code does not run to completion.
the last run_python call exited with an error
The model hit its output-token limit mid-script; a capacity limit of the model/config, not a reasoning error.
a completion ended with finish_reason=length
A cell ran past the resource limit.
an execution hit the wall-clock or CPU limit
The loop did not converge.
the agent never called submit within max_steps or the episode wall-clock budget
Infrastructure, not the agent.
the model API failed repeatedly
Planted traps (task tags)
The inherited script targets a retired column schema documented in CHANGELOG.md.
task-level trap tag
An inherited script contains a wrong kinematic formula.
task-level trap tag
A column is stored in different units from the rest of the table.
task-level trap tag
A bookkeeping column encodes the truth label.
task-level trap tag
Yields require cross-section x luminosity x weight normalisation.
task-level trap tag