Data-quality audit of a jet table
Count duplicates and missing values in a synthetic jet table whose data card documents a sentinel value and a known merge bug.
t1-missing-auditinspectionsynth_jetstrap: missing_valuestrap: sentinel_valuetrap: duplicated_eventsload_csvread_data_cardmissing_valuesdeduplication
Task prompt (what the agent sees)
Audit the data quality of `data/synth_jets.csv` (one row per jet). Read `README.md` first: it is the data card and documents the file's conventions. Write `result.json` with exactly these keys: - `n_rows`: number of rows in the raw file (int) - `n_duplicate_rows`: number of rows that are exact repeats of an earlier row (int) - `n_unique_jets`: number of distinct jets, identified as described in the data card (int) - `n_events`: number of distinct events (int) - `n_missing_pt`: number of distinct jets whose transverse momentum is missing (int) - `n_missing_eta`: number of distinct jets whose pseudorapidity is missing, following the data card's convention (int) - `columns_with_missing`: sorted list of column names that contain any missing value (including the documented sentinel) - `max_jets_per_event`: maximum number of distinct jets in one event (int) - `frac_jets_missing_pt`: n_missing_pt / n_unique_jets (float) Remove duplicate rows (keeping the first occurrence) before computing every quantity except `n_rows` and `n_duplicate_rows`. Save the complete analysis as `solution.py`; running `python solution.py` from a clean copy of this directory must regenerate `result.json`.
Data card (README.md in the workdir)
Data card: data/synth_jets.csv (SYNTHETIC)
Simulated jet table for 20,000 events, one row per jet, produced by a toy generator with a known seed. This is synthetic data; it is not a measurement.
| column | meaning | unit |
|---|---|---|
| event_id | event number | – |
| jet_index | index of the jet within its event, 0 = leading (highest-pt) jet | – |
| jet_pt | transverse momentum. NaN when jet reconstruction failed. | GeV |
| jet_eta | pseudorapidity. The value -999.0 is a sentinel meaning "not measured". | – |
| jet_phi | azimuthal angle | rad |
| jet_e_mev | jet energy. Note the unit: MeV, not GeV. | MeV |
| btag_score | b-tagging discriminant in [0, 1] | – |
Known issues
- A jet is uniquely identified by the pair
(event_id, jet_index). - Because of a bug in the ntuple merger, some rows are exact duplicates. Duplicates must be removed (keep the first occurrence) before any physics quantity is computed.
- Jets with missing
jet_ptfailed reconstruction and must be excluded from any sum over jet pt. Jets with the eta sentinel are otherwise valid (their pt and energy are measured).
Expected artifacts
- result.json json
- solution.py script
Deterministic checks and tolerances
| Check | Type | Target | Tolerance | Weight | Category | Critical | Failure implies |
|---|---|---|---|---|---|---|---|
| result_exists | file_exists | result.json | 1 | artifact | critical | no_output | |
| solution_exists | file_exists | solution.py | 1 | artifact | no_output | ||
| n_rows | json_value | result.json › n_rows | exact | 1 | numeric | wrong_dataset | |
| n_dup | json_value | result.json › n_duplicate_rows | exact | 2 | numeric | critical | duplicated_events |
| n_unique | json_value | result.json › n_unique_jets | exact | 2 | numeric | critical | duplicated_events |
| n_events | json_value | result.json › n_events | exact | 1 | numeric | wrong_variable | |
| n_missing_pt | json_value | result.json › n_missing_pt | exact | 2 | numeric | missing_values | |
| n_missing_eta | json_value | result.json › n_missing_eta | exact | 3 | numeric | critical | missing_valuessentinel_value |
| cols_missing | json_list_equal | result.json › columns_with_missing | exact | 2 | numeric | missing_valuessentinel_value | |
| max_jets | json_value | result.json › max_jets_per_event | exact | 1 | numeric | duplicated_events | |
| frac_missing | json_value | result.json › frac_jets_missing_pt | atol 0.000001 | 1 | numeric | missing_values | |
| reruns | script_runs | solution.py | 1 | reproducibility | non_reproducible | ||
| not_hardcoded | no_hardcoded_result | solution.py | 1 | compliance | fabricated_result |
Tolerance rationale. All counts are exact integers; the fraction uses atol 1e-6.
Ground truth (produced by the reference in the sandbox)
result.json
{
"n_rows": 65779,
"n_duplicate_rows": 1915,
"n_unique_jets": 63864,
"n_events": 20000,
"n_missing_pt": 1277,
"n_missing_eta": 957,
"columns_with_missing": [
"jet_eta",
"jet_pt"
],
"max_jets_per_event": 6,
"frac_jets_missing_pt": 0.01999561568332707
}reference.py
import json
import numpy as np
import pandas as pd
raw = pd.read_csv("data/synth_jets.csv")
n_rows = len(raw)
dup_mask = raw.duplicated(keep="first")
df = raw[~dup_mask].copy()
eta_missing = df["jet_eta"] == -999.0
missing_cols = sorted([c for c in df.columns if df[c].isna().any()] + (["jet_eta"] if eta_missing.any() else []))
result = {
"n_rows": int(n_rows),
"n_duplicate_rows": int(dup_mask.sum()),
"n_unique_jets": int(len(df.drop_duplicates(["event_id", "jet_index"]))),
"n_events": int(df["event_id"].nunique()),
"n_missing_pt": int(df["jet_pt"].isna().sum()),
"n_missing_eta": int(eta_missing.sum()),
"columns_with_missing": sorted(set(missing_cols)),
"max_jets_per_event": int(df.groupby("event_id")["jet_index"].nunique().max()),
}
result["frac_jets_missing_pt"] = result["n_missing_pt"] / result["n_unique_jets"]
json.dump(result, open("result.json", "w"), indent=2)
print(result)
Results on this task
| Agent | Model | Runs | Strict success | Mean score |
|---|---|---|---|---|
| Self-debugging | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.35 |
| Planner / executor | Qwen3-8B (gariyuu gateway) | 1 | 100% | 1 |
| ReAct | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.92 |
| Single-shot | Qwen3-8B (gariyuu gateway) | 1 | 0% | 0.15 |
| Run | Agent | Model | Result | Score | Labels |
|---|---|---|---|---|---|
| 20260906T171852…r0 | Single-shot | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.15 | no_outputwrong_datasetduplicated_eventswrong_variablemissing_valuessentinel_valuenon_reproducibleexecution_failed |
| 20260906T172011…r0 | ReAct | Qwen3-8B (gariyuu gateway) | ✗ core only | 0.92 | missing_valuessentinel_value |
| 20260906T173433…r0 | Planner / executor | Qwen3-8B (gariyuu gateway) | ✓ strict | 1 | |
| 20260906T174829…r0 | Self-debugging | Qwen3-8B (gariyuu gateway) | ✗ fail | 0.35 | no_outputwrong_datasetduplicated_eventswrong_variablemissing_valuessentinel_valuenon_reproducible |