Fail-Before-Fix in AI Coding Agents: Speculative Patching vs. a Reproduction-First Gate (-95.3% Input Tokens, 100% Accuracy)

💡 AI Labo Benchmark Series — Experiment E22 (Verification Gates & Bugfix Workflows)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

---

Metadata & Experiment Identity

• Experiment ID: E22 (Backlog Candidate C018)
• Run ID: e22-20261028-vm226-01
• Sandbox Environment: Proxmox Isolated VM VM226 (ai-e22-226, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp22 / labjudge22)
• Model: gemini-3.8-flash (thinkingLevel: LOW, temperature=0.0)
• Pattern Under Test: "SWE-agent & Claude Code Bugfix Workflow: Fail-Before-Fix Reproduction Script Gate" — Direct Speculative Patching (DirectSpeculativePatchRunner) vs. Fail-Before-Fix Reproduction Gate (ReproductionFirstGateRunner)

---

1. Executive Summary (TL;DR)

When an AI coding agent receives a bug report and immediately has write access to production source files, its default behavior is Direct Speculative Patching: it guesses at a fix, edits the code, and runs the repository's existing smoke tests.

The problem? Existing test suites are almost always weak on the exact edge cases that caused the bug in the first place. When happy-path smoke tests return green (PASSED_FALSE_POSITIVE), the agent gets lured into a false sense of security or thrashing multi-turn partial edits—missing subtle boundary contracts like zero-dollar authorizations (unit_cents == 0) or strict input key whitelisting.

To test the Reproduction-First Gate (Fail-Before-Fix) pattern pioneered in SWE-agent and modern TDD harnesses, we ran a controlled A/B experiment inside isolated VM VM226 on a 3-module payment validation, chargeback reserve, and settlement statement pipeline (validator.py ➔ settlement.py ➔ statement.py):

• Mode A (DirectSpeculativePatchRunner): Allows immediate speculative code edits against the existing weak smoke suite, producing 2 turns of noisy false-positive logs (61,860 tool output chars) before a Turn 3 submission.
• Mode B (ReproductionFirstGateRunner): Enforces a Fail-Before-Fix Reproduction Gate (REPRODUCTION_FIRST_GATE_MD) that executes 5 minimal bug-reproducing assertions (RED_REPRO_01–05), confirms all 5 fail (Red) with exact expected vs. actual contract diagnostics, and only then unlocks production files for a single-shot fix.

Key Result

Mode A stalled at 17 / 20 (85.0%) after 3 turns and 5 file edits—failing 3 hidden cases because dict(item) leaked extra keys and cents <= 0 rejected valid $0 authorizations. Mode B achieved a 20 / 20 (100%) pass in a single turn, cutting LLM calls by -66.7% (3 ➔ 1), cumulative input tokens by -95.3% (49,801 ➔ 2,323 tokens—a 21.44x reduction), API cost by -83.9% ($0.0480 ➔ $0.0077), and running 2.18x faster (27.36s ➔ 12.55s, -54.2%).

Metric	Mode A: Speculative Patching (DirectSpeculativePatch)	Mode B: Fail-Before-Fix Gate (ReproductionFirstGate)	Delta / Improvement
Hidden Grading Suite Score (labjudge22)	17 / 20 (85.0% — 3 failures)	20 / 20 (100% — Perfect)	+15.0% accuracy (+3 edge cases)
Rework Traps Hit (gotcha_traps_hit)	2 traps (Turns 1 & 2)	0 traps	-100% (zero rework)
LLM Calls (model_calls)	3 calls	1 call (1-shot)	-66.7% (3 ➔ 1 turn)
Cumulative File Edits (file_edits)	5 edits	3 edits (1 per module)	-40.0% fewer edits
Cumulative Tool Output (tool_output_chars)	61,860 chars	103 chars	-99.8% (600.6x noise reduction)
Turn 1 Tokens (In / Out)	1,646 in / 11 out	2,323 in / 1,596 out	+677 in for 5 Red Repro assertions
Turn 2 Tokens (In / Out)	16,223 in / 1,244 out	— (completed on Turn 1)	Avoided false-positive smoke log bloat
Turn 3 Tokens (In / Out)	31,932 in / 1,596 out	— (completed on Turn 1)	Avoided secondary context explosion
Cumulative Input Tokens (promptTokenCount)	49,801 tokens	2,323 tokens	-95.3% (21.44x efficiency)
Cumulative Output Tokens (candidatesTokenCount)	2,851 tokens	1,596 tokens	-44.0% output reduction
Elapsed Execution Time (elapsed_seconds)	27.36 sec	12.55 sec	-54.2% (2.18x faster)
Estimated API Cost (estimated_micro_usd)	$0.0480 (48,043 micro-USD)	$0.0077 (7,728 micro-USD)	-83.9% (6.22x cheaper)

---

2. Experiment Architecture: Speculative Patching vs. Reproduction-First Gate

+-------------------------------------------------------------------------+
| Proxmox Isolated VM226 (0 NICs, InaccessiblePaths, grader labjudge22)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py ➔ settlement.py ➔ statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: DirectSpeculativePatchRunner (Edit First, Weak Smoke)     |  |
|  |  - Turn 1 (1,646 in):  Calls `apply_speculative_patch` ➔ 74 weak  |  |
|  |                        happy-path smoke runs pass while masking   |  |
|  |                        4 contract violations (30,943 chars log)   |  |
|  |  - Turn 2 (16,223 in): Calls `stage_unverified_fix` ➔ partial     |  |
|  |                        2-file edit leaves 2 violations (30,597c)  |  |
|  |  - Turn 3 (31,932 in): Final submit still misses `extra` key drop |  |
|  |                        & `unit_cents == 0` zero-dollar auth       |  |
|  |  --> 3 calls | 49,801 in | $0.0480 | 27.36s | 17/20 (3 FAILED)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ReproductionFirstGateRunner (Red Repro Before Unlock)     |  |
|  |  - Pre-Edit Gate: Runs 5 minimal bug-reproduction assertions      |  |
|  |    (`RED_REPRO_01`..`05`) ➔ confirms all 5 FAIL (Red) and unlocks |  |
|  |    production files with exact contract diffs (+677 input tokens) |  |
|  |  - Turn 1 (2,323 in): Submits all 3 files in 1 shot, turning all  |  |
|  |                       5 Red Repros Green on the first try         |  |
|  |  --> 1 call  | 2,323 in  | $0.0077 | 12.55s | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

---

3. Inside the Fail-Before-Fix Gate (input-03.json)

In Mode B, production code edits are physically locked until the harness verifies that 5 minimal reproduction assertions fail (Red) against the buggy codebase. Once verified, the gate injects this compact 677-token REPRODUCTION_FIRST_GATE_MD block:

# FAIL-BEFORE-FIX REPRODUCTION GATE (Verified Minimal Red-Test Suite — Unlocked for Production Fix)
The Fail-Before-Fix Reproduction Gate executed the 5 minimal bug-reproduction assertions against the current workspace and confirmed ALL 5 FAIL (Red) as expected:
1. `[RED_REPRO_01 / validator.py]`: `validate_txns([{"merchant_id": " m-1 ", "currency": " jpy ", "tx_count": 2, "unit_cents": 0, "chargeback_bps": 1000, "extra": 1}]) == [{"merchant_id": "M-1", "currency": "JPY", "tx_count": 2, "unit_cents": 0, "chargeback_bps": 1000}]` -> FAILED (`ALLOWED_CURRENCIES` must be `("USD", "EUR", "JPY")`, empty `records` list must raise `ValueError`, `isinstance(v, bool)` must be rejected before `isinstance(v, int)`, `unit_cents >= 0`, `0 <= chargeback_bps <= 1000`, and `out.append` MUST construct a brand-new 5-key dict dropping any extra keys like `"extra"` and never mutating caller dicts).
2. `[RED_REPRO_02 / settlement.py]`: `settle_batch` ordering & reserve boundary -> FAILED (Must preserve first-seen `(merchant_id, currency)` order via `for k in order:` instead of `sorted(order)`, and 10% reserve `(gross * 10) // 100` applies STRICTLY when `chargeback_bps > 300`, so at `300` bps `reserve == 0`).
3. `[RED_REPRO_03 / settlement.py]`: Tier discount & rebate cap -> FAILED (`subtotal >= 15000` is INCLUSIVE: `GROWTH` gets 10% at `>= 15000` else 0; `ENTERPRISE` gets 20% at `>= 15000` else 10%; and `rebate_applied = min(fee_rebate, after_discount)`).
4. `[RED_REPRO_04 / statement.py]`: Header delimiter -> FAILED (Must format each line as `{merchant_id}#{currency}: tx={tx_count} unit={unit_cents} bps={chargeback_bps} = {line_total}` using `#` instead of `:`).
5. `[RED_REPRO_05 / statement.py]`: Signed-zero discount/rebate lines -> FAILED (Must always format `TIER_DISCOUNT: -{batch['tier_discount']}` and `REBATE_APPLIED: -{batch['rebate_applied']}` so zero renders as `-0`).

---

4. Code Diff Analysis: Why Speculative Patching Missed 3 Hidden Cases (17/20)

Comparing Mode A's final Turn 3 validator.py (speculative-patch-bundle.json) against Mode B's Turn 1 validator.py (reproduction-gate-bundle.json) shows the exact 2 bugs that survived Mode A's 3-turn loop:

# ❌ Mode A (DirectSpeculativePatchRunner — Turn 3 Final Submission, 17/20)
        if cnt <= 0 or cents <= 0 or not (0 <= bps <= 1000):  # <-- BUG 1: rejects unit_cents == 0 ($0 auth)!
            raise ValueError("numeric bounds violated")
        new_item = dict(item)                                 # <-- BUG 2: shallow copy leaks unknown "extra" keys!
        new_item["merchant_id"] = mid.strip().upper()
        new_item["currency"] = cur.strip().upper()
        out.append(new_item)

# ✅ Mode B (ReproductionFirstGateRunner — Turn 1 Single-Shot Submission, 20/20)
        if cnt <= 0 or cents < 0 or not (0 <= bps <= 1000):   # <-- Allows unit_cents == 0 ($0 auth)
            raise ValueError("numeric bounds violated")
        out.append({                                          # <-- Strict 5-key dict drops "extra" keys cleanly
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
            "tx_count": cnt,
            "unit_cents": cents,
            "chargeback_bps": bps,
        })

Because RED_REPRO_01 explicitly included "unit_cents": 0 and "extra": 1 in its failing reproduction input before a single line of production code was edited, Mode B fixed both edge cases on Turn 1—whereas Mode A drowned in 61,860 characters of noisy smoke-test output and missed validator_01, validator_05, and e2e_03.

---

5. Practical Takeaway: Lock Production Code Until a Reproduction Test Fails

1. Enforce a mechanical Red-Before-Green gate on bugfix tasks: Block edit_file / apply_search_replace on src/ files until the agent (or harness) runs a minimal reproduction command that exits non-zero (Red) on the reported edge case.
2. Feed the failing reproduction input/output directly into the edit turn: A concrete 5-item failing assertion list (+677 tokens) focuses model attention far better than 30,000 tokens of raw test runner logs.

---

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥈 Previous Sequential Post (E21 — Rank #2): Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons
• 📦 Next Sequential Post (E23): Stopping Edit-Test Thrashing Across Coupled Modules: Sequential Single-File Edits vs. Atomic Multi-File Batches (2.26x Faster)
• 🧪 Related TDD Deep Dive (E06): Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)