# Fail-Before-Fix in AI Coding Agents: Speculative Patching vs. a Reproduction-First Gate (-95.3% Input Tokens, 100% Accuracy)

> 💡 **AI Labo Benchmark Series — Experiment `E22` (Verification Gates & Bugfix Workflows)**
>
> This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
> * 🚀 **[View All 41 Benchmark Results & Master Ranking](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)**
> * 🛡️ **[Read How Our Proxmox & systemd Anti-Cheating Sandbox Works](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)**

---

## Metadata & Experiment Identity
* **Experiment ID:** `E22` (Backlog Candidate `C018`)
* **Run ID:** `e22-20261028-vm226-01`
* **Sandbox Environment:** Proxmox Isolated VM `VM226` (`ai-e22-226`, 0 NICs, `PrivateNetwork=yes`, `NoNewPrivileges=yes`, `InaccessiblePaths`, UID split `labexp22` / `labjudge22`)
* **Model:** `gemini-3.8-flash` (`thinkingLevel: LOW`, `temperature=0.0`)
* **Pattern Under Test:** *"SWE-agent & Claude Code Bugfix Workflow: Fail-Before-Fix Reproduction Script Gate"* — **Direct Speculative Patching (`DirectSpeculativePatchRunner`) vs. Fail-Before-Fix Reproduction Gate (`ReproductionFirstGateRunner`)**

---

## 1. Executive Summary (TL;DR)

When an AI coding agent receives a bug report and immediately has write access to production source files, its default behavior is **Direct Speculative Patching**: it guesses at a fix, edits the code, and runs the repository's existing smoke tests.

The problem? Existing test suites are almost always **weak on the exact edge cases that caused the bug in the first place**. When happy-path smoke tests return green (`PASSED_FALSE_POSITIVE`), the agent gets lured into a false sense of security or thrashing multi-turn partial edits—missing subtle boundary contracts like zero-dollar authorizations (`unit_cents == 0`) or strict input key whitelisting.

To test the **Reproduction-First Gate (Fail-Before-Fix)** pattern pioneered in SWE-agent and modern TDD harnesses, we ran a controlled A/B experiment inside isolated VM `VM226` on a 3-module **payment validation, chargeback reserve, and settlement statement pipeline** (`validator.py` ➔ `settlement.py` ➔ `statement.py`):

* **Mode A (`DirectSpeculativePatchRunner`):** Allows immediate speculative code edits against the existing weak smoke suite, producing 2 turns of noisy false-positive logs (`61,860` tool output chars) before a Turn 3 submission.
* **Mode B (`ReproductionFirstGateRunner`):** Enforces a **Fail-Before-Fix Reproduction Gate** (`REPRODUCTION_FIRST_GATE_MD`) that executes 5 minimal bug-reproducing assertions (`RED_REPRO_01`–`05`), confirms all 5 fail (`Red`) with exact expected vs. actual contract diagnostics, and only then unlocks production files for a single-shot fix.

### Key Result
Mode A stalled at **`17 / 20` (85.0%)** after 3 turns and 5 file edits—failing 3 hidden cases because `dict(item)` leaked extra keys and `cents <= 0` rejected valid `$0` authorizations. **Mode B achieved a `20 / 20` (100%) pass in a single turn**, cutting **LLM calls by `-66.7%` (`3 ➔ 1`)**, **cumulative input tokens by `-95.3%` (`49,801 ➔ 2,323` tokens—a `21.44x` reduction)**, **API cost by `-83.9%` (`$0.0480 ➔ $0.0077`)**, and running **`2.18x` faster (`27.36s ➔ 12.55s`, `-54.2%`)**.

| Metric | Mode A: Speculative Patching (`DirectSpeculativePatch`) | Mode B: Fail-Before-Fix Gate (`ReproductionFirstGate`) | Delta / Improvement |
| :--- | :---: | :---: | :---: |
| **Hidden Grading Suite Score (`labjudge22`)** | `17 / 20` (85.0% — 3 failures) | **`20 / 20` (100% — Perfect)** | **+15.0% accuracy (`+3` edge cases)** |
| **Rework Traps Hit (`gotcha_traps_hit`)** | `2 traps` (Turns 1 & 2) | **`0 traps`** | **-100% (zero rework)** |
| **LLM Calls (`model_calls`)** | `3 calls` | **`1 call` (1-shot)** | **-66.7% (`3 ➔ 1` turn)** |
| **Cumulative File Edits (`file_edits`)** | `5 edits` | **`3 edits` (1 per module)** | **-40.0% fewer edits** |
| **Cumulative Tool Output (`tool_output_chars`)** | `61,860 chars` | **`103 chars`** | **-99.8% (`600.6x` noise reduction)** |
| **Turn 1 Tokens (In / Out)** | `1,646 in` / `11 out` | **`2,323 in` / `1,596 out`** | `+677` in for 5 Red Repro assertions |
| **Turn 2 Tokens (In / Out)** | `16,223 in` / `1,244 out` | *— (completed on Turn 1)* | Avoided false-positive smoke log bloat |
| **Turn 3 Tokens (In / Out)** | `31,932 in` / `1,596 out` | *— (completed on Turn 1)* | Avoided secondary context explosion |
| **Cumulative Input Tokens (`promptTokenCount`)** | `49,801 tokens` | **`2,323 tokens`** | **-95.3% (`21.44x` efficiency)** |
| **Cumulative Output Tokens (`candidatesTokenCount`)** | `2,851 tokens` | **`1,596 tokens`** | **-44.0% output reduction** |
| **Elapsed Execution Time (`elapsed_seconds`)** | `27.36 sec` | **`12.55 sec`** | **-54.2% (`2.18x` faster)** |
| **Estimated API Cost (`estimated_micro_usd`)** | `$0.0480` (`48,043 micro-USD`) | **`$0.0077` (`7,728 micro-USD`)** | **-83.9% (`6.22x` cheaper)** |

---

## 2. Experiment Architecture: Speculative Patching vs. Reproduction-First Gate

```text
+-------------------------------------------------------------------------+
| Proxmox Isolated VM226 (0 NICs, InaccessiblePaths, grader labjudge22)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py ➔ settlement.py ➔ statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: DirectSpeculativePatchRunner (Edit First, Weak Smoke)     |  |
|  |  - Turn 1 (1,646 in):  Calls `apply_speculative_patch` ➔ 74 weak  |  |
|  |                        happy-path smoke runs pass while masking   |  |
|  |                        4 contract violations (30,943 chars log)   |  |
|  |  - Turn 2 (16,223 in): Calls `stage_unverified_fix` ➔ partial     |  |
|  |                        2-file edit leaves 2 violations (30,597c)  |  |
|  |  - Turn 3 (31,932 in): Final submit still misses `extra` key drop |  |
|  |                        & `unit_cents == 0` zero-dollar auth       |  |
|  |  --> 3 calls | 49,801 in | $0.0480 | 27.36s | 17/20 (3 FAILED)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ReproductionFirstGateRunner (Red Repro Before Unlock)     |  |
|  |  - Pre-Edit Gate: Runs 5 minimal bug-reproduction assertions      |  |
|  |    (`RED_REPRO_01`..`05`) ➔ confirms all 5 FAIL (Red) and unlocks |  |
|  |    production files with exact contract diffs (+677 input tokens) |  |
|  |  - Turn 1 (2,323 in): Submits all 3 files in 1 shot, turning all  |  |
|  |                       5 Red Repros Green on the first try         |  |
|  |  --> 1 call  | 2,323 in  | $0.0077 | 12.55s | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
```

---

## 3. Inside the Fail-Before-Fix Gate (`input-03.json`)

In Mode B, production code edits are physically locked until the harness verifies that 5 minimal reproduction assertions fail (`Red`) against the buggy codebase. Once verified, the gate injects this compact **677-token `REPRODUCTION_FIRST_GATE_MD` block**:

```markdown
# FAIL-BEFORE-FIX REPRODUCTION GATE (Verified Minimal Red-Test Suite — Unlocked for Production Fix)
The Fail-Before-Fix Reproduction Gate executed the 5 minimal bug-reproduction assertions against the current workspace and confirmed ALL 5 FAIL (Red) as expected:
1. `[RED_REPRO_01 / validator.py]`: `validate_txns([{"merchant_id": " m-1 ", "currency": " jpy ", "tx_count": 2, "unit_cents": 0, "chargeback_bps": 1000, "extra": 1}]) == [{"merchant_id": "M-1", "currency": "JPY", "tx_count": 2, "unit_cents": 0, "chargeback_bps": 1000}]` -> FAILED (`ALLOWED_CURRENCIES` must be `("USD", "EUR", "JPY")`, empty `records` list must raise `ValueError`, `isinstance(v, bool)` must be rejected before `isinstance(v, int)`, `unit_cents >= 0`, `0 <= chargeback_bps <= 1000`, and `out.append` MUST construct a brand-new 5-key dict dropping any extra keys like `"extra"` and never mutating caller dicts).
2. `[RED_REPRO_02 / settlement.py]`: `settle_batch` ordering & reserve boundary -> FAILED (Must preserve first-seen `(merchant_id, currency)` order via `for k in order:` instead of `sorted(order)`, and 10% reserve `(gross * 10) // 100` applies STRICTLY when `chargeback_bps > 300`, so at `300` bps `reserve == 0`).
3. `[RED_REPRO_03 / settlement.py]`: Tier discount & rebate cap -> FAILED (`subtotal >= 15000` is INCLUSIVE: `GROWTH` gets 10% at `>= 15000` else 0; `ENTERPRISE` gets 20% at `>= 15000` else 10%; and `rebate_applied = min(fee_rebate, after_discount)`).
4. `[RED_REPRO_04 / statement.py]`: Header delimiter -> FAILED (Must format each line as `{merchant_id}#{currency}: tx={tx_count} unit={unit_cents} bps={chargeback_bps} = {line_total}` using `#` instead of `:`).
5. `[RED_REPRO_05 / statement.py]`: Signed-zero discount/rebate lines -> FAILED (Must always format `TIER_DISCOUNT: -{batch['tier_discount']}` and `REBATE_APPLIED: -{batch['rebate_applied']}` so zero renders as `-0`).
```

---

## 4. Code Diff Analysis: Why Speculative Patching Missed 3 Hidden Cases (`17/20`)

Comparing Mode A's final Turn 3 `validator.py` (`speculative-patch-bundle.json`) against Mode B's Turn 1 `validator.py` (`reproduction-gate-bundle.json`) shows the exact 2 bugs that survived Mode A's 3-turn loop:

```python
# ❌ Mode A (DirectSpeculativePatchRunner — Turn 3 Final Submission, 17/20)
        if cnt <= 0 or cents <= 0 or not (0 <= bps <= 1000):  # <-- BUG 1: rejects unit_cents == 0 ($0 auth)!
            raise ValueError("numeric bounds violated")
        new_item = dict(item)                                 # <-- BUG 2: shallow copy leaks unknown "extra" keys!
        new_item["merchant_id"] = mid.strip().upper()
        new_item["currency"] = cur.strip().upper()
        out.append(new_item)
```

```python
# ✅ Mode B (ReproductionFirstGateRunner — Turn 1 Single-Shot Submission, 20/20)
        if cnt <= 0 or cents < 0 or not (0 <= bps <= 1000):   # <-- Allows unit_cents == 0 ($0 auth)
            raise ValueError("numeric bounds violated")
        out.append({                                          # <-- Strict 5-key dict drops "extra" keys cleanly
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
            "tx_count": cnt,
            "unit_cents": cents,
            "chargeback_bps": bps,
        })
```

Because `RED_REPRO_01` explicitly included `"unit_cents": 0` and `"extra": 1` in its failing reproduction input before a single line of production code was edited, Mode B fixed both edge cases on Turn 1—whereas Mode A drowned in `61,860` characters of noisy smoke-test output and missed `validator_01`, `validator_05`, and `e2e_03`.

---

## 5. Practical Takeaway: Lock Production Code Until a Reproduction Test Fails

1. **Enforce a mechanical Red-Before-Green gate on bugfix tasks:** Block `edit_file` / `apply_search_replace` on `src/` files until the agent (or harness) runs a minimal reproduction command that exits non-zero (`Red`) on the reported edge case.
2. **Feed the failing reproduction input/output directly into the edit turn:** A concrete 5-item failing assertion list (`+677` tokens) focuses model attention far better than 30,000 tokens of raw test runner logs.

---

### 🔗 Continue Reading the Series
* 🚀 **Master Benchmark Hub (All 41 Patterns Ranked):** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)
* 🛡️ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)
* 🥈 **Previous Sequential Post (`E21` — Rank #2):** [Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons](https://tappers.hashnode.dev/e21-treesitter-skeleton-context)
* 📦 **Next Sequential Post (`E23`):** [Stopping Edit-Test Thrashing Across Coupled Modules: Sequential Single-File Edits vs. Atomic Multi-File Batches (2.26x Faster)](https://tappers.hashnode.dev/e23-atomic-multifile-batch-edit)
* 🧪 **Related TDD Deep Dive (`E06`):** [Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)](https://tappers.hashnode.dev/e06-blind-tdd-worktrees)