# Why Self-Review Fails in AI Coding Agents: Single-Agent Self-Review vs. an Adversarial Critic (0% vs. 100% Mutant Kill Rate)

> 💡 **AI Labo Benchmark Series — Experiment `E10` (Verification & Adversarial Sensors)**
>
> This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite and mutation audits.
> * 🚀 **[View All 41 Benchmark Results & Master Ranking](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)**
> * 🛡️ **[Read How Our Proxmox & systemd Anti-Cheating Sandbox Works](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)**

---

## Metadata & Experiment Identity
* **Experiment ID:** `E10`
* **Run ID:** `e10-vm214-20261028-01`
* **Sandbox Environment:** Proxmox Isolated VM `VM214` (0 NICs, `virtio-serial` QGA relay, isolated grader `labjudge10`)
* **Model:** `gemini-3.8-flash`
* **Pattern Under Test:** **Single-Agent Self-Review vs. Adversarial Critic / Maker-Checker (`Maker ➔ Read-Only Locked Critic ➔ Finalizer`)**

---

## 1. Executive Summary (TL;DR)

One of the most common prompts in agent frameworks is the **Self-Review** step: after an agent finishes writing code, the harness sends one more message in the same conversation—*"Carefully review your implementation, add thorough edge-case tests, and submit when ready."*

In practice, asking an agent to review its own code in the same context triggers **Self-Preferential Confirmation Bias**: the agent inherits every assumption and blind spot from Turn 1, writes confirmatory happy-path checks (or unexecuted test code), and rubber-stamps its own work.

To test whether separating verification into an **Adversarial Critic (Maker-Checker)** fixes this blind spot, we ran a controlled A/B experiment on a 3-module **marketplace escrow & dispute reserve pipeline** (`sanitizer.py` ➔ `escrow.py` ➔ `voucher.py` + `adversarial_suite.py`) inside isolated VM `VM214`:

* **Mode A (`SelfReviewRunner`):** A single agent patches the 3 modules on Turn 1, then self-reviews its code and writes `adversarial_suite.py` in the same conversation on Turn 2 before submitting.
* **Mode B (`AdversarialMakerCheckerRunner`):** **Role 1 (`Initial Maker`)** patches the code; **Role 2 (`Adversarial Critic`)** enters with a **fresh context** whose sole objective is falsification—writing a pure-Python `adversarial_suite.py` targeting every boundary invariant in `TASK.md`, which the harness immediately locks as **Read-Only**; **Role 3 (`Maker Finalizer`)** verifies and finalizes the implementation under the locked adversarial suite.

### Key Result
While both implementations passed the 20 functional cases, **Mode A's self-reviewed audit suite crashed on execution (`ModuleNotFoundError: No module named 'pytest'`, `returncode=1`) and achieved a `0 / 6 (0.0%)` mutation kill rate**. Mode B's fresh Adversarial Critic produced a 50-line zero-dependency Python falsification suite (`returncode=0`) that **killed `6 / 6 (100.0%)` boundary mutants**—buying complete boundary verification for just **`+$0.0052` (half a cent)** and **`+6.92s`**.

| Metric | Mode A: Single-Agent Self-Review (`SelfReview`) | Mode B: Adversarial Maker-Checker (`AdversarialCritic`) | Delta / Improvement |
| :--- | :---: | :---: | :---: |
| **Hidden Grading Suite Score (20 Cases)** | `20 / 20` (100%) | **`20 / 20` (100%)** | Both implementations pass |
| **Submitted `adversarial_suite.py` Execution** | **FAILED (`returncode=1`: missing `pytest`)** | **PASSED (`returncode=0`)** | **Eliminated unverified environment assumptions** |
| **Valid Against Golden Reference Implementation** | `False` | **`True`** | **100% pure-stdlib compliance** |
| **Adversarial Mutant Kill Rate (`M1`–`M6`)** | **`0 / 6` (0.0%)** | **`6 / 6` (100.0%)** | **+100 percentage points (`0% ➔ 100%`)** |
| **LLM Calls (`model_calls`)** | `2 calls` | `3 calls` (Maker ➔ Critic ➔ Finalizer) | `+1` call (dedicated Critic pass) |
| **Cumulative Input Tokens (`promptTokenCount`)** | `6,512 tokens` | `10,289 tokens` | `+3,777` tokens |
| **Cumulative Output Tokens (`candidatesTokenCount`)** | `4,373 tokens` | `5,003 tokens` | `+630` tokens |
| **Elapsed Execution Time (`elapsed_seconds`)** | `34.05 sec` | `40.97 sec` | `+6.92 sec` |
| **Estimated API Cost (`estimated_micro_usd`)** | `$0.0213` (`21,283 micro-USD`) | **`$0.0265` (`26,479 micro-USD`)** | **`+$0.0052` (~0.5 cents for 100% mutant kill)** |

---

## 2. Experiment Architecture: Self-Review vs. Adversarial Maker-Checker

```text
+-------------------------------------------------------------------------+
| Proxmox Isolated VM214 (0 NICs, offline stdlib-only Python, labjudge10) |
|  Target Task: Marketplace Escrow & Dispute Reserve Pipeline             |
|               (sanitizer.py ➔ escrow.py ➔ voucher.py + suite)           |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: SelfReviewRunner (Same Context Reviews Its Own Homework)  |  |
|  |  - Turn 1: Patches sanitizer.py, escrow.py, voucher.py            |  |
|  |  - Turn 2 [Same History]: Self-reviews & rewrites suite using     |  |
|  |    `import pytest` (forgetting stdlib-only rule) + immediately    |  |
|  |    calls SUBMIT without re-running; skips boundary pairs          |  |
|  |  --> Suite crashes (returncode=1) | Mutant Kill Rate: 0/6 (0.0%)  |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: AdversarialMakerCheckerRunner (Fresh Critic + File Lock)  |  |
|  |  - Role 1 [Maker]:  Patches sanitizer.py, escrow.py, voucher.py   |  |
|  |  - Role 2 [Critic]: Fresh context writes 50-line pure-Python      |  |
|  |    falsification suite (100 vs 101 bps, 7999 vs 8000, bool, -0)   |  |
|  |    --> Harness locks adversarial_suite.py as READ-ONLY            |  |
|  |  - Role 3 [Finalizer]: Runs locked suite (returncode=0) & submits |  |
|  |  --> Suite green (returncode=0)   | Mutant Kill Rate: 6/6 (100%)  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
```

---

## 3. Mutation Kill Audit (`labjudge10`): `0/6` vs. `6/6`

To measure whether each mode's submitted `adversarial_suite.py` could actually catch subtle regressions, the isolated `labjudge10` grader injected **6 classic off-by-one, type-coercion, and formatting mutants (`M1`–`M6`)** into a clean reference implementation and ran both test suites against each mutant:

| Mutant ID | Injected Boundary Bug | Mode A (`SelfReview`) Suite | Mode B (`AdversarialCritic`) Suite |
| :--- | :--- | :---: | :---: |
| **`M1_dispute_bps_ge_100_instead_of_gt_100`** | Mutates `dispute_bps > 100` to `>= 100` (wrongly holding reserve at exact `100` bps) | ❌ **Survived** (crashed on `pytest`; never tested `100` vs `101` boundary) | ✅ **Killed** (asserted exact `100` bps ➔ `10000` and `101` bps ➔ `9000`) |
| **`M2_premier_rebate_strict_gt_8000`** | Mutates `subtotal >= 8000` to `> 8000` (denying rebate on exact `8,000` orders) | ❌ **Survived** (only checked `subtotal=13500`; missed `7999` vs `8000`) | ✅ **Killed** (asserted `7999` ➔ `0` and `8000` ➔ `800` / `1600`) |
| **`M3_bool_accepted_as_int_in_quantity`** | Removes `isinstance(qty, bool)` guard so `True` is accepted as `1` | ❌ **Survived** (`pytest.raises` dependency crashed in offline sandbox) | ✅ **Killed** (tested `quantity=True`, `unit_cents=False`, `dispute_bps=True`, `holdback_waiver=True`) |
| **`M4_unit_cents_zero_rejected`** | Mutates `unit_cents < 0` to `<= 0` (rejecting free `0`-cent promotional items) | ❌ **Survived** (suite crashed on import) | ✅ **Killed** (verified `unit_cents=0` passes and `-1` raises `ValueError`) |
| **`M5_conflicting_cents_or_bps_ignored`** | Silently ignores conflicting `unit_cents` or `dispute_bps` on duplicate `(merchant_id, category)` | ❌ **Survived** (conflict test completely omitted from Mode A's suite) | ✅ **Killed** (verified both price conflict and bps conflict raise `ValueError`) |
| **`M6_zero_tier_rebate_missing_minus_sign`** | Formats zero rebate as `TIER_REBATE: 0` instead of mandatory `TIER_REBATE: -0` | ❌ **Survived** (suite crashed on import) | ✅ **Killed** (asserted exact `TIER_REBATE: -0` and `WAIVER_APPLIED: -0` lines) |

---

## 4. Deep Dive: Two Failure Modes of Single-Agent Self-Review

Inspecting `self-bundle.json` and `execution-traces.json` from `VM214` exposes two structural traps that occur whenever an implementing agent reviews its own work:

### Trap 1: Unverified Last-Mile Edits & Constraint Drift
On Turn 1, Mode A spent its attention fixing the business logic across `sanitizer.py`, `escrow.py`, and `voucher.py`. On Turn 2, when asked to review its work and strengthen `adversarial_suite.py`, it forgot the system constraint that the offline sandbox only had Python's standard library (`assert` / `unittest`), imported `pytest`, and immediately emitted `"op": "SUBMIT"` in the same turn—submitting a broken test file (`returncode=1`) with complete confidence.

### Trap 2: Representative-Value Confirmation Bias
Even setting aside the `pytest` import error, inspecting the assertions inside Mode A's `adversarial_suite.py` revealed pure **confirmatory testing**:
* Instead of testing the exact boundary pair `dispute_bps=100` (no reserve) vs. `dispute_bps=101` (10% reserve), Mode A only tested `dispute_bps=101`.
* Instead of testing `subtotal=7999` vs. `subtotal=8000`, Mode A only tested a comfortable middle value (`subtotal=13500`).
* It completely omitted any test for conflicting `unit_cents` / `dispute_bps` across duplicate `(merchant_id, category)` entries (`M5`).

By contrast, Mode B's **Adversarial Critic (`role_2_adversarial_critic`)** had zero emotional or contextual attachment to the Maker's code. Given a compact 55-line budget and an explicit mandate to break flawed implementations using pure Python, it wrote a dense 50-line (`3,532`-character) falsification suite (`bad(fn, *a, **kw)` helper + exact boundary pairs) that caught all 6 mutants on the first try.

---

## 5. Practical Takeaway: Never Let the Maker Own the Test File

1. **Replace "Review your own code" prompts with a fresh Adversarial Critic context:** If a task has non-trivial boundary math (`<` vs. `<=`), type edge cases, or strict formatting contracts, spawn a separate context whose prompt rewards falsification rather than completion.
2. **Enforce Read-Only locks on Critic tests:** When handing control back to the Maker (`role_3_maker_finalizer`) to fix any failing assertions, physically block write permissions on `adversarial_suite.py` (see also **Blind TDD (`E06`)** and **Cryptographic Test Locks (`E38`)**) so the Maker cannot weaken the tests to pass.

---

### 🔗 Continue Reading the Series
* 🚀 **Master Benchmark Hub (All 41 Patterns Ranked):** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)
* 🛡️ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)
* 🗺️ **Previous Post (`E09`):** [Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster)](https://tappers.hashnode.dev/e09-dependency-map-vs-raw-grep)
* 🛡️ **Next Companion Post (`E11`):** [Why Long System Prompts Can't Enforce Code Architecture: Prompt Rules vs. Deterministic Pre-Commit AST Hooks](https://tappers.hashnode.dev/e11-architecture-guardrail-hooks)
* 🧪 **Related Verification Deep Dive (`E06`):** [Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)](https://tappers.hashnode.dev/e06-blind-tdd-worktrees)