# Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)

> 💡 **AI Labo Benchmark Series — Experiment `E06` (Verification & Mutation Testing)**
>
> This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
> * 🚀 **[View All 41 Benchmark Results & Master Ranking](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)**
> * 🛡️ **[Read How Our Proxmox & systemd Anti-Cheating Sandbox Works](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)**

---

## Metadata & Experiment Identity
* **Experiment ID:** `E06` (Backlog Candidate `C003`)
* **Run ID:** `e06-vm210-20261028-01`
* **Sandbox Environment:** Proxmox Isolated VM `VM210` (`ai-e06-210`, 0 NICs, `virtio-serial` QGA relay, `systemd` UID separation)
* **Model:** `gemini-3.8-flash` (`thinkingLevel: LOW`)
* **Pattern Under Test:** *"Two Agents, Two Worktrees: Blind Tester Writes the Tests, Implementer Makes Them Pass"* — **Single-Context Self-TDD vs. Two-Agent Blind Worktree TDD**

---

## 1. Executive Summary (TL;DR)

Telling a coding agent *"Use TDD—write unit tests and fix the code"* inside a single conversation has a fatal flaw: **Self-Validation Blindness**.

When the exact same LLM context sees the existing buggy code and writes both `test_suite.py` and the production modules, the model unconsciously anchors on its own implementation assumptions—writing a weak, happy-path-only test that rubber-stamps its own bugs.

To measure this effect rigorously, we benchmarked a 3-module SaaS subscription proration pipeline (`normalizer.py`, `proration.py`, `invoice.py`) inside isolated VM `VM210`, evaluating both **hidden 20-case functional accuracy** and **Mutation Testing (`M1`–`M6` boundary mutant kill rate)** on the agent-authored `test_suite.py`:

* **Mode A (`SingleContextTddRunner`):** One agent receives `TASK.md`, the 3 buggy implementation files, and `test_suite.py`, updating both tests and code together until its own `test_suite.py` passes.
* **Mode B (`BlindWorktreeTddRunner`):** Physically splits the task across two isolated workspaces:
  1. **Worktree 1 (`role_1_blind_tester`):** Sees **only `TASK.md`** (zero bytes of implementation code) and writes `test_suite.py`. The harness verifies that `test_suite.py` fails against the initial buggy code (**TDD Red Phase**), then locks `test_suite.py` read-only.
  2. **Worktree 2 (`role_2_implementer`):** Receives the locked `test_suite.py` and edits only the 3 production modules until all assertions pass (**TDD Green Phase**).

### Key Result
Mode A wrote a 28-line happy-path test that **killed `0 / 6 (0.0%)` boundary mutants**, declared 100% success, and **leaked a type-coercion bug (`19 / 20` on the hidden suite)**. Mode B's Blind Tester wrote a ruthless 52-line specification test suite that **killed `6 / 6 (100.0%)` mutants**, forcing the Implementer to pass **`20 / 20` (100%)** on the hidden grading suite for less than one extra cent (`+$0.0098`).

| Metric | Mode A: Single-Context Self-TDD (`SingleContextTDD`) | Mode B: Two-Agent Blind TDD (`BlindWorktreeTDD`) | Delta / Improvement |
| :--- | :---: | :---: | :---: |
| **Hidden Grading Suite Score (20 Cases)** | `19 / 20` (95.0% — **❌ leaked bug**) | **`20 / 20` (100% Perfect Pass)** | **+5.0 pt (Zero leaked defects)** |
| **Test Suite Mutant Kill Rate (6 Mutants)** | **`0 / 6` (0.0% — all 6 survived!)** | **`6 / 6` (100.0% — all 6 killed!)** | **+100.0 pt mutation coverage** |
| **TDD Red Phase Verified on Initial Bugs** | No (simultaneous edit) | **Yes (`red_phase_caught_initial_bugs: true`)** | True Red ➔ Green enforcement |
| **Agent's Self-Reported Test Status** | `PASS` *(False Positive!)* | **`PASS` *(Verified True Positive)*** | Eliminated rubber-stamp green |
| **LLM Calls (`model_calls`)** | `1 call` | **`2 calls` (1 Blind Tester + 1 Implementer)** | +1 call for independent verification |
| **Cumulative Input Tokens (`promptTokenCount`)** | `2,035 tokens` | **`7,493 tokens` (`2,448` + `5,045`)** | Lean 2-worktree isolation |
| **Elapsed Execution Time (`elapsed_seconds`)** | `14.70 sec` | **`25.61 sec`** | +10.91 sec for 100% defect-free code |
| **Estimated API Cost (`estimated_micro_usd`)** | `$0.0080` (`7,965 micro-USD`) | **`$0.0178` (`17,835 micro-USD`)** | **+$0.0098 (< 1 cent) for 100% kill rate** |

---

## 2. Architecture Comparison: Self-TDD Collusion vs. Blind Worktree Isolation

```text
+-------------------------------------------------------------------------+
| Proxmox Isolated VM210 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Subscription Proration Pipeline                  |
|               (normalizer.py ➔ proration.py ➔ invoice.py)               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: SingleContextTddRunner (Same Agent Writes Tests & Code)   |  |
|  |  - Sees existing buggy `str(item.get("account_id"))`              |  |
|  |  - Writes 1 happy-path test + misses non-string rejection bug     |  |
|  |  - Self-test: PASS! ➔ Hidden Judge: 19/20 FAIL | Mutants: 0/6     |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: BlindWorktreeTddRunner (2 Worktrees, Physically Split)    |  |
|  |  - Worktree 1 [Blind Tester]: Sees ONLY TASK.md (0 impl files!)   |  |
|  |      Writes 52-line adversarial `test_suite.py`                   |  |
|  |      Red Phase Check: FAILS on initial buggy code (Verified!)     |  |
|  |  - Worktree 2 [Implementer]:  `test_suite.py` LOCKED read-only;   |  |
|  |      Patches normalizer/proration/invoice until Green             |  |
|  |  - Hidden Judge: 20/20 PASS (100%) | Mutants Killed: 6/6 (100%)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
```

---

## 3. Code-Level Postmortem: Implementation Anchoring Caught on Tape

Comparing `single-bundle.json` (Mode A) and `blind-bundle.json` (Mode B) demonstrates **Implementation Anchoring** in action.

In the initial buggy `normalizer.py`, account IDs were coerced using `str(...)`:
```python
# Initial buggy fixture in normalizer.py:
item["account_id"] = str(item.get("account_id", "")).strip()
```

Because **Mode A** saw that line in its prompt, it carried the `str(...)` coercion habit straight into its "fixed" implementation—silently converting integer `account_id: 999` into `"999"` instead of raising `ValueError` as required by `TASK.md`:

```python
# ❌ Mode A (`SingleContextTDD`): Anchored on str() coercion & wrote 0 negative tests!
acc = str(item.get("account_id", "")).strip().upper()
```

Worse still, Mode A's `test_suite.py` tested only a single valid string `"acc-1"` with a `SAVE10` coupon—never testing non-string IDs, `bool` values, `9,999` vs. `10,000` discount boundaries, or `DISCOUNT: -0`.

In contrast, **Mode B's Blind Tester** never saw the `str(...)` bug. Reading `TASK.md` cold, it immediately built a table-driven adversarial helper `bad(fn, *a, **k)` probing booleans (`[True, False, 1.5]`), integer account IDs, `days_active` bounds (`0` and `31`), `unit_price == 0` free trials, exact `9,999` vs. `10,000` thresholds, compounding `SAVE10`/`VIP20` order, and `DISCOUNT: -0`—forcing Worktree 2's Implementer to enforce strict `isinstance(account_id, str)` checks.

---

## 4. Mutation Testing Audit (`M1`–`M6`) & 20-Case Grading Breakdown

Our hidden evaluator (`labjudge06`) injected 6 subtle boundary bugs (**Mutants `M1`–`M6`**) into a reference implementation to measure whether each agent's `test_suite.py` could actually catch regressions:

| Mutant ID | Injected Edge-Case Bug | Mode A `test_suite.py` (`Single TDD`) | Mode B `test_suite.py` (`Blind TDD`) |
| :--- | :--- | :---: | :---: |
| **`M1`** | Off-by-one volume discount (`> 10000` instead of `>= 10000`) | ❌ **Survived** *(only tested 9,000)* | ✅ **Killed** *(tested 9,999 & 10,000)* |
| **`M2`** | Coupon applied to `gross_subtotal` instead of `after_volume` | ❌ **Survived** | ✅ **Killed** *(verified 10,000 + `SAVE10` = 900)* |
| **`M3`** | Python `bool` (`seats=True`) accepted as integer `1` | ❌ **Survived** | ✅ **Killed** *(tested `[True, False, 1.5]`)* |
| **`M4`** | Free trial (`unit_price == 0`) erroneously rejected | ❌ **Survived** | ✅ **Killed** *(verified `unit_price: 0`)* |
| **`M5`** | Upper bound `days_active == 31` (> 30 max) accepted | ❌ **Survived** | ✅ **Killed** *(verified `0` and `31` raise `ValueError`)* |
| **`M6`** | Zero discount formatted as `DISCOUNT: 0` instead of `-0` | ❌ **Survived** *(only tested `-1400`)* | ✅ **Killed** *(explicitly asserted `DISCOUNT: -0`)* |
| **Total** | **Mutation Kill Score** | **`0 / 6` (0.0%)** | **`6 / 6` (100.0%)** |

On the **20 hidden grading cases**, Mode A failed `normalizer_03_blank_account_or_unknown_plan_rejected` (`19 / 20`), while Mode B passed all **`20 / 20` (100%)**.

---

## 5. Practical Takeaway: Fold Implementation (`E05`), Isolate Verification (`E06`)

Together, **`E05`** and **`E06`** resolve the biggest debate in multi-agent engineering:
* **Never split implementation roles** (`Architect ➔ Coder A ➔ Coder B`) via lossy text summaries (`E05`).
* **Always isolate the Test Author** into a **Blind Worktree** that sees *only* the specification (`SPEC.md` / issue contract) and zero lines of implementation (`E06`), then lock the resulting test files read-only before launching your folded implementation loop.

---

### 🔗 Continue Reading the Series
* 🚀 **Master Benchmark Hub (All 41 Patterns Ranked):** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)
* 🛡️ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)
* 🪆 **Previous (`E05`):** [Why Role-Based Subagents Slow Down Coding Agents: Subagent Folding + Progressive Skill Disclosure (2.66x Faster)](https://tappers.hashnode.dev/e05-subagent-folding)
* ⚙️ **Next (`E07`):** [Eliminating Goal Drift in Multi-File Coding Agents: In-Chat Loops vs. Code-Mode Dynamic Workflows](https://tappers.hashnode.dev/e07-code-mode-workflows)