Skip to main content
Hashnode
Tappers Tech Blog

Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)

Updated 8 min readMarkdown
T
tappers

💡 AI Labo Benchmark Series — Experiment E06 (Verification & Mutation Testing)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

Metadata & Experiment Identity

  • Experiment ID: E06 (Backlog Candidate C003)
  • Run ID: e06-vm210-20261028-01
  • Sandbox Environment: Proxmox Isolated VM VM210 (ai-e06-210, 0 NICs, virtio-serial QGA relay, systemd UID separation)
  • Model: gemini-3.8-flash (thinkingLevel: LOW)
  • Pattern Under Test: "Two Agents, Two Worktrees: Blind Tester Writes the Tests, Implementer Makes Them Pass" — Single-Context Self-TDD vs. Two-Agent Blind Worktree TDD

1. Executive Summary (TL;DR)

Telling a coding agent "Use TDD—write unit tests and fix the code" inside a single conversation has a fatal flaw: Self-Validation Blindness.

When the exact same LLM context sees the existing buggy code and writes both test_suite.py and the production modules, the model unconsciously anchors on its own implementation assumptions—writing a weak, happy-path-only test that rubber-stamps its own bugs.

To measure this effect rigorously, we benchmarked a 3-module SaaS subscription proration pipeline (normalizer.py, proration.py, invoice.py) inside isolated VM VM210, evaluating both hidden 20-case functional accuracy and Mutation Testing (M1–M6 boundary mutant kill rate) on the agent-authored test_suite.py:

  • Mode A (SingleContextTddRunner): One agent receives TASK.md, the 3 buggy implementation files, and test_suite.py, updating both tests and code together until its own test_suite.py passes.
  • Mode B (BlindWorktreeTddRunner): Physically splits the task across two isolated workspaces:
    1. Worktree 1 (role_1_blind_tester): Sees only TASK.md (zero bytes of implementation code) and writes test_suite.py. The harness verifies that test_suite.py fails against the initial buggy code (TDD Red Phase), then locks test_suite.py read-only.
    2. Worktree 2 (role_2_implementer): Receives the locked test_suite.py and edits only the 3 production modules until all assertions pass (TDD Green Phase).

Key Result

Mode A wrote a 28-line happy-path test that killed 0 / 6 (0.0%) boundary mutants, declared 100% success, and leaked a type-coercion bug (19 / 20 on the hidden suite). Mode B's Blind Tester wrote a ruthless 52-line specification test suite that killed 6 / 6 (100.0%) mutants, forcing the Implementer to pass 20 / 20 (100%) on the hidden grading suite for less than one extra cent (+$0.0098).

Metric Mode A: Single-Context Self-TDD (SingleContextTDD) Mode B: Two-Agent Blind TDD (BlindWorktreeTDD) Delta / Improvement
Hidden Grading Suite Score (20 Cases) 19 / 20 (95.0% — ❌ leaked bug) 20 / 20 (100% Perfect Pass) +5.0 pt (Zero leaked defects)
Test Suite Mutant Kill Rate (6 Mutants) 0 / 6 (0.0% — all 6 survived!) 6 / 6 (100.0% — all 6 killed!) +100.0 pt mutation coverage
TDD Red Phase Verified on Initial Bugs No (simultaneous edit) Yes (red_phase_caught_initial_bugs: true) True Red ➔ Green enforcement
Agent's Self-Reported Test Status PASS (False Positive!) PASS (Verified True Positive) Eliminated rubber-stamp green
LLM Calls (model_calls) 1 call 2 calls (1 Blind Tester + 1 Implementer) +1 call for independent verification
Cumulative Input Tokens (promptTokenCount) 2,035 tokens 7,493 tokens (2,448 + 5,045) Lean 2-worktree isolation
Elapsed Execution Time (elapsed_seconds) 14.70 sec 25.61 sec +10.91 sec for 100% defect-free code
Estimated API Cost (estimated_micro_usd) $0.0080 (7,965 micro-USD) $0.0178 (17,835 micro-USD) +$0.0098 (< 1 cent) for 100% kill rate

2. Architecture Comparison: Self-TDD Collusion vs. Blind Worktree Isolation

+-------------------------------------------------------------------------+
| Proxmox Isolated VM210 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Subscription Proration Pipeline                  |
|               (normalizer.py ➔ proration.py ➔ invoice.py)               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: SingleContextTddRunner (Same Agent Writes Tests & Code)   |  |
|  |  - Sees existing buggy `str(item.get("account_id"))`              |  |
|  |  - Writes 1 happy-path test + misses non-string rejection bug     |  |
|  |  - Self-test: PASS! ➔ Hidden Judge: 19/20 FAIL | Mutants: 0/6     |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: BlindWorktreeTddRunner (2 Worktrees, Physically Split)    |  |
|  |  - Worktree 1 [Blind Tester]: Sees ONLY TASK.md (0 impl files!)   |  |
|  |      Writes 52-line adversarial `test_suite.py`                   |  |
|  |      Red Phase Check: FAILS on initial buggy code (Verified!)     |  |
|  |  - Worktree 2 [Implementer]:  `test_suite.py` LOCKED read-only;   |  |
|  |      Patches normalizer/proration/invoice until Green             |  |
|  |  - Hidden Judge: 20/20 PASS (100%) | Mutants Killed: 6/6 (100%)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Code-Level Postmortem: Implementation Anchoring Caught on Tape

Comparing single-bundle.json (Mode A) and blind-bundle.json (Mode B) demonstrates Implementation Anchoring in action.

In the initial buggy normalizer.py, account IDs were coerced using str(...):

# Initial buggy fixture in normalizer.py:
item["account_id"] = str(item.get("account_id", "")).strip()

Because Mode A saw that line in its prompt, it carried the str(...) coercion habit straight into its "fixed" implementation—silently converting integer account_id: 999 into "999" instead of raising ValueError as required by TASK.md:

# ❌ Mode A (`SingleContextTDD`): Anchored on str() coercion & wrote 0 negative tests!
acc = str(item.get("account_id", "")).strip().upper()

Worse still, Mode A's test_suite.py tested only a single valid string "acc-1" with a SAVE10 coupon—never testing non-string IDs, bool values, 9,999 vs. 10,000 discount boundaries, or DISCOUNT: -0.

In contrast, Mode B's Blind Tester never saw the str(...) bug. Reading TASK.md cold, it immediately built a table-driven adversarial helper bad(fn, *a, **k) probing booleans ([True, False, 1.5]), integer account IDs, days_active bounds (0 and 31), unit_price == 0 free trials, exact 9,999 vs. 10,000 thresholds, compounding SAVE10/VIP20 order, and DISCOUNT: -0—forcing Worktree 2's Implementer to enforce strict isinstance(account_id, str) checks.

4. Mutation Testing Audit (M1–M6) & 20-Case Grading Breakdown

Our hidden evaluator (labjudge06) injected 6 subtle boundary bugs (Mutants M1–M6) into a reference implementation to measure whether each agent's test_suite.py could actually catch regressions:

Mutant ID Injected Edge-Case Bug Mode A test_suite.py (Single TDD) Mode B test_suite.py (Blind TDD)
M1 Off-by-one volume discount (> 10000 instead of >= 10000) ❌ Survived (only tested 9,000) ✅ Killed (tested 9,999 & 10,000)
M2 Coupon applied to gross_subtotal instead of after_volume ❌ Survived ✅ Killed (verified 10,000 + SAVE10 = 900)
M3 Python bool (seats=True) accepted as integer 1 ❌ Survived ✅ Killed (tested [True, False, 1.5])
M4 Free trial (unit_price == 0) erroneously rejected ❌ Survived ✅ Killed (verified unit_price: 0)
M5 Upper bound days_active == 31 (> 30 max) accepted ❌ Survived ✅ Killed (verified 0 and 31 raise ValueError)
M6 Zero discount formatted as DISCOUNT: 0 instead of -0 ❌ Survived (only tested -1400) ✅ Killed (explicitly asserted DISCOUNT: -0)
Total Mutation Kill Score 0 / 6 (0.0%) 6 / 6 (100.0%)

On the 20 hidden grading cases, Mode A failed normalizer_03_blank_account_or_unknown_plan_rejected (19 / 20), while Mode B passed all 20 / 20 (100%).

5. Practical Takeaway: Fold Implementation (E05), Isolate Verification (E06)

Together, E05 and E06 resolve the biggest debate in multi-agent engineering:

  • Never split implementation roles (Architect ➔ Coder A ➔ Coder B) via lossy text summaries (E05).
  • Always isolate the Test Author into a Blind Worktree that sees only the specification (SPEC.md / issue contract) and zero lines of implementation (E06), then lock the resulting test files read-only before launching your folded implementation loop.

🔗 Continue Reading the Series

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog