💡 AI Labo Benchmark Series — Experiment
E10(Verification & Adversarial Sensors)This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite and mutation audits.
Metadata & Experiment Identity
- Experiment ID:
E10 - Run ID:
e10-vm214-20261028-01 - Sandbox Environment: Proxmox Isolated VM
VM214(0 NICs,virtio-serialQGA relay, isolated graderlabjudge10) - Model:
gemini-3.8-flash - Pattern Under Test: Single-Agent Self-Review vs. Adversarial Critic / Maker-Checker (
Maker ➔ Read-Only Locked Critic ➔ Finalizer)
1. Executive Summary (TL;DR)
One of the most common prompts in agent frameworks is the Self-Review step: after an agent finishes writing code, the harness sends one more message in the same conversation—"Carefully review your implementation, add thorough edge-case tests, and submit when ready."
In practice, asking an agent to review its own code in the same context triggers Self-Preferential Confirmation Bias: the agent inherits every assumption and blind spot from Turn 1, writes confirmatory happy-path checks (or unexecuted test code), and rubber-stamps its own work.
To test whether separating verification into an Adversarial Critic (Maker-Checker) fixes this blind spot, we ran a controlled A/B experiment on a 3-module marketplace escrow & dispute reserve pipeline (sanitizer.py ➔ escrow.py ➔ voucher.py + adversarial_suite.py) inside isolated VM VM214:
- Mode A (
SelfReviewRunner): A single agent patches the 3 modules on Turn 1, then self-reviews its code and writesadversarial_suite.pyin the same conversation on Turn 2 before submitting. - Mode B (
AdversarialMakerCheckerRunner): Role 1 (Initial Maker) patches the code; Role 2 (Adversarial Critic) enters with a fresh context whose sole objective is falsification—writing a pure-Pythonadversarial_suite.pytargeting every boundary invariant inTASK.md, which the harness immediately locks as Read-Only; Role 3 (Maker Finalizer) verifies and finalizes the implementation under the locked adversarial suite.
Key Result
While both implementations passed the 20 functional cases, Mode A's self-reviewed audit suite crashed on execution (ModuleNotFoundError: No module named 'pytest', returncode=1) and achieved a 0 / 6 (0.0%) mutation kill rate. Mode B's fresh Adversarial Critic produced a 50-line zero-dependency Python falsification suite (returncode=0) that killed 6 / 6 (100.0%) boundary mutants—buying complete boundary verification for just +$0.0052 (half a cent) and +6.92s.
| Metric | Mode A: Single-Agent Self-Review (SelfReview) |
Mode B: Adversarial Maker-Checker (AdversarialCritic) |
Delta / Improvement |
|---|---|---|---|
| Hidden Grading Suite Score (20 Cases) | 20 / 20 (100%) |
20 / 20 (100%) |
Both implementations pass |
Submitted adversarial_suite.py Execution |
FAILED (returncode=1: missing pytest) |
PASSED (returncode=0) |
Eliminated unverified environment assumptions |
| Valid Against Golden Reference Implementation | False |
True |
100% pure-stdlib compliance |
Adversarial Mutant Kill Rate (M1–M6) |
0 / 6 (0.0%) |
6 / 6 (100.0%) |
+100 percentage points (0% ➔ 100%) |
LLM Calls (model_calls) |
2 calls |
3 calls (Maker ➔ Critic ➔ Finalizer) |
+1 call (dedicated Critic pass) |
Cumulative Input Tokens (promptTokenCount) |
6,512 tokens |
10,289 tokens |
+3,777 tokens |
Cumulative Output Tokens (candidatesTokenCount) |
4,373 tokens |
5,003 tokens |
+630 tokens |
Elapsed Execution Time (elapsed_seconds) |
34.05 sec |
40.97 sec |
+6.92 sec |
Estimated API Cost (estimated_micro_usd) |
$0.0213 (21,283 micro-USD) |
$0.0265 (26,479 micro-USD) |
+$0.0052 (~0.5 cents for 100% mutant kill) |
2. Experiment Architecture: Self-Review vs. Adversarial Maker-Checker
+-------------------------------------------------------------------------+
| Proxmox Isolated VM214 (0 NICs, offline stdlib-only Python, labjudge10) |
| Target Task: Marketplace Escrow & Dispute Reserve Pipeline |
| (sanitizer.py ➔ escrow.py ➔ voucher.py + suite) |
| |
| +-------------------------------------------------------------------+ |
| | Mode A: SelfReviewRunner (Same Context Reviews Its Own Homework) | |
| | - Turn 1: Patches sanitizer.py, escrow.py, voucher.py | |
| | - Turn 2 [Same History]: Self-reviews & rewrites suite using | |
| | `import pytest` (forgetting stdlib-only rule) + immediately | |
| | calls SUBMIT without re-running; skips boundary pairs | |
| | --> Suite crashes (returncode=1) | Mutant Kill Rate: 0/6 (0.0%) | |
| +-------------------------------------------------------------------+ |
| |
| +-------------------------------------------------------------------+ |
| | Mode B: AdversarialMakerCheckerRunner (Fresh Critic + File Lock) | |
| | - Role 1 [Maker]: Patches sanitizer.py, escrow.py, voucher.py | |
| | - Role 2 [Critic]: Fresh context writes 50-line pure-Python | |
| | falsification suite (100 vs 101 bps, 7999 vs 8000, bool, -0) | |
| | --> Harness locks adversarial_suite.py as READ-ONLY | |
| | - Role 3 [Finalizer]: Runs locked suite (returncode=0) & submits | |
| | --> Suite green (returncode=0) | Mutant Kill Rate: 6/6 (100%) | |
| +-------------------------------------------------------------------+ |
+-------------------------------------------------------------------------+
3. Mutation Kill Audit (labjudge10): 0/6 vs. 6/6
To measure whether each mode's submitted adversarial_suite.py could actually catch subtle regressions, the isolated labjudge10 grader injected 6 classic off-by-one, type-coercion, and formatting mutants (M1–M6) into a clean reference implementation and ran both test suites against each mutant:
| Mutant ID | Injected Boundary Bug | Mode A (SelfReview) Suite |
Mode B (AdversarialCritic) Suite |
|---|---|---|---|
M1_dispute_bps_ge_100_instead_of_gt_100 |
Mutates dispute_bps > 100 to >= 100 (wrongly holding reserve at exact 100 bps) |
❌ Survived (crashed on pytest; never tested 100 vs 101 boundary) |
✅ Killed (asserted exact 100 bps ➔ 10000 and 101 bps ➔ 9000) |
M2_premier_rebate_strict_gt_8000 |
Mutates subtotal >= 8000 to > 8000 (denying rebate on exact 8,000 orders) |
❌ Survived (only checked subtotal=13500; missed 7999 vs 8000) |
✅ Killed (asserted 7999 ➔ 0 and 8000 ➔ 800 / 1600) |
M3_bool_accepted_as_int_in_quantity |
Removes isinstance(qty, bool) guard so True is accepted as 1 |
❌ Survived (pytest.raises dependency crashed in offline sandbox) |
✅ Killed (tested quantity=True, unit_cents=False, dispute_bps=True, holdback_waiver=True) |
M4_unit_cents_zero_rejected |
Mutates unit_cents < 0 to <= 0 (rejecting free 0-cent promotional items) |
❌ Survived (suite crashed on import) | ✅ Killed (verified unit_cents=0 passes and -1 raises ValueError) |
M5_conflicting_cents_or_bps_ignored |
Silently ignores conflicting unit_cents or dispute_bps on duplicate (merchant_id, category) |
❌ Survived (conflict test completely omitted from Mode A's suite) | ✅ Killed (verified both price conflict and bps conflict raise ValueError) |
M6_zero_tier_rebate_missing_minus_sign |
Formats zero rebate as TIER_REBATE: 0 instead of mandatory TIER_REBATE: -0 |
❌ Survived (suite crashed on import) | ✅ Killed (asserted exact TIER_REBATE: -0 and WAIVER_APPLIED: -0 lines) |
4. Deep Dive: Two Failure Modes of Single-Agent Self-Review
Inspecting self-bundle.json and execution-traces.json from VM214 exposes two structural traps that occur whenever an implementing agent reviews its own work:
Trap 1: Unverified Last-Mile Edits & Constraint Drift
On Turn 1, Mode A spent its attention fixing the business logic across sanitizer.py, escrow.py, and voucher.py. On Turn 2, when asked to review its work and strengthen adversarial_suite.py, it forgot the system constraint that the offline sandbox only had Python's standard library (assert / unittest), imported pytest, and immediately emitted "op": "SUBMIT" in the same turn—submitting a broken test file (returncode=1) with complete confidence.
Trap 2: Representative-Value Confirmation Bias
Even setting aside the pytest import error, inspecting the assertions inside Mode A's adversarial_suite.py revealed pure confirmatory testing:
- Instead of testing the exact boundary pair
dispute_bps=100(no reserve) vs.dispute_bps=101(10% reserve), Mode A only testeddispute_bps=101. - Instead of testing
subtotal=7999vs.subtotal=8000, Mode A only tested a comfortable middle value (subtotal=13500). - It completely omitted any test for conflicting
unit_cents/dispute_bpsacross duplicate(merchant_id, category)entries (M5).
By contrast, Mode B's Adversarial Critic (role_2_adversarial_critic) had zero emotional or contextual attachment to the Maker's code. Given a compact 55-line budget and an explicit mandate to break flawed implementations using pure Python, it wrote a dense 50-line (3,532-character) falsification suite (bad(fn, *a, **kw) helper + exact boundary pairs) that caught all 6 mutants on the first try.
5. Practical Takeaway: Never Let the Maker Own the Test File
- Replace "Review your own code" prompts with a fresh Adversarial Critic context: If a task has non-trivial boundary math (
<vs.<=), type edge cases, or strict formatting contracts, spawn a separate context whose prompt rewards falsification rather than completion. - Enforce Read-Only locks on Critic tests: When handing control back to the Maker (
role_3_maker_finalizer) to fix any failing assertions, physically block write permissions onadversarial_suite.py(see also Blind TDD (E06) and Cryptographic Test Locks (E38)) so the Maker cannot weaken the tests to pass.
🔗 Continue Reading the Series
- 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
- 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
- 🗺️ Previous Post (
E09): Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster) - 🛡️ Next Companion Post (
E11): Why Long System Prompts Can't Enforce Code Architecture: Prompt Rules vs. Deterministic Pre-Commit AST Hooks - 🧪 Related Verification Deep Dive (
E06): Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)
Comments
Join the discussionNo comments yet. Be the first to comment.