Why Hand-Written Example Tests Fail AI Coding Agents: Fixed Sample Tests vs. a Property-Based Invariant Fuzzing Sensor (-95.1% Input Tokens)

💡 AI Labo Benchmark Series — Experiment E33 (Verification & Property Fuzzing)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

🚀 View All 41 Benchmark Results & Master Ranking

🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

Experiment ID: E33 (Backlog Candidate C029)
Run ID: e33-20261029-vm237-01
Sandbox Environment: Proxmox Isolated VM VM237 (ai-e33-237, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp33 / labjudge33)
Model: gemini-3.8-flash (thinkingLevel: LOW, temperature=0.0)
Pattern Under Test: "Generator-Evaluator Harness with Property-Based Invariant Fuzzing (Finding Edge-Case Counterexamples Automatically)" — Hand-Written Example Tests Only (HandWrittenExampleTestRunner) vs. Property-Based Invariant Fuzzing Sensor (PropertyBasedFuzzingSensorRunner)

1. Executive Summary (TL;DR)

When an AI coding agent validates its code solely against 2 or 3 hand-written "happy-path" unit test examples (e.g., unit_cents=500, chargeback_bps=100), it falls into the Happy-Path Illusion: the example tests pass, while boundary counterexamples—such as tx_count=True (since issubclass(bool, int) is True in Python), unit_cents=0, chargeback_bps=300 vs. 301, subtotal=15000, or extra dictionary keys ({"extra": 1})—slip straight through.

Equipping the harness with a Property-Based Invariant Fuzzing Sensor tests domain invariants across boundary and type-spoofing vectors automatically, surfacing minimal falsifying counterexamples up front.

We tested this head-to-head inside isolated VM VM237 on the 3-module enterprise payment settlement pipeline (validator.py ➔ settlement.py ➔ statement.py):

Mode A (HandWrittenExampleTestRunner): Relies on fixed example tests first, hitting 2 blind-spot rework traps and ingesting 77,172 chars of counterexample escape logs across 3 turns before finally converging.
Mode B (PropertyBasedFuzzingSensorRunner): Injects the Property-Based Invariant Fuzzing Sensor specification (PROPERTY_BASED_FUZZING_SENSOR_MD + pass_property_fuzzer_and_submit) with explicit falsifying counterexamples for all 6 domain invariants, solving the entire pipeline in 1 shot.

Key Result

While both modes reached 20 / 20 on the hidden grading suite, Mode A burned 3 turns, 5 file edits, and 47,536 cumulative input tokens. Mode B solved 20 / 20 (100%) in 1 turn on the first shot, cutting LLM calls by -66.7% (3 ➔ 1), cumulative input tokens by -95.1% (47,536 ➔ 2,338 tokens—a 20.33x reduction), API cost by -83.5% ($0.0471 ➔ $0.0078), and running 2.27x faster (30.33s ➔ 13.35s, -56.0%).

Metric	Mode A: Fixed Example Tests (`HandWrittenExampleTest`)	Mode B: Property-Based Fuzzer (`PropertyBasedFuzzingSensor`)	Delta / Improvement
Hidden Grading Suite Score (`labjudge33`)	`20 / 20` (100% — Turn 3)	`20 / 20` (100% — Turn 1 One-Shot)	1-shot convergence (`3 ➔ 1` turn)
Example Blind-Spot Traps (`gotcha_traps_hit`)	`2 traps` (Turns 1 & 2)	`0 traps`	-100% (zero blind spots)
LLM Calls (`model_calls`)	`3 calls`	`1 call` (1-shot)	-66.7% (`3.00x` faster convergence)
Cumulative File Edits (`file_edits`)	`5 edits`	`3 edits` (1 per module)	-40.0% fewer edits
Cumulative Tool Output (`tool_output_chars`)	`77,172 chars`	`101 chars`	-99.9% (`764.1x` noise reduction)
Turn 1 Tokens (In / Out)	`1,658 in` / `12 out`	`2,338 in` / `1,610 out`	`+680` in for property invariants
Turn 2 Tokens (In / Out)	`15,447 in` / `1,355 out`	— (completed on Turn 1)	Avoided blind-spot rediscovery logs
Turn 3 Tokens (In / Out)	`30,431 in` / `1,693 out`	— (completed on Turn 1)	Avoided secondary rework bloat
Cumulative Input Tokens (`promptTokenCount`)	`47,536 tokens`	`2,338 tokens`	-95.1% (`20.33x` efficiency)
Cumulative Output Tokens (`candidatesTokenCount`)	`3,060 tokens`	`1,610 tokens`	-47.4% output reduction
Elapsed Execution Time (`elapsed_seconds`)	`30.33 sec`	`13.35 sec`	-56.0% (`2.27x` faster)
Estimated API Cost (`estimated_micro_usd`)	`$0.0471` (`47,128 micro-USD`)	`$0.0078` (`7,791 micro-USD`)	-83.5% (`6.05x` cheaper)

2. Experiment Architecture: Fixed Example Tests vs. Property-Based Invariant Fuzzer

+-------------------------------------------------------------------------+
| Proxmox Isolated VM237 (0 NICs, InaccessiblePaths, grader labjudge33)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py ➔ settlement.py ➔ statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: HandWrittenExampleTestRunner (Happy-Path Samples Only)    |  |
|  |  - Turn 1 (1,658 in):  Calls `run_fixed_example_tests` ➔ misses   |  |
|  |                        boundary counterexamples (38,795 chars)    |  |
|  |  - Turn 2 (15,447 in): Calls `patch_example_blindspot` on 2 files |  |
|  |                        ➔ `statement.py` still fails (38,278 chars)|  |
|  |  - Turn 3 (30,431 in): Final submission passes after 2 traps      |  |
|  |  --> 3 calls | 47,536 in | $0.0471 | 30.33s | 20/20 (2 Traps Hit) |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: PropertyBasedFuzzingSensorRunner (Invariant Fuzzing)      |  |
|  |  - Injects `PROPERTY_BASED_FUZZING_SENSOR_MD` (+680 tokens):      |  |
|  |    Falsifying counterexamples (`tx_count=True`, `unit_cents=0`,   |  |
|  |    `chargeback_bps=300/1000`, `{"extra":1}`, `subtotal=15000`,    |  |
|  |    `tier_discount=0`) verified against P1/P2/P3 invariants        |  |
|  |  - Turn 1 (2,338 in): Calls `pass_property_fuzzer_and_submit`     |  |
|  |  --> 1 call  | 2,338 in  | $0.0078 | 13.35s | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Drop-In Implementation Recipe: Zero-Dependency Property Fuzz Sensor

Even without external packages like hypothesis, you can build a deterministic boundary and type-spoofing property sensor in pure Python:

from typing import Callable

BOUNDARY_VECTORS = [
    {"merchant_id": " m-1 ", "currency": " jpy ", "tx_count": 1, "unit_cents": 0, "chargeback_bps": 1000, "extra": 9},
    {"merchant_id": "M-1", "currency": "USD", "tx_count": True, "unit_cents": 100, "chargeback_bps": 0},
    {"merchant_id": "M-1", "currency": "USD", "tx_count": 1, "unit_cents": 100, "chargeback_bps": 300},
    {"merchant_id": "M-1", "currency": "USD", "tx_count": 1, "unit_cents": 15000, "chargeback_bps": 0},
]


def run_property_fuzz_sensor(validate_fn: Callable, settle_fn: Callable) -> list[str]:
    """Probes boundary & type-spoofing vectors against domain invariants and returns minimal counterexamples."""
    counterexamples: list[str] = []
    # Invariant P1a: Zero unit_cents allowed & extra keys stripped
    res = validate_fn([BOUNDARY_VECTORS[0]])
    if list(res[0].keys()) != ["merchant_id", "currency", "tx_count", "unit_cents", "chargeback_bps"]:
        counterexamples.append(f"Counterexample #1 (extra key leaked): {res[0]!r}")
    # Invariant P1b: bool must be rejected even though issubclass(bool, int) is True
    try:
        validate_fn([BOUNDARY_VECTORS[1]])
        counterexamples.append("Counterexample #2: tx_count=True was accepted as int")
    except ValueError:
        pass
    return counterexamples

4. Practical Takeaway: Feed Minimal Falsifying Counterexamples, Not Happy-Path Confirmations

Example tests only prove code works on the examples you thought of: LLMs routinely overfit to the 2 or 3 sample inputs visible in a test file.
Probe 0, -1, exact threshold boundaries (300, 1000, 15000), bool-as-int, and extra dict keys automatically: Returning a single falsifying counterexample (Counterexample: tx_count=True was accepted) gives the agent everything it needs to fix the invariant in 1 turn.

🔗 Continue Reading the Series

🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
🔍 Previous Sequential Post (E32): Why Coding Agents Need a Pre-Submit Diff Audit Gate: Blind Unchecked Submits vs. Blast-Radius & Entropy Auditing (-95.0% Input Tokens)
📸 Next Sequential Post (E34): Eliminating Print-Debugging Loops in Coding Agents: Bare Stack Traces vs. Runtime Frame Variable-Snapshot Injection (-95.0% Input Tokens)
🧪 Related Blind TDD Deep Dive (E06): Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)

Why Hand-Written Example Tests Fail AI Coding Agents: Fixed Sample Tests vs. a Property-Based Invariant Fuzzing Sensor (-95.1% Input Tokens)

Metadata & Experiment Identity

1. Executive Summary (TL;DR)

Key Result

2. Experiment Architecture: Fixed Example Tests vs. Property-Based Invariant Fuzzer

3. Drop-In Implementation Recipe: Zero-Dependency Property Fuzz Sensor

4. Practical Takeaway: Feed Minimal Falsifying Counterexamples, Not Happy-Path Confirmations

🔗 Continue Reading the Series

Comments

More from this blog

Why a 500-Line AGENTS.md Hurts Coding Agent Accuracy: Monolithic Encyclopedia Prompts vs. a 30-Line Directory-Pointer Index (3.13x Faster)

Eliminating Print-Debugging Loops in Coding Agents: Bare Stack Traces vs. Runtime Frame Variable-Snapshot Injection (-95.0% Input Tokens)

Why Coding Agents Need a Pre-Submit Diff Audit Gate: Blind Unchecked Submits vs. Blast-Radius & Entropy Auditing (-95.0% Input Tokens)

Stop Letting Coding Agents Do Manual Multi-Spot String Replacements: Manual Text Edits vs. Parameterized AST Codemod Tools (-94.8% Input Tokens)