# Why Hand-Written Example Tests Fail AI Coding Agents: Fixed Sample Tests vs. a Property-Based Invariant Fuzzing Sensor (-95.1% Input Tokens) > ๐Ÿ’ก **AI Labo Benchmark Series โ€” Experiment `E33` (Verification & Property Fuzzing)** > > This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite. > * ๐Ÿš€ **[View All 41 Benchmark Results & Master Ranking](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)** > * ๐Ÿ›ก๏ธ **[Read How Our Proxmox & systemd Anti-Cheating Sandbox Works](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)** --- ## Metadata & Experiment Identity * **Experiment ID:** `E33` (Backlog Candidate `C029`) * **Run ID:** `e33-20261029-vm237-01` * **Sandbox Environment:** Proxmox Isolated VM `VM237` (`ai-e33-237`, 0 NICs, `PrivateNetwork=yes`, `NoNewPrivileges=yes`, `InaccessiblePaths`, UID split `labexp33` / `labjudge33`) * **Model:** `gemini-3.8-flash` (`thinkingLevel: LOW`, `temperature=0.0`) * **Pattern Under Test:** *"Generator-Evaluator Harness with Property-Based Invariant Fuzzing (Finding Edge-Case Counterexamples Automatically)"* โ€” **Hand-Written Example Tests Only (`HandWrittenExampleTestRunner`) vs. Property-Based Invariant Fuzzing Sensor (`PropertyBasedFuzzingSensorRunner`)** --- ## 1. Executive Summary (TL;DR) When an AI coding agent validates its code solely against 2 or 3 hand-written "happy-path" unit test examples (e.g., `unit_cents=500, chargeback_bps=100`), it falls into the **Happy-Path Illusion**: the example tests pass, while boundary counterexamplesโ€”such as `tx_count=True` (since `issubclass(bool, int)` is `True` in Python), `unit_cents=0`, `chargeback_bps=300` vs. `301`, `subtotal=15000`, or extra dictionary keys (`{"extra": 1}`)โ€”slip straight through. Equipping the harness with a **Property-Based Invariant Fuzzing Sensor** tests domain invariants across boundary and type-spoofing vectors automatically, surfacing minimal falsifying counterexamples up front. We tested this head-to-head inside isolated VM `VM237` on the 3-module enterprise payment settlement pipeline (`validator.py` โž” `settlement.py` โž” `statement.py`): * **Mode A (`HandWrittenExampleTestRunner`):** Relies on fixed example tests first, hitting 2 blind-spot rework traps and ingesting `77,172` chars of counterexample escape logs across 3 turns before finally converging. * **Mode B (`PropertyBasedFuzzingSensorRunner`):** Injects the **Property-Based Invariant Fuzzing Sensor** specification (`PROPERTY_BASED_FUZZING_SENSOR_MD` + `pass_property_fuzzer_and_submit`) with explicit falsifying counterexamples for all 6 domain invariants, solving the entire pipeline in 1 shot. ### Key Result While both modes reached `20 / 20` on the hidden grading suite, Mode A burned **3 turns, 5 file edits, and `47,536` cumulative input tokens**. **Mode B solved `20 / 20` (100%) in 1 turn on the first shot**, cutting **LLM calls by `-66.7%` (`3 โž” 1`)**, **cumulative input tokens by `-95.1%` (`47,536 โž” 2,338` tokensโ€”a `20.33x` reduction)**, **API cost by `-83.5%` (`$0.0471 โž” $0.0078`)**, and running **`2.27x` faster (`30.33s โž” 13.35s`, `-56.0%`)**. | Metric | Mode A: Fixed Example Tests (`HandWrittenExampleTest`) | Mode B: Property-Based Fuzzer (`PropertyBasedFuzzingSensor`) | Delta / Improvement | | :--- | :---: | :---: | :---: | | **Hidden Grading Suite Score (`labjudge33`)** | `20 / 20` (100% โ€” Turn 3) | **`20 / 20` (100% โ€” Turn 1 One-Shot)** | **1-shot convergence (`3 โž” 1` turn)** | | **Example Blind-Spot Traps (`gotcha_traps_hit`)** | `2 traps` (Turns 1 & 2) | **`0 traps`** | **-100% (zero blind spots)** | | **LLM Calls (`model_calls`)** | `3 calls` | **`1 call` (1-shot)** | **-66.7% (`3.00x` faster convergence)** | | **Cumulative File Edits (`file_edits`)** | `5 edits` | **`3 edits` (1 per module)** | **-40.0% fewer edits** | | **Cumulative Tool Output (`tool_output_chars`)** | `77,172 chars` | **`101 chars`** | **-99.9% (`764.1x` noise reduction)** | | **Turn 1 Tokens (In / Out)** | `1,658 in` / `12 out` | **`2,338 in` / `1,610 out`** | `+680` in for property invariants | | **Turn 2 Tokens (In / Out)** | `15,447 in` / `1,355 out` | *โ€” (completed on Turn 1)* | Avoided blind-spot rediscovery logs | | **Turn 3 Tokens (In / Out)** | `30,431 in` / `1,693 out` | *โ€” (completed on Turn 1)* | Avoided secondary rework bloat | | **Cumulative Input Tokens (`promptTokenCount`)** | `47,536 tokens` | **`2,338 tokens`** | **-95.1% (`20.33x` efficiency)** | | **Cumulative Output Tokens (`candidatesTokenCount`)** | `3,060 tokens` | **`1,610 tokens`** | **-47.4% output reduction** | | **Elapsed Execution Time (`elapsed_seconds`)** | `30.33 sec` | **`13.35 sec`** | **-56.0% (`2.27x` faster)** | | **Estimated API Cost (`estimated_micro_usd`)** | `$0.0471` (`47,128 micro-USD`) | **`$0.0078` (`7,791 micro-USD`)** | **-83.5% (`6.05x` cheaper)** | --- ## 2. Experiment Architecture: Fixed Example Tests vs. Property-Based Invariant Fuzzer ```text +-------------------------------------------------------------------------+ | Proxmox Isolated VM237 (0 NICs, InaccessiblePaths, grader labjudge33) | | Target Task: 3-Module Enterprise Payment Settlement Pipeline | | validator.py โž” settlement.py โž” statement.py | | | | +-------------------------------------------------------------------+ | | | Mode A: HandWrittenExampleTestRunner (Happy-Path Samples Only) | | | | - Turn 1 (1,658 in): Calls `run_fixed_example_tests` โž” misses | | | | boundary counterexamples (38,795 chars) | | | | - Turn 2 (15,447 in): Calls `patch_example_blindspot` on 2 files | | | | โž” `statement.py` still fails (38,278 chars)| | | | - Turn 3 (30,431 in): Final submission passes after 2 traps | | | | --> 3 calls | 47,536 in | $0.0471 | 30.33s | 20/20 (2 Traps Hit) | | | +-------------------------------------------------------------------+ | | | | +-------------------------------------------------------------------+ | | | Mode B: PropertyBasedFuzzingSensorRunner (Invariant Fuzzing) | | | | - Injects `PROPERTY_BASED_FUZZING_SENSOR_MD` (+680 tokens): | | | | Falsifying counterexamples (`tx_count=True`, `unit_cents=0`, | | | | `chargeback_bps=300/1000`, `{"extra":1}`, `subtotal=15000`, | | | | `tier_discount=0`) verified against P1/P2/P3 invariants | | | | - Turn 1 (2,338 in): Calls `pass_property_fuzzer_and_submit` | | | | --> 1 call | 2,338 in | $0.0078 | 13.35s | 20/20 (100% PASS) | | | +-------------------------------------------------------------------+ | +-------------------------------------------------------------------------+ ``` --- ## 3. Drop-In Implementation Recipe: Zero-Dependency Property Fuzz Sensor Even without external packages like `hypothesis`, you can build a deterministic boundary and type-spoofing property sensor in pure Python: ```python from typing import Callable BOUNDARY_VECTORS = [ {"merchant_id": " m-1 ", "currency": " jpy ", "tx_count": 1, "unit_cents": 0, "chargeback_bps": 1000, "extra": 9}, {"merchant_id": "M-1", "currency": "USD", "tx_count": True, "unit_cents": 100, "chargeback_bps": 0}, {"merchant_id": "M-1", "currency": "USD", "tx_count": 1, "unit_cents": 100, "chargeback_bps": 300}, {"merchant_id": "M-1", "currency": "USD", "tx_count": 1, "unit_cents": 15000, "chargeback_bps": 0}, ] def run_property_fuzz_sensor(validate_fn: Callable, settle_fn: Callable) -> list[str]: """Probes boundary & type-spoofing vectors against domain invariants and returns minimal counterexamples.""" counterexamples: list[str] = [] # Invariant P1a: Zero unit_cents allowed & extra keys stripped res = validate_fn([BOUNDARY_VECTORS[0]]) if list(res[0].keys()) != ["merchant_id", "currency", "tx_count", "unit_cents", "chargeback_bps"]: counterexamples.append(f"Counterexample #1 (extra key leaked): {res[0]!r}") # Invariant P1b: bool must be rejected even though issubclass(bool, int) is True try: validate_fn([BOUNDARY_VECTORS[1]]) counterexamples.append("Counterexample #2: tx_count=True was accepted as int") except ValueError: pass return counterexamples ``` --- ## 4. Practical Takeaway: Feed Minimal Falsifying Counterexamples, Not Happy-Path Confirmations 1. **Example tests only prove code works on the examples you thought of:** LLMs routinely overfit to the 2 or 3 sample inputs visible in a test file. 2. **Probe `0`, `-1`, exact threshold boundaries (`300`, `1000`, `15000`), `bool`-as-`int`, and extra dict keys automatically:** Returning a single falsifying counterexample (`Counterexample: tx_count=True was accepted`) gives the agent everything it needs to fix the invariant in 1 turn. --- ### ๐Ÿ”— Continue Reading the Series * ๐Ÿš€ **Master Benchmark Hub (All 41 Patterns Ranked):** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark) * ๐Ÿ›ก๏ธ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox) * ๐Ÿ” **Previous Sequential Post (`E32`):** [Why Coding Agents Need a Pre-Submit Diff Audit Gate: Blind Unchecked Submits vs. Blast-Radius & Entropy Auditing (-95.0% Input Tokens)](https://tappers.hashnode.dev/e32-pre-submit-blast-radius-diff-audit) * ๐Ÿ“ธ **Next Sequential Post (`E34`):** [Eliminating Print-Debugging Loops in Coding Agents: Bare Stack Traces vs. Runtime Frame Variable-Snapshot Injection (-95.0% Input Tokens)](https://tappers.hashnode.dev/e34-runtime-variable-snapshot-injection) * ๐Ÿงช **Related Blind TDD Deep Dive (`E06`):** [Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)](https://tappers.hashnode.dev/e06-blind-tdd-worktrees)