💡 AI Labo Benchmark Series — Experiment E18 (Ranked #7 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

• Experiment ID: E18 (Backlog Candidate C014)
• Run ID: e18-20261028-vm222-01
• Sandbox Environment: Proxmox Isolated VM VM222 (ai-e18-222, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW)
• Pattern Under Test: Claude Code /advisor & Upfront Architectural Routing — "Solo Executor Trial-and-Error Loop vs. Upfront Advisor Blueprint + Executor One-Shot"

1. Executive Summary (TL;DR)

When an AI coding agent is tasked with fixing a bug that spans multiple coupled modules, running a single worker loop without upfront architectural guidance (Solo Executor Loop) often leads to myopic Whack-a-Mole patching: the agent edits Module 1, runs a trial, discovers a downstream contract mismatch in Module 2, patches Module 2, and repeats—accumulating trial logs and regenerating code across multiple turns.

Anthropic's Claude Code /advisor pattern (pairing a high-level architectural Advisor memo with a fast Executor) proposes a cleaner division of labor: consult the Advisor once at the start of the task to produce a concise cross-module Architectural Blueprint Memo, then let the Executor implement all coupled files in a single pass.

To test this pattern empirically, we benchmarked our 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) inside isolated Proxmox VM VM222 across two modes:

• Mode A (SoloExecutorLoopRunner): No Advisor consultation (advisor_calls = 0). The Executor runs an initial trial (run_solo_trial), applies piecemeal patches (stage_executor_patch on Turn 2 and Turn 3), and finally submits all 3 modules on Turn 4 (4 LLM calls, 5 file edits).
• Mode B (AdvisorExecutorRoutingRunner): Calls consult_advisor on Turn 1 (advisor_calls = 1) to retrieve a 1,143-character 5-point cross-module execution blueprint, allowing the Executor to implement and submit all 3 modules in one shot on Turn 2 (2 LLM calls, 3 file edits).

Key Result

Injecting a 1,143-character upfront Advisor blueprint eliminated all unplanned partial-patch retries (2 ➔ 0), halved the number of LLM calls (4 ➔ 2), cut cumulative input tokens by -84.9% (27,699 ➔ 4,172 tokens, 6.64x efficiency), reduced cumulative output tokens by -45.9% (2,897 ➔ 1,568 tokens), lowered API cost by -71.5% ($0.0316 ➔ $0.0090), and sped up execution by 1.78x (33.45s ➔ 18.78s)—with both modes passing 20 / 20 (100%) on the hidden grading suite.

2. Experiment Architecture: Solo Trial Loop vs. Advisor Blueprint Routing

+-------------------------------------------------------------------------+
| Proxmox Isolated VM222 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Bugfix                       |
|               (validator.py / settlement.py / statement.py)             |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: SoloExecutorLoopRunner (No Upfront Design Blueprint)      |  |
|  |  - Turn 1 (1,909 in):  run_solo_trial (+7,500 chars trial log)    |  |
|  |  - Turn 2 (5,000 in):  stage_executor_patch (validator.py only)   |  |
|  |  - Turn 3 (8,527 in):  stage_executor_patch (settlement.py only)  |  |
|  |  - Turn 4 (12,263 in): submit_settlement_bundle (All 3 modules)   |  |
|  |  --> Total: 27,699 in / 2,897 out / 5 file edits / 33.45 sec      |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: AdvisorExecutorRoutingRunner (Claude Code /advisor Style) |  |
|  |  - Turn 1 (1,846 in / 8 out):                                     |  |
|  |      Calls `consult_advisor` --> Receives 1,143-char 5-point      |  |
|  |      cross-module `one_shot_execution_blueprint`                  |  |
|  |  - Turn 2 (2,326 in / 1,560 out):                                 |  |
|  |      Executor implements & submits all 3 modules in 1 shot!       |  |
|  |  --> Total: 4,172 in / 1,568 out / 3 file edits / 18.78 sec       |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Why Advisor-Executor Routing Beats Both Solo Loops and Role Subagents

3.1 The Dual Token Penalty of Solo Trial-and-Error (Mode A)

Analyzing usage.json and execution-traces.json shows why piecemeal patching without an upfront architectural plan is so expensive:

1. Input Token Snowballing: Every time Mode A staged a partial single-file patch (validator.py on Turn 2, settlement.py on Turn 3), the harness ran a trial diagnostic that appended roughly 7,500 characters of output to the chat history (1,909 ➔ 5,000 ➔ 8,527 ➔ 12,263 input tokens).
2. Duplicate Output Token Generation (+84.8% extra output): Because Mode A generated validator.py on Turn 2 (488 output tokens), settlement.py on Turn 3 (795 output tokens), and then re-emitted all 3 modules in the final Turn 4 bundle (1,604 output tokens), it generated 2,897 output tokens compared to Mode B's 1,568 output tokens. Since LLM output tokens cost 3x to 5x more per token than input tokens, avoiding duplicate code generation directly slashes API cost and latency.

3.2 How E18 (Advisor-Executor) Differs from E05 (Role Subagents)

A natural question is: "In E05 (Subagent Folding), you showed that splitting a task across Architect ➔ Coder ➔ QA subagents was 2.66x slower. Why did E18 (Advisor-Executor) succeed?"

The difference lies in who holds the context and how the handoff works:

• In E05 (Role Subagent Handoff — Anti-Pattern), the Architect subagent ran in an isolated context and passed a lossy natural-language summary to the Coder, hiding the original codebase and dropping fine-grained rules (like signed zero -0).
• In E18 (Advisor Blueprint Injection — Golden Pattern), the Executor retains direct visibility into the actual source files, and the Advisor injects a concise 1,143-character checklist of cross-module invariants (one_shot_execution_blueprint) directly into Turn 1 before the Executor writes code. Nothing is lost in translation, and the Executor completes all 3 files in a single pass.

4. All 20 Hidden Grading Suite Results (labjudge18)

Both modes passed all 20 hidden unit and E2E test cases verified by labjudge18:

5. Practical Takeaway: Inject an Architectural Blueprint Before First Edit

When using Claude Code (/advisor), Antigravity (/plan), or your own custom agent harness on multi-file tasks:

1. Never let an Executor start editing File 1 before mapping File 2 and File 3's contracts: Piecemeal file-by-file trial loops double your output token generation and bloat input history with intermediate failure logs.
2. Generate a 5-bullet Cross-Module Blueprint on Turn 1: Have your Advisor model (or planning step) output a compact <1,500-character contract checklist specifying the exact data types, boundary inequalities (> vs. >=), and formatting invariants across all affected files before unlocking file edit tools.

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md
• 🥈 Rank #2 Deep Dive (E21): Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons
• 🥉 Rank #3 Deep Dive (E16): Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%
• 🌿 Rank #4 Deep Dive (E19): Why Keeping Failed Code Attempts in Chat History Poisons Your Coding Agent: Error-Trace Pruning vs. Accumulated History
• 🔍 Rank #5 Deep Dive (E14): The Hidden "Schema Tax" of MCP Tools: Dumping 32 Tool Schemas vs. On-Demand Tool Search (-92.7% Input Tokens)
• 🗜️ Rank #6 Deep Dive (E12): How Save-Game Context Compaction Fixes Spec Drift in Multi-Step Coding Agents (-91.9% Input Tokens)

💡 AI Labo Benchmark Series — Experiment E12 (Ranked #6 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

• Experiment ID: E12 (Backlog Candidate C008)
• Run ID: e12-20261028-vm216-01
• Sandbox Environment: Proxmox Isolated VM VM216 (ai-e12-216, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW)
• Pattern Under Test: Save-Game Context Compaction (/compact at Milestones) — "Unbounded History Accumulation vs. Milestone JSON State Compaction"

1. Executive Summary (TL;DR)

When an AI coding agent tackles a multi-step engineering task across several modules, keeping every prior file dump, telemetry log, and intermediate code output inside a single growing messages[] array (Unbounded Context Accumulation) causes two compounding problems:

1. Quadratic Token Inflation: Every past turn's tool output is re-billed on every subsequent turn (6,556 ➔ 11,769 ➔ 17,544 input tokens).
2. Mid-Task Spec Drift (Lost-in-the-Middle): By Turn 2 or Turn 3, fine-grained string normalization rules and boundary conditions buried in the middle of the history get drowned out by verbose telemetry logs—causing subtle bugs that break end-to-end integration tests.

To test the solution—Milestone Context Compaction (the "Save-Game" Pattern)—we benchmarked a 3-milestone power-grid telemetry and carbon-credit clearing pipeline (Milestone 1: collector.py ➔ Milestone 2: clearing.py ➔ Milestone 3: dispatch.py) inside isolated Proxmox VM VM216:

• Mode A (AccumulatedHistoryRunner): Appends all 3 milestone specifications, code outputs, and verbose telemetry audit dumps (VERBOSE_TELEMETRY_AUDIT_DUMP) into one continuous chat thread (0 compactions).
• Mode B (MilestoneCompactionRunner): Immediately after each milestone passes its deterministic verification gate, executes a /compact reset—discarding raw logs and replacing the chat history with a 120-token structured JSON checkpoint (SAVE_GAME_COMPACTION_V1) plus only the next milestone's specification.

Key Result

Mode A's accumulated history (35,869 cumulative input tokens) suffered a classic Lost-in-the-Middle failure on Milestone 2 (clearing.py), forgetting to apply .strip().upper() to contract_tier and failing 2 out of 20 hidden E2E tests (18 / 20, 90.0%). Mode B's Save-Game Milestone Compaction kept every turn under 1,400 input tokens (2,922 cumulative input tokens, -91.9% reduction), cut API cost by -72.2% ($0.0340 ➔ $0.0095), and achieved a flawless 20 / 20 (100%) pass.

2. Experiment Architecture: Unbounded History vs. Save-Game Compaction

+-------------------------------------------------------------------------+
| Proxmox Isolated VM216 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Milestone Power Grid & Carbon Clearing Pipeline         |
|               (M1: collector.py --> M2: clearing.py --> M3: dispatch.py)|
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: AccumulatedHistoryRunner (No Compaction)                  |  |
|  |  - Turn 1 (M1): 6,556 in  --> collector.py PASS                   |  |
|  |  - Turn 2 (M2): 11,769 in --> Audit logs drown out M2 spec!       |  |
|  |                               Forgets contract_tier.strip().upper()|  |
|  |  - Turn 3 (M3): 17,544 in --> Cumulative 35,869 input tokens      |  |
|  |  --> Final Score: 18 / 20 (Crashes on lowercase/padded tiers!)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: MilestoneCompactionRunner (Save-Game JSON Compaction)     |  |
|  |  - Turn 1 (M1): 650 in   --> collector.py PASS                    |  |
|  |  - [Compaction #1]: Discards raw logs; writes 120-tok JSON state  |  |
|  |  - Turn 2 (M2): 1,388 in --> Fresh context nails all M2 rules!    |  |
|  |  - [Compaction #2]: Updates verified_contracts in JSON checkpoint |  |
|  |  - Turn 3 (M3): 884 in   --> Cumulative 2,922 input tokens (-91.9%)| |
|  |  --> Final Score: 20 / 20 (100% Perfect Pass)                     |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Code-Level Postmortem: How Unbounded History Caused a Lost-in-the-Middle Bug

3.1 Mode A (AccumulatedHistory): The Missing .strip().upper() on Turn 2

Inspecting accum-bundle.json (Mode A's final submitted code) reveals a textbook example of Attention Dilution:

• On Turn 1 (collector.py), Mode A properly implemented .strip().upper() normalization for station_id and grid_zone.
• By Turn 2 (clearing.py), however, 45 lines of VERBOSE_TELEMETRY_AUDIT_DUMP_M1 plus the full Turn 1 code had bloated the prompt to 11,769 tokens. Buried in the middle of the prompt was the requirement that clearing.py must also normalize contract_tier via .strip().upper().
• Mode A completely missed that normalization rule in clearing.py:

# ❌ Mode A (accum-bundle.json — clearing.py generated at 11,769 input tokens)
def settle_grid(readings, contract_tier="BASE", credit_offset=0):
    if not isinstance(readings, list) or len(readings) == 0:
        raise ValueError("readings must be a non-empty list")
    # BUG: Forgot isinstance(contract_tier, str) and contract_tier.strip().upper()!
    if contract_tier not in ALLOWED_TIERS:
        raise ValueError(f"contract_tier must be one of {ALLOWED_TIERS}")

Because of that single omitted .strip().upper(), Mode A crashed with a ValueError on two hidden E2E integration tests that passed padded/lowercase tier strings (contract_tier=" green " in e2e_01 and contract_tier="hydro" in e2e_02).

3.2 Mode B (MilestoneCompaction): The Exact Save-Game JSON Checkpoint

In Mode B, as soon as Milestone 1 (collector.py) passed verification, the harness wiped the raw conversation history and started Turn 2 with this compact SAVE_GAME_COMPACTION_V1 JSON checkpoint:

{
  "checkpoint_type": "SAVE_GAME_COMPACTION_V1",
  "completed_modules": ["collector.py"],
  "history_compacted": true,
  "milestone_completed": 1,
  "next_target_module": "clearing.py",
  "verified_contracts": {
    "collector.collect_readings": "PASS (returns list of dicts with station_id, grid_zone in NORTH/SOUTH/WEST, mwh_units>0, rate_cents>=0, 0<=carbon_bps<=1500, non-mutating)"
  }
}

Because Turn 2 ran in a fresh 1,388-token context containing only the verified upstream contract and Milestone 2's specification, Mode B had zero attention dilution—properly implementing clean_tier = contract_tier.strip().upper() and passing all 20 / 20 test cases:

4. Practical Takeaway: Compact to Structured State, Not Prose Summaries

When building multi-step agent workflows (or using /compact in interactive coding agents):

1. Don't wait until the 200k context window is 95% full to compact: Attention degradation (Lost-in-the-Middle) starts happening well above 10,000–15,000 tokens of noisy tool logs. Compact at every milestone boundary as soon as a module's tests turn green.
2. Use deterministic JSON checkpoints instead of free-form prose summaries: As we saw in E05 (Subagent Folding), natural-language summaries often drop exact type contracts. Instead, record a structured JSON object containing completed_modules, verified_contracts (exact function signatures & invariants), and next_target_module.

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md
• 🥈 Rank #2 Deep Dive (E21): Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons
• 🥉 Rank #3 Deep Dive (E16): Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%
• 🌿 Rank #4 Deep Dive (E19): Why Keeping Failed Code Attempts in Chat History Poisons Your Coding Agent: Error-Trace Pruning vs. Accumulated History
• 🔍 Rank #5 Deep Dive (E14): The Hidden "Schema Tax" of MCP Tools: Dumping 32 Tool Schemas vs. On-Demand Tool Search (-92.7% Input Tokens)

💡 AI Labo Benchmark Series — Experiment E14 (Ranked #5 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

• Experiment ID: E14 (Backlog Candidate C010)
• Run ID: e14-20261028-vm218-01
• Sandbox Environment: Proxmox Isolated VM VM218 (ai-e14-218, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW)
• Pattern Under Test: Claude Code ENABLE_TOOL_SEARCH=auto & Lazy MCP Tool Loading — "Upfront Monolithic Schema Dump (32 Tools) vs. Progressive On-Demand Tool Search"

1. Executive Summary (TL;DR)

Connecting Model Context Protocol (MCP) servers—Kubernetes, AWS, GitHub, Datadog, PagerDuty, Stripe, Terraform—makes an AI coding agent vastly more capable, but it introduces two severe failure modes when all tool definitions are injected upfront:

1. The "Schema Tax": Detailed JSON Schemas average 700 to 800 tokens per tool. Injecting 32 MCP tool schemas into the system prompt forces you to pay 25,000+ input tokens on every single turn—even before the agent reads a single file.
2. Tool Confusion (Decoy Misfires): When 32 tool schemas sit flat in the context window, the model frequently calls similarly named deprecated or staging tools (e.g., verify_legacy_v1_router or patch_staging_mirror_files) instead of the active production tool.

To quantify how Progressive Tool Schema Disclosure (ENABLE_TOOL_SEARCH=auto) solves both problems, we benchmarked a 3-module cloud incident triage and SLA remediation pipeline (router.py, remediator.py, dossier.py) inside isolated Proxmox VM VM218:

• Mode A (MonolithicSchema): Injects all 32 enterprise MCP tool JSON Schemas into the prompt on every turn.
• Mode B (ProgressiveToolSearch): Injects only 1 lightweight meta-tool (search_tools, 120 tokens) on Turn 1. The agent queries search_tools(query="remediation") (22 output tokens) to dynamically load only the 2 active production tool schemas needed for the task.

Key Result

Replacing the 32-tool upfront schema dump with search_tools cut Turn 1 input tokens by -90.1% (25,646 ➔ 2,544) and cumulative input tokens by -92.7% (79,773 ➔ 5,844 tokens, 13.65x reduction). Even more importantly, filtering out 28 irrelevant and 2 deprecated decoy tools reduced decoy tool misfires from 2 ➔ 0, cutting API cost by -84.8% ($0.0737 ➔ $0.0112) and execution time by -48.5% (1.94x faster).

2. Experiment Architecture & Tool Discovery Workflow

+-------------------------------------------------------------------------+
| Proxmox Isolated VM218 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Enterprise MCP Tool Registry (32 Total Tools):                         |
|   - 2 Active Production Tools (verify_pipeline_suite, submit_bundle)    |
|   - 2 Deprecated / Staging Decoy Tools (legacy_v1, staging_mirror)      |
|   - 28 Unrelated Cloud/Infra MCP Tools (K8s, AWS, Stripe, Kafka, etc.)  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: MonolithicSchema (All 32 JSON Schemas Injected Every Turn)|  |
|  |  - Turn 1 (25,646 in): Calls decoy `verify_legacy_v1_router`      |  |
|  |  - Turn 2 (26,397 in): Calls decoy `patch_staging_mirror_files`   |  |
|  |  - Turn 3 (27,730 in): Calls `submit_remediation_bundle`          |  |
|  |  --> Total: 79,773 input tokens / 2 decoy misfires / 37.00 sec    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ProgressiveToolSearch (ENABLE_TOOL_SEARCH=auto)           |  |
|  |  - Turn 1 (2,544 in, 1 schema):                                   |  |
|  |      Calls `search_tools(query="remediation")` (22 output tokens) |  |
|  |      --> Loads ONLY 2 active production schemas (filters 30 out)  |  |
|  |  - Turn 2 (3,300 in, 2 schemas):                                  |  |
|  |      Calls `submit_remediation_bundle` (All 3 files in 1 shot)    |  |
|  |  --> Total: 5,844 input tokens / 0 decoy misfires / 19.06 sec     |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Turn-by-Turn Token & Tool Execution Breakdown

Inspecting usage.json and execution-traces.json across all 5 LLM calls (seq 0..4) shows how the upfront schema tax compounds—and how on-demand search eliminates it:

Two Critical Takeaways from the Trace

1. You Pay the Schema Tax on Every Turn—Not Just Turn 1: In Mode A, having 32 detailed MCP schemas in the system prompt cost 25,646 tokens before the agent wrote a single line of code. Because the agent needed 3 turns to recover from decoy tool calls, those exact same 32 schemas were billed three times—burning over 70,000 input tokens on unused JSON schema definitions alone.
2. A 22-Token Search Call Is Faster Than a 0-Search Monolithic Prompt: A common objection to lazy tool loading is "Won't an extra search_tools round-trip slow the agent down?" The empirical data proves the exact opposite: Mode B's Turn 1 generated just 22 output tokens ({"query": "remediation", "tool": "search_tools"}), and because it avoided prefilling 25k tokens per turn and eliminated 2 decoy misfires, Mode B finished in 19.06 seconds vs. 37.00 seconds (1.94x faster).

4. All 20 Hidden Grading Suite Results (labjudge14)

Both modes achieved 20 / 20 (100%) on the hidden grading suite verifying router.py, remediator.py, and dossier.py:

5. Practical Takeaway: When and How to Enable Lazy Tool Loading

1. Never inject more than 10 full JSON tool schemas upfront: Once your connected MCP servers exceed 10–15 tools, switch to Progressive Tool Disclosure (such as ENABLE_TOOL_SEARCH=auto in Claude Code, or lazy MCP schema files loaded on demand via a search_tools / call_mcp_tool dispatcher).
2. Filter out DEPRECATED and non-production tools inside the search index: Instead of letting the LLM guess between verify_legacy_v1_router and verify_pipeline_suite, enforce status == "PRODUCTION_ACTIVE" filtering inside your search_tools handler so decoy tools never enter the context window.

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md
• 🥈 Rank #2 Deep Dive (E21): Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons
• 🥉 Rank #3 Deep Dive (E16): Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%
• 🌿 Rank #4 Deep Dive (E19): Why Keeping Failed Code Attempts in Chat History Poisons Your Coding Agent: Error-Trace Pruning vs. Accumulated History

💡 AI Labo Benchmark Series — Experiment E19 (Ranked #4 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

• Experiment ID: E19 (Backlog Candidate C015)
• Run ID: e19-20261028-vm223-01
• Sandbox Environment: Proxmox Isolated VM VM223 (ai-e19-223, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW)
• Pattern Under Test: Context Poisoning Prevention — "Error-Trace Pruning (Failed-Attempt Rollback) vs. Accumulated Failed History"

1. Executive Summary (TL;DR)

When a coding agent attempts a multi-file fix and the first patch fails tests, standard agent loops simply append the failed code diff (role: "model") and the verbose error stack trace (role: "user") to the conversation history (messages[]) and ask the LLM to try again.

This creates Self-Conditioning Bias (Context Poisoning): LLMs strongly anchor on their own prior generated code in the context window. Instead of rethinking the cross-module architecture, the agent keeps making narrow incremental tweaks on top of its broken first attempt while stack traces snowball across turns.

To test the antidote—Error-Trace Pruning (Failed-Attempt Rollback)—we benchmarked our 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) inside isolated Proxmox VM VM223 across two modes:

• Mode A (AccumulatedFailedHistoryRunner): Retains failed partial patches and full error stack traces (36,606 chars on Turn 1; 36,264 chars on Turn 2) in the conversation history across retries.
• Mode B (PrunedFailedAttemptRunner): Immediately rolls back the workspace on test failure (git checkout), prunes the failed model output and raw stack trace from messages[], and appends only a 927-character (+392-token) structured negative lesson to the clean initial prompt.

Key Result

While both modes eventually passed all 20 / 20 (100%) hidden test cases, pruning failed turns in Mode B eliminated the second failed retry (3 calls ➔ 2 calls), compressed final-turn input tokens by -94.0% (38,134 ➔ 2,301 tokens), cut cumulative input tokens by -93.0% (59,944 ➔ 4,210 tokens, 14.24x efficiency), reduced API cost by -80.3% ($0.0558 ➔ $0.0110), and finished 1.57x faster.

2. Experiment Architecture & Rollback-Prune Workflow

+-------------------------------------------------------------------------+
| Proxmox Isolated VM223 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Bugfix                       |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: AccumulatedFailedHistoryRunner (Keeps Failed Turns)       |  |
|  |  - Turn 1: Partial patch (validator.py) FAILS (+36,606 char trace)|  |
|  |  - Turn 2: Anchors on Turn 1; patches only settlement.py          |  |
|  |            FAILS again (+36,264 char trace; 19,912 in tokens)     |  |
|  |  - Turn 3: Submits all 3 files at 38,134 input tokens (59,944 sum)|  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: PrunedFailedAttemptRunner (Rollback + Prune Failed Turn)  |  |
|  |  - Turn 1: Partial patch (validator.py) FAILS (1,909 in tokens)   |  |
|  |  - Harness Action:                                                |  |
|  |      1. Roll back workspace files to clean snapshot               |  |
|  |      2. Delete Turn 1 model output & 36,606-char trace from chat  |  |
|  |      3. Append 927-char structured lesson (+392 tokens)           |  |
|  |  - Turn 2: Fresh retry at 2,301 input tokens --> Submits all 3    |  |
|  |            files in 1 shot (20 / 20 PASS, 4,210 cumulative in)    |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Turn-by-Turn Token Progression & Prompt Comparison

3.1 Turn-by-Turn Token & Cost Breakdown (usage.json)

3.2 What Mode A Kept in History (input-01.json / input-02.json)

In Mode A, after Turn 1 patched only validator.py, the harness appended 36,606 characters of unpruned diff frames and local variable dumps (MSG[2]), followed by another failed partial attempt (MSG[3]) and 36,264 characters of traces (MSG[4]):

=== FAILED ATTEMPT #1 FULL DIFF & TRACEBACK DUMP (UNPRUNED) ===
Status: FAILED (3 invariants broken)
[FAILED_PATCH_CONTEXT_FRAME_001] attempt=1 rejected_hunk_sha256=0000...0001 locals={'merchant_id': 'M-1', 'currency': 'USD', 'bps': 300, 'tier': 'ENTERPRISE'}
  File '/var/lib/ai-lab-e19-vm223-01/work_accumulated/settlement.py', line 26, in settle_batch
    AssertionError: Failed attempt #1 left residual bug in pipeline state.
... (FRAME_002 through FRAME_094: 36,000+ chars of repetitive frame dumps) ...

3.3 What Mode B Injected After Pruning (input-04.json)

In Mode B, as soon as Turn 1 failed, error_trace_pruner_v1 restored the workspace, deleted MSG[1] (the failed partial patch) and MSG[2] (the 36,606-char traceback) from messages[], and appended only this 927-character (+392-token) structured lesson to the original prompt:

{
  "concise_lessons_learned": [
    "Lesson 1 (validator.py): Partial patch failed because `settlement.py` and `statement.py` were not updated together; in `validator.py` use `ALLOWED_CURRENCIES=('USD','EUR','JPY')`, reject empty list & `bool`, allow `unit_cents>=0` & `0<=chargeback_bps<=1000`, and return a new 5-key dict.",
    "Lesson 2 (settlement.py): Preserve first-seen order (`for k in order:`), charge 10% reserve strictly when `chargeback_bps > 300`, apply `>= 15000` discount boundary (`GROWTH` 10%, `ENTERPRISE` 20% else 10%), and cap `rebate_applied = min(fee_rebate, after_discount)`.",
    "Lesson 3 (statement.py): Format item lines with `#` (`{merchant_id}#{currency}: ...`) and always prefix `-` on `TIER_DISCOUNT: -{tier_discount}` and `REBATE_APPLIED: -{rebate_applied}`."
  ],
  "pruning_engine": "error_trace_pruner_v1 (failed_code_rolled_back=true, stack_traces_pruned=true)",
  "remaining_issues": 4
}

With a clean 2,301-token context freed from its failed first attempt, Mode B immediately generated a complete, unified 3-module patch on Turn 2.

4. Practical Implementation Recipe: A 20-Line Rollback-and-Prune Loop

If you maintain a custom coding agent loop in Python, you can implement Error-Trace Pruning in 20 lines by combining git checkout with a lesson_log list instead of appending failed turns to messages:

import subprocess

def run_step_with_error_pruning(messages: list[dict], lesson_log: list[str], call_llm, run_tests) -> bool:
    # 1. Build prompt from clean base messages + 1-line negative lessons from prior failed attempts
    prompt_msgs = list(messages)
    if lesson_log:
        prompt_msgs[-1] = {
            **prompt_msgs[-1],
            "text": prompt_msgs[-1]["text"] + "\n\n[PRUNED_FAILED_ATTEMPT_LESSONS]:\n" + "\n".join(lesson_log),
        }
    model_reply = call_llm(prompt_msgs)

    # 2. Apply patch and run deterministic test suite
    passed, short_failure_reason, _raw_trace = run_tests(model_reply)
    if passed:
        messages.append({"role": "model", "text": model_reply})
        return True

    # 3. On failure: roll back workspace and DO NOT append model_reply or _raw_trace to messages!
    subprocess.run(["git", "checkout", "--", "."], check=True)
    lesson_log.append(f"- Attempt #{len(lesson_log) + 1} rolled back: {short_failure_reason}")
    return False

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md
• 🥈 Rank #2 Deep Dive (E21): Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons
• 🥉 Rank #3 Deep Dive (E16): Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%

💡 AI Labo Benchmark Series — Experiment E16 (Ranked #3 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

• Experiment ID: E16 (Backlog Candidate C012)
• Run ID: e16-20261028-vm220-01
• Sandbox Environment: Proxmox Isolated VM VM220 (ai-e16-220, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW)
• Pattern Under Test: RTK (rtk-ai/rtk, Rust Token Killer) & SWE-agent Output Compression — "Raw CLI Terminal Output vs. Semantic JSON Failure-Diff Filter"

1. Executive Summary (TL;DR)

What is the single biggest token sink in an autonomous coding agent loop? Surprisingly, it isn't the source code itself—it is unfiltered CLI test runner output.

When an agent executes pytest -vv --tb=long, cargo test, or npm test on a realistic repository where 76 tests pass and 4 fail, dumping the raw terminal output (ANSI color codes, environment banners, 76 [PASS] lines, captured DEBUG stdout, and multi-frame internal tracebacks) directly into chat history adds 60,000+ characters (28,000+ tokens) per test run.

To measure the impact of deterministic CLI output filtering, we benchmarked a 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) across two harness modes inside isolated VM VM220:

• Mode A (RawTerminalOutputRunner): Appends raw pytest terminal stdout (60,749 chars on Turn 1; 117,391 chars cumulative) directly to the conversation history.
• Mode B (TokenKillerFilterRunner): Wraps the test runner in a deterministic preprocessor that strips ANSI escape codes, environment banners, all 76 passing tests, and internal framework frames—returning only a 1,506-character structured JSON summary of the 4 failed assertions.

Key Result

Compressing raw CLI test output from 117,391 chars ➔ 1,604 chars (-98.6% reduction, 73.2x compression) enabled Mode B to fix all three modules in a single edit turn (2 LLM calls total vs. 3 in Mode A). Mode B cut cumulative input tokens by -94.1% (92,385 ➔ 5,445 tokens, 16.97x efficiency), API cost by -87.0% ($0.0770 ➔ $0.0100), and wall-clock time by -35.0% (1.54x faster) while achieving 20 / 20 (100%) on the hidden grading suite.

2. Experiment Architecture & Workflow Comparison

+-------------------------------------------------------------------------+
| Proxmox Isolated VM220 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Suite: 80-Test Suite (76 PASS / 4 FAIL across 3 modules)        |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: RawTerminalOutputRunner (Unfiltered pytest -vv --tb=long) |  |
|  |  - Turn 1: run_raw_pytest     --> Returns 60,749 chars raw log    |  |
|  |  - Turn 2: stage_partial_fix  --> Input jumps to 31,170 tokens!   |  |
|  |                                   Returns 56,544 chars raw log    |  |
|  |  - Turn 3: submit_settlement  --> Input hits 58,809 tokens!       |  |
|  |                                   (Total: 92,385 input tokens)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: TokenKillerFilterRunner (RTK-Style Semantic JSON Filter)  |  |
|  |  - Turn 1: run_filtered_tests --> Strips 76 PASS & ANSI noise;    |  |
|  |                                   returns 1,506-char failure JSON |  |
|  |  - Turn 2: submit_settlement  --> Input is only 3,030 tokens!     |  |
|  |                                   Fixes all 3 files in 1 shot     |  |
|  |                                   (Total: 5,445 input tokens)     |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Inside the Trace: How Raw CLI Logs Snowball Across Turns

3.1 Mode A (RawTerminalOutput): How 2 Test Runs Burned 92,385 Input Tokens

Tracking execution-traces.json and usage.json reveals how fast raw terminal output compounds in an agent's message array:

1. Turn 1 (2,406 in / 10 out): The agent calls {"tool": "run_raw_pytest"}. The runner executes the 80-test suite and returns 60,749 characters containing ANSI color codes, platform headers, 76 verbose [PASS] lines with AUDIT_TRACE dumps, and long multi-frame tracebacks for the 4 failures.
2. Turn 2 (31,170 in / 484 out): Appending that single raw stdout dump causes the prompt size to jump by +28,764 tokens (2,406 ➔ 31,170). Distracted by the 60KB wall of text, the agent only patches validator.py (stage_partial_fix), triggering a second test run that appends another 56,544 characters (now 77 PASS lines + 3 FAIL tracebacks) to the conversation history.
3. Turn 3 (58,809 in / 1,574 out): Now carrying 117,293 characters of raw terminal logs across two turns, Turn 3 alone requires 58,809 input tokens before the agent finally submits all 3 modules (92,385 cumulative input tokens).

3.2 Mode B (TokenKillerFilter): 1,506 Characters of Pure Signal

When Mode B called {"tool": "run_filtered_tests"} on Turn 1, the harness preprocessor stripped all 76 passing tests, ANSI escape sequences, and DEBUG noise—returning a 1,506-character (615-token) JSON payload containing only the 4 failing tests (test, file, line, assertion, expected_vs_actual):

{
  "summary": {"total": 80, "passed": 76, "failed": 4, "filtered_noise_chars_removed": 59243},
  "failures": [
    {
      "test": "test_validator_contract",
      "file": "validator.py",
      "line": 14,
      "assertion": "ALLOWED_CURRENCIES must include 'JPY'; reject empty list & bool; allow unit_cents==0 & chargeback_bps==1000; do not mutate input dict."
    },
    {
      "test": "test_settlement_ordering_and_reserve",
      "file": "settlement.py",
      "line": 22,
      "assertion": "Preserve first-seen (merchant_id, currency) order (never sorted(order)); apply 10% reserve strictly when chargeback_bps > 300."
    },
    {
      "test": "test_settlement_discounts_and_rebate_cap",
      "file": "settlement.py",
      "line": 39,
      "assertion": "Discount boundary is subtotal >= 15000 (GROWTH 10%, ENTERPRISE 20% else 10%); cap rebate_applied = min(fee_rebate, after_discount)."
    },
    {
      "test": "test_statement_format_and_signed_zero",
      "file": "statement.py",
      "line": 11,
      "assertion": "Use '#' delimiter ({merchant_id}#{currency}:) and preserve leading '-' on zero discount/rebate (TIER_DISCOUNT: -0, REBATE_APPLIED: -0)."
    }
  ]
}

Because all four failure root causes were immediately visible without scrolling through 76 passing tests, Mode B fixed all three files in a single Turn 2 call (3,030 input tokens)—eliminating the intermediate rework turn entirely.

4. All 20 Hidden Grading Suite Results (labjudge16)

Both modes passed all 20 hidden unit and E2E test cases verified by labjudge16:

5. Practical Takeaway: Wrap Your Agent's Test Runner Today

Never wire raw subprocess.run(["pytest"]) stdout directly into your LLM agent's tool response. Instead:

1. Use machine-readable reporters: Run pytest --tb=short -q --no-header --no-summary (or pytest-json-report) inside your tool wrapper.
2. Strip ANSI codes and passing tests: If 500 tests pass and 2 fail, the LLM only needs "passed": 500 as a single integer plus the exact file, line number, and assertion diff for the 2 failures.
3. Adopt CLI proxy filters: Tools like rtk (Rust Token Killer) or a 30-line Python wrapper around your test runner immediately cut agent token bills by 85%–95% while speeding up multi-file bug fixes.

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md
• 🥈 Rank #2 Deep Dive (E21): Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons

💡 AI Labo Benchmark Series — Experiment E21 (Ranked #2 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

• Experiment ID: E21 (Backlog Candidate C017)
• Run ID: e21-20261028-vm225-01
• Sandbox Environment: Proxmox Isolated VM VM225 (ai-e21-225, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW, temperature: 0.0)
• Pattern Under Test: Aider Repo-Map & Tree-sitter Skeleton Compression — "Full Dependency Implementation Context vs. Interface-Only Skeleton Context"

1. Executive Summary (TL;DR)

When an AI coding agent edits a module that depends on several peripheral files, a common approach is to dump the entire source code of all imported dependencies into the prompt so the model "has full context."

In practice, feeding full implementation bodies of peripheral dependencies is actively counterproductive. Legacy helper logic, internal comments, and verbose function bodies bloat the context window and trigger Attention Dilution, causing the model to miss strict boundary contracts in the files it actually needs to edit.

To quantify this, we benchmarked a 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) inside isolated Proxmox VM VM225 across two harness modes:

• Mode A (FullDependencyContextRunner): Loads the full implementation bodies of peripheral dependency modules and raw diagnostics (78,719 chars of tool output across 3 turns).
• Mode B (TreeSitterSkeletonContextRunner): Elides function implementation bodies (...) using an AST / Tree-sitter projection, injecting only public function signatures, type annotations, and contract docstrings (+673 tokens on Turn 1).

Key Result

Passing only the Tree-sitter Interface Skeleton (2,316 input tokens) enabled a 1-turn 20 / 20 (100%) pass on the hidden test suite. In contrast, Mode A's full dependency dump consumed 45,270 input tokens across 3 turns and stalled at 17 / 20 (85.0%) due to attention dilution. Mode B cut cumulative input tokens by -94.9% (19.55x reduction), API cost by -82.6% ($0.0446 ➔ $0.0078), and execution time by -57.2% (2.34x faster).

2. Experiment Architecture & Harness Workflow

+-------------------------------------------------------------------------+
| Proxmox Isolated VM225 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Refactoring                  |
|               (validator.py / settlement.py / statement.py)             |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: FullDependencyContextRunner (Unfiltered Full Source Dump) |  |
|  |  - Turn 1: load_full_dependency_dump (1,643 in / 39,373 chars)    |  |
|  |  - Turn 2: stage_dependency_patch    (14,712 in / 39,027 chars)   |  |
|  |  - Turn 3: submit_skeleton_guided_bundle (28,915 in --> 17 / 20)  |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: TreeSitterSkeletonContextRunner (Signature & Contract)    |  |
|  |  - Turn 1: Reads AST Interface Skeleton (+673 tok) & submits all  |  |
|  |            3 modules (2,316 in / 1,611 out --> 20 / 20 on Turn 1) |  |
|  +-------------------------------------------------------------------+  |
|                                    |                                    |
|                                    v                                    |
|  +-------------------------------------------------------------------+  |
|  | Independent Grader (labjudge21): 20 Hidden Unit & E2E Test Cases  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Inside the Prompts: Tree-sitter Skeleton vs. Full Implementation Dump

3.1 The Exact Tree-sitter Interface Skeleton Injected into Mode B (input-03.json)

Instead of dumping internal helper functions, Mode B's harness projected the target modules and their contracts into a compact Interface Skeleton (2,316 total input tokens on Turn 1) with implementation bodies folded into ...:

# REPO-MAP INTERFACE SKELETON (AST / Tree-sitter Signature & Contract Projection — Implementation Bodies Elided)

# module: validator.py (PUBLIC INTERFACE SKELETON)
ALLOWED_CURRENCIES: tuple[str, ...] = ("USD", "EUR", "JPY")
def validate_txns(raw_txns: list[dict]) -> list[dict]:
    """Contract:
    1. Reject empty list or non-list with ValueError.
    2. Strip & uppercase `merchant_id` (non-empty str) and `currency` (must be in ALLOWED_CURRENCIES: USD, EUR, JPY).
    3. Reject `isinstance(v, bool)` before `isinstance(v, int)` on `tx_count`, `unit_cents`, `chargeback_bps`.
    4. Bounds: `tx_count > 0`, `unit_cents >= 0` (zero allowed!), `0 <= chargeback_bps <= 1000` (1000 inclusive!).
    5. Return a brand-new 5-key dict list without mutating caller dicts.
    """
    ...

# module: settlement.py (PUBLIC INTERFACE SKELETON)
ALLOWED_TIERS: tuple[str, ...] = ("STANDARD", "GROWTH", "ENTERPRISE")
def settle_batch(txns: list[dict], tier: str = "STANDARD", fee_rebate: int = 0) -> dict:
    """Contract:
    1. Preserve first-seen `(merchant_id, currency)` order (`for k in order:`, NEVER `sorted(order)`).
    2. Chargeback reserve: 10% (`(gross * 10) // 100`) STRICTLY when `chargeback_bps > 300` (at 300 bps, reserve == 0).
    3. Tier discount threshold `subtotal >= 15000` is INCLUSIVE:
       - `GROWTH`: 10% if `subtotal >= 15000` else 0.
       - `ENTERPRISE`: 20% if `subtotal >= 15000` else 10%.
    4. Rebate cap: `rebate_applied = min(fee_rebate, after_discount)`.
    """
    ...

# module: statement.py (PUBLIC INTERFACE SKELETON)
def render_statement(batch: dict) -> str:
    """Contract:
    1. Line format: `{merchant_id}#{currency}: tx={tx_count} unit={unit_cents} bps={chargeback_bps} = {line_total}` (delimiter is `#`, NOT `:`).
    2. Discount & rebate lines MUST always carry a leading `-` sign even when 0 (`TIER_DISCOUNT: -0`, `REBATE_APPLIED: -0`).
    """
    ...

3.2 What Mode A Received: 78,719 Characters of Unfiltered Dependency Implementation Noise

In Mode A (FullDependencyContextRunner), calling load_full_dependency_dump on Turn 1 and stage_dependency_patch on Turn 2 flooded the context with 39,373 chars and 39,027 chars of internal helper bodies, bitwise checksum loops, and deprecated v1 comments:

=== FULL DEPENDENCY IMPLEMENTATION DUMP & INTERFACE MISMATCH LOG (STAGE #1) ===
Status: MODE_A_TRAP_HIT (4 contract invariants unresolved in FullDependencyContextRunner)
# RAW DEPENDENCY IMPLEMENTATION DUMP (UNFILTERED FULL SOURCE OF 12 INTERNAL HELPER MODULES)
# WARNING: Contains legacy internal helpers, deprecated rounding tables, and verbose implementation bodies.

# --- internal_dep_module_01.py (Full Implementation Body) ---
def _internal_helper_pipeline_stage_01(payload_buffer, flags=17):
    """Internal helper implementation body #01 (non-public implementation details)."""
    acc_state = 1024
    for step_i in range(16):
        acc_state = ((acc_state ^ (step_i * 31)) + 97) & 0xFFFFFFFF
    # Legacy note: old v1 parser used sorted(keys) and >= 300 bps, superseded by v2 public contract
    return {'stage': 1, 'checksum': hex(acc_state), 'legacy_delimiter': ':'}
... (37,500+ chars of internal helper implementation bodies) ...

Turn-by-Turn Token & Cost Breakdown (usage.json)

4. Code-Level Postmortem: Why Mode A Failed 3 out of 20 Test Cases

Inspecting the final submitted bundles (full-dep-bundle.json vs. treesitter-skeleton-bundle.json) shows how 28,915 tokens of noisy dependency implementations diluted Mode A's attention on Turn 3:

# ❌ Mode A (FullDependencyContextRunner, Turn 3 Final Submission — validator.py)
        if cnt <= 0 or cents <= 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")
        out.append({
            **item,
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
        })

# ✅ Mode B (TreeSitterSkeletonContextRunner, Turn 1 Final Submission — validator.py)
        if cnt <= 0 or cents < 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")
        out.append({
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
            "tx_count": cnt,
            "unit_cents": cents,
            "chargeback_bps": bps,
        })

Buried under 78KB of irrelevant helper bodies, Mode A missed two critical boundary contracts in validator.py:

1. Unvalidated Extra Key Leakage (validator_01_normalize_and_non_mutation FAIL): Using {**item, ...} copied unvalidated extra keys (e.g., "extra": 99) from the input dictionary into the normalized output instead of returning a strict 5-key dictionary.
2. Zero-Amount Card Auth Rejection (validator_05 & e2e_03 FAIL): Mode A left cents <= 0 instead of cents < 0, rejecting valid $0 card authorization probes (unit_cents == 0).

Mode B, operating on a crystal-clear 2,316-token AST interface skeleton, passed all 20 / 20 test cases on its very first turn.

5. Practical Takeaway: How to Build a Skeleton Context Filter in Python

When your agent needs to call or integrate with neighboring modules that it is not actively modifying, never dump their full source code into the prompt. Instead, use Python's ast module (or tree-sitter for multi-language repos) to fold function bodies into ...:

import ast

def extract_interface_skeleton(source_code: str) -> str:
    """Fold function and method bodies into '...' while preserving signatures, types, and docstrings."""
    tree = ast.parse(source_code)
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            # Keep docstring if present, replace remaining body statements with Ellipsis (...)
            has_doc = (
                node.body
                and isinstance(node.body[0], ast.Expr)
                and isinstance(node.body[0].value, ast.Constant)
                and isinstance(node.body[0].value.value, str)
            )
            ellipsis_stmt = ast.Expr(value=ast.Constant(value=...))
            node.body = [node.body[0], ellipsis_stmt] if has_doc else [ellipsis_stmt]
    return ast.unparse(tree)

By exposing interfaces and contracts while hiding internal implementation noise, you cut input tokens by over 90% and eliminate Lost-in-the-Middle bugs.

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md

💡 AI Labo Benchmark Series — Experiment E20 (Ranked #1 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

• Experiment ID: E20 (Backlog Candidate C016)
• Run ID: e20-20261028-vm224-01
• Sandbox Environment: Proxmox Isolated VM VM224 (ai-e20-224, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW)
• Pattern Under Test: Letta / MemGPT & Claude Code CLAUDE.md / MEMORY.md Pattern — "Episodic Gotchas Memory vs. Cold-Start Exploration"

1. Executive Summary (TL;DR)

When running multiple tasks sequentially on the same repository, starting every agent session from scratch (Cold-Start Exploration) forces the model to rediscover non-obvious domain invariants and repository quirks ("gotchas") through repeated test failures—burning tens of thousands of tokens on stack traces and diluting model attention.

To test whether a compact Episodic Gotchas Memory (MEMORY.md) solves this, we benchmarked a 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) containing 5 subtle repository gotchas (7 domain invariants) across two modes:

• Mode A (Cold-Start Exploration): No MEMORY.md injected; the agent explores the codebase and diagnoses failures via standard test/audit feedback.
• Mode B (Episodic Gotchas Memory): A 5-item MEMORY.md (+442 tokens) distilled from prior session failures is injected into the initial prompt.

Key Result

Injecting just +442 tokens of MEMORY.md reduced repository trap hits from 2 ➔ 0, achieving a 1-turn 20 / 20 (100%) pass on the hidden grading suite. Compared to Mode A (17 / 20 pass across 3 turns), Mode B cut cumulative input tokens by -95.5% (46,560 ➔ 2,078 tokens), API cost by -83.7% ($0.0454 ➔ $0.0074), and wall-clock time by -54.8% (2.21x faster).

2. Experiment Architecture & Repository Gotchas Under Test

+-------------------------------------------------------------------------+
| Proxmox Isolated VM224 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Bugfix                       |
|               (validator.py / settlement.py / statement.py)             |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: ColdStartRunner (No MEMORY.md — Cold-Start Exploration)   |  |
|  |  - Turn 1: probe_repo_gotchas   (1,636 in / 28,317 chars trap log)|  |
|  |  - Turn 2: stage_partial_fix    (15,153 in / 27,975 chars log)    |  |
|  |  - Turn 3: submit_settlement    (29,771 in --> 17 / 20 passed)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: EpisodicGotchasMemoryRunner (5-Item MEMORY.md [+442 tok]) |  |
|  |  - Turn 1: Reads 5-item MEMORY.md & submits all 3 modules         |  |
|  |            (2,078 in / 1,560 out --> 20 / 20 passed on 1st try!)  |  |
|  +-------------------------------------------------------------------+  |
|                                    |                                    |
|                                    v                                    |
|  +-------------------------------------------------------------------+  |
|  | Independent Grader (labjudge20): 20 Hidden Unit & E2E Test Cases  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

The 5 Non-Obvious Repository Gotchas in the Initial Codebase

The initial workspace inside VM224 contains three payment modules with subtle domain bugs and Python traps that are easy to miss on a cold read:

1. validator.py (Transaction Validation & Normalization):
   • ALLOWED_CURRENCIES = ("USD", "EUR") is missing "JPY".
   • Empty list [] rejection is missing, and because bool is a subclass of int in Python (isinstance(True, int) == True), isinstance(v, bool) must be explicitly rejected before checking isinstance(v, int).
   • The bounds check if cnt <= 0 or cents <= 0 or not (0 <= bps < 1000): erroneously rejects unit_cents == 0 ($0 card authorization probes) and chargeback_bps == 1000 (inclusive upper bound).
   • It mutates caller dictionaries in-place (item["merchant_id"] = ...; out.append(item)) and leaks extra unvalidated keys (extra) into the output.
2. settlement.py (Batch Aggregation, Reserve & Tier Discounts):
   • Iterates with for k in sorted(order):, destroying first-seen (merchant_id, currency) ordering.
   • Applies the 10% chargeback reserve at chargeback_bps >= 300 instead of strictly > 300 (at exactly 300 bps, reserve must be 0).
   • Uses subtotal > 15000 instead of inclusive >= 15000, omits the 10% discount for ENTERPRISE below 15000, and forgets to cap rebate_applied = min(fee_rebate, after_discount).
3. statement.py (Canonical Statement Formatter):
   • Uses : instead of # between merchant ID and currency ({merchant_id}#{currency}:).
   • Formats zero discounts as "0" instead of preserving the mandatory leading minus sign (TIER_DISCOUNT: -0, REBATE_APPLIED: -0).

3. Inside the Prompts: What Mode B Saw vs. What Mode A Endured

3.1 The Exact 5-Item MEMORY.md Injected into Mode B (+442 tokens)

In Mode B (input-03.json), the harness prepended this exact 5-bullet # MEMORY.md section to the initial prompt:

# MEMORY.md — Repository Episodic Gotchas & Non-Obvious Invariants (Distilled from Session #1)
1. [GOTCHA-01 / validator.py] `ALLOWED_CURRENCIES` MUST include `"JPY"` (`("USD", "EUR", "JPY")`), empty `records` list MUST raise `ValueError`, and `isinstance(v, bool)` MUST be rejected before `isinstance(v, int)` because `bool` is a subclass of `int` in Python.
2. [GOTCHA-02 / validator.py] Zero `unit_cents == 0` and boundary `chargeback_bps == 1000` are VALID (`tx_count > 0`, `unit_cents >= 0`, `0 <= chargeback_bps <= 1000`). Always return a brand-new 5-key dict without mutating caller dicts.
3. [GOTCHA-03 / settlement.py] Preserve first-seen `(merchant_id, currency)` order (`for k in order:`, NEVER `sorted(order)`). The 10% reserve applies STRICTLY when `chargeback_bps > 300` (at `300` bps, `reserve_cents == 0`).
4. [GOTCHA-04 / settlement.py] Tier discount threshold `subtotal >= 15000` is INCLUSIVE: `GROWTH` gets 10% (`>= 15000`), `ENTERPRISE` gets 20% (`>= 15000`) or 10% (`< 15000`), and `rebate_applied = min(fee_rebate, after_discount)`.
5. [GOTCHA-05 / statement.py] Header delimiter between merchant and currency is `#` (`{merchant_id}#{currency}: tx=...`) and discount/rebate lines MUST always carry `-` even when zero (`TIER_DISCOUNT: -0`, `REBATE_APPLIED: -0`).

3.2 The 56,605-Character Failure Trace That Flooded Mode A (Cold-Start)

Without MEMORY.md, Mode A called probe_repo_gotchas on Turn 1 and stage_partial_fix on Turn 2, accumulating 28,317 characters on Turn 1 and 27,975 characters on Turn 2 (totaling 56,605 characters of verbose audit stack traces) inside its conversation history:

=== COLD-START REPOSITORY GOTCHA TRAP LOG (STAGE #1) ===
Status: FAILED (4 repository gotchas triggered due to missing cross-session MEMORY.md)
[COLD_START_GOTCHA_AUDIT_001] stage=1 invariant_probe=GOTCHA_02 trace_digest=0000000000000000000000000000000000000000000000000000000000000001
  File '/var/lib/ai-lab-e20-vm224-01/work_cold_start/settlement.py', line 21, in verify_repo_invariants
    AssertionError: Cold-start session rediscovered repository gotcha #2 via trial-and-error failure.
... (28,000+ chars of repetitive invariant audit traces) ...
TRIGGERED_GOTCHA [validator.py] (validator_contract_violation): Failed JPY currency, empty list check, bool rejection, zero unit_cents, 1000 bps, or mutated input dict.
TRIGGERED_GOTCHA [settlement.py] (settlement_order_or_reserve_boundary): Lines reordered by sorted(order) or 10% reserve charged at chargeback_bps == 300.
TRIGGERED_GOTCHA [settlement.py] (settlement_tier_discount_or_rebate_cap): Missed >=15000 inclusive discount boundary, ENTERPRISE 10% below 15000, or min(fee_rebate, after_discount) cap.
TRIGGERED_GOTCHA [statement.py] (statement_delimiter_or_signed_zero): Used ':' instead of '#' between merchant and currency or omitted '-' sign when discount/rebate is 0.

4. Code-Level Postmortem: Why Mode A Failed 3 out of 20 Test Cases

Comparing the final submitted code bundles (cold-start-bundle.json vs. episodic-memory-bundle.json) reveals the exact mechanism of Attention Dilution:

# ❌ Mode A (Cold-Start, Turn 3 Final Submission — from cold-start-bundle.json)
        if cnt <= 0 or cents <= 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")
        out.append({
            **item,
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
        })

# ✅ Mode B (Episodic Gotchas Memory, Turn 1 Final Submission — from episodic-memory-bundle.json)
        if cnt <= 0 or cents < 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")

        out.append({
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
            "tx_count": cnt,
            "unit_cents": cents,
            "chargeback_bps": bps,
        })

By Turn 3, Mode A was carrying 29,771 input tokens of noisy logs. While it managed to fix settlement.py, statement.py, "JPY", bool rejection, and chargeback_bps <= 1000, its diluted attention missed two fine-grained details in validator.py:

1. Left cents <= 0 instead of cents < 0: Rejecting unit_cents == 0 ($0 auth probes) caused both unit test validator_05_zero_unit_cents_allowed_negative_rejected and E2E test e2e_03_zero_amount_auth_statement to fail.
2. Used {**item, ...} spread instead of constructing a clean 5-key dict: Extra unvalidated keys from the input dict leaked into the normalized output, failing validator_01_normalize_and_non_mutation.

Mode B, reading [GOTCHA-02 / validator.py] inside a clean 2,078-token Turn 1 prompt, nailed every single invariant on the first try.

All 20 Hidden Test Case Results (Mode A vs. Mode B)

5. Practical Takeaway: How to Structure Your Own MEMORY.md

A 442-token investment saved 44,482 input tokens (a 100x+ token ROI) while eliminating 3 subtle edge-case bugs.

To get the same result in your own repositories with Claude Code, Cursor, or custom agents:

• Don't turn CLAUDE.md or MEMORY.md into a 500-line textbook. (As we proved in E35, monolithic instruction files hurt performance.)
• Instead, maintain a 10-to-20 line MEMORY.md containing only [GOTCHA-ID / filename] 1-line invariant entries distilled from actual test failures or PR review catches:

# MEMORY.md (Repository Gotchas — Keep under 20 lines)
- [GOTCHA-01 / validator.py] `isinstance(x, bool)` is True for `int` in Python; always reject `bool` before checking `isinstance(x, int)`.
- [GOTCHA-02 / validator.py] `unit_cents == 0` ($0 auth) and `chargeback_bps == 1000` are valid; construct a new 5-key dict (never `**item` or mutate in-place).
- [GOTCHA-03 / settlement.py] Keep first-seen order (`for k in order:`); apply 10% reserve strictly when `chargeback_bps > 300`.
- [GOTCHA-04 / statement.py] Use `#` between merchant and currency (`{mid}#{cur}:`), and format zero discount/rebate with a leading minus (`-0`).

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox

💡 Methodology & Eval Infrastructure Deep Dive

This article is the Methodology & Architecture companion to our AI Labo Agent Harness Engineering Benchmark Series (41 Patterns Tested). Here we explain how we designed an isolated evaluation infrastructure combining Proxmox VE (0-NIC VMs), QEMU Guest Agent (virtio-serial), a budget-enforcing Mac relay, and Linux UID / systemd sandboxing to prevent test peeking, reward hacking, and API key leaks—along with a candid look at the current security boundaries and remaining limitations of our setup.

TL;DR

When benchmarking autonomous AI coding agents fairly and safely, engineers inevitably run into a three-way dilemma (trilemma):

1. Eval Cheating / Reward Hacking: If the AI agent and the grading test suite share the same user privileges and filesystem view, a capable model may inspect the test cases to hardcode edge-case branches or modify the test files (pytest.skip(), weakening assertions) instead of fixing the underlying bug.
2. API Key Exposure & Runaway Spend: Passing GEMINI_API_KEY or OPENAI_API_KEY directly into the guest environment creates a risk of credential leakage via logs/prompts and runaway billing if an agent enters an infinite retry loop.
3. Network Isolation vs. Cloud LLM Access: Stripping network interfaces (NICs) from the sandbox VM prevents unauthorized external traffic, but also cuts off direct HTTPS access to cloud LLM APIs.

To solve these challenges simultaneously, we combined a mini-PC running Proxmox VE with a macOS controller over a 1-to-1 Tailscale link to build the AI Labo Evaluation Sandbox:

• Network-Isolated VMs (0 NICs) + Inspected Relay Proxy: Experiment VMs have zero network interfaces. The sole external data path is structured JSON passed over QEMU Guest Agent (virtio-serial) to a macOS relay that loads the API key from the OS Keychain into memory once and enforces strict call, token, and USD budget caps (zero API keys inside the VM).
• OS-Level Separation via Linux UIDs and systemd (ProtectSystem=strict / InaccessiblePaths): While the agent (labexp) runs, ProtectSystem=strict mounts the filesystem read-only (except for pre-created target .py files bound via ReadWritePaths), and InaccessiblePaths blocks access to the grading suite (evaluation/).
• 0.8s Cached ISO Generation + Disposable Full-Clone VMs: Pre-verified .deb packages (verified via InRelease ➔ Packages SHA-256 chains) and experiment payloads are burned into read-only iso9660 CD-ROM images in 0.8 seconds, mounted onto a fresh full clone of a stopped Debian 13 base VM, and executed end-to-end in **60–90 seconds per experiment**.

1. Why Naive Single-User Execution Distorts Agent Benchmarks

Anyone who has evaluated coding agents at scale has seen Reward Hacking firsthand: the agent reports that all tests pass on Turn 1, but git diff reveals that instead of fixing the bug, it modified the test runner or weakened an assertion.

Running the Agent (Generator) and the Grader (Verifier) under the same user and writable filesystem introduces four structural failure modes:

💡 Why Proxmox VMs + systemd Instead of Docker Containers? You can achieve similar logical separation in Docker by splitting agent and grader containers, passing --network none and --read-only, and keeping secrets on a host proxy. We chose Proxmox VMs + systemd for two reasons:

1. Kernel Isolation: Avoiding shared host kernels between the controller/host and disposable guest environments.
2. Full-Clone & ISO Reproducibility: LVM full clones combined with immutable iso9660 virtual CD-ROMs make it trivial to pin and audit exact VM configuration SHA-256 digests before and after every run.

2. System Architecture (Mac × Tailscale × Proxmox × 0-NIC VM)

The infrastructure is split across a macOS Controller/Relay and a Proxmox VE Host connected via a 1-to-1 peer-restricted Tailscale VPN:

+-------------------------------------------------------------------------+
| 1. Controller & Relay Host (macOS)                                      |
|  - OS Keychain / 0400 Secret --> Loaded once into memory at startup     |
|  - Experiment Orchestrator   --> Drives Phase 1 (ISO) & Phase 2 (Run)   |
|  - Private API Relay         --> Enforces call, token & micro-USD caps  |
|  - Evidence Store            --> Verifies & saves 7 artifacts + SHA256  |
+------------------------------------+------------------------------------+
                                     |
         1-to-1 Peer-Restricted VPN  |  (SSH Control + Peer-Pinned HTTP)
                                     v
+-------------------------------------------------------------------------+
| 2. Virtualization Host (Proxmox VE Mini-PC)                             |
|  - Host Controller (driver.py)  --> Allowlisted qm/pvesh + Watchdogs    |
|  - Local ISO Builder            --> Verified .deb cache -> iso9660 CDs  |
|  - Golden Base VM (VM9300)      --> Stopped Debian 13 clone source      |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | 3. Disposable Sandbox VM (Full Clone per Experiment / 0 NICs)     |  |
|  |  - serial0 (/dev/ttyS0) : One-way boot & QGA readiness signal     |  |
|  |  - virtio-serial (QGA)  : Bidirectional RPC via `qm guest exec`   |  |
|  |  - ide2 & ide3 (CD-ROM) : Read-only offline packages & payload    |  |
|  |                                                                   |  |
|  |  +-----------------------------+  +----------------------------+  |  |
|  |  | Agent Sandbox (UID: labexp) |  | Grader Sandbox (labjudge)  |  |  |
|  |  | - PrivateNetwork=yes        |  | - Runs AFTER agent exits   |  |  |
|  |  | - ProtectSystem=strict      |  | - Evaluates 20 hidden cases|  |  |
|  |  | - InaccessiblePaths=/opt/.. |  |   + 6 boundary mutants     |  |  |
|  |  |   /evaluation (EACCES)      |  | - Writes grader-output.json|  |  |
|  |  +-----------------------------+  +----------------------------+  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

Full Component Matrix (15 Components, Privileges, and Security Roles)

3. Core Mechanism 1: OS-Level Separation via Linux UIDs and systemd

Inside each experiment VM, the Agent (labexp) and the Independent Grader (labjudge) run under separate Linux UIDs and separate systemd mount namespaces.

Before starting the agent, the root installer (guest/install.py) sets strict filesystem ownership and permissions:

• Grading directory (/opt/.../evaluation): Owned by root:labjudge with mode 0750 (files 0640), preventing labexp from reading it even at the basic DAC level.
• Work directories (work_cold_start/, work_episodic_memory/): Owned by root:root with mode 0555 (read/traverse only). Only the 3 target Python files inside each directory (validator.py, settlement.py, statement.py) are pre-created with labexp ownership (0644).

Below are the actual systemd unit files from experiment E20 (VM224):

Agent Execution Unit (ai-lab-e20-01-agent.service)

[Unit]
Description=AI Lab E20 Episodic Gotchas Memory Isolated Agent Runner (VM224)

[Service]
Type=oneshot
User=labexp20
Group=labexp20
WorkingDirectory=/var/lib/ai-lab-e20-vm224-01
ExecStart=/usr/bin/python3 -I -B /opt/ai-lab-e20-vm224-01/guest/run_guest.py
Restart=no
KillMode=control-group
TimeoutStartSec=240
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
PrivateNetwork=yes
ProtectSystem=strict
ProtectHome=yes
CapabilityBoundingSet=
AmbientCapabilities=
# Under ProtectSystem=strict, ONLY outbox and the 6 pre-created target files (3 files x 2 modes) are writable
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/outbox /var/lib/ai-lab-e20-vm224-01/work_cold_start/validator.py /var/lib/ai-lab-e20-vm224-01/work_cold_start/settlement.py /var/lib/ai-lab-e20-vm224-01/work_cold_start/statement.py /var/lib/ai-lab-e20-vm224-01/work_episodic_memory/validator.py /var/lib/ai-lab-e20-vm224-01/work_episodic_memory/settlement.py /var/lib/ai-lab-e20-vm224-01/work_episodic_memory/statement.py
# Grading code (evaluation/), initial fixtures, and evidence directories are overlaid with empty 0000 mounts
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/root /var/lib/ai-lab-e20-vm224-01/evidence /var/lib/ai-lab-e20-vm224-01/grade /opt/ai-lab-e20-vm224-01/evaluation /opt/ai-lab-e20-vm224-01/fixtures

Independent Grader Unit (ai-lab-e20-01-grade.service)

Only after agent.service terminates and the root RPC handler (guest_rpc.py) copies a snapshot of the edited files into grade/ does the grader service launch as labjudge20:

[Unit]
Description=AI Lab E20 Episodic Gotchas Memory Isolated Grader (VM224)

[Service]
Type=oneshot
User=labjudge20
Group=labjudge20
WorkingDirectory=/var/lib/ai-lab-e20-vm224-01/grade
ExecStart=/usr/bin/python3 -I -B /opt/ai-lab-e20-vm224-01/evaluation/grade_comparison.py
Restart=no
KillMode=control-group
TimeoutStartSec=30
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
PrivateNetwork=yes
ProtectSystem=strict
ProtectHome=yes
CapabilityBoundingSet=
AmbientCapabilities=
# The grader can only write to the single grader-output.json file
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/grader-output.json
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/root /var/lib/ai-lab-e20-vm224-01/evidence /var/lib/ai-lab-e20-vm224-01/inbox /var/lib/ai-lab-e20-vm224-01/outbox /var/lib/ai-lab-e20-vm224-01/work_cold_start /var/lib/ai-lab-e20-vm224-01/work_episodic_memory

This layered setup enforces three protections during agent execution:

1. Blocking Access to Grading Contents (DAC 0750 + InaccessiblePaths):InaccessiblePaths bind-mounts an empty directory with 0000 permissions over the target path in the unit's mount namespace. While listing /opt/ai-lab-.../ still shows the directory entry evaluation, any attempt to read or traverse into it fails immediately with Permission denied (EACCES).
2. Preventing File Creation, Deletion, or Rename (ProtectSystem=strict + Root-Owned 0555 Workdir): The primary mechanism blocking file creation or deletion is ProtectSystem=strict, which mounts the entire filesystem read-only and bind-mounts only the 6 listed .py files as writable (backed up by root-owned 0555 directory permissions so labexp cannot chmod the parent directory). Running Python with -I (isolated mode) also disables user site-packages and PYTHONPATH injection.
3. Structured JSON Actions Only: In our benchmark suite (E04–E44), the agent does not execute arbitrary shell commands; both the Mac relay (serve_once.py) and guest harness (pipeline_checks.py) validate structured JSON tool calls against a strict schema, verify ast.parse() syntax, and overwrite existing file inodes in-place via os.open(..., os.O_WRONLY | os.O_TRUNC | os.O_NOFOLLOW).

💡 How We Use Mutation Testing (Mutation Kill Rate) In experiments where the agent submits both production code and a test suite (test_suite.py in E06 Blind TDD, E07, E08, E10), passing functional tests isn't enough—we also need to know if the agent wrote weak, self-serving tests. Our grader (judges.py) runs the agent's test_suite.py against 6 deliberate boundary mutants of the reference implementation (e.g., changing >= 10000 to > 10000, removing isinstance(x, bool) checks, omitting -0 formatting) to measure the exact Mutation Kill Rate (0/6 to 6/6).

4. Core Mechanism 2: Dual Serial Channels (serial0 vs. virtio-serial) & Budget-Gated Relay

Because the experiment VM has zero network interfaces (verified at boot by asserting that /sys/class/net contains only lo), it is a network-isolated VM whose sole external channel is an inspected host-mediated relay.

We use two distinct serial mechanisms between the Proxmox host and the guest VM for different phases of the lifecycle:

1. serial0 (/dev/ttyS0 inside guest — emulated 16550 UART):
   • Role: Used exclusively for one-way boot progress notifications during the initial 25-second offline bootstrap (before qemu-guest-agent is installed from the CD-ROM). The host controller (driver.py) monitors /var/run/qemu-server/<vmid>.serial0 until it receives the common_prepared_QGA_acceptance_pending JSON marker.
2. virtio-serial (org.qemu.guest_agent.0 / .qga socket — QEMU Guest Agent):
   • Role: Used for all subsequent bidirectional RPC (qm guest exec <vmid>), including payload installation, polling the agent's Outbox, delivering LLM responses to the Inbox, and retrieving the 7 final evidence files.

End-to-End Execution Lifecycle (~60–90 Seconds per Experiment)

External Token Accounting & Pre-Call Budget Enforcement (usage.json)

Because the Mac relay (serve_once.py) mediates every LLM call outside the VM, the agent cannot tamper with token accounting. Before each generateContent call, the relay calls countTokens to verify that cumulative_inputs + prompt_tokens <= MAX_INPUT_TOKENS and cumulative_outputs + maxOutputTokens <= MAX_OUTPUT_TOKENS.

Upon receiving the response, the relay extracts promptTokenCount, candidatesTokenCount, and thoughtsTokenCount (reasoning tokens) from usageMetadata, computing billed_out = candidatesTokenCount + thoughtsTokenCount and recording exact costs in micro-USD (1e-6 USD) against a hard session cap (e.g., 350,000 micro-USD = $0.35).

5. Known Limitations & Future Security Roadmap

While this architecture is highly effective for benchmarking structured-action coding agents, evaluating untrusted, adversarial agents with arbitrary code/shell execution requires addressing several remaining security boundaries. We document these limitations openly for anyone building on this design:

Limitation 1: [Critical] The Grader (labjudge) Directly execs Agent-Submitted Code in the Same Process

• Current Behavior & Risk: During agent execution (labexp), InaccessiblePaths prevents the agent from reading evaluation/judges.py. However, once the agent finishes, judges.py runs as labjudge and calls exec(compile(code_bytes, ...), ns) on the submitted validator.py inside the grader process. If an adversarial agent planted a Trojan payload inside validator.py that executes upon import/call—such as reading /opt/.../evaluation/judges.py, inspecting the caller stack frame, or writing a fake passing JSON directly to /var/lib/.../grader-output.json (which is writable by labjudge via ReadWritePaths)—that payload would execute with labjudge privileges during grading. (In our 41 experiments, all submitted code bundles are archived in mode-a/b-bundle.json and verified to contain only pure domain logic, but the architectural boundary itself is porous against active malice.)
• Planned Remediation (Decoupling Untrusted Runner from Trusted Scorer):
  1. Split grading into two separate processes/UIDs: an Untrusted Test Runner (which imports the candidate code with zero access to judges.py or grader-output.json) and a Trusted Scorer (root or a separate UID).
  2. Pass test input vectors to the Untrusted Runner over a pipe (stdin/stdout), compare the returned JSON outputs against expected values in the Trusted Scorer, and have only the Trusted Scorer write grader-output.json.

Limitation 2: Denylist (InaccessiblePaths) Blind Spots & Mount Visibility

• Current Behavior & Risk:InaccessiblePaths is a denylist mechanism. If any unredacted copy of the experiment payload remains accessible elsewhere (for example, on the mounted iso9660 CD-ROM path if not unmounted or restricted before agent launch), a shell-capable agent could read grading assets from the mount source. Additionally, the current unit file can be hardened further:
  • Adding ProtectProc=invisible to hide /proc command-line arguments of other processes (such as root's guest_rpc.py).
  • Adding RestrictAddressFamilies=none (since PrivateNetwork=yes blocks TCP/IP and abstract sockets, but filesystem UNIX domain sockets remain reachable unless address families are restricted).
  • Adding MemoryMax=, TasksMax=, and LimitFSIZE= to prevent fork bombs or disk-filling DoS writes to outbox/.
• Planned Remediation: Transition from a denylist (InaccessiblePaths) to an allowlist root filesystem (TemporaryFileSystem=/ + minimal BindReadOnlyPaths=) and audit the unit with systemd-analyze security.

Limitation 3: Tamper-Evident vs. Tamper-Proof Evidence

• The SHA-256 manifest (manifest.json) makes the 7 evidence files tamper-evident across transit and storage, but it is not cryptographically tamper-proof against a host administrator who recalculates both the file and manifest hashes.

6. Two Real-World Operational Gotchas We Hit (and How We Fixed Them)

While running 40+ consecutive VM benchmarks, we encountered two non-obvious operational traps worth sharing:

Gotcha 1: QGA Timeouts When Passing Non-ASCII Strings Directly to qm guest exec -c

What We Observed

When invoking /usr/sbin/qm guest exec <vmid> -- /usr/bin/python3 -c <script_source> from the Proxmox host, early builds of E21 that included Japanese UTF-8 characters inside Python docstrings consistently failed with VM qga command 'guest-exec' failed - got timeout (exit code 255), leaving guest-ping unresponsive afterward.

Analysis & Best Practice Workaround

While virtio-serial itself is a byte stream and Proxmox's QGA client prepends guest-sync-delimited (\xff delimiter resynchronization), passing large multi-kilobyte scripts containing multi-byte UTF-8 characters directly as command-line arguments to Proxmox's /usr/sbin/qm guest exec CLI (which is implemented in Perl under the hood) triggered encoding/buffer handling stalls.

• Our Fix: We enforced a unit-tested 100% Pure ASCII (all(b < 128)) policy on the bootstrap scripts passed via -c (guest_rpc.py and install.py) and passed all experiment payloads as Base64-encoded JSON (payload_b64) over stdin. Across 40+ subsequent VM runs, we observed zero QGA timeouts.
• General Recommendation: Avoid passing raw multi-line script strings to qm guest exec -- python3 -c "...". Instead, pass a tiny Base64 decoder one-liner (python3 -c "import base64,sys;exec(base64.b64decode(sys.stdin.read()))") or write the file via guest-file-write first.

Gotcha 2: False-Positive LLM Safety Filter Blocks (PROHIBITED_CONTENT) During Infra Debugging

What We Observed

While using an AI coding assistant on our development machine to debug Proxmox socket states and systemd sandbox rules (not inside the benchmarked VMs themselves), the Gemini API occasionally blocked responses with:

This request was blocked by Gemini's filters. They can occasionally trigger by mistake on safe coding, security, or biology-related queries.

Analysis & Workaround

This is a known false positive (promptFeedback.blockReason: "PROHIBITED_CONTENT") documented on the Google AI Developers Forum that cannot be disabled via standard safety threshold sliders. Because the safety classifier evaluates the full conversation history, accumulating multiple inline snippets referencing raw hypervisor sockets (*.qmp, *.qga) and namespace boundary checks (InaccessiblePaths, runuser) likely resembled VM-escape or privilege-escalation exploit development. Switching our debugging workflow to invoke only standard /usr/sbin/qm guest exec CLI wrappers and pre-committed scripts eliminated these false positives completely (and zero filter blocks ever occurred inside the 41 benchmark runs).

7. The 7 Tamper-Evident Artifacts Collected per Run

Every metric reported in our benchmark series (E04–E44) is extracted directly from the 7 JSON files collected from the VM and verified against manifest.json:

1. run-summary.json: High-level comparison of Mode A vs. Mode B (tests passed, turns, elapsed seconds, tool output chars).
2. usage.json: External relay ledger recording promptTokenCount, candidatesTokenCount, thoughtsTokenCount, and estimated_micro_usd per call.
3. execution-traces.json: Turn-by-turn log of tool calls, output character counts, and sensor/hook triggers.
4. independent-tests.json: Individual pass/fail results for all 20 hidden test cases (plus mutation kill audits where applicable).
5. mode-a-bundle.json: Full snapshot of the final Python files submitted by the baseline agent.
6. mode-b-bundle.json: Full snapshot of the final Python files submitted by the harnessed agent.
7. actions.json: Normalized JSON action payloads returned by the model on every turn.

🔗 Explore the Benchmark Results

Ready to see what we discovered when we ran 41 agent harness patterns through this sandbox? Check out the master benchmark index and our #1 ranked pattern deep dive:

• 🚀 Master Benchmark Hub: I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md

💡 Living Benchmark Index (Last Updated: October 31, 2026 | Verified in Isolated Sandbox VMs: 41 Patterns)

Across X/Twitter, Hacker News, and engineering blogs from OpenAI, Anthropic, Google, Martin Fowler (Thoughtworks), and Databricks, dozens of "Agent Harness Engineering" patterns have been proposed to make coding agents (Claude Code, Cursor, Codex, custom agents) faster, cheaper, and more reliable.

To find out which patterns actually work—and which ones make things worse, we built an isolated Proxmox VM benchmark infrastructure (AI Labo) and ran controlled A/B experiments across 41 harness patterns (E04–E44) using identical models, identical initial codebases, a hidden 20-case grading suite, and mutation testing.

0. Introduction: What Is "Harness Engineering"?

Every engineer deploying AI coding agents in production eventually hits the same wall: "Why does the exact same frontier model (Claude, GPT, or Gemini) write a flawless patch in one turn today, and tomorrow burn 80,000 tokens editing the wrong files before breaking the test suite?"

When we started this benchmark series (E04–E24), we weren't setting out to define a new discipline—we were simply testing practical agent workflow ideas from developer communities bottom-up. As the data accumulated, we realized that our findings converged directly on what OpenAI, Anthropic, Google, and Thoughtworks call "Harness Engineering". We then expanded the suite to cover industry-standard patterns (E25–E42) and cross-model / cross-generation scaling laws (E43–E44).

At its core, Harness Engineering boils down to a single equation:

Agent = Model (Probabilistic Reasoning Core) + Harness (Deterministic Control System)

An LLM is a non-deterministic token predictor. Turning it into a reliable engineering system requires wrapping it in a Harness composed of three control layers:

1. 🧭 Guides (Feed-Forward Control — Before Writing Code): Steering the agent onto the right path before it generates a patch (e.g., directory-pointer AGENTS.md, AST call-graph injection, two-phase read-only plan locks, spec-first gates).
2. 🛡️ Sensors (Feedback Control — After Writing Code): Deterministically catching errors, regressions, and reward-hacking after an edit so the agent can self-correct (e.g., AST pre-commit hooks, actionable fix-hint linters, cryptographic test locks, loop-detection circuit breakers).
3. 🧹 Context Hygiene (Attention & Memory Management): Ruthlessly stripping noise that dilutes model attention (e.g., CLI test output token killers, failed-attempt rollback pruning, Tree-sitter skeleton context, episodic MEMORY.md gotchas).

🛡️ Methodology & Anti-Cheating Sandbox Architecture All 41 A/B experiments were executed inside disposable Debian 13 Proxmox VMs with zero network interfaces (0 NICs), controlled via QEMU Guest Agent (virtio-serial) and a budget-enforcing Mac relay, with Linux UID + systemd (ProtectSystem=strict / InaccessiblePaths) isolation separating the AI agent (labexp) from the hidden grading suite (labjudge). 👉 For a full technical deep dive into our evaluation infrastructure—including actual systemd unit files and security boundary limitations—read our companion post: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox.

1. Top 7 High-Impact Patterns (Ranked by Token Reduction & Accuracy)

Among all 41 verified patterns (E04–E44), these 7 patterns delivered the most dramatic improvements in input token reduction, first-turn pass rate (zero rework), and defect/mutant detection:

2. Why Does the "Harness" Matter More Than the Prompt?

Across all 41 experiments, five structural anti-patterns consistently caused token explosions and accuracy failures—and five deterministic harness mechanisms solved them:

Key Insight 1: Stage-Gated Graphs Work, but Role-Based Subagents Backfire (E04 vs. E05)

• In E04 (Loops vs. Graphs), replacing a single free-form loop with a stage-gated Directed Graph (resetting context after each module's deterministic verifier passes) cut input tokens by -74.4% and execution time by -48.0%.
• Conversely, in E05 (Subagent Folding), splitting the exact same pipeline into role-based subagents (Architect ➔ Implementer ➔ QA) passing natural-language handoff summaries caused a "Telephone Game" effect: subtle formatting invariants (like signed zero -0) were lost in summarization, triggering rework that made the multi-agent setup 2.66x slower and 2.2x more expensive than a folded single loop with on-demand skill loading.

Key Insight 2: Self-Testing and Self-Review Yield a 0% Mutation Kill Rate (E06, E08, E10, E22)

• In E06 (Blind TDD) and E10 (Adversarial Critic), when a single agent context wrote both the implementation and the unit tests (or reviewed its own code), the agent unconsciously wrote weak "happy-path" tests tailored to pass its own bugs—resulting in a 0/6 (0%) mutation kill rate against boundary mutants.
• Isolating the test author into a Blind Tester (E06) that sees only SPEC.md (never the implementation), enforcing Spec-First SDD (E08), using an Adversarial Critic (E10) rewarded for finding counterexamples, or locking production files until a failing reproduction test is recorded (Reproduction-First Gate E22) pushed test mutation kill rates and hidden-suite accuracy to 100%.

Key Insight 3: 90% of Input Tokens Are Wasted on Exploration, Unused Schemas, and Raw Logs (E09, E14, E16, E19, E21)

• Across every experiment, the actual fixed Python code generated by the LLM was only 1,200 to 1,800 output tokens.
• Why did baseline agents consume 30,000 to 92,000 input tokens? Because of blind grep exploration (E09), 30 unused MCP tool schemas (E14), 76 passing test lines in raw terminal output (E16), failed code diffs left in context (E19), and full source bodies of dependency files (E21).
• Filtering these deterministically in the harness cuts API costs by 80%–95% while simultaneously preventing Lost-in-the-Middle attention degradation.

Key Insight 4: A Cheap Model + Great Harness Beats an Unharnessed Frontier Model (E43 & E44)

• In E43 (Cross-Model Giant Killing), we pitted an unharnessed frontier model (gemini-4-argon) in a raw loop against a model costing 1/8th as much (gemini-3.1-flash-lite) equipped with our combined golden harness (E09 AST Map + E20 MEMORY.md + E21 Skeleton + E11 Hook + E16 Token Killer + E39 Remediation Sensor).
• The harnessed lightweight model achieved 20/20 (100%) in 1 turn (5.6s, $0.0029), beating the unharnessed frontier model (15/20 in 3 turns, 39.1s, $0.1538)—delivering better accuracy at 6.95x the speed and -98.1% lower cost.
• In E44 (Model Evolution vs. Harness Boundary), testing across four model generations (2.5-flash ➔ 3.5-flash ➔ 3.8-flash ➔ 4-argon) revealed that while syntax-assist harnesses (like diff format helpers) gradually become less critical as models improve, repository-knowledge & physical guardrail harnesses (MEMORY.md, output filters, circuit breakers, cryptographic test locks) remain indispensable even for the latest frontier models.

3. Master Benchmark Table: All 41 Patterns (E04–E44)

(Note: Each experiment represents a controlled n=1 A/B case study on a 3-module domain pipeline evaluated against 20 hidden test cases. Detailed deep-dive posts are linked as they are published.)

4. A 5-Point Checklist You Can Apply to Your Agent Harness Tomorrow

1. Never return raw CLI test or linter stdout to the LLM (E16 / E17) Wrap your test runner to strip passing tests and ANSI codes. Return only a structured JSON of failed assertions plus a 1-line 💡 Fix Hint showing the expected helper or pattern.
2. Replace negative prompt prohibitions with Pre-Commit Hooks and 1 Golden Example (E11 / E24) Listing 20 "DO NOT do X" rules in CLAUDE.md triggers the Pink Elephant effect. Show one 20-line affirmative code example and enforce constraints mechanically via AST save/commit hooks.
3. Maintain a 5-to-15 line MEMORY.md of repository-specific gotchas (E20 / E35) Instead of a 500-line encyclopedia, keep AGENTS.md as a 30-line directory pointer and maintain a tiny MEMORY.md listing only non-obvious domain invariants that previously caused test failures.
4. Use line-number-free SEARCH/REPLACE blocks + Atomic Multi-File Batches (E15 / E23) Unified diffs (@@ -l,s +l,s @@) frequently fail due to LLM line-counting errors. Apply multi-file interface changes atomically in a single turn before running verification tests.
5. Separate the Test Author context from the Implementer context (E06 / E22) Never trust an agent to grade its own homework in the same context. Lock production files until a reproduction test fails (E22), or generate tests in a blind worktree that only sees the spec (E06).

🔗 Explore the Series

• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md
• 🥈 Rank #2 Deep Dive (E21): Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons
• 🥉 Rank #3 Deep Dive (E16): Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%
• 🌿 Rank #4 Deep Dive (E19): Why Keeping Failed Code Attempts in Chat History Poisons Your Coding Agent: Error-Trace Pruning vs. Accumulated History
• 🔍 Rank #5 Deep Dive (E14): The Hidden "Schema Tax" of MCP Tools: Dumping 32 Tool Schemas vs. On-Demand Tool Search (-92.7% Input Tokens)
• 🗜️ Rank #6 Deep Dive (E12): How Save-Game Context Compaction Fixes Spec Drift in Multi-Step Coding Agents (-91.9% Input Tokens)
• 🧭 Rank #7 Deep Dive (E18): Benchmarking the Claude Code /advisor Pattern: Solo Executor Loops vs. Upfront Advisor Blueprint Routing