π‘ AI Labo Benchmark Series β Experiment
E26(Orchestration & Phase Gates)This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
Metadata & Experiment Identity
- Experiment ID:
E26(Backlog CandidateC022) - Run ID:
e26-20261029-vm230-01 - Sandbox Environment: Proxmox Isolated VM
VM230(ai-e26-230, 0 NICs,PrivateNetwork=yes,NoNewPrivileges=yes,InaccessiblePaths, UID splitlabexp26/labjudge26) - Model:
gemini-3.8-flash(thinkingLevel: LOW,temperature=0.0) - Pattern Under Test: "Claude Code Plan Mode (
Shift+TabRead-Only Enforcement) & Two-Phase Exploration Before Write Unlock" β Interleaved Read-Write Loop (InterleavedReadWriteRunner) vs. Two-Phase Read-Only Plan Lock (TwoPhaseReadOnlyPlanLockRunner)
1. Executive Summary (TL;DR)
When an AI coding agent has unrestricted access to both Read (read_file, grep) and Write (edit_file, write_file) tools from Turn 1, its natural reflex on multi-file tasks is Premature Interleaved Patching: it reads the first upstream file (validator.py), immediately writes a local patch before inspecting downstream consumers (settlement.py and statement.py), and triggers a cascade of cross-module schema mismatches.
This is the exact failure mode that inspired Plan Mode (Shift+Tab) in Claude Code and two-phase exploration gates in production coding agents: physically disable all Write tools during Phase 1 until the agent has inspected every affected module and locked a cross-module impact plan, then unlock Write tools in Phase 2 for a single, coordinated execution pass.
We tested this head-to-head inside isolated VM VM230 on the 3-module enterprise payment settlement pipeline (validator.py β settlement.py β statement.py):
- Mode A (
InterleavedReadWriteRunner): Allows interleaved read/write tool calls from Turn 1, resulting in premature upstream edits before reading downstream consumers (77,467chars of rework cascade logs across Turns 1β2). - Mode B (
TwoPhaseReadOnlyPlanLockRunner): Enforces a Phase 1 Read-Only Lock (chmod 0555/ middleware gate) until all 3 modules are inspected and a verified cross-module modification plan (TWO_PHASE_READONLY_PLAN_LOCK_MD) is locked, unlocking Phase 2 for a 1-shot atomic implementation.
Key Result
Mode A stalled at 17 / 20 (85.0%) after 3 turns (43,300 cumulative input tokens), missing extra key filtering (item = dict(raw_item)) and $0 authorizations (cents <= 0). Mode B achieved a 20 / 20 (100%) pass in 1 turn, cutting LLM calls by -66.7% (3 β 1), cumulative input tokens by -94.8% (43,300 β 2,262 tokensβa 19.14x reduction), API cost by -82.2% ($0.0430 β $0.0076), and running 2.22x faster (29.27s β 13.16s, -55.1%).
| Metric | Mode A: Interleaved Read-Write (InterleavedReadWrite) |
Mode B: Read-Only Plan Lock (TwoPhaseReadOnlyPlanLock) |
Delta / Improvement |
|---|---|---|---|
Hidden Grading Suite Score (labjudge26) |
17 / 20 (85.0% β 3 failures) |
20 / 20 (100% β Perfect) |
+15.0% accuracy (+3 cases) |
Premature Write Traps (gotcha_traps_hit) |
2 traps (Turns 1 & 2) |
0 traps |
-100% (zero rework) |
LLM Calls (model_calls) |
3 calls |
1 call (1-shot) |
-66.7% (3 β 1 turn) |
Cumulative File Edits (file_edits) |
5 edits |
3 edits (1 per module) |
-40.0% fewer edits |
Cumulative Tool Output (tool_output_chars) |
77,467 chars |
104 chars |
-99.9% (744.9x noise reduction) |
| Turn 1 Tokens (In / Out) | 1,655 in / 12 out |
2,262 in / 1,586 out |
+607 in for Phase 1 locked plan |
| Turn 2 Tokens (In / Out) | 14,063 in / 1,225 out |
β (completed on Turn 1) | Avoided premature patch cascade |
| Turn 3 Tokens (In / Out) | 27,582 in / 1,571 out |
β (completed on Turn 1) | Avoided secondary rework bloat |
Cumulative Input Tokens (promptTokenCount) |
43,300 tokens |
2,262 tokens |
-94.8% (19.14x efficiency) |
Cumulative Output Tokens (candidatesTokenCount) |
2,808 tokens |
1,586 tokens |
-43.5% output reduction |
Elapsed Execution Time (elapsed_seconds) |
29.27 sec |
13.16 sec |
-55.1% (2.22x faster) |
Estimated API Cost (estimated_micro_usd) |
$0.0430 (43,006 micro-USD) |
$0.0076 (7,644 micro-USD) |
-82.2% (5.63x cheaper) |
2. Experiment Architecture: Interleaved Read-Write vs. Two-Phase Plan Lock
+-------------------------------------------------------------------------+
| Proxmox Isolated VM230 (0 NICs, InaccessiblePaths, grader labjudge26) |
| Target Task: 3-Module Enterprise Payment Settlement Pipeline |
| validator.py β settlement.py β statement.py |
| |
| +-------------------------------------------------------------------+ |
| | Mode A: InterleavedReadWriteRunner (Write Unlocked From Turn 1) | |
| | - Turn 1 (1,655 in): Calls `interleaved_premature_write` before | |
| | inspecting downstream consumers (38,748c) | |
| | - Turn 2 (14,063 in): Calls `stage_rework_patch` β partial 2-file| |
| | edit leaves statement desynced (38,402c) | |
| | - Turn 3 (27,582 in): Final submit misses `item = dict(raw_item)`| |
| | extra-key drop & `cents <= 0` $0 auth | |
| | --> 3 calls | 43,300 in | $0.0430 | 29.27s | 17/20 (3 FAILED) | |
| +-------------------------------------------------------------------+ |
| |
| +-------------------------------------------------------------------+ |
| | Mode B: TwoPhaseReadOnlyPlanLockRunner (Phase 1 Lock β Phase 2) | |
| | - Phase 1 [Read-Only Enforcement]: Write tools physically locked | |
| | until all 3 modules & cross-module contracts are mapped into | |
| | `TWO_PHASE_READONLY_PLAN_LOCK_MD` (+607 tokens) | |
| | - Phase 2 [Write Unlocked, Turn 1 (2,262 in)]: | |
| | Executes `unlock_plan_and_submit_bundle` across all 3 files | |
| | --> 1 call | 2,262 in | $0.0076 | 13.16s | 20/20 (100% PASS) | |
| +-------------------------------------------------------------------+ |
+-------------------------------------------------------------------------+
3. Drop-In Implementation Recipe: TwoPhasePlanLockMiddleware
Asking an LLM in the system prompt "Please read all files before editing" is a soft instruction that models routinely ignore when they think a fix looks obvious. Instead, enforce the two-phase boundary deterministically in your tool dispatcher:
class TwoPhasePlanLockMiddleware:
"""Physically blocks Write tools in Phase 1 until a cross-module impact plan is locked."""
READ_ONLY_TOOLS = {"read_file", "grep_search", "submit_impact_plan"}
WRITE_TOOLS = {"edit_file", "write_file", "apply_batch_patch"}
def __init__(self, required_modules: set[str]):
self.required_modules = set(required_modules)
self.plan_locked = False
self.locked_plan: dict[str, str] = {}
def dispatch_tool(self, tool_name: str, args: dict) -> dict:
if not self.plan_locked and tool_name in self.WRITE_TOOLS:
raise PermissionError(
f"[PLAN LOCK ACTIVE] Tool '{tool_name}' is blocked in Phase 1. "
"Complete read-only inspection and call 'submit_impact_plan' covering all modules first."
)
if tool_name == "submit_impact_plan":
plan = args.get("module_plan", {})
missing = self.required_modules - set(plan.keys())
if missing:
return {"locked": False, "error": f"Plan missing modules: {sorted(missing)}"}
self.plan_locked = True
self.locked_plan = plan
return {"locked": True, "message": "Phase 2 Write tools unlocked."}
return {"allowed": True}
4. Practical Takeaway: Enforce Read-Only Exploration via Tool Permissions, Not Prose
- Strip Write tool schemas (or gate them with
PermissionError/chmod 0555) during initial exploration: When an agent cannot calledit_fileon Turn 1, it is forced to trace upstream producers (validator.py) all the way to downstream consumers (settlement.pyandstatement.py) before committing to a data structure. - Require a structured module-by-module impact plan to unlock Phase 2: Pair
TwoPhasePlanLockMiddleware(E26) with Atomic Multi-File Batch Edits (E23) so that once Phase 2 unlocks, all cross-module edits land in a single transaction.
π Continue Reading the Series
- π Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
- π‘οΈ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
- β‘ Previous Sequential Post (
E25): Stop Invalidating Your Coding Agent's KV Cache: Dynamic System Prompt Mutation vs. Append-Only Prefix-Stable Prompt Caching (2.63x Faster) - ποΈ Next Sequential Post (
E27): Stop Your Coding Agent From Cycling Through Refuted Guesses: Free-Text Monologue vs. a Structured Hypothesis Matrix (2.54x Faster) - π Related Spec-Lock Deep Dive (
E08): Vibe Coding vs. Spec-First SDD: How Locking an Executable Spec Eliminates Rework and Cuts Input Tokens by 32.9%
Comments
Join the discussionNo comments yet. Be the first to comment.