π‘ AI Labo Benchmark Series β Experiment
E44(Model Evolution vs. Irreplaceable Harness Boundary)This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
Metadata & Experiment Identity
- Experiment ID:
E44(Backlog CandidateC040) - Run ID:
e44-20261030-vm248-01 - Sandbox Environment: Proxmox Isolated VM
VM248(ai-e44-248, 0 NICs,PrivateNetwork=yes,NoNewPrivileges=yes,InaccessiblePaths, UID splitlabexp40/labjudge44) - Model:
gemini-3.8-flash(thinkingLevel: LOW,temperature=0.0) - Pattern Under Test: "What Harnesses Become Obsolete as Models Scale (
2.5 β 3.5 β 3.8 β 4-Argon), and What Remains Irreplaceable?" β Syntax-Only Obsolete Harness (SyntaxOnlyObsoleteHarnessRunner) vs. Irreplaceable Core Boundary Harness (IrreplaceableCoreBoundaryRunner)
1. Executive Summary (TL;DR)
As foundation models evolve across generations (2.5 β 3.5 β 3.8 β 4-Argon), a critical architectural question arises: Which agent harnesses become obsolete "scaffolding" absorbed by better model weights, and which harnesses remain permanently irreplaceable?
Our cross-generation analysis reveals a sharp boundary:
- Fading Syntax & Reasoning Scaffolding (
E15,E17,E27): Harnesses that compensate for basic diff syntax errors or simple linter syntax mistakes gradually fade in importance as newer models master patch formatting. - The 4 Irreplaceable Core Harnesses (
E16,E20/E35,E28,E38): No matter how smart a model's pre-trained weights become, it cannot read your team's private, undocumented repository invariants by telepathy (E20/E35), nor can it avoid paying token costs on 85,000 characters of unfiltered CLI stdout (E16), nor can prompt weights replace OS/cryptographic test-tampering locks (E38).
We tested this boundary head-to-head inside isolated VM VM248 on the 3-module enterprise payment settlement pipeline (validator.py β settlement.py β statement.py):
- Mode A (
SyntaxOnlyObsoleteHarnessRunner): Retains only syntax scaffolding (E15/E17/E27) while omitting the Irreplaceable Core (E16Output Filter,E20/E35Domain Memory,E28Circuit Breaker,E38Crypto Lock), burning 3 turns,85,395chars of raw tool output, and51,285cumulative input tokens. - Mode B (
IrreplaceableCoreBoundaryRunner): Integrates the 4 Irreplaceable Core Harnesses (IRREPLACEABLE_CORE_HARNESS_MD+submit_irreplaceable_core_bundle), solving20 / 20(100%) in 1 turn with2,333input tokens.
Key Result
While both modes reached 20 / 20 on the hidden grading suite, Mode B cut LLM calls by -66.7% (3 β 1), cumulative input tokens by -95.5% (51,285 β 2,333 tokensβa 21.98x reduction), API cost by -84.4% ($0.0497 β $0.0077), and ran 2.39x faster (31.24s β 13.08s, -58.1%).
| Metric | Mode A: Syntax-Only Scaffolding (SyntaxOnlyObsoleteHarness) |
Mode B: Irreplaceable Core (IrreplaceableCoreBoundary) |
Delta / Improvement |
|---|---|---|---|
Hidden Grading Suite Score (labjudge44) |
20 / 20 (100% β Turn 3) |
20 / 20 (100% β Turn 1 One-Shot) |
1-shot convergence (3 β 1 turn) |
Missing-Core Rework Traps (gotcha_traps_hit) |
2 traps (Turns 1 & 2) |
0 traps |
-100% (zero rediscovery loops) |
LLM Calls (model_calls) |
3 calls |
1 call (1-shot) |
-66.7% (3.00x faster convergence) |
Cumulative File Edits (file_edits) |
5 edits |
3 edits (1 per module) |
-40.0% fewer edits |
Cumulative Tool Output (tool_output_chars) |
85,395 chars |
104 chars |
-99.9% (821.1x noise reduction) |
| Turn 1 Tokens (In / Out) | 1,667 in / 15 out |
2,333 in / 1,597 out |
+666 in for 4-pillar core spec |
| Turn 2 Tokens (In / Out) | 16,695 in / 1,321 out |
β (completed on Turn 1) | Avoided 42K-char unfiltered log dump |
| Turn 3 Tokens (In / Out) | 32,923 in / 1,660 out |
β (completed on Turn 1) | Avoided secondary rework bloat |
Cumulative Input Tokens (promptTokenCount) |
51,285 tokens |
2,333 tokens |
-95.5% (21.98x efficiency) |
Cumulative Output Tokens (candidatesTokenCount) |
2,996 tokens |
1,597 tokens |
-46.7% output reduction |
Elapsed Execution Time (elapsed_seconds) |
31.24 sec |
13.08 sec |
-58.1% (2.39x faster) |
Estimated API Cost (estimated_micro_usd) |
$0.0497 (49,700 micro-USD) |
$0.0077 (7,739 micro-USD) |
-84.4% (6.42x cheaper) |
2. Experiment Architecture: Fading Syntax Scaffolding vs. The 4 Irreplaceable Core Harnesses
+-------------------------------------------------------------------------+
| Proxmox Isolated VM248 (0 NICs, InaccessiblePaths, grader labjudge44) |
| Target Task: 3-Module Enterprise Payment Settlement Pipeline |
| validator.py β settlement.py β statement.py |
| |
| +-------------------------------------------------------------------+ |
| | Mode A: SyntaxOnlyObsoleteHarnessRunner (Missing Core 4) | |
| | - Turn 1 (1,667 in): Calls `run_syntax_only_scaffold_check` | |
| | β 42,818 chars of unfiltered test logs | |
| | - Turn 2 (16,695 in): Calls `stage_missing_core_memory_fix` | |
| | β 42,472 chars of secondary log bloat | |
| | - Turn 3 (32,923 in): Converges only after rediscovering repo | |
| | invariants by trial-and-error | |
| | --> 3 calls | 51,285 in | $0.0497 | 31.24s | 20/20 (2 Traps Hit) | |
| +-------------------------------------------------------------------+ |
| |
| +-------------------------------------------------------------------+ |
| | Mode B: IrreplaceableCoreBoundaryRunner (The 4 Core Pillars) | |
| | - Integrates `IRREPLACEABLE_CORE_HARNESS_MD` (+666 tokens): | |
| | 1. `E20`/`E35` Scoped Domain Memory (`MEMORY.md` + `AGENTS.md`)| |
| | 2. `E16` CLI Output Token Killer Filter | |
| | 3. `E28` Loop-Detection Circuit Breaker | |
| | 4. `E38` Cryptographic SHA-256 Test Integrity Lock | |
| | - Turn 1 (2,333 in): Calls `submit_irreplaceable_core_bundle` | |
| | --> 1 call | 2,333 in | $0.0077 | 13.08s | 20/20 (100% PASS) | |
| +-------------------------------------------------------------------+ |
+-------------------------------------------------------------------------+
3. Drop-In Implementation Recipe: IrreplaceableCoreHarness
Build your long-term agent infrastructure around the 4 pillars that never become obsolete when upgrading models:
import hashlib
class IrreplaceableCoreHarness:
"""Combines the 4 model-agnostic core pillars: Scoped Domain Memory, Output Filter, Circuit Breaker, and Test Lock."""
def __init__(self, scoped_domain_memory: dict[str, str], canonical_tests: dict[str, str]):
self.domain_memory = scoped_domain_memory
self.test_hashes = {
k: hashlib.sha256(v.encode("utf-8")).hexdigest() for k, v in canonical_tests.items()
}
self.failure_history: list[str] = []
def prepare_turn_context(self, target_files: list[str], raw_tool_output: str) -> str:
scoped_rules = [self.domain_memory[f] for f in target_files if f in self.domain_memory]
filtered_output = "\n".join(
line
for line in raw_tool_output.splitlines()
if any(tag in line for tag in ("ERROR", "FAIL", "BOUNDARY", "MISMATCH"))
)[:1200]
return "# SCOPED DOMAIN MEMORY\n" + "\n".join(scoped_rules) + "\n\n# FILTERED SENSOR LOG\n" + filtered_output
def verify_immutable_tests(self, current_tests: dict[str, str]) -> bool:
return all(
hashlib.sha256(current_tests.get(k, "").encode("utf-8")).hexdigest() == sha
for k, sha in self.test_hashes.items()
)
4. Practical Takeaway: Prune Syntax Crutches, Double Down on Context Hygiene & Physical Locks
- Pre-training weights never contain your private repository's implicit contracts: Even a future GPT-6 or Gemini 5 cannot guess whether your internal settlement engine charges a reserve at
>= 300bps or> 300bps unlessMEMORY.md(E20) and a Directory-PointerAGENTS.md(E35) tell it. - Keep your harness modular so you can subtract syntax crutches as models improve: Retire heavy reasoning templates when upgrading to frontier reasoning models, while permanently keeping
E16(Output Filter),E20/E35(Domain Memory),E28(Circuit Breaker), andE38(Cryptographic Test Lock).
π Continue Reading the Series
- π Master Benchmark Hub: I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
- π‘οΈ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
- βοΈ Previous Sequential Post (
E43): Cross-Model Giant Killing: How 1/8th-Cost Flash-Lite + Golden Harnesses Beats Unharnessed Frontier Pro (-98.1% Cost, 6.95x Faster) - π§Ύ Next Sequential Post (
E45): Stop Accepting 'Done' Without Proof: Unsupported Completion Claims vs. a Risk-Tiered Change Receipt Gate (-95.1% Tokens) - π₯ Related Rank #1 Deep Dive (
E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-ItemMEMORY.md
Comments
Join the discussionNo comments yet. Be the first to comment.