Skip to main content

Which Harnesses Survive Model Scaling? Syntax Scaffolding vs. the 4 Irreplaceable Core Harnesses (-95.5% Input Tokens)

Updated 7 min readMarkdown

πŸ’‘ AI Labo Benchmark Series β€” Experiment E44 (Model Evolution vs. Irreplaceable Harness Boundary)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.


Metadata & Experiment Identity

  • Experiment ID: E44 (Backlog Candidate C040)
  • Run ID: e44-20261030-vm248-01
  • Sandbox Environment: Proxmox Isolated VM VM248 (ai-e44-248, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp40 / labjudge44)
  • Model: gemini-3.8-flash (thinkingLevel: LOW, temperature=0.0)
  • Pattern Under Test: "What Harnesses Become Obsolete as Models Scale (2.5 βž” 3.5 βž” 3.8 βž” 4-Argon), and What Remains Irreplaceable?" β€” Syntax-Only Obsolete Harness (SyntaxOnlyObsoleteHarnessRunner) vs. Irreplaceable Core Boundary Harness (IrreplaceableCoreBoundaryRunner)

1. Executive Summary (TL;DR)

As foundation models evolve across generations (2.5 βž” 3.5 βž” 3.8 βž” 4-Argon), a critical architectural question arises: Which agent harnesses become obsolete "scaffolding" absorbed by better model weights, and which harnesses remain permanently irreplaceable?

Our cross-generation analysis reveals a sharp boundary:

  • Fading Syntax & Reasoning Scaffolding (E15, E17, E27): Harnesses that compensate for basic diff syntax errors or simple linter syntax mistakes gradually fade in importance as newer models master patch formatting.
  • The 4 Irreplaceable Core Harnesses (E16, E20/E35, E28, E38): No matter how smart a model's pre-trained weights become, it cannot read your team's private, undocumented repository invariants by telepathy (E20/E35), nor can it avoid paying token costs on 85,000 characters of unfiltered CLI stdout (E16), nor can prompt weights replace OS/cryptographic test-tampering locks (E38).

We tested this boundary head-to-head inside isolated VM VM248 on the 3-module enterprise payment settlement pipeline (validator.py βž” settlement.py βž” statement.py):

  • Mode A (SyntaxOnlyObsoleteHarnessRunner): Retains only syntax scaffolding (E15/E17/E27) while omitting the Irreplaceable Core (E16 Output Filter, E20/E35 Domain Memory, E28 Circuit Breaker, E38 Crypto Lock), burning 3 turns, 85,395 chars of raw tool output, and 51,285 cumulative input tokens.
  • Mode B (IrreplaceableCoreBoundaryRunner): Integrates the 4 Irreplaceable Core Harnesses (IRREPLACEABLE_CORE_HARNESS_MD + submit_irreplaceable_core_bundle), solving 20 / 20 (100%) in 1 turn with 2,333 input tokens.

Key Result

While both modes reached 20 / 20 on the hidden grading suite, Mode B cut LLM calls by -66.7% (3 βž” 1), cumulative input tokens by -95.5% (51,285 βž” 2,333 tokensβ€”a 21.98x reduction), API cost by -84.4% ($0.0497 βž” $0.0077), and ran 2.39x faster (31.24s βž” 13.08s, -58.1%).

Metric Mode A: Syntax-Only Scaffolding (SyntaxOnlyObsoleteHarness) Mode B: Irreplaceable Core (IrreplaceableCoreBoundary) Delta / Improvement
Hidden Grading Suite Score (labjudge44) 20 / 20 (100% β€” Turn 3) 20 / 20 (100% β€” Turn 1 One-Shot) 1-shot convergence (3 βž” 1 turn)
Missing-Core Rework Traps (gotcha_traps_hit) 2 traps (Turns 1 & 2) 0 traps -100% (zero rediscovery loops)
LLM Calls (model_calls) 3 calls 1 call (1-shot) -66.7% (3.00x faster convergence)
Cumulative File Edits (file_edits) 5 edits 3 edits (1 per module) -40.0% fewer edits
Cumulative Tool Output (tool_output_chars) 85,395 chars 104 chars -99.9% (821.1x noise reduction)
Turn 1 Tokens (In / Out) 1,667 in / 15 out 2,333 in / 1,597 out +666 in for 4-pillar core spec
Turn 2 Tokens (In / Out) 16,695 in / 1,321 out β€” (completed on Turn 1) Avoided 42K-char unfiltered log dump
Turn 3 Tokens (In / Out) 32,923 in / 1,660 out β€” (completed on Turn 1) Avoided secondary rework bloat
Cumulative Input Tokens (promptTokenCount) 51,285 tokens 2,333 tokens -95.5% (21.98x efficiency)
Cumulative Output Tokens (candidatesTokenCount) 2,996 tokens 1,597 tokens -46.7% output reduction
Elapsed Execution Time (elapsed_seconds) 31.24 sec 13.08 sec -58.1% (2.39x faster)
Estimated API Cost (estimated_micro_usd) $0.0497 (49,700 micro-USD) $0.0077 (7,739 micro-USD) -84.4% (6.42x cheaper)

2. Experiment Architecture: Fading Syntax Scaffolding vs. The 4 Irreplaceable Core Harnesses

+-------------------------------------------------------------------------+
| Proxmox Isolated VM248 (0 NICs, InaccessiblePaths, grader labjudge44)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py βž” settlement.py βž” statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: SyntaxOnlyObsoleteHarnessRunner (Missing Core 4)          |  |
|  |  - Turn 1 (1,667 in):  Calls `run_syntax_only_scaffold_check`     |  |
|  |                        βž” 42,818 chars of unfiltered test logs     |  |
|  |  - Turn 2 (16,695 in): Calls `stage_missing_core_memory_fix`      |  |
|  |                        βž” 42,472 chars of secondary log bloat      |  |
|  |  - Turn 3 (32,923 in): Converges only after rediscovering repo    |  |
|  |                        invariants by trial-and-error              |  |
|  |  --> 3 calls | 51,285 in | $0.0497 | 31.24s | 20/20 (2 Traps Hit) |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: IrreplaceableCoreBoundaryRunner (The 4 Core Pillars)      |  |
|  |  - Integrates `IRREPLACEABLE_CORE_HARNESS_MD` (+666 tokens):      |  |
|  |    1. `E20`/`E35` Scoped Domain Memory (`MEMORY.md` + `AGENTS.md`)|  |
|  |    2. `E16` CLI Output Token Killer Filter                        |  |
|  |    3. `E28` Loop-Detection Circuit Breaker                        |  |
|  |    4. `E38` Cryptographic SHA-256 Test Integrity Lock             |  |
|  |  - Turn 1 (2,333 in): Calls `submit_irreplaceable_core_bundle`    |  |
|  |  --> 1 call  | 2,333 in  | $0.0077 | 13.08s | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Drop-In Implementation Recipe: IrreplaceableCoreHarness

Build your long-term agent infrastructure around the 4 pillars that never become obsolete when upgrading models:

import hashlib


class IrreplaceableCoreHarness:
    """Combines the 4 model-agnostic core pillars: Scoped Domain Memory, Output Filter, Circuit Breaker, and Test Lock."""

    def __init__(self, scoped_domain_memory: dict[str, str], canonical_tests: dict[str, str]):
        self.domain_memory = scoped_domain_memory
        self.test_hashes = {
            k: hashlib.sha256(v.encode("utf-8")).hexdigest() for k, v in canonical_tests.items()
        }
        self.failure_history: list[str] = []

    def prepare_turn_context(self, target_files: list[str], raw_tool_output: str) -> str:
        scoped_rules = [self.domain_memory[f] for f in target_files if f in self.domain_memory]
        filtered_output = "\n".join(
            line
            for line in raw_tool_output.splitlines()
            if any(tag in line for tag in ("ERROR", "FAIL", "BOUNDARY", "MISMATCH"))
        )[:1200]
        return "# SCOPED DOMAIN MEMORY\n" + "\n".join(scoped_rules) + "\n\n# FILTERED SENSOR LOG\n" + filtered_output

    def verify_immutable_tests(self, current_tests: dict[str, str]) -> bool:
        return all(
            hashlib.sha256(current_tests.get(k, "").encode("utf-8")).hexdigest() == sha
            for k, sha in self.test_hashes.items()
        )

4. Practical Takeaway: Prune Syntax Crutches, Double Down on Context Hygiene & Physical Locks

  1. Pre-training weights never contain your private repository's implicit contracts: Even a future GPT-6 or Gemini 5 cannot guess whether your internal settlement engine charges a reserve at >= 300 bps or > 300 bps unless MEMORY.md (E20) and a Directory-Pointer AGENTS.md (E35) tell it.
  2. Keep your harness modular so you can subtract syntax crutches as models improve: Retire heavy reasoning templates when upgrading to frontier reasoning models, while permanently keeping E16 (Output Filter), E20/E35 (Domain Memory), E28 (Circuit Breaker), and E38 (Cryptographic Test Lock).

πŸ”— Continue Reading the Series

No comments yet. Be the first to comment.

More from this blog