π‘ AI Labo Benchmark Series β Experiment
E43(Cross-Model Scaling & Giant Killing)This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
Metadata & Experiment Identity
- Experiment ID:
E43(Backlog CandidateC039) - Run ID:
e43-20261030-vm247-01 - Sandbox Environment: Proxmox Isolated VM
VM247(ai-e43-247, 0 NICs,PrivateNetwork=yes,NoNewPrivileges=yes,InaccessiblePaths, UID splitlabexp40/labjudge43) - Models Tested: Mode A:
gemini-3.1-pro-preview($2.00 / $12.00per 1M tokens) vs. Mode B:gemini-3.1-flash-lite($0.25 / $1.50per 1M tokens β 1/8th unit price) - Pattern Under Test: "AI Labo Scaling Law & Harness Engineering Meta-Analysis: Does a Lightweight Model Equipped with Golden Harnesses Outperform an Unharnessed Frontier Model?" β Unharnessed Frontier Pro (
UnharnessedFrontierProRunner) vs. Harnessed Flash-Lite Giant Killing (HarnessedFlashLiteGiantKillingRunner)
1. Executive Summary (TL;DR)
When an AI coding agent underperforms in production, the most common instinct is to throw a more expensive frontier model at the problemβupgrading from a fast lightweight tier ($0.25 / $1.50 per 1M tokens) to an 8x more expensive Pro/Frontier tier ($2.00 / $12.00 per 1M tokens) while keeping the exact same unguarded, noisy agent loop.
To test whether Harness Engineering beats raw model price, we pitted an unharnessed frontier model (gemini-3.1-pro-preview) in a raw loop against 1/8th-cost gemini-3.1-flash-lite equipped with our combined Golden Harnesses (E09 AST Map + E11 Pre-Commit Hook + E16 Output Token Killer + E20 MEMORY.md + E21 Tree-sitter Skeletons + E39 Guides & Sensors) inside isolated VM VM247 on the 3-module enterprise payment settlement pipeline (validator.py β settlement.py β statement.py):
- Mode A (
UnharnessedFrontierProRunnerβgemini-3.1-pro-preview): Runs without harness Guides or Sensors, ingesting73,962chars of raw terminal logs across 3 turns (48,097input tokens,1,909thinking tokens,39.10s,$0.1538), and stalls at15 / 20(75.0%) after missingextrakey stripping (new_item = dict(item)) and E2E statement formatting/ordering contracts. - Mode B (
HarnessedFlashLiteGiantKillingRunnerβgemini-3.1-flash-lite): Runs the 1/8th-price lightweight model inside the Golden Guides & Sensors Harness (CROSS_MODEL_GIANT_KILLING_HARNESS_MD+flash_lite_harnessed_submit), achieving20 / 20(100% pass) in 1 turn (5.62s,$0.0029).
Key Result: A 52.6x Cost Reduction + +25.0% Accuracy Gain
Equipped with the Golden Harness, 1/8th-cost gemini-3.1-flash-lite beat unharnessed gemini-3.1-pro-preview by +25.0 percentage points (20/20 vs. 15/20), while cutting cumulative input tokens by -95.2% (48,097 β 2,321), execution time by -85.6% (39.10s β 5.62s β 6.95x faster), and API cost by -98.1% ($0.1538 β $0.0029 β 52.60x cheaper).
| Metric | Mode A: Unharnessed Frontier Pro (gemini-3.1-pro-preview) |
Mode B: Harnessed Flash-Lite (gemini-3.1-flash-lite + Golden Harness) |
Delta / Improvement |
|---|---|---|---|
| Model Unit Pricing (In / Out per 1M) | $2.00 in / $12.00 out |
$0.25 in / $1.50 out |
1/8th (87.5% lower) unit price |
Hidden Grading Suite Score (labjudge43) |
15 / 20 (75.0% β 5 failures) |
20 / 20 (100% β Perfect) |
+25.0% accuracy (+5 cases) |
Rework Traps Hit (gotcha_traps_hit) |
2 traps (Turns 1 & 2) |
0 traps |
-100% (zero rework) |
LLM Calls (model_calls) |
3 calls |
1 call (1-shot) |
-66.7% (3.00x faster convergence) |
Cumulative Tool Output (tool_output_chars) |
73,962 chars |
106 chars |
-99.9% (697.8x noise reduction) |
| Turn 1 Tokens (In / Out / Thinking) | 1,675 in / 20 out / 43 think |
2,321 in / 1,421 out / 141 think |
1-shot completion on Turn 1 |
| Turn 2 Tokens (In / Out / Thinking) | 15,651 in / 1,278 out / 811 think |
β (completed on Turn 1) | Avoided 37K-char raw log flood |
| Turn 3 Tokens (In / Out / Thinking) | 30,771 in / 1,594 out / 1,055 think |
β (completed on Turn 1) | Avoided secondary rework bloat |
Cumulative Input Tokens (promptTokenCount) |
48,097 tokens |
2,321 tokens |
-95.2% (20.72x efficiency) |
Cumulative Output Tokens (candidatesTokenCount) |
2,892 tokens |
1,421 tokens |
-50.9% output reduction |
Elapsed Execution Time (elapsed_seconds) |
39.10 sec |
5.62 sec |
-85.6% (6.95x faster) |
Estimated API Cost (estimated_micro_usd) |
$0.1538 (153,806 micro-USD) |
$0.0029 (2,924 micro-USD) |
-98.1% (52.60x cheaper) |
2. Experiment Architecture: Unharnessed Frontier Pro vs. Harnessed Flash-Lite
+-------------------------------------------------------------------------+
| Proxmox Isolated VM247 (0 NICs, InaccessiblePaths, grader labjudge43) |
| Target Task: 3-Module Enterprise Payment Settlement Pipeline |
| validator.py β settlement.py β statement.py |
| |
| +-------------------------------------------------------------------+ |
| | Mode A: UnharnessedFrontierProRunner (`gemini-3.1-pro-preview`) | |
| | - Turn 1 (1,675 in): Calls `pro_unharnessed_raw_probe` | |
| | β 37,154 chars of unfiltered audit logs | |
| | - Turn 2 (15,651 in): Calls `pro_stage_unverified_patch` | |
| | β partial 2-file edit (36,808 chars) | |
| | - Turn 3 (30,771 in): Fails `validator_01` (`dict(item)` leak) | |
| | and 4 E2E statement cases (`15/20`) | |
| | --> 3 calls | 48,097 in | $0.1538 | 39.10s | 15/20 (5 FAILED) | |
| +-------------------------------------------------------------------+ |
| |
| +-------------------------------------------------------------------+ |
| | Mode B: HarnessedFlashLiteGiantKillingRunner (`3.1-flash-lite`) | |
| | - Equips 1/8th-cost `gemini-3.1-flash-lite` with Golden Harness: | |
| | * Guides (`E09` + `E20` + `E21`): `[SPEC-VAL/SET/STM]` | |
| | * Sensors (`E11` + `E16` + `E39`): Deterministic pre-submit | |
| | contract & format verification | |
| | - Turn 1 (2,321 in): Solves all 20/20 cases in 5.62 seconds | |
| | --> 1 call | 2,321 in | $0.0029 | 5.62s | 20/20 (100% PASS) | |
| +-------------------------------------------------------------------+ |
+-------------------------------------------------------------------------+
3. Drop-In Implementation Recipe: run_giant_killing_harness
Instead of routing every repository task to a slow, expensive frontier model in a raw loop, wrap your fast lightweight model in a deterministic Guides & Sensors harness:
def run_giant_killing_harness(
task_files: dict[str, str], scoped_guides: str, call_flash_lite_fn, sensor_fn
) -> dict:
"""Pairs 1/8th-cost gemini-3.1-flash-lite with scoped Guides and deterministic Sensors."""
prompt = (
"# GUIDES & SENSORS HARNESS FOR FLASH-LITE\n"
f"{scoped_guides}\n\n"
"# WORKSPACE FILES\n"
+ "\n".join(f"### {k}\n```python\n{v}\n```" for k, v in sorted(task_files.items()))
)
candidate_files = call_flash_lite_fn(model="gemini-3.1-flash-lite", prompt=prompt)
sensor_errors = sensor_fn(candidate_files)
return {
"model": "gemini-3.1-flash-lite",
"sensor_passed": len(sensor_errors) == 0,
"errors": sensor_errors,
"files": candidate_files,
}
4. Practical Takeaway: Invest in Your Harness Before Upgrading Your Model Tier
- A frontier model in a bad harness loses to a small model in a great harness: Raw terminal noise (
73,962chars) and undocumented repository gotchas degrade even top-tier models (15 / 20). - Use Advisor-Executor Routing (
E18) when frontier reasoning is genuinely needed: Consult a frontier model once on Turn 1 to produce a concise architectural blueprint (E18), then execute all file edits and verification loops onflash-lite/flashinside the Golden Harness (E43).
π Continue Reading the Series
- π Master Benchmark Hub: I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
- π‘οΈ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
- π§Ή Previous Sequential Post (
E42): Why 'Stop-at-Green' Accumulates Technical Debt: Immediate Commit on Green vs. a Post-Green Entropy Sweep Pass (2.34x Faster) - 𧬠Next Sequential Post (
E44): Which Harnesses Survive Model Scaling? Syntax Scaffolding vs. the 4 Irreplaceable Core Harnesses (-95.5% Input Tokens) - π§ Related Advisor-Executor Routing Deep Dive (
E18): Benchmarking the Claude Code/advisorPattern: Solo Executor Loops vs. Upfront Advisor Blueprint Routing
Comments
Join the discussionNo comments yet. Be the first to comment.