Skip to main content

Cross-Model Giant Killing: How 1/8th-Cost Flash-Lite + Golden Harnesses Beats Unharnessed Frontier Pro (-98.1% Cost, 6.95x Faster)

Updated 7 min readMarkdown

πŸ’‘ AI Labo Benchmark Series β€” Experiment E43 (Cross-Model Scaling & Giant Killing)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.


Metadata & Experiment Identity

  • Experiment ID: E43 (Backlog Candidate C039)
  • Run ID: e43-20261030-vm247-01
  • Sandbox Environment: Proxmox Isolated VM VM247 (ai-e43-247, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp40 / labjudge43)
  • Models Tested: Mode A: gemini-3.1-pro-preview ($2.00 / $12.00 per 1M tokens) vs. Mode B: gemini-3.1-flash-lite ($0.25 / $1.50 per 1M tokens β€” 1/8th unit price)
  • Pattern Under Test: "AI Labo Scaling Law & Harness Engineering Meta-Analysis: Does a Lightweight Model Equipped with Golden Harnesses Outperform an Unharnessed Frontier Model?" β€” Unharnessed Frontier Pro (UnharnessedFrontierProRunner) vs. Harnessed Flash-Lite Giant Killing (HarnessedFlashLiteGiantKillingRunner)

1. Executive Summary (TL;DR)

When an AI coding agent underperforms in production, the most common instinct is to throw a more expensive frontier model at the problemβ€”upgrading from a fast lightweight tier ($0.25 / $1.50 per 1M tokens) to an 8x more expensive Pro/Frontier tier ($2.00 / $12.00 per 1M tokens) while keeping the exact same unguarded, noisy agent loop.

To test whether Harness Engineering beats raw model price, we pitted an unharnessed frontier model (gemini-3.1-pro-preview) in a raw loop against 1/8th-cost gemini-3.1-flash-lite equipped with our combined Golden Harnesses (E09 AST Map + E11 Pre-Commit Hook + E16 Output Token Killer + E20 MEMORY.md + E21 Tree-sitter Skeletons + E39 Guides & Sensors) inside isolated VM VM247 on the 3-module enterprise payment settlement pipeline (validator.py βž” settlement.py βž” statement.py):

  • Mode A (UnharnessedFrontierProRunner β€” gemini-3.1-pro-preview): Runs without harness Guides or Sensors, ingesting 73,962 chars of raw terminal logs across 3 turns (48,097 input tokens, 1,909 thinking tokens, 39.10s, $0.1538), and stalls at 15 / 20 (75.0%) after missing extra key stripping (new_item = dict(item)) and E2E statement formatting/ordering contracts.
  • Mode B (HarnessedFlashLiteGiantKillingRunner β€” gemini-3.1-flash-lite): Runs the 1/8th-price lightweight model inside the Golden Guides & Sensors Harness (CROSS_MODEL_GIANT_KILLING_HARNESS_MD + flash_lite_harnessed_submit), achieving 20 / 20 (100% pass) in 1 turn (5.62s, $0.0029).

Key Result: A 52.6x Cost Reduction + +25.0% Accuracy Gain

Equipped with the Golden Harness, 1/8th-cost gemini-3.1-flash-lite beat unharnessed gemini-3.1-pro-preview by +25.0 percentage points (20/20 vs. 15/20), while cutting cumulative input tokens by -95.2% (48,097 βž” 2,321), execution time by -85.6% (39.10s βž” 5.62s β€” 6.95x faster), and API cost by -98.1% ($0.1538 βž” $0.0029 β€” 52.60x cheaper).

Metric Mode A: Unharnessed Frontier Pro (gemini-3.1-pro-preview) Mode B: Harnessed Flash-Lite (gemini-3.1-flash-lite + Golden Harness) Delta / Improvement
Model Unit Pricing (In / Out per 1M) $2.00 in / $12.00 out $0.25 in / $1.50 out 1/8th (87.5% lower) unit price
Hidden Grading Suite Score (labjudge43) 15 / 20 (75.0% β€” 5 failures) 20 / 20 (100% β€” Perfect) +25.0% accuracy (+5 cases)
Rework Traps Hit (gotcha_traps_hit) 2 traps (Turns 1 & 2) 0 traps -100% (zero rework)
LLM Calls (model_calls) 3 calls 1 call (1-shot) -66.7% (3.00x faster convergence)
Cumulative Tool Output (tool_output_chars) 73,962 chars 106 chars -99.9% (697.8x noise reduction)
Turn 1 Tokens (In / Out / Thinking) 1,675 in / 20 out / 43 think 2,321 in / 1,421 out / 141 think 1-shot completion on Turn 1
Turn 2 Tokens (In / Out / Thinking) 15,651 in / 1,278 out / 811 think β€” (completed on Turn 1) Avoided 37K-char raw log flood
Turn 3 Tokens (In / Out / Thinking) 30,771 in / 1,594 out / 1,055 think β€” (completed on Turn 1) Avoided secondary rework bloat
Cumulative Input Tokens (promptTokenCount) 48,097 tokens 2,321 tokens -95.2% (20.72x efficiency)
Cumulative Output Tokens (candidatesTokenCount) 2,892 tokens 1,421 tokens -50.9% output reduction
Elapsed Execution Time (elapsed_seconds) 39.10 sec 5.62 sec -85.6% (6.95x faster)
Estimated API Cost (estimated_micro_usd) $0.1538 (153,806 micro-USD) $0.0029 (2,924 micro-USD) -98.1% (52.60x cheaper)

2. Experiment Architecture: Unharnessed Frontier Pro vs. Harnessed Flash-Lite

+-------------------------------------------------------------------------+
| Proxmox Isolated VM247 (0 NICs, InaccessiblePaths, grader labjudge43)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py βž” settlement.py βž” statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: UnharnessedFrontierProRunner (`gemini-3.1-pro-preview`)   |  |
|  |  - Turn 1 (1,675 in):  Calls `pro_unharnessed_raw_probe`          |  |
|  |                        βž” 37,154 chars of unfiltered audit logs    |  |
|  |  - Turn 2 (15,651 in): Calls `pro_stage_unverified_patch`         |  |
|  |                        βž” partial 2-file edit (36,808 chars)       |  |
|  |  - Turn 3 (30,771 in): Fails `validator_01` (`dict(item)` leak)   |  |
|  |                        and 4 E2E statement cases (`15/20`)        |  |
|  |  --> 3 calls | 48,097 in | $0.1538 | 39.10s | 15/20 (5 FAILED)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: HarnessedFlashLiteGiantKillingRunner (`3.1-flash-lite`)   |  |
|  |  - Equips 1/8th-cost `gemini-3.1-flash-lite` with Golden Harness: |  |
|  |    * Guides (`E09` + `E20` + `E21`): `[SPEC-VAL/SET/STM]`         |  |
|  |    * Sensors (`E11` + `E16` + `E39`): Deterministic pre-submit    |  |
|  |      contract & format verification                               |  |
|  |  - Turn 1 (2,321 in): Solves all 20/20 cases in 5.62 seconds      |  |
|  |  --> 1 call  | 2,321 in  | $0.0029 | 5.62s  | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Drop-In Implementation Recipe: run_giant_killing_harness

Instead of routing every repository task to a slow, expensive frontier model in a raw loop, wrap your fast lightweight model in a deterministic Guides & Sensors harness:

def run_giant_killing_harness(
    task_files: dict[str, str], scoped_guides: str, call_flash_lite_fn, sensor_fn
) -> dict:
    """Pairs 1/8th-cost gemini-3.1-flash-lite with scoped Guides and deterministic Sensors."""
    prompt = (
        "# GUIDES & SENSORS HARNESS FOR FLASH-LITE\n"
        f"{scoped_guides}\n\n"
        "# WORKSPACE FILES\n"
        + "\n".join(f"### {k}\n```python\n{v}\n```" for k, v in sorted(task_files.items()))
    )
    candidate_files = call_flash_lite_fn(model="gemini-3.1-flash-lite", prompt=prompt)
    sensor_errors = sensor_fn(candidate_files)
    return {
        "model": "gemini-3.1-flash-lite",
        "sensor_passed": len(sensor_errors) == 0,
        "errors": sensor_errors,
        "files": candidate_files,
    }

4. Practical Takeaway: Invest in Your Harness Before Upgrading Your Model Tier

  1. A frontier model in a bad harness loses to a small model in a great harness: Raw terminal noise (73,962 chars) and undocumented repository gotchas degrade even top-tier models (15 / 20).
  2. Use Advisor-Executor Routing (E18) when frontier reasoning is genuinely needed: Consult a frontier model once on Turn 1 to produce a concise architectural blueprint (E18), then execute all file edits and verification loops on flash-lite / flash inside the Golden Harness (E43).

πŸ”— Continue Reading the Series

No comments yet. Be the first to comment.

More from this blog