π‘ AI Labo Benchmark Series β Experiment
E41(Minimal Tool Primitives & Harness Tax)This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
Metadata & Experiment Identity
- Experiment ID:
E41(Backlog CandidateC037) - Run ID:
e41-20261029-vm245-01 - Sandbox Environment: Proxmox Isolated VM
VM245(ai-e41-245, 0 NICs,PrivateNetwork=yes,NoNewPrivileges=yes,InaccessiblePaths, UID splitlabexp40/labjudge41) - Model:
gemini-3.8-flash(thinkingLevel: LOW,temperature=0.0) - Pattern Under Test: "Databricks Blog: Eliminating the 'Harness Tax' β Heavy Multi-Tool Wrapper Schemas vs. Pi-Style Minimal Primitives (
bash/verify+file_edit)" β Heavy Multi-Tool Wrapper Harness (HeavyMultiToolWrapperRunner) vs. Zero-Tax Minimal Primitives (ZeroTaxMinimalPrimitivesRunner)
1. Executive Summary (TL;DR)
When teams build custom coding agent harnesses, a common anti-pattern is creating a Heavy Multi-Tool Wrapper Harness: 15 to 20 specialized tool schemas (validate_currency_schema_tool, inspect_tier_matrix_wrapper, patch_ast_field_wrapper, etc.) with deeply nested JSON parameters and verbose wrapper outputs.
As highlighted in Databricks' research on Harness Tax and Pi-style minimal agent architectures, bloated wrapper schemas tax every single LLM turn with thousands of schema tokens and distract the model's attention away from subtle domain invariants. Stripping the tool surface down to 2 Zero-Tax Minimal Primitives (verify_workspace + apply_minimal_primitive_bundle) eliminates wrapper overhead completely.
We tested this head-to-head inside isolated VM VM245 on the 3-module enterprise payment settlement pipeline (validator.py β settlement.py β statement.py):
- Mode A (
HeavyMultiToolWrapperRunner): Uses an 18-tool wrapper schema surface (invoke_heavy_wrapper_schemaβstage_wrapper_tax_patch), accumulating73,018chars of wrapper tax logs across Turns 1β2 and failingvalidator_01_normalize_and_non_mutation(normalized = dict(item)leaks"extra"keys) for a final score of19 / 20(95.0%). - Mode B (
ZeroTaxMinimalPrimitivesRunner): Strips away wrapper schemas in favor of Zero-Tax Minimal Primitives (ZERO_TAX_MINIMAL_PRIMITIVES_MD+exec_minimal_primitives_submit), solving20 / 20(100%) in 1 turn.
Key Result
Mode A stalled at 19 / 20 (95.0%) after 3 turns (45,968 cumulative input tokens). Mode B achieved a 20 / 20 (100%) pass in 1 turn, cutting LLM calls by -66.7% (3 β 1), cumulative input tokens by -95.2% (45,968 β 2,224 tokensβa 20.67x reduction), API cost by -83.1% ($0.0453 β $0.0076), and running 2.21x faster (29.25s β 13.25s, -54.7%).
| Metric | Mode A: 18-Tool Wrapper (HeavyMultiToolWrapper) |
Mode B: 2 Minimal Primitives (ZeroTaxMinimalPrimitives) |
Delta / Improvement |
|---|---|---|---|
Hidden Grading Suite Score (labjudge41) |
19 / 20 (95.0% β 1 failure) |
20 / 20 (100% β Perfect) |
+5.0% accuracy (19/20 β 20/20) |
Wrapper-Tax Rework Traps (gotcha_traps_hit) |
2 traps (Turns 1 & 2) |
0 traps |
-100% (zero wrapper drift) |
LLM Calls (model_calls) |
3 calls |
1 call (1-shot) |
-66.7% (3.00x faster convergence) |
Cumulative File Edits (file_edits) |
5 edits |
3 edits (1 per module) |
-40.0% fewer edits |
Cumulative Tool Output (tool_output_chars) |
73,018 chars |
104 chars |
-99.9% (702.1x noise reduction) |
| Turn 1 Tokens (In / Out) | 1,659 in / 12 out |
2,224 in / 1,594 out |
+565 in for minimal contract |
| Turn 2 Tokens (In / Out) | 14,944 in / 1,252 out |
β (completed on Turn 1) | Avoided 36K-char wrapper tax dump |
| Turn 3 Tokens (In / Out) | 29,365 in / 1,619 out |
β (completed on Turn 1) | Avoided secondary rework bloat |
Cumulative Input Tokens (promptTokenCount) |
45,968 tokens |
2,224 tokens |
-95.2% (20.67x efficiency) |
Cumulative Output Tokens (candidatesTokenCount) |
2,883 tokens |
1,594 tokens |
-44.7% output reduction |
Elapsed Execution Time (elapsed_seconds) |
29.25 sec |
13.25 sec |
-54.7% (2.21x faster) |
Estimated API Cost (estimated_micro_usd) |
$0.0453 (45,288 micro-USD) |
$0.0076 (7,646 micro-USD) |
-83.1% (5.92x cheaper) |
2. Experiment Architecture: 18-Tool Wrapper Tax vs. Zero-Tax Minimal Primitives
+-------------------------------------------------------------------------+
| Proxmox Isolated VM245 (0 NICs, InaccessiblePaths, grader labjudge41) |
| Target Task: 3-Module Enterprise Payment Settlement Pipeline |
| validator.py β settlement.py β statement.py |
| |
| +-------------------------------------------------------------------+ |
| | Mode A: HeavyMultiToolWrapperRunner (18-Tool Wrapper Schemas) | |
| | - Turn 1 (1,659 in): Calls `invoke_heavy_wrapper_schema` | |
| | β 36,524 chars of wrapper schema tax logs | |
| | - Turn 2 (14,944 in): Calls `stage_wrapper_tax_patch` on 2 files | |
| | β 36,178 chars of secondary wrapper logs | |
| | - Turn 3 (29,365 in): Submits `normalized = dict(item)` (leaks | |
| | `"extra"` key β fails `validator_01`) | |
| | --> 3 calls | 45,968 in | $0.0453 | 29.25s | 19/20 (1 FAILED) | |
| +-------------------------------------------------------------------+ |
| |
| +-------------------------------------------------------------------+ |
| | Mode B: ZeroTaxMinimalPrimitivesRunner (Pi-Style 2 Primitives) | |
| | - Enforces `ZERO_TAX_MINIMAL_PRIMITIVES_MD` (+565 tokens): | |
| | Replaces 18 custom tool wrappers with 2 minimal primitives | |
| | (`verify_workspace` + `exec_minimal_primitives_submit`) and | |
| | scoped `[SPEC-VAL]`, `[SPEC-SET]`, `[SPEC-STM]` contracts | |
| | - Turn 1 (2,224 in): 1-shot atomic submission passes 20/20 | |
| | --> 1 call | 2,224 in | $0.0076 | 13.25s | 20/20 (100% PASS) | |
| +-------------------------------------------------------------------+ |
+-------------------------------------------------------------------------+
3. Drop-In Implementation Recipe: Auditing and Stripping Your Harness Schema Tax
Measure how many characters your registered tool schemas consume on every turn, and collapse redundant wrappers into 2 core primitives:
import json
MINIMAL_PRIMITIVE_SCHEMAS = [
{
"name": "verify_workspace",
"description": "Run deterministic boundary checks across validator.py, settlement.py, and statement.py.",
"parameters": {"type": "object", "properties": {}},
},
{
"name": "apply_minimal_primitive_bundle",
"description": "Atomically update validator.py, settlement.py, and statement.py and run verification.",
"parameters": {
"type": "object",
"required": ["files"],
"properties": {"files": {"type": "object"}},
},
},
]
def compute_harness_schema_tax(tool_schemas: list[dict]) -> int:
"""Computes the per-turn character overhead (Harness Tax) of registered tool definitions."""
return len(json.dumps(tool_schemas, ensure_ascii=False))
4. Practical Takeaway: Fewer Tools, Stronger Sensors
- Every custom tool schema you add is taxed on every turn: If a tool wrapper can be expressed as a standard
bash/verifycall or folded into a singlefile_editprimitive, delete the wrapper. - Pair Zero-Tax Minimal Primitives (
E41) with Progressive Tool Search (E14): Keep the base toolset at 2β3 primitives (E41), and if specialized MCP tools are needed, load their schemas on demand viasearch_tools(E14).
π Continue Reading the Series
- π Master Benchmark Hub: I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
- π‘οΈ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
- πΎ Previous Sequential Post (
E40): Why Outcome-Only Evals Let Coding Agents Guess Blindly: Outcome Evals vs. a Behavioral Action-Sequence Gate (-95.2% Tokens) - π§Ή Next Sequential Post (
E42): Why 'Stop-at-Green' Accumulates Technical Debt: Immediate Commit on Green vs. a Post-Green Entropy Sweep Pass (2.34x Faster) - π Related Tool Schema Tax Deep Dive (
E14): The Hidden "Schema Tax" of MCP Tools: Dumping 32 Tool Schemas vs. On-Demand Tool Search (-92.7% Input Tokens)
Comments
Join the discussionNo comments yet. Be the first to comment.