π‘ AI Labo Benchmark Series β Experiment
E42(Continuous Garbage Collection & Entropy Sweeps)This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
Metadata & Experiment Identity
- Experiment ID:
E42(Backlog CandidateC038) - Run ID:
e42-20261029-vm246-01 - Sandbox Environment: Proxmox Isolated VM
VM246(ai-e42-246, 0 NICs,PrivateNetwork=yes,NoNewPrivileges=yes,InaccessiblePaths, UID splitlabexp40/labjudge42) - Model:
gemini-3.8-flash(thinkingLevel: LOW,temperature=0.0) - Pattern Under Test: "OpenAI Harness Engineering & Continuous Garbage Collection: Post-Green Entropy Sweep Pass vs. Stop-at-Green Baseline" β Stop-at-Green Baseline (
StopAtGreenBaselineRunner) vs. Post-Green Entropy Sweep Pass (PostGreenEntropySweepRunner)
1. Executive Summary (TL;DR)
When an AI coding agent optimizes solely to make the immediate smoke test turn "Green" (Stop-at-Green), it frequently leaves behind subtle codebase entropy: shallow dictionary copies (new_item = dict(item)) that retain unvalidated extra keys, redundant helper functions, dead branches, or naming drift.
Enforcing a Post-Green Entropy Sweep Pass (continuous garbage collection integrated directly into the verification gate) audits and eliminates structural entropy before final submission.
We tested this head-to-head inside isolated VM VM246 on the 3-module enterprise payment settlement pipeline (validator.py β settlement.py β statement.py):
- Mode A (
StopAtGreenBaselineRunner): Stops as soon as surface-level checks pass (run_stop_at_green_checkβstage_redundant_helper_patch), accumulating73,496chars of unswept debt logs and submittingnew_item = dict(item)invalidator.pyβfailingvalidator_01_normalize_and_non_mutation(19 / 20, 95.0%). - Mode B (
PostGreenEntropySweepRunner): Enforces the Post-Green Entropy Sweep Pass (POST_GREEN_ENTROPY_SWEEP_MD+run_entropy_sweep_and_submit), eliminating shallow-copy key leaks and achieving20 / 20(100%) in 1 turn.
Key Result
Mode A stalled at 19 / 20 (95.0%) after 3 turns (47,114 cumulative input tokens). Mode B achieved a 20 / 20 (100%) pass in 1 turn, cutting LLM calls by -66.7% (3 β 1), cumulative input tokens by -95.3% (47,114 β 2,217 tokensβa 21.25x reduction), API cost by -83.3% ($0.0461 β $0.0077), and running 2.34x faster (29.83s β 12.75s, -57.3%).
| Metric | Mode A: Stop-at-Green (StopAtGreenBaseline) |
Mode B: Post-Green Sweep (PostGreenEntropySweep) |
Delta / Improvement |
|---|---|---|---|
Hidden Grading Suite Score (labjudge42) |
19 / 20 (95.0% β 1 failure) |
20 / 20 (100% β Perfect) |
+5.0% accuracy (19/20 β 20/20) |
Unswept-Entropy Traps (gotcha_traps_hit) |
2 traps (Turns 1 & 2) |
0 traps |
-100% (zero technical debt) |
LLM Calls (model_calls) |
3 calls |
1 call (1-shot) |
-66.7% (3.00x faster convergence) |
Cumulative File Edits (file_edits) |
5 edits |
3 edits (1 per module) |
-40.0% fewer edits |
Cumulative Tool Output (tool_output_chars) |
73,496 chars |
99 chars |
-99.9% (742.4x noise reduction) |
| Turn 1 Tokens (In / Out) | 1,662 in / 14 out |
2,217 in / 1,612 out |
+555 in for entropy sweep spec |
| Turn 2 Tokens (In / Out) | 15,324 in / 1,260 out |
β (completed on Turn 1) | Avoided 36K-char debt log dump |
| Turn 3 Tokens (In / Out) | 30,128 in / 1,594 out |
β (completed on Turn 1) | Avoided secondary rework bloat |
Cumulative Input Tokens (promptTokenCount) |
47,114 tokens |
2,217 tokens |
-95.3% (21.25x efficiency) |
Cumulative Output Tokens (candidatesTokenCount) |
2,868 tokens |
1,612 tokens |
-43.8% output reduction |
Elapsed Execution Time (elapsed_seconds) |
29.83 sec |
12.75 sec |
-57.3% (2.34x faster) |
Estimated API Cost (estimated_micro_usd) |
$0.0461 (46,091 micro-USD) |
$0.0077 (7,708 micro-USD) |
-83.3% (5.98x cheaper) |
2. Experiment Architecture: Stop-at-Green vs. Post-Green Entropy Sweep
+-------------------------------------------------------------------------+
| Proxmox Isolated VM246 (0 NICs, InaccessiblePaths, grader labjudge42) |
| Target Task: 3-Module Enterprise Payment Settlement Pipeline |
| validator.py β settlement.py β statement.py |
| |
| +-------------------------------------------------------------------+ |
| | Mode A: StopAtGreenBaselineRunner (Immediate Commit on Green) | |
| | - Turn 1 (1,662 in): Calls `run_stop_at_green_check` | |
| | β 36,763 chars of unswept debt warnings | |
| | - Turn 2 (15,324 in): Calls `stage_redundant_helper_patch` | |
| | β partial 2-file edit (36,417 chars) | |
| | - Turn 3 (30,128 in): Submits `new_item = dict(item)` (retains | |
| | `"extra"` key β fails `validator_01`) | |
| | --> 3 calls | 47,114 in | $0.0461 | 29.83s | 19/20 (1 FAILED) | |
| +-------------------------------------------------------------------+ |
| |
| +-------------------------------------------------------------------+ |
| | Mode B: PostGreenEntropySweepRunner (Zero-Entropy Consolidation) | |
| | - Enforces `POST_GREEN_ENTROPY_SWEEP_MD` (+555 tokens): | |
| | Sweeps redundant shallow copies (`dict(item)`), dead branches, | |
| | and formatting drift across `validator.py`, `settlement.py`, | |
| | and `statement.py` before final commit | |
| | - Turn 1 (2,217 in): Calls `run_entropy_sweep_and_submit` once | |
| | --> 1 call | 2,217 in | $0.0077 | 12.75s | 20/20 (100% PASS) | |
| +-------------------------------------------------------------------+ |
+-------------------------------------------------------------------------+
3. Drop-In Implementation Recipe: run_post_green_entropy_sweep
Run a deterministic AST entropy sweep immediately after functional smoke tests pass:
import ast
def run_post_green_entropy_sweep(files: dict[str, str]) -> list[str]:
"""Audits green candidate files for shallow-copy key leaks and unnecessary imports before commit."""
entropy_issues: list[str] = []
val_src = files.get("validator.py", "")
if "dict(item)" in val_src or "item.copy()" in val_src:
entropy_issues.append("validator.py: shallow copy retains unknown keys; construct explicit 5-key dict")
for name, src in files.items():
tree = ast.parse(src)
for node in ast.walk(tree):
if isinstance(node, (ast.Import, ast.ImportFrom)):
entropy_issues.append(f"{name}: unnecessary import statement increases module entropy")
return entropy_issues
4. Practical Takeaway: Treat "Tests Green" as the Start of Cleanup, Not the End of the Task
- Automated agents generate technical debt faster than humans if left unswept: Without a post-green entropy check, agents leave behind
dict(item)shallow copies, duplicate helper functions, and dead code branches. - Pair Post-Green Entropy Sweeps (
E42) with Pre-Submit Blast-Radius Audits (E32): Ensure every patch that passes functional tests also passes zero-entropy structural hygiene before merging.
π Continue Reading the Series
- π Master Benchmark Hub: I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
- π‘οΈ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
- πͺΆ Previous Sequential Post (
E41): Eliminating the 'Harness Tax' in Coding Agents: 18-Tool Wrapper Schemas vs. Zero-Tax Minimal Primitives (-95.2% Input Tokens) - βοΈ Next Sequential Post (
E43): Cross-Model Giant Killing: How 1/8th-Cost Flash-Lite + Golden Harnesses Beats Unharnessed Frontier Pro (-98.1% Cost, 6.95x Faster) - π Related Pre-Submit Diff Audit Deep Dive (
E32): Why Coding Agents Need a Pre-Submit Diff Audit Gate: Blind Unchecked Submits vs. Blast-Radius & Entropy Auditing (-95.0% Input Tokens)
Comments
Join the discussionNo comments yet. Be the first to comment.