π‘ AI Labo Benchmark Series β Experiment
E28(Feedback Sensors & Loop Circuit Breakers)This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
Metadata & Experiment Identity
- Experiment ID:
E28(Backlog CandidateC024) - Run ID:
e28-20261029-vm232-01 - Sandbox Environment: Proxmox Isolated VM
VM232(ai-e28-232, 0 NICs,PrivateNetwork=yes,NoNewPrivileges=yes,InaccessiblePaths, UID splitlabexp28/labjudge28) - Model:
gemini-3.8-flash(thinkingLevel: LOW,temperature=0.0) - Pattern Under Test: "Agent Harness Loop-Detection & Automated Strategy-Shift Circuit Breaker (Defeating Sunk-Cost Oscillation)" β Unguarded Retry Loop (
UnguardedRetryOscillationRunner) vs. Loop-Detection Circuit Breaker + Strategy Shift (CircuitBreakerStrategyShiftRunner)
1. Executive Summary (TL;DR)
Every developer who has watched an autonomous coding agent run unattended has seen the dreaded A β B β A Edit Oscillation Loop:
- The agent tweaks a local expression in
settlement.py, and an integration test fails. - On the next turn, it tweaks
validator.pyandsettlement.pywhile leavingstatement.pyuntouchedβand the integration test fails again with the same downstream signature. - Without an external circuit breaker, the agent burns 40,000+ input tokens repeating local tweaks inside a local minimum.
To test how a deterministic Loop-Detection Circuit Breaker eliminates sunk-cost oscillation, we ran a controlled A/B experiment inside isolated VM VM232 on the 3-module enterprise payment settlement pipeline (validator.py β settlement.py β statement.py):
- Mode A (
UnguardedRetryOscillationRunner): Lets the agent retry local edits across turns without loop interception, accumulating80,874characters of repetitive failure telemetry across Turns 1β2 before finally converging on Turn 3. - Mode B (
CircuitBreakerStrategyShiftRunner): Hashes(edited_files, failure_signatures)on every turn; the instant a repeated local retry pattern is detected, the Circuit Breaker trips, rolls back the partial local state, and injects a Cross-Layer Strategy-Shift Directive (CIRCUIT_BREAKER_STRATEGY_SHIFT_MD) forcing a holistic 3-layer fix.
Key Result
Both modes achieved 20 / 20 (100%) on the hidden grading suite. However, tripping the Circuit Breaker to force a cross-layer strategy shift cut LLM calls by -66.7% (3 β 1), slashed cumulative input tokens by -94.9% (43,900 β 2,230 tokensβa 19.69x reduction), lowered API cost by -82.6% ($0.0439 β $0.0077), and ran 2.46x faster (32.38s β 13.16s, -59.4%).
| Metric | Mode A: Unguarded Retry Loop (UnguardedRetryOscillation) |
Mode B: Circuit Breaker (CircuitBreakerStrategyShift) |
Delta / Improvement |
|---|---|---|---|
Hidden Grading Suite Score (labjudge28) |
20 / 20 (100% β on Turn 3) |
20 / 20 (100% β on Turn 1) |
3x faster convergence to 100% |
Oscillation Traps Hit (gotcha_traps_hit) |
2 traps (Turns 1 & 2) |
0 traps |
-100% (zero oscillation waste) |
LLM Calls (model_calls) |
3 calls |
1 call (1-shot) |
-66.7% (3 β 1 turn) |
Cumulative File Edits (file_edits) |
5 edits (2 partial + 3 final) |
3 edits (1 per module) |
-40.0% fewer edits |
Cumulative Tool Output (tool_output_chars) |
80,874 chars |
101 chars |
-99.9% (800.7x noise reduction) |
| Turn 1 Tokens (In / Out) | 1,660 in / 16 out |
2,230 in / 1,595 out |
+570 in for Strategy-Shift Directive |
| Turn 2 Tokens (In / Out) | 14,260 in / 1,298 out |
β (completed on Turn 1) | Avoided Turn 1 oscillation log |
| Turn 3 Tokens (In / Out) | 27,980 in / 1,617 out |
β (completed on Turn 1) | Avoided Turn 2 partial-edit log |
Cumulative Input Tokens (promptTokenCount) |
43,900 tokens |
2,230 tokens |
-94.9% (19.69x efficiency) |
Cumulative Output Tokens (candidatesTokenCount) |
2,931 tokens |
1,595 tokens |
-45.6% output reduction |
Elapsed Execution Time (elapsed_seconds) |
32.38 sec |
13.16 sec |
-59.4% (2.46x faster) |
Estimated API Cost (estimated_micro_usd) |
$0.0439 (43,917 micro-USD) |
$0.0077 (7,654 micro-USD) |
-82.6% (5.74x cheaper) |
2. Experiment Architecture: Unguarded Retries vs. Loop-Detection Circuit Breaker
+-------------------------------------------------------------------------+
| Proxmox Isolated VM232 (0 NICs, InaccessiblePaths, grader labjudge28) |
| Target Task: 3-Module Enterprise Payment Settlement Pipeline |
| validator.py β settlement.py β statement.py |
| |
| +-------------------------------------------------------------------+ |
| | Mode A: UnguardedRetryOscillationRunner (No Loop Interception) | |
| | - Turn 1 (1,660 in): Calls `repeat_oscillating_edit` β 40,645 | |
| | chars of local oscillation logs | |
| | - Turn 2 (14,260 in): Calls `stage_sunk_cost_retry` β patches | |
| | only `validator.py` & `settlement.py`, | |
| | leaving `statement.py` broken (40,128c) | |
| | - Turn 3 (27,980 in): Finally patches all 3 files after burning | |
| | 43,900 cumulative input tokens | |
| | --> 3 calls | 43,900 in | $0.0439 | 32.38s | 20/20 (Turn 3) | |
| +-------------------------------------------------------------------+ |
| |
| +-------------------------------------------------------------------+ |
| | Mode B: CircuitBreakerStrategyShiftRunner (Hash-Based Breaker) | |
| | - Breaker Sensor: Intercepts local oscillation, rolls back, and | |
| | injects `CIRCUIT_BREAKER_STRATEGY_SHIFT_MD` (+570 tokens) | |
| | - Turn 1 (2,230 in): Pivots to all 3 layers simultaneously and | |
| | passes 20/20 on the first model call | |
| | --> 1 call | 2,230 in | $0.0077 | 13.16s | 20/20 (Turn 1) | |
| +-------------------------------------------------------------------+ |
+-------------------------------------------------------------------------+
3. Drop-In Implementation Recipe: LoopDetectionCircuitBreaker
You can build a deterministic loop-detection circuit breaker in 20 lines of pure Python by hashing the sorted list of edited files and normalized error codes after every turn:
import hashlib
class LoopDetectionCircuitBreaker:
"""Detects repeated failure signatures or oscillating edits and forces a strategy shift."""
def __init__(self, threshold: int = 2):
self.threshold = threshold
self.failure_signatures: list[str] = []
def record_turn_outcome(self, edited_files: list[str], error_codes: list[str]) -> str | None:
if not error_codes:
self.failure_signatures.clear()
return None
sig_raw = f"{sorted(edited_files)}::{sorted(error_codes)}"
sig_hash = hashlib.sha256(sig_raw.encode("utf-8")).hexdigest()[:16]
self.failure_signatures.append(sig_hash)
if len(self.failure_signatures) >= self.threshold:
return (
"[CIRCUIT BREAKER TRIPPED]: Repeated failure signature detected across consecutive turns. "
"STOP incremental patching. Re-verify cross-module invariants across validator -> settlement -> statement."
)
return None
4. Practical Takeaway: Never Let an Agent Fail the Same Way Twice
- Hash
(edited_files, failure_signature)and DIFF hashes on every turn: Track both the error signature and the AST/file hash of modified files. If the agent produces the same error twiceβor reverts a file to a previously seen SHA-256 (A β B β A)βtrip the breaker immediately. - Pair the Circuit Breaker (
E28) with Error-Trace Pruning (E19) or a Hypothesis-Pivot Ladder (E37): When the breaker trips,git checkout -- .the broken partial edits, prune the failed stack traces from chat history (E19), and require the agent to formulate a fresh cross-layer plan before unlocking writes again.
π Continue Reading the Series
- π Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
- π‘οΈ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
- ποΈ Previous Sequential Post (
E27): Stop Your Coding Agent From Cycling Through Refuted Guesses: Free-Text Monologue vs. a Structured Hypothesis Matrix (2.54x Faster) - π― Next Sequential Post (
E29): Deduplicating Cascading Test Failures for Coding Agents: Raw Error Dumps vs. AST Root-Cause Error Clustering (-95.3% Input Tokens) - π Related Tournament Deep Dive (
E13): Escaping Sunk-Cost Retry Loops in Coding Agents: Single-Session Retries vs. a Parallel Best-of-N Tournament (-67.2% Input Tokens)
Comments
Join the discussionNo comments yet. Be the first to comment.