Eliminating Print-Debugging Loops in Coding Agents: Bare Stack Traces vs. Runtime Frame Variable-Snapshot Injection (-95.0% Input Tokens)

💡 AI Labo Benchmark Series — Experiment E34 (Runtime Introspection & Debugging Sensors)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

---

Metadata & Experiment Identity

• Experiment ID: E34 (Backlog Candidate C030)
• Run ID: e34-20261029-vm238-01
• Sandbox Environment: Proxmox Isolated VM VM238 (ai-e34-238, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp34 / labjudge34)
• Model: gemini-3.8-flash (thinkingLevel: LOW, temperature=0.0)
• Pattern Under Test: "Runtime State Introspection in Agent Harnesses: Injecting Frame locals() Snapshots on Test Failure (pytest --showlocals / tb.tb_frame.f_locals)" — Traceback-Only Debugging (TracebackOnlyDebuggingRunner) vs. Runtime Variable-Snapshot Injection (RuntimeVariableSnapshotRunner)

---

1. Executive Summary (TL;DR)

A standard Python stack trace tells an AI coding agent which line raised an exception (File "settlement.py", line 42, in settle_batch -> AssertionError), but hides what values the local variables held at that exact moment (g['chargeback_bps'] = 300, subtotal = 15000, item['unit_cents'] = 0).

Deprived of live runtime state, the agent is forced into Speculative Print-Debugging Loops: it guesses intermediate variable values, inserts temporary print() statements, re-runs the test suite, and pollutes its context window with tens of thousands of characters of debug output.

By capturing tb.tb_frame.f_locals at the failing frame and injecting a compact Runtime Variable Snapshot directly into the harness prompt, the agent sees the exact failing state transition immediately.

We tested this head-to-head inside isolated VM VM238 on the 3-module enterprise payment settlement pipeline (validator.py ➔ settlement.py ➔ statement.py):

• Mode A (TracebackOnlyDebuggingRunner): Receives bare stack traces without frame.f_locals, triggering speculative print-debugging across Turns 1–2 (77,735 chars of tool output) and finishing at 16 / 20 (80.0%) due to unfixed dict(item) extra-key leaks, cents <= 0, and bps < 1000.
• Mode B (RuntimeVariableSnapshotRunner): Injects captured frame.f_locals variable snapshots (RUNTIME_VARIABLE_SNAPSHOT_MD) at each failure point, achieving 20 / 20 (100%) in a single turn.

Key Result

Mode A stalled at 16 / 20 (80.0%) after 3 turns (46,581 cumulative input tokens). Mode B achieved a 20 / 20 (100%) pass in 1 turn, cutting LLM calls by -66.7% (3 ➔ 1), cumulative input tokens by -95.0% (46,581 ➔ 2,322 tokens—a 20.06x reduction), API cost by -83.1% ($0.0456 ➔ $0.0077), and running 2.18x faster (28.65s ➔ 13.16s, -54.1%).

Metric	Mode A: Bare Traceback (TracebackOnlyDebugging)	Mode B: Variable Snapshot (RuntimeVariableSnapshot)	Delta / Improvement
Hidden Grading Suite Score (labjudge34)	16 / 20 (80.0% — 4 failures)	20 / 20 (100% — Perfect)	+20.0% accuracy (+4 cases)
Print-Debug Rework Traps (gotcha_traps_hit)	2 traps (Turns 1 & 2)	0 traps	-100% (zero print-debugging)
LLM Calls (model_calls)	3 calls	1 call (1-shot)	-66.7% (3.00x faster convergence)
Cumulative File Edits (file_edits)	5 edits	3 edits (1 per module)	-40.0% fewer edits
Cumulative Tool Output (tool_output_chars)	77,735 chars	103 chars	-99.9% (754.7x noise reduction)
Turn 1 Tokens (In / Out)	1,654 in / 11 out	2,322 in / 1,587 out	+668 in for f_locals snapshots
Turn 2 Tokens (In / Out)	15,151 in / 1,245 out	— (completed on Turn 1)	Avoided print-debug log flood
Turn 3 Tokens (In / Out)	29,776 in / 1,592 out	— (completed on Turn 1)	Avoided secondary rework bloat
Cumulative Input Tokens (promptTokenCount)	46,581 tokens	2,322 tokens	-95.0% (20.06x efficiency)
Cumulative Output Tokens (candidatesTokenCount)	2,848 tokens	1,587 tokens	-44.3% output reduction
Elapsed Execution Time (elapsed_seconds)	28.65 sec	13.16 sec	-54.1% (2.18x faster)
Estimated API Cost (estimated_micro_usd)	$0.0456 (45,616 micro-USD)	$0.0077 (7,693 micro-USD)	-83.1% (5.93x cheaper)

---

2. Experiment Architecture: Bare Traceback vs. Runtime Variable-Snapshot Injection

+-------------------------------------------------------------------------+
| Proxmox Isolated VM238 (0 NICs, InaccessiblePaths, grader labjudge34)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py ➔ settlement.py ➔ statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: TracebackOnlyDebuggingRunner (Line Numbers Only)          |  |
|  |  - Turn 1 (1,654 in):  Calls `inspect_traceback_only` ➔ `f_locals`|  |
|  |                        hidden; 38,882 chars of speculation logs   |  |
|  |  - Turn 2 (15,151 in): Calls `insert_speculative_print_debug`     |  |
|  |                        ➔ 38,536 chars of print-debug logs         |  |
|  |  - Turn 3 (29,776 in): Submits `dict(item)` (extra key leak),     |  |
|  |                        `cents <= 0`, and `0 <= bps < 1000`        |  |
|  |  --> 3 calls | 46,581 in | $0.0456 | 28.65s | 16/20 (4 FAILED)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: RuntimeVariableSnapshotRunner (`tb.tb_frame.f_locals`)    |  |
|  |  - Injects `RUNTIME_VARIABLE_SNAPSHOT_MD` (+668 tokens):          |  |
|  |    * Frame #1 (`validate_txns`): `tx_count=True, unit_cents=0,    |  |
|  |      chargeback_bps=1000, extra=1`                                |  |
|  |    * Frame #2 (`settle_batch`): `chargeback_bps=300, gross=10000, |  |
|  |      subtotal=15000, fee_rebate=99999`                            |  |
|  |    * Frame #3 (`render_statement`): `tier_discount=0`             |  |
|  |  - Turn 1 (2,322 in): Calls `submit_snapshot_verified_bundle`     |  |
|  |  --> 1 call  | 2,322 in  | $0.0077 | 13.16s | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

---

3. Drop-In Implementation Recipe: run_with_variable_snapshot

Capture bounded local variable snapshots from the inner stack frames whenever a test or tool call raises an exception:

import sys
from typing import Any, Callable


def run_with_variable_snapshot(fn: Callable, *args: Any, **kwargs: Any) -> dict[str, Any]:
    """Captures `frame.f_locals` scalar/collection snapshots from the failing stack frames on exception."""
    try:
        return {"ok": True, "result": fn(*args, **kwargs)}
    except Exception as exc:
        _, _, tb = sys.exc_info()
        frames = []
        while tb is not None:
            frame = tb.tb_frame
            safe_locals = {
                k: repr(v)[:120]
                for k, v in frame.f_locals.items()
                if not k.startswith("__") and isinstance(v, (int, float, str, bool, list, dict, tuple))
            }
            frames.append({
                "file": frame.f_code.co_filename,
                "line": tb.tb_lineno,
                "func": frame.f_code.co_name,
                "locals": safe_locals,
            })
            tb = tb.tb_next
        return {"ok": False, "error": repr(exc), "frame_snapshots": frames[-3:]}

---

4. Practical Takeaway: Always Include Bounded f_locals in Failure Reports

1. Line numbers tell the agent where it crashed; f_locals tells it why: Seeing chargeback_bps=300, reserve=1000 in the failing frame snapshot immediately reveals an off-by-one >= 300 vs. > 300 bug without a single print-debugging turn.
2. Pair Runtime Variable Snapshots (E34) with the CLI Test Output Token Killer (E16): Strip passing test noise (E16) and enrich the remaining failure JSON with the top 3 frames' f_locals snapshots (E34) truncated to 120 chars per variable.

---

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🎲 Previous Sequential Post (E33): Why Hand-Written Example Tests Fail AI Coding Agents: Fixed Sample Tests vs. a Property-Based Invariant Fuzzing Sensor (-95.1% Input Tokens)
• 📖 Next Sequential Post (E35): Why a 500-Line AGENTS.md Hurts Coding Agent Accuracy: Monolithic Encyclopedia Prompts vs. a 30-Line Directory-Pointer Index (3.13x Faster)
• 🧹 Related Test Filter Deep Dive (E16): Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%