Stop Your Coding Agent From Cycling Through Refuted Guesses: Free-Text Monologue vs. a Structured Hypothesis Matrix (2.54x Faster)

💡 AI Labo Benchmark Series — Experiment E27 (Structured Reasoning & Diagnostic Scratchpads)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

🚀 View All 41 Benchmark Results & Master Ranking

🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

Experiment ID: E27 (Backlog Candidate C023)
Run ID: e27-20261029-vm231-01
Sandbox Environment: Proxmox Isolated VM VM231 (ai-e27-231, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp27 / labjudge27)
Model: gemini-3.8-flash (thinkingLevel: LOW, temperature=0.0)
Pattern Under Test: "Anthropic <think> Tool Research & SWE-agent Structured Diagnostic Scratchpad (Hypothesis Elimination Matrix)" — Unstructured Free-Text Monologue (FreeTextMonologueRunner) vs. Structured THINK.md Hypothesis Elimination Table (StructuredHypothesisTableRunner)

1. Executive Summary (TL;DR)

When debugging a multi-module pipeline with 7+ interacting defects, letting an AI coding agent reason via unstructured Free-Text Monologue ("Let me think... maybe the settlement total is off because of integer division? Or wait, did we check that earlier?") leads to Hypothesis Cycling: as tool logs grow across turns, the model loses track of which candidate root causes were already falsified and which boundary contracts remain unverified.

Inspired by Anthropic's <think> scratchpad research and SWE-agent diagnostic matrices, we tested replacing free-text debugging monologues with a mandatory Structured Hypothesis Elimination Table (THINK.md) inside isolated VM VM231 on the 3-module enterprise payment settlement pipeline (validator.py ➔ settlement.py ➔ statement.py):

Mode A (FreeTextMonologueRunner): Reasons through unstructured free-text monologue logs (92,463 tool output chars across Turns 1–2) before submitting on Turn 3.
Mode B (StructuredHypothesisTableRunner): Requires a fixed-schema Hypothesis Elimination Matrix (STRUCTURED_HYPOTHESIS_TABLE_MD) mapping Hyp ID ➔ Target Module ➔ Suspected Defect ➔ Diagnostic Probe ➔ Status (CONFIRMED/REFUTED) ➔ Canonical Fix before any code edit.

Key Result

Mode A missed validator_01 (19 / 20, 95.0%) after 3 turns (48,712 input tokens) because it over-constrained set(item.keys()) != required_keys instead of stripping extra keys into a clean 5-key dictionary. Mode B achieved a 20 / 20 (100%) pass in 1 turn, cutting LLM calls by -66.7% (3 ➔ 1), cumulative input tokens by -95.0% (48,712 ➔ 2,450 tokens—a 19.88x reduction), API cost by -83.6% ($0.0478 ➔ $0.0079), and running 2.54x faster (33.66s ➔ 13.28s, -60.6%).

Metric	Mode A: Free-Text Monologue (`FreeTextMonologue`)	Mode B: Hypothesis Matrix (`StructuredHypothesisTable`)	Delta / Improvement
Hidden Grading Suite Score (`labjudge27`)	`19 / 20` (95.0% — 1 failure)	`20 / 20` (100% — Perfect)	+5.0% accuracy (`19/20 ➔ 20/20`)
Hypothesis Cycling Traps (`gotcha_traps_hit`)	`2 traps` (Turns 1 & 2)	`0 traps`	-100% (zero cycling)
LLM Calls (`model_calls`)	`3 calls`	`1 call` (1-shot)	-66.7% (`3 ➔ 1` turn)
Cumulative File Edits (`file_edits`)	`5 edits`	`3 edits` (1 per module)	-40.0% fewer edits
Cumulative Tool Output (`tool_output_chars`)	`92,463 chars`	`102 chars`	-99.9% (`906.5x` noise reduction)
Turn 1 Tokens (In / Out)	`1,655 in` / `18 out`	`2,450 in` / `1,604 out`	`+795` in for `H-01`..`H-07` Matrix
Turn 2 Tokens (In / Out)	`15,832 in` / `1,333 out`	— (completed on Turn 1)	Avoided monologue rambling bloat
Turn 3 Tokens (In / Out)	`31,225 in` / `1,641 out`	— (completed on Turn 1)	Avoided secondary cycling bloat
Cumulative Input Tokens (`promptTokenCount`)	`48,712 tokens`	`2,450 tokens`	-95.0% (`19.88x` efficiency)
Cumulative Output Tokens (`candidatesTokenCount`)	`2,992 tokens`	`1,604 tokens`	-46.4% output reduction
Elapsed Execution Time (`elapsed_seconds`)	`33.66 sec`	`13.28 sec`	-60.6% (`2.54x` faster)
Estimated API Cost (`estimated_micro_usd`)	`$0.0478` (`47,755 micro-USD`)	`$0.0079` (`7,853 micro-USD`)	-83.6% (`6.08x` cheaper)

2. The Verified `THINK.md` Hypothesis Elimination Matrix (`input-03.json`)

Instead of rambling in prose, Mode B locked this 7-row structured diagnostic matrix (+795 tokens) before touching a single line of code:

Hyp ID	Target Module	Suspected Defect / Boundary	Diagnostic Probe & Elimination Result	Status	Required Canonical Fix
`H-01`	`validator.py`	`ALLOWED_CURRENCIES` missing `"JPY"` & empty list `[]` allowed	`validate_txns([])` did not raise; `"jpy"` rejected	CONFIRMED	Set `ALLOWED_CURRENCIES = ("USD", "EUR", "JPY")` and reject `len(raw_txns) == 0`
`H-02`	`validator.py`	`isinstance(True, int)` passes in Python & `cents <= 0` rejects `$0` auth	`tx_count=True` accepted; `unit_cents=0` raised `ValueError`	CONFIRMED	Check `isinstance(x, bool)` before `isinstance(x, int)`; enforce `cnt > 0`, `cents >= 0`, `0 <= bps <= 1000`
`H-03`	`validator.py`	In-place mutation `item["merchant_id"] = ...` & extra keys retained	Caller dict mutated; `"extra": 1` leaked into output	CONFIRMED	Construct a fresh 5-key dict (`merchant_id`, `currency`, `tx_count`, `unit_cents`, `chargeback_bps`)
`H-04`	`settlement.py`	`sorted(order)` destroys first-seen merchant-currency order	`M-Z` before `M-A` reordered to `M-A`, `M-Z`	CONFIRMED	Iterate `for k in order:` (never `sorted(order)`)
`H-05`	`settlement.py`	Chargeback reserve `>= 300` vs strictly `> 300` bps	`300 bps` charged 10% reserve instead of `0`	CONFIRMED	Charge `(gross * 10) // 100` strictly when `g["chargeback_bps"] > 300`
`H-06`	`settlement.py`	Tier discount `> 15000` vs inclusive `>= 15000` & rebate cap	`subtotal == 15000` missed discount; `rebate > after_discount` went negative	CONFIRMED	Use `subtotal >= 15000` (`GROWTH` 10%/0%; `ENTERPRISE` 20%/10%) and `min(fee_rebate, after_discount)`
`H-07`	`statement.py`	`:` separator and unsigned `"0"` on zero discount/rebate	Output had `M-1:USD` and `TIER_DISCOUNT: 0`	CONFIRMED	Format `{merchant_id}#{currency}` and always prefix `-`: `TIER_DISCOUNT: -{batch['tier_discount']}`

3. Why Free-Text Monologue Missed `validator_01` (`19/20`)

In Mode A (freetext-monologue-bundle.json), without H-03 explicitly distinguishing "rejecting missing required keys" from "stripping unapproved extra keys into a fresh 5-key dict", Mode A wrote:

# ❌ Mode A (FreeTextMonologueRunner — Turn 3 Final validator.py, 19/20)
        required_keys = {"merchant_id", "currency", "tx_count", "unit_cents", "chargeback_bps"}
        if set(item.keys()) != required_keys:  # <-- BUG: Raises ValueError when caller passes {"extra": 1} instead of stripping it!
            raise ValueError("invalid keys")

Mode B, guided by row H-03 ("extra": 1 leaked into output ➔ Construct a fresh 5-key dict), extracted the 5 required fields and dropped extra keys cleanly without raising a false-positive ValueError.

4. Drop-In Implementation Recipe: `HypothesisTableGate`

from dataclasses import dataclass

@dataclass
class HypothesisRow:
    hyp_id: str
    target_module: str
    suspected_defect: str
    probe_result: str
    status: str  # "CONFIRMED" or "REFUTED"
    canonical_fix: str

class HypothesisTableGate:
    """Blocks code edits until all target modules have confirmed/refuted rows in THINK.md."""
    def __init__(self, required_modules: set[str]):
        self.required_modules = required_modules
        self.confirmed_rows: list[HypothesisRow] = []

    def register_table(self, rows: list[HypothesisRow]) -> str:
        covered = {r.target_module for r in rows if r.status in {"CONFIRMED", "REFUTED"}}
        missing = self.required_modules - covered
        if missing:
            raise ValueError(f"THINK.md table incomplete; missing modules: {sorted(missing)}")
        self.confirmed_rows = [r for r in rows if r.status == "CONFIRMED"]
        return f"THINK.md locked with {len(self.confirmed_rows)} confirmed root causes."

5. Practical Takeaway: Force Tabular State for Multi-Bug Debugging

Replace unstructured CoT monologues with a fixed-column Markdown table (THINK.md) on multi-bug tasks: A 6-column table acts as an external working-memory register where every confirmed bug maps 1-to-1 to a required code change.
Keep REFUTED rows visible: Recording why a hypothesis was ruled out (status: REFUTED) prevents the agent from re-trying the same failed guess three turns later.

🔗 Continue Reading the Series

🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
🔒 Previous Sequential Post (E26): Why Claude Code's Plan Mode Works: Interleaved Read-Write Loops vs. a Two-Phase Read-Only Plan Lock (-94.8% Input Tokens)
🔌 Next Sequential Post (E28): Breaking A➔B➔A Edit Oscillations in Coding Agents: Unguarded Retry Loops vs. a Loop-Detection Circuit Breaker (-94.9% Input Tokens)
🌿 Related Context Hygiene Deep Dive (E19 — Rank #4): Why Keeping Failed Code Attempts in Chat History Poisons Your Coding Agent: Error-Trace Pruning vs. Accumulated History

Stop Your Coding Agent From Cycling Through Refuted Guesses: Free-Text Monologue vs. a Structured Hypothesis Matrix (2.54x Faster)

Metadata & Experiment Identity

1. Executive Summary (TL;DR)

Key Result

2. The Verified `THINK.md` Hypothesis Elimination Matrix (`input-03.json`)

3. Why Free-Text Monologue Missed `validator_01` (`19/20`)

4. Drop-In Implementation Recipe: `HypothesisTableGate`

5. Practical Takeaway: Force Tabular State for Multi-Bug Debugging

🔗 Continue Reading the Series

Comments

More from this blog

Stop Running Unrelated Monorepo Test Suites on Every Agent Edit: Full-Suite Runs vs. AST Dependency-Scoped Incremental Tests (-95.7% Input Tokens)

Deduplicating Cascading Test Failures for Coding Agents: Raw Error Dumps vs. AST Root-Cause Error Clustering (-95.3% Input Tokens)

Breaking A➔B➔A Edit Oscillations in Coding Agents: Unguarded Retry Loops vs. a Loop-Detection Circuit Breaker (-94.9% Input Tokens)

Why Claude Code's Plan Mode Works: Interleaved Read-Write Loops vs. a Two-Phase Read-Only Plan Lock (-94.8% Input Tokens)

Metadata & Experiment Identity

1. Executive Summary (TL;DR)

Key Result

2. The Verified THINK.md Hypothesis Elimination Matrix (input-03.json)

3. Why Free-Text Monologue Missed validator_01 (19/20)

4. Drop-In Implementation Recipe: HypothesisTableGate

5. Practical Takeaway: Force Tabular State for Multi-Bug Debugging

🔗 Continue Reading the Series

Comments

More from this blog

Stop Running Unrelated Monorepo Test Suites on Every Agent Edit: Full-Suite Runs vs. AST Dependency-Scoped Incremental Tests (-95.7% Input Tokens)

Deduplicating Cascading Test Failures for Coding Agents: Raw Error Dumps vs. AST Root-Cause Error Clustering (-95.3% Input Tokens)

Breaking A➔B➔A Edit Oscillations in Coding Agents: Unguarded Retry Loops vs. a Loop-Detection Circuit Breaker (-94.9% Input Tokens)

Why Claude Code's Plan Mode Works: Interleaved Read-Write Loops vs. a Two-Phase Read-Only Plan Lock (-94.8% Input Tokens)

2. The Verified `THINK.md` Hypothesis Elimination Matrix (`input-03.json`)

3. Why Free-Text Monologue Missed `validator_01` (`19/20`)

4. Drop-In Implementation Recipe: `HypothesisTableGate`