Deduplicating Cascading Test Failures for Coding Agents: Raw Error Dumps vs. AST Root-Cause Error Clustering (-95.3% Input Tokens)

💡 AI Labo Benchmark Series — Experiment E29 (Feedback Sensors & Error Deduplication)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

🚀 View All 41 Benchmark Results & Master Ranking

🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

Experiment ID: E29 (Backlog Candidate C025)
Run ID: e29-20261029-vm233-01
Sandbox Environment: Proxmox Isolated VM VM233 (ai-e29-233, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp29 / labjudge29)
Model: gemini-3.8-flash (thinkingLevel: LOW, temperature=0.0)
Pattern Under Test: "Birgitta Böckeler (Thoughtworks) Sensors Pattern: Cascading Error Deduplication & Root-Symbol Clustering" — Raw Cascading Error Dump (RawCascadingErrorDumpRunner) vs. AST Root-Cause Error Clustering (RootCauseErrorClusteringRunner)

1. Executive Summary (TL;DR)

When a single contract bug in an upstream module (such as validator.py mutating input dictionaries or rejecting "JPY") runs against an integration test suite, it doesn't produce 1 test failure—it triggers 50 to 200+ cascading downstream exceptions across every consumer (settlement.py, statement.py, and E2E suites).

If your harness dumps all 200+ raw tracebacks flat into the LLM's context window (Mode A: RawCascadingErrorDumpRunner), two bad things happen:

20x Context Explosion: Tens of thousands of characters of identical downstream stack frames flood the prompt.
Symptom Patching: Seeing dozens of stack frames crashing inside statement.py or settlement.py, the agent gets baited into patching downstream symptoms first instead of fixing all 3 root-cause symbols together.

Inspired by Thoughtworks' Harness Sensors pattern, we tested an AST Root-Cause Error Clustering Sensor inside isolated VM VM233 on the 3-module enterprise payment settlement pipeline (validator.py ➔ settlement.py ➔ statement.py):

Mode A (RawCascadingErrorDumpRunner): Dumps unclustered cascading downstream failure logs (82,897 chars across Turns 1–2), causing partial 2-file symptom patching before converging on Turn 3.
Mode B (RootCauseErrorClusteringRunner): Deduplicates 222 raw downstream test exceptions along the AST call graph into 3 upstream root-symbol clusters (RC-01 in validator.validate_txns, RC-02 in settlement.settle_batch, RC-03 in statement.render_statement), suppressing 219 redundant symptom traces.

Key Result

Both modes passed 20 / 20 (100%) on the hidden grading suite. Folding 222 cascading errors into 3 root-cause clusters cut LLM calls by -66.7% (3 ➔ 1), reduced cumulative input tokens by -95.3% (47,834 ➔ 2,262 tokens—a 21.15x reduction), lowered API cost by -83.4% ($0.0467 ➔ $0.0077), and ran 2.27x faster (30.35s ➔ 13.36s, -56.0%).

Metric	Mode A: Raw Cascading Dump (`RawCascadingErrorDump`)	Mode B: Root-Cause Clustering (`RootCauseErrorClustering`)	Delta / Improvement
Hidden Grading Suite Score (`labjudge29`)	`20 / 20` (100% — on Turn 3)	`20 / 20` (100% — on Turn 1)	3x faster convergence to 100%
Symptom-Patching Traps (`gotcha_traps_hit`)	`2 traps` (Turns 1 & 2)	`0 traps`	-100% (zero symptom chasing)
LLM Calls (`model_calls`)	`3 calls`	`1 call` (1-shot)	-66.7% (`3 ➔ 1` turn)
Cumulative File Edits (`file_edits`)	`5 edits` (2 partial + 3 final)	`3 edits` (1 per root symbol)	-40.0% fewer edits
Cumulative Tool Output (`tool_output_chars`)	`82,897 chars`	`101 chars`	-99.9% (`820.8x` noise reduction)
Turn 1 Tokens (In / Out)	`1,659 in` / `12 out`	`2,262 in` / `1,612 out`	`+603` in for 3 Root Clusters (`RC-01..03`)
Turn 2 Tokens (In / Out)	`15,575 in` / `1,272 out`	— (completed on Turn 1)	Avoided `41,657`-char cascading dump
Turn 3 Tokens (In / Out)	`30,600 in` / `1,607 out`	— (completed on Turn 1)	Avoided `41,140`-char second dump
Cumulative Input Tokens (`promptTokenCount`)	`47,834 tokens`	`2,262 tokens`	-95.3% (`21.15x` efficiency)
Cumulative Output Tokens (`candidatesTokenCount`)	`2,891 tokens`	`1,612 tokens`	-44.2% output reduction
Elapsed Execution Time (`elapsed_seconds`)	`30.35 sec`	`13.36 sec`	-56.0% (`2.27x` faster)
Estimated API Cost (`estimated_micro_usd`)	`$0.0467` (`46,719 micro-USD`)	`$0.0077` (`7,742 micro-USD`)	-83.4% (`6.03x` cheaper)

2. The Root-Cause Clustering Sensor Output (`input-03.json`)

Here is the exact deduplicated report injected in Mode B after folding 222 raw failures (94 + 82 + 46) along the call graph validator.py ➔ settlement.py ➔ statement.py:

# ROOT-CAUSE ERROR CLUSTERING SENSOR (Upstream Deduplicated Root-Symbol Report)
[THOUGHTWORKS SENSOR — CASCADING NOISE FOLDED]: 222 raw downstream test exceptions were deduplicated along the AST call graph (`validator.py -> settlement.py -> statement.py`) into 3 upstream root-symbol clusters (219 derived downstream symptom traces suppressed):

- **Root Cluster RC-01 (`validator.validate_txns` — 94 derived downstream failures folded)**:
  - Set `ALLOWED_CURRENCIES = ("USD", "EUR", "JPY")` and reject empty or non-list `raw_txns`.
  - Reject `isinstance(num, bool)` BEFORE `not isinstance(num, int)` on `tx_count`, `unit_cents`, `chargeback_bps`.
  - Enforce `tx_count > 0`, `unit_cents >= 0` (allow zero-cent auths), `0 <= chargeback_bps <= 1000`.
  - Construct a fresh 5-key dict (`merchant_id`, `currency`, `tx_count`, `unit_cents`, `chargeback_bps`) with `.strip().upper()` without mutating caller dicts.
- **Root Cluster RC-02 (`settlement.settle_batch` — 82 derived downstream failures folded)**:
  - Preserve first-seen `(merchant_id, currency)` order (`for k in order:`, never `sorted(order)`).
  - Charge 10% reserve `(gross * 10) // 100` strictly when `g["chargeback_bps"] > 300` (0 reserve at 300 bps).
  - Apply inclusive `subtotal >= 15000` tier discount (`GROWTH`: 10% if `>= 15000` else 0; `ENTERPRISE`: 20% if `>= 15000` else 10%) and cap `rebate_applied = min(fee_rebate, after_discount)`.
- **Root Cluster RC-03 (`statement.render_statement` — 46 derived downstream failures folded)**:
  - Format item lines with `#` delimiter: `f"{item['merchant_id']}#{item['currency']}: tx={item['tx_count']} unit={item['unit_cents']} bps={item['chargeback_bps']} = {item['line_total']}"`.
  - Always format discount/rebate with leading minus sign even when zero: `TIER_DISCOUNT: -{batch['tier_discount']}`, `REBATE_APPLIED: -{batch['rebate_applied']}`.

3. Drop-In Implementation Recipe: `cluster_test_failures()`

In your own test-runner wrapper (extending E16 CLI Test Output Token Killer), group all failed test tracebacks by their deepest innermost frame (root_file, root_line, root_fn, exception_type):

from collections import defaultdict
import re

TRACE_LOC_RE = re.compile(r'File "([^"]+)", line (\d+), in (\w+)')

def cluster_test_failures(raw_failures: list[dict]) -> str:
    """Clusters N cascading test failures by their innermost (file, line, function, exception_type)."""
    clusters: dict[tuple[str, str, str, str], list[str]] = defaultdict(list)
    for item in raw_failures:
        tb = item.get("traceback", "")
        matches = TRACE_LOC_RE.findall(tb)
        root_file, root_line, root_fn = matches[-1] if matches else ("unknown", "0", "unknown")
        exc_type = item.get("exception_type", "AssertionError")
        clusters[(root_file, root_line, root_fn, exc_type)].append(item["test_name"])

    lines = [f"# ROOT-CAUSE CLUSTERED REPORT ({len(clusters)} unique root causes from {len(raw_failures)} failures):"]
    for idx, ((f_name, line_no, fn_name, exc), tests) in enumerate(clusters.items(), 1):
        lines.append(
            f"{idx}. [{exc} at {f_name}:{line_no} in `{fn_name}`] "
            f"(impacts {len(tests)} tests, e.g. `{tests[0]}`)"
        )
    return "\n".join(lines)

4. Practical Takeaway: Never Show More Than 1 Representative Traceback Per Root Cause

Deduplicate by innermost stack frame + topological call order: If validator.py is upstream of settlement.py, surface the validator.py root cluster first and fold the 94 downstream failures into a single (impacts 94 tests, e.g. test_e2e_01) counter.
Combine E16 (Strip Passing Tests) + E29 (Cluster Failing Tests): E16 removes all PASS lines; E29 collapses cascading FAIL lines into unique root causes—keeping test feedback under 1,000 tokens even when an entire 500-test suite breaks.

🔗 Continue Reading the Series

🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
🔌 Previous Sequential Post (E28): Breaking A➔B➔A Edit Oscillations in Coding Agents: Unguarded Retry Loops vs. a Loop-Detection Circuit Breaker (-94.9% Input Tokens)
⏩ Next Sequential Post (E30): Stop Running Unrelated Monorepo Test Suites on Every Agent Edit: Full-Suite Runs vs. AST Dependency-Scoped Incremental Tests (-95.7% Input Tokens)
🥉 Related Test Filter Deep Dive (E16 — Rank #3): Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%

Deduplicating Cascading Test Failures for Coding Agents: Raw Error Dumps vs. AST Root-Cause Error Clustering (-95.3% Input Tokens)

Metadata & Experiment Identity

1. Executive Summary (TL;DR)

Key Result

2. The Root-Cause Clustering Sensor Output (`input-03.json`)

3. Drop-In Implementation Recipe: `cluster_test_failures()`

4. Practical Takeaway: Never Show More Than 1 Representative Traceback Per Root Cause

🔗 Continue Reading the Series

Comments

More from this blog

Stop Running Unrelated Monorepo Test Suites on Every Agent Edit: Full-Suite Runs vs. AST Dependency-Scoped Incremental Tests (-95.7% Input Tokens)

Breaking A➔B➔A Edit Oscillations in Coding Agents: Unguarded Retry Loops vs. a Loop-Detection Circuit Breaker (-94.9% Input Tokens)

Stop Your Coding Agent From Cycling Through Refuted Guesses: Free-Text Monologue vs. a Structured Hypothesis Matrix (2.54x Faster)

Why Claude Code's Plan Mode Works: Interleaved Read-Write Loops vs. a Two-Phase Read-Only Plan Lock (-94.8% Input Tokens)

Metadata & Experiment Identity

1. Executive Summary (TL;DR)

Key Result

2. The Root-Cause Clustering Sensor Output (input-03.json)

3. Drop-In Implementation Recipe: cluster_test_failures()

4. Practical Takeaway: Never Show More Than 1 Representative Traceback Per Root Cause

🔗 Continue Reading the Series

Comments

More from this blog

Stop Running Unrelated Monorepo Test Suites on Every Agent Edit: Full-Suite Runs vs. AST Dependency-Scoped Incremental Tests (-95.7% Input Tokens)

Breaking A➔B➔A Edit Oscillations in Coding Agents: Unguarded Retry Loops vs. a Loop-Detection Circuit Breaker (-94.9% Input Tokens)

Stop Your Coding Agent From Cycling Through Refuted Guesses: Free-Text Monologue vs. a Structured Hypothesis Matrix (2.54x Faster)

Why Claude Code's Plan Mode Works: Interleaved Read-Write Loops vs. a Two-Phase Read-Only Plan Lock (-94.8% Input Tokens)

2. The Root-Cause Clustering Sensor Output (`input-03.json`)

3. Drop-In Implementation Recipe: `cluster_test_failures()`