# Deduplicating Cascading Test Failures for Coding Agents: Raw Error Dumps vs. AST Root-Cause Error Clustering (-95.3% Input Tokens)

> 💡 **AI Labo Benchmark Series — Experiment `E29` (Feedback Sensors & Error Deduplication)**
>
> This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
> * 🚀 **[View All 41 Benchmark Results & Master Ranking](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)**
> * 🛡️ **[Read How Our Proxmox & systemd Anti-Cheating Sandbox Works](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)**

---

## Metadata & Experiment Identity
* **Experiment ID:** `E29` (Backlog Candidate `C025`)
* **Run ID:** `e29-20261029-vm233-01`
* **Sandbox Environment:** Proxmox Isolated VM `VM233` (`ai-e29-233`, 0 NICs, `PrivateNetwork=yes`, `NoNewPrivileges=yes`, `InaccessiblePaths`, UID split `labexp29` / `labjudge29`)
* **Model:** `gemini-3.8-flash` (`thinkingLevel: LOW`, `temperature=0.0`)
* **Pattern Under Test:** *"Birgitta Böckeler (Thoughtworks) Sensors Pattern: Cascading Error Deduplication & Root-Symbol Clustering"* — **Raw Cascading Error Dump (`RawCascadingErrorDumpRunner`) vs. AST Root-Cause Error Clustering (`RootCauseErrorClusteringRunner`)**

---

## 1. Executive Summary (TL;DR)

When a single contract bug in an upstream module (such as `validator.py` mutating input dictionaries or rejecting `"JPY"`) runs against an integration test suite, it doesn't produce 1 test failure—it triggers **50 to 200+ cascading downstream exceptions** across every consumer (`settlement.py`, `statement.py`, and E2E suites).

If your harness dumps all 200+ raw tracebacks flat into the LLM's context window (**Mode A: `RawCascadingErrorDumpRunner`**), two bad things happen:
1. **20x Context Explosion:** Tens of thousands of characters of identical downstream stack frames flood the prompt.
2. **Symptom Patching:** Seeing dozens of stack frames crashing inside `statement.py` or `settlement.py`, the agent gets baited into patching downstream symptoms first instead of fixing all 3 root-cause symbols together.

Inspired by Thoughtworks' **Harness Sensors** pattern, we tested an **AST Root-Cause Error Clustering Sensor** inside isolated VM `VM233` on the 3-module enterprise payment settlement pipeline (`validator.py` ➔ `settlement.py` ➔ `statement.py`):

* **Mode A (`RawCascadingErrorDumpRunner`):** Dumps unclustered cascading downstream failure logs (`82,897` chars across Turns 1–2), causing partial 2-file symptom patching before converging on Turn 3.
* **Mode B (`RootCauseErrorClusteringRunner`):** Deduplicates `222` raw downstream test exceptions along the AST call graph into **3 upstream root-symbol clusters (`RC-01` in `validator.validate_txns`, `RC-02` in `settlement.settle_batch`, `RC-03` in `statement.render_statement`)**, suppressing `219` redundant symptom traces.

### Key Result
Both modes passed **`20 / 20` (100%)** on the hidden grading suite. Folding `222` cascading errors into `3` root-cause clusters cut **LLM calls by `-66.7%` (`3 ➔ 1`)**, reduced **cumulative input tokens by `-95.3%` (`47,834 ➔ 2,262` tokens—a `21.15x` reduction)**, lowered **API cost by `-83.4%` (`$0.0467 ➔ $0.0077`)**, and ran **`2.27x` faster (`30.35s ➔ 13.36s`, `-56.0%`)**.

| Metric | Mode A: Raw Cascading Dump (`RawCascadingErrorDump`) | Mode B: Root-Cause Clustering (`RootCauseErrorClustering`) | Delta / Improvement |
| :--- | :---: | :---: | :---: |
| **Hidden Grading Suite Score (`labjudge29`)** | `20 / 20` (100% — on Turn 3) | **`20 / 20` (100% — on Turn 1)** | **3x faster convergence to 100%** |
| **Symptom-Patching Traps (`gotcha_traps_hit`)** | `2 traps` (Turns 1 & 2) | **`0 traps`** | **-100% (zero symptom chasing)** |
| **LLM Calls (`model_calls`)** | `3 calls` | **`1 call` (1-shot)** | **-66.7% (`3 ➔ 1` turn)** |
| **Cumulative File Edits (`file_edits`)** | `5 edits` (2 partial + 3 final) | **`3 edits` (1 per root symbol)** | **-40.0% fewer edits** |
| **Cumulative Tool Output (`tool_output_chars`)** | `82,897 chars` | **`101 chars`** | **-99.9% (`820.8x` noise reduction)** |
| **Turn 1 Tokens (In / Out)** | `1,659 in` / `12 out` | **`2,262 in` / `1,612 out`** | `+603` in for 3 Root Clusters (`RC-01..03`) |
| **Turn 2 Tokens (In / Out)** | `15,575 in` / `1,272 out` | *— (completed on Turn 1)* | Avoided `41,657`-char cascading dump |
| **Turn 3 Tokens (In / Out)** | `30,600 in` / `1,607 out` | *— (completed on Turn 1)* | Avoided `41,140`-char second dump |
| **Cumulative Input Tokens (`promptTokenCount`)** | `47,834 tokens` | **`2,262 tokens`** | **-95.3% (`21.15x` efficiency)** |
| **Cumulative Output Tokens (`candidatesTokenCount`)** | `2,891 tokens` | **`1,612 tokens`** | **-44.2% output reduction** |
| **Elapsed Execution Time (`elapsed_seconds`)** | `30.35 sec` | **`13.36 sec`** | **-56.0% (`2.27x` faster)** |
| **Estimated API Cost (`estimated_micro_usd`)** | `$0.0467` (`46,719 micro-USD`) | **`$0.0077` (`7,742 micro-USD`)** | **-83.4% (`6.03x` cheaper)** |

---

## 2. The Root-Cause Clustering Sensor Output (`input-03.json`)

Here is the exact deduplicated report injected in Mode B after folding `222` raw failures (`94 + 82 + 46`) along the call graph `validator.py ➔ settlement.py ➔ statement.py`:

```markdown
# ROOT-CAUSE ERROR CLUSTERING SENSOR (Upstream Deduplicated Root-Symbol Report)
[THOUGHTWORKS SENSOR — CASCADING NOISE FOLDED]: 222 raw downstream test exceptions were deduplicated along the AST call graph (`validator.py -> settlement.py -> statement.py`) into 3 upstream root-symbol clusters (219 derived downstream symptom traces suppressed):

- **Root Cluster RC-01 (`validator.validate_txns` — 94 derived downstream failures folded)**:
  - Set `ALLOWED_CURRENCIES = ("USD", "EUR", "JPY")` and reject empty or non-list `raw_txns`.
  - Reject `isinstance(num, bool)` BEFORE `not isinstance(num, int)` on `tx_count`, `unit_cents`, `chargeback_bps`.
  - Enforce `tx_count > 0`, `unit_cents >= 0` (allow zero-cent auths), `0 <= chargeback_bps <= 1000`.
  - Construct a fresh 5-key dict (`merchant_id`, `currency`, `tx_count`, `unit_cents`, `chargeback_bps`) with `.strip().upper()` without mutating caller dicts.
- **Root Cluster RC-02 (`settlement.settle_batch` — 82 derived downstream failures folded)**:
  - Preserve first-seen `(merchant_id, currency)` order (`for k in order:`, never `sorted(order)`).
  - Charge 10% reserve `(gross * 10) // 100` strictly when `g["chargeback_bps"] > 300` (0 reserve at 300 bps).
  - Apply inclusive `subtotal >= 15000` tier discount (`GROWTH`: 10% if `>= 15000` else 0; `ENTERPRISE`: 20% if `>= 15000` else 10%) and cap `rebate_applied = min(fee_rebate, after_discount)`.
- **Root Cluster RC-03 (`statement.render_statement` — 46 derived downstream failures folded)**:
  - Format item lines with `#` delimiter: `f"{item['merchant_id']}#{item['currency']}: tx={item['tx_count']} unit={item['unit_cents']} bps={item['chargeback_bps']} = {item['line_total']}"`.
  - Always format discount/rebate with leading minus sign even when zero: `TIER_DISCOUNT: -{batch['tier_discount']}`, `REBATE_APPLIED: -{batch['rebate_applied']}`.
```

---

## 3. Drop-In Implementation Recipe: `cluster_test_failures()`

In your own test-runner wrapper (extending **`E16` CLI Test Output Token Killer**), group all failed test tracebacks by their deepest innermost frame `(root_file, root_line, root_fn, exception_type)`:

```python
from collections import defaultdict
import re

TRACE_LOC_RE = re.compile(r'File "([^"]+)", line (\d+), in (\w+)')

def cluster_test_failures(raw_failures: list[dict]) -> str:
    """Clusters N cascading test failures by their innermost (file, line, function, exception_type)."""
    clusters: dict[tuple[str, str, str, str], list[str]] = defaultdict(list)
    for item in raw_failures:
        tb = item.get("traceback", "")
        matches = TRACE_LOC_RE.findall(tb)
        root_file, root_line, root_fn = matches[-1] if matches else ("unknown", "0", "unknown")
        exc_type = item.get("exception_type", "AssertionError")
        clusters[(root_file, root_line, root_fn, exc_type)].append(item["test_name"])

    lines = [f"# ROOT-CAUSE CLUSTERED REPORT ({len(clusters)} unique root causes from {len(raw_failures)} failures):"]
    for idx, ((f_name, line_no, fn_name, exc), tests) in enumerate(clusters.items(), 1):
        lines.append(
            f"{idx}. [{exc} at {f_name}:{line_no} in `{fn_name}`] "
            f"(impacts {len(tests)} tests, e.g. `{tests[0]}`)"
        )
    return "\n".join(lines)
```

---

## 4. Practical Takeaway: Never Show More Than 1 Representative Traceback Per Root Cause

1. **Deduplicate by innermost stack frame + topological call order:** If `validator.py` is upstream of `settlement.py`, surface the `validator.py` root cluster first and fold the 94 downstream failures into a single `(impacts 94 tests, e.g. test_e2e_01)` counter.
2. **Combine `E16` (Strip Passing Tests) + `E29` (Cluster Failing Tests):** `E16` removes all `PASS` lines; `E29` collapses cascading `FAIL` lines into unique root causes—keeping test feedback under `1,000` tokens even when an entire 500-test suite breaks.

---

### 🔗 Continue Reading the Series
* 🚀 **Master Benchmark Hub (All 41 Patterns Ranked):** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)
* 🛡️ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)
* 🔌 **Previous Sequential Post (`E28`):** [Breaking A➔B➔A Edit Oscillations in Coding Agents: Unguarded Retry Loops vs. a Loop-Detection Circuit Breaker (-94.9% Input Tokens)](https://tappers.hashnode.dev/e28-loop-detection-circuit-breaker)
* ⏩ **Next Sequential Post (`E30`):** [Stop Running Unrelated Monorepo Test Suites on Every Agent Edit: Full-Suite Runs vs. AST Dependency-Scoped Incremental Tests (-95.7% Input Tokens)](https://tappers.hashnode.dev/e30-dependency-scoped-incremental-test)
* 🥉 **Related Test Filter Deep Dive (`E16` — Rank #3):** [Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%](https://tappers.hashnode.dev/e16-cli-test-output-token-killer)