# Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%

> 💡 **AI Labo Benchmark Series — Experiment `E16` (Ranked #3 out of 41 Patterns)**
>
> This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
> * 🚀 **[View All 41 Benchmark Results & Master Ranking](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)**
> * 🛡️ **[Read How Our Proxmox & systemd Anti-Cheating Sandbox Works](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)**

---

## Metadata & Experiment Identity
* **Experiment ID:** `E16` (Backlog Candidate `C012`)
* **Run ID:** `e16-20261028-vm220-01`
* **Sandbox Environment:** Proxmox Isolated VM `VM220` (`ai-e16-220`, 0 NICs, `virtio-serial` QGA relay, `systemd` UID separation)
* **Model:** `gemini-3.8-flash` (`thinkingLevel: LOW`)
* **Pattern Under Test:** RTK (`rtk-ai/rtk`, Rust Token Killer) & SWE-agent Output Compression — *"Raw CLI Terminal Output vs. Semantic JSON Failure-Diff Filter"*

---

## 1. Executive Summary (TL;DR)

What is the single biggest token sink in an autonomous coding agent loop? Surprisingly, it isn't the source code itself—it is **unfiltered CLI test runner output**.

When an agent executes `pytest -vv --tb=long`, `cargo test`, or `npm test` on a realistic repository where 76 tests pass and 4 fail, dumping the raw terminal output (ANSI color codes, environment banners, 76 `[PASS]` lines, captured `DEBUG` stdout, and multi-frame internal tracebacks) directly into chat history adds **60,000+ characters (`28,000+` tokens) per test run**.

To measure the impact of deterministic CLI output filtering, we benchmarked a 3-module payment settlement pipeline (`validator.py`, `settlement.py`, `statement.py`) across two harness modes inside isolated VM `VM220`:

* **Mode A (`RawTerminalOutputRunner`):** Appends raw `pytest` terminal stdout (`60,749 chars` on Turn 1; `117,391 chars` cumulative) directly to the conversation history.
* **Mode B (`TokenKillerFilterRunner`):** Wraps the test runner in a deterministic preprocessor that strips ANSI escape codes, environment banners, all 76 passing tests, and internal framework frames—returning **only a `1,506-character` structured JSON summary of the 4 failed assertions**.

### Key Result
Compressing raw CLI test output from **`117,391 chars` ➔ `1,604 chars` (`-98.6%` reduction, `73.2x` compression)** enabled Mode B to fix all three modules in a single edit turn (`2` LLM calls total vs. `3` in Mode A). Mode B cut cumulative input tokens by **`-94.1%` (`92,385` ➔ `5,445` tokens, `16.97x` efficiency)**, API cost by **`-87.0%` (`$0.0770` ➔ `$0.0100`)**, and wall-clock time by **`-35.0%` (`1.54x` faster)** while achieving **`20 / 20` (100%)** on the hidden grading suite.

| Metric | Mode A: Raw Terminal Output (`RawTerminalOutput`) | Mode B: Semantic JSON Filter (`TokenKillerFilter`) | Delta / Improvement |
| :--- | :---: | :---: | :---: |
| **Hidden Grading Suite Score (20 Cases)** | `20 / 20` (100%) | **`20 / 20` (100%)** | Both passed all 20 cases |
| **Cumulative CLI Tool Output (`tool_output_chars`)** | `117,391 chars` | **`1,604 chars`** | **-98.6% (73.19x compression)** |
| **First Test Run Output Size** | `60,749 chars` | **`1,506 chars`** | **-97.5% (76 PASS lines & ANSI stripped)** |
| **LLM Calls (`model_calls`)** | `3 calls` | **`2 calls` (1 test run + 1 full fix)** | **-33.3% (No intermediate partial patch)** |
| **Turn 1 Input / Output Tokens** | `2,406` in / `10` out | **`2,415` in / `10` out** | Identical initial test invocation |
| **Turn 2 Input / Output Tokens** | `31,170` in / `484` out | **`3,030` in / `1,561` out** | **Turn 2 input cut by -90.3%** |
| **Turn 3 Input / Output Tokens** | `58,809` in / `1,574` out | — *(Completed on Turn 2)* | **Avoided `58,809`-token log snowball** |
| **Cumulative Input Tokens (`promptTokenCount`)** | `92,385 tokens` | **`5,445 tokens`** | **-94.1% (16.97x token efficiency)** |
| **Cumulative Output Tokens (`candidatesTokenCount`)** | `2,068 tokens` | **`1,571 tokens`** | **-24.0% reduction** |
| **Elapsed Execution Time (`elapsed_seconds`)** | `25.94 sec` | **`16.88 sec`** | **-35.0% (1.54x faster)** |
| **Estimated API Cost (`estimated_micro_usd`)** | `$0.0770` (`77,045 micro-USD`) | **`$0.0100` (`9,976 micro-USD`)** | **-87.0% (7.72x cheaper)** |

---

## 2. Experiment Architecture & Workflow Comparison

```text
+-------------------------------------------------------------------------+
| Proxmox Isolated VM220 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Suite: 80-Test Suite (76 PASS / 4 FAIL across 3 modules)        |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: RawTerminalOutputRunner (Unfiltered pytest -vv --tb=long) |  |
|  |  - Turn 1: run_raw_pytest     --> Returns 60,749 chars raw log    |  |
|  |  - Turn 2: stage_partial_fix  --> Input jumps to 31,170 tokens!   |  |
|  |                                   Returns 56,544 chars raw log    |  |
|  |  - Turn 3: submit_settlement  --> Input hits 58,809 tokens!       |  |
|  |                                   (Total: 92,385 input tokens)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: TokenKillerFilterRunner (RTK-Style Semantic JSON Filter)  |  |
|  |  - Turn 1: run_filtered_tests --> Strips 76 PASS & ANSI noise;    |  |
|  |                                   returns 1,506-char failure JSON |  |
|  |  - Turn 2: submit_settlement  --> Input is only 3,030 tokens!     |  |
|  |                                   Fixes all 3 files in 1 shot     |  |
|  |                                   (Total: 5,445 input tokens)     |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
```

---

## 3. Inside the Trace: How Raw CLI Logs Snowball Across Turns

### 3.1 Mode A (`RawTerminalOutput`): How 2 Test Runs Burned 92,385 Input Tokens

Tracking `execution-traces.json` and `usage.json` reveals how fast raw terminal output compounds in an agent's message array:

1. **Turn 1 (`2,406 in / 10 out`):** The agent calls `{"tool": "run_raw_pytest"}`. The runner executes the 80-test suite and returns **`60,749 characters`** containing ANSI color codes, platform headers, 76 verbose `[PASS]` lines with `AUDIT_TRACE` dumps, and long multi-frame tracebacks for the 4 failures.
2. **Turn 2 (`31,170 in / 484 out`):** Appending that single raw stdout dump causes the prompt size to jump by **`+28,764 tokens`** (`2,406` ➔ `31,170`). Distracted by the 60KB wall of text, the agent only patches `validator.py` (`stage_partial_fix`), triggering a second test run that appends another **`56,544 characters`** (now 77 PASS lines + 3 FAIL tracebacks) to the conversation history.
3. **Turn 3 (`58,809 in / 1,574 out`):** Now carrying **117,293 characters** of raw terminal logs across two turns, Turn 3 alone requires **`58,809 input tokens`** before the agent finally submits all 3 modules (`92,385` cumulative input tokens).

### 3.2 Mode B (`TokenKillerFilter`): 1,506 Characters of Pure Signal

When Mode B called `{"tool": "run_filtered_tests"}` on Turn 1, the harness preprocessor stripped all 76 passing tests, ANSI escape sequences, and `DEBUG` noise—returning a **`1,506-character` (`615-token`) JSON payload** containing only the 4 failing tests (`test`, `file`, `line`, `assertion`, `expected_vs_actual`):

```json
{
  "summary": {"total": 80, "passed": 76, "failed": 4, "filtered_noise_chars_removed": 59243},
  "failures": [
    {
      "test": "test_validator_contract",
      "file": "validator.py",
      "line": 14,
      "assertion": "ALLOWED_CURRENCIES must include 'JPY'; reject empty list & bool; allow unit_cents==0 & chargeback_bps==1000; do not mutate input dict."
    },
    {
      "test": "test_settlement_ordering_and_reserve",
      "file": "settlement.py",
      "line": 22,
      "assertion": "Preserve first-seen (merchant_id, currency) order (never sorted(order)); apply 10% reserve strictly when chargeback_bps > 300."
    },
    {
      "test": "test_settlement_discounts_and_rebate_cap",
      "file": "settlement.py",
      "line": 39,
      "assertion": "Discount boundary is subtotal >= 15000 (GROWTH 10%, ENTERPRISE 20% else 10%); cap rebate_applied = min(fee_rebate, after_discount)."
    },
    {
      "test": "test_statement_format_and_signed_zero",
      "file": "statement.py",
      "line": 11,
      "assertion": "Use '#' delimiter ({merchant_id}#{currency}:) and preserve leading '-' on zero discount/rebate (TIER_DISCOUNT: -0, REBATE_APPLIED: -0)."
    }
  ]
}
```

Because all four failure root causes were immediately visible without scrolling through 76 passing tests, Mode B fixed all three files in a single Turn 2 call (`3,030` input tokens)—eliminating the intermediate rework turn entirely.

---

## 4. All 20 Hidden Grading Suite Results (`labjudge16`)

Both modes passed all 20 hidden unit and E2E test cases verified by `labjudge16`:

| # | Test Case ID | Invariant / Boundary Verified | Mode A (`Raw CLI`) | Mode B (`Token Killer`) |
| :---: | :--- | :--- | :---: | :---: |
| 1–6 | `validator_01` to `validator_06` | Trim/uppercase, non-mutation, `JPY`, `bool` rejection, `0` cents & `1000` bps bounds | `6 / 6` PASS | **`6 / 6` PASS** |
| 7–15 | `settlement_01` to `settlement_09` | First-seen order, conflict check, `300 vs 301` bps, `14999 vs 15000` tier, rebate cap | `9 / 9` PASS | **`9 / 9` PASS** |
| 16–20 | `e2e_01` to `e2e_05` | `#` separator, signed zero `-0`, `$0` auth statement, line order, error propagation | `5 / 5` PASS | **`5 / 5` PASS** |
| **Total** | **20 Hidden Cases** | — | **`20 / 20` (100%)** | **`20 / 20` (100%)** |

---

## 5. Practical Takeaway: Wrap Your Agent's Test Runner Today

Never wire raw `subprocess.run(["pytest"])` stdout directly into your LLM agent's tool response. Instead:

1. **Use machine-readable reporters:** Run `pytest --tb=short -q --no-header --no-summary` (or `pytest-json-report`) inside your tool wrapper.
2. **Strip ANSI codes and passing tests:** If 500 tests pass and 2 fail, the LLM only needs `"passed": 500` as a single integer plus the exact file, line number, and assertion diff for the 2 failures.
3. **Adopt CLI proxy filters:** Tools like `rtk` (`Rust Token Killer`) or a 30-line Python wrapper around your test runner immediately cut agent token bills by **85%–95%** while speeding up multi-file bug fixes.

---

### 🔗 Continue Reading the Series
* 🚀 **Master Benchmark Hub (All 41 Patterns Ranked):** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)
* 🛡️ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)
* 🥇 **Rank #1 Deep Dive (`E20`):** [Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item `MEMORY.md`](https://tappers.hashnode.dev/e20-episodic-gotchas-memory)
* 🥈 **Rank #2 Deep Dive (`E21`):** [Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons](https://tappers.hashnode.dev/e21-treesitter-skeleton-context)