# Stop Running Unrelated Monorepo Test Suites on Every Agent Edit: Full-Suite Runs vs. AST Dependency-Scoped Incremental Tests (-95.7% Input Tokens)

> 💡 **AI Labo Benchmark Series — Experiment `E30` (Feedback Sensors & Incremental Verification)**
>
> This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
> * 🚀 **[View All 41 Benchmark Results & Master Ranking](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)**
> * 🛡️ **[Read How Our Proxmox & systemd Anti-Cheating Sandbox Works](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)**

---

## Metadata & Experiment Identity
* **Experiment ID:** `E30` (Backlog Candidate `C026`)
* **Run ID:** `e30-20261029-vm234-01`
* **Sandbox Environment:** Proxmox Isolated VM `VM234` (`ai-e30-234`, 0 NICs, `PrivateNetwork=yes`, `NoNewPrivileges=yes`, `InaccessiblePaths`, UID split `labexp30` / `labjudge30`)
* **Model:** `gemini-3.8-flash` (`thinkingLevel: LOW`, `temperature=0.0`)
* **Pattern Under Test:** *"OpenAI 1M-LOC Agent Harness Feedback Loop Optimization: AST Dependency-Scoped Incremental Test Execution"* — **Unfiltered Full-Repository Test Suite (`FullSuiteTestExecutionRunner`) vs. AST Dependency-Scoped Incremental Runner (`DependencyScopedIncrementalRunner`)**

---

## 1. Executive Summary (TL;DR)

In a large monorepo or multi-package service, running `pytest` across the entire repository after every agent edit—including `148` completely unrelated packages (`auth_ledger`, `fx_rates`, `webhook_dispatch`, `audit_archive`)—creates a massive **Signal-to-Noise Ratio (SNR) collapse**:
1. **Wall-Clock Lag:** The agent waits for 150+ unaffected test files to import and run.
2. **Attention Dilution:** When `82,343` characters of unfiltered full-suite `PASS`/`SKIP` telemetry flood the context window, the model loses sight of subtle edge cases in the 3 modules that were actually modified.

We tested **AST Dependency-Scoped Incremental Test Execution** (the Python AST equivalent of `pytest-testmon` / `bazel test` affected-target scoping) inside isolated VM `VM234` on the 3-module enterprise payment settlement pipeline (`validator.py` ➔ `settlement.py` ➔ `statement.py`):

* **Mode A (`FullSuiteTestExecutionRunner`):** Runs all 168 repository suites without dependency scoping on intermediate turns (`82,343` chars of monorepo test output across Turns 1–2).
* **Mode B (`DependencyScopedIncrementalRunner`):** Uses a reverse AST import graph to skip `148` unaffected monorepo subtrees during inner-loop verification, scoping feedback strictly to the **20 affected tests (`6` in `validator.py`, `9` in `settlement.py`, `5` E2E in `statement.py`)** before running the full gate at final submit.

### Key Result
Mode A dropped to **`16 / 20` (80.0% — 4 failed cases)** after 3 turns (`52,146` cumulative input tokens)—missing `extra` key filtering (`new_item = dict(item)`), rejecting `1000` bps (`0 <= bps < 1000`), and rejecting `$0` authorizations (`cents <= 0`). **Mode B achieved a `20 / 20` (100%) pass in 1 turn**, cutting **LLM calls by `-66.7%` (`3 ➔ 1`)**, **cumulative input tokens by `-95.7%` (`52,146 ➔ 2,240` tokens—a `23.28x` reduction)**, **API cost by `-84.4%` (`$0.0499 ➔ $0.0078`)**, and running **`2.11x` faster (`28.97s ➔ 13.76s`, `-52.5%`)**.

| Metric | Mode A: Unfiltered Full Suite (`FullSuiteTestExecution`) | Mode B: AST Dependency-Scoped (`DependencyScopedIncremental`) | Delta / Improvement |
| :--- | :---: | :---: | :---: |
| **Hidden Grading Suite Score (`labjudge30`)** | `16 / 20` (80.0% — **4 failures**) | **`20 / 20` (100% — Perfect)** | **+20.0% accuracy (`+4` cases)** |
| **Full-Suite Noise Traps (`gotcha_traps_hit`)** | `2 traps` (Turns 1 & 2) | **`0 traps`** | **-100% (zero rework)** |
| **LLM Calls (`model_calls`)** | `3 calls` | **`1 call` (1-shot)** | **-66.7% (`3 ➔ 1` turn)** |
| **Cumulative File Edits (`file_edits`)** | `5 edits` | **`3 edits` (1 per module)** | **-40.0% fewer edits** |
| **Cumulative Tool Output (`tool_output_chars`)** | `82,343 chars` (`168` suites) | **`104 chars` (`20` scoped tests)** | **-99.9% (`791.8x` noise reduction)** |
| **Turn 1 Tokens (In / Out)** | `1,657 in` / `13 out` | **`2,240 in` / `1,624 out`** | `+583` in for scoped contract matrix |
| **Turn 2 Tokens (In / Out)** | `17,002 in` / `1,256 out` | *— (completed on Turn 1)* | Avoided `41,188`-char Turn 1 suite log |
| **Turn 3 Tokens (In / Out)** | `33,487 in` / `1,608 out` | *— (completed on Turn 1)* | Avoided `40,842`-char Turn 2 suite log |
| **Cumulative Input Tokens (`promptTokenCount`)** | `52,146 tokens` | **`2,240 tokens`** | **-95.7% (`23.28x` efficiency)** |
| **Cumulative Output Tokens (`candidatesTokenCount`)** | `2,877 tokens` | **`1,624 tokens`** | **-43.6% output reduction** |
| **Elapsed Execution Time (`elapsed_seconds`)** | `28.97 sec` | **`13.76 sec`** | **-52.5% (`2.11x` faster)** |
| **Estimated API Cost (`estimated_micro_usd`)** | `$0.0499` (`49,900 micro-USD`) | **`$0.0078` (`7,770 micro-USD`)** | **-84.4% (`6.42x` cheaper)** |

---

## 2. Why Full-Suite Noise Caused 4 Missed Boundary Cases (`16/20`)

Inspecting Mode A's final `validator.py` (`full-suite-bundle.json`) reveals how burying 3 target contracts inside `82,343` characters of unrelated monorepo test logs degraded code quality:

```python
# ❌ Mode A (FullSuiteTestExecutionRunner — Turn 3 Final validator.py, 16/20)
        if cnt <= 0 or cents <= 0 or not (0 <= bps < 1000):  # <-- Fails validator_05 & e2e_03 ($0 auth rejected)
                                                             # <-- Fails validator_04 (1000 bps rejected by `< 1000`)
            raise ValueError("numeric bounds violated")
        new_item = dict(item)                                # <-- Fails validator_01 (leaks unknown "extra" keys)
        new_item["merchant_id"] = mid.strip().upper()
        new_item["currency"] = cur.strip().upper()
        out.append(new_item)
```

---

## 3. Drop-In Implementation Recipe: Reverse AST Import Graph + Transitive Test Selector

With zero third-party dependencies, you can build an exact reverse-import DAG in Python's standard `ast` module and select only the test files transitively impacted by `git diff --name-only`:

```python
import ast
from collections import defaultdict
import os

def build_reverse_import_graph(src_files: list[str]) -> dict[str, set[str]]:
    """Builds a reverse dependency map `imported_module -> set(dependent_modules)` via Python AST."""
    rev_graph: dict[str, set[str]] = defaultdict(set)
    for path in src_files:
        mod_name = os.path.splitext(os.path.basename(path))[0]
        with open(path, "r", encoding="utf-8") as f:
            tree = ast.parse(f.read(), filename=path)
        for node in ast.walk(tree):
            if isinstance(node, ast.Import):
                for alias in node.names:
                    rev_graph[alias.name.split(".")[0]].add(mod_name)
            elif isinstance(node, ast.ImportFrom) and node.module:
                rev_graph[node.module.split(".")[0]].add(mod_name)
    return rev_graph

def select_affected_modules(changed_modules: list[str], rev_graph: dict[str, set[str]]) -> set[str]:
    """Finds all modules and test suites transitively affected by `changed_modules` via BFS."""
    affected = set(changed_modules)
    queue = list(changed_modules)
    while queue:
        cur = queue.pop(0)
        for dep in rev_graph.get(cur, ()):
            if dep not in affected:
                affected.add(dep)
                queue.append(dep)
    return affected
```

---

## 4. Practical Takeaway: Two-Tier Test Execution (Scoped Inner Loop, Full Outer Gate)

1. **Scope inner-loop test runs strictly to the transitive closure of modified files:** During iterative editing turns, run only `select_affected_modules(git_diff_files, rev_graph)` so feedback arrives in milliseconds with 100% relevant assertions.
2. **Run the full repository test suite only at the final `SUBMIT` gate (filtered via `E16` + `E29`):** This gives you fast, high-SNR inner-loop iteration without ever sacrificing full-repo regression safety at merge time.

---

### 🔗 Continue Reading the Series
* 🚀 **Master Benchmark Hub (All 41 Patterns Ranked):** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)
* 🛡️ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)
* 🎯 **Previous Sequential Post (`E29`):** [Deduplicating Cascading Test Failures for Coding Agents: Raw Error Dumps vs. AST Root-Cause Error Clustering (-95.3% Input Tokens)](https://tappers.hashnode.dev/e29-root-cause-error-clustering)
* 🗺️ **Related AST Graph Deep Dive (`E09`):** [Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster)](https://tappers.hashnode.dev/e09-dependency-map-vs-raw-grep)
* 🥇 **Rank #1 Deep Dive (`E20`):** [Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item `MEMORY.md`](https://tappers.hashnode.dev/e20-episodic-gotchas-memory)