Stop Running Unrelated Monorepo Test Suites on Every Agent Edit: Full-Suite Runs vs. AST Dependency-Scoped Incremental Tests (-95.7% Input Tokens)

💡 AI Labo Benchmark Series — Experiment E30 (Feedback Sensors & Incremental Verification)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

---

Metadata & Experiment Identity

• Experiment ID: E30 (Backlog Candidate C026)
• Run ID: e30-20261029-vm234-01
• Sandbox Environment: Proxmox Isolated VM VM234 (ai-e30-234, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp30 / labjudge30)
• Model: gemini-3.8-flash (thinkingLevel: LOW, temperature=0.0)
• Pattern Under Test: "OpenAI 1M-LOC Agent Harness Feedback Loop Optimization: AST Dependency-Scoped Incremental Test Execution" — Unfiltered Full-Repository Test Suite (FullSuiteTestExecutionRunner) vs. AST Dependency-Scoped Incremental Runner (DependencyScopedIncrementalRunner)

---

1. Executive Summary (TL;DR)

In a large monorepo or multi-package service, running pytest across the entire repository after every agent edit—including 148 completely unrelated packages (auth_ledger, fx_rates, webhook_dispatch, audit_archive)—creates a massive Signal-to-Noise Ratio (SNR) collapse:

1. Wall-Clock Lag: The agent waits for 150+ unaffected test files to import and run.
2. Attention Dilution: When 82,343 characters of unfiltered full-suite PASS/SKIP telemetry flood the context window, the model loses sight of subtle edge cases in the 3 modules that were actually modified.

We tested AST Dependency-Scoped Incremental Test Execution (the Python AST equivalent of pytest-testmon / bazel test affected-target scoping) inside isolated VM VM234 on the 3-module enterprise payment settlement pipeline (validator.py ➔ settlement.py ➔ statement.py):

• Mode A (FullSuiteTestExecutionRunner): Runs all 168 repository suites without dependency scoping on intermediate turns (82,343 chars of monorepo test output across Turns 1–2).
• Mode B (DependencyScopedIncrementalRunner): Uses a reverse AST import graph to skip 148 unaffected monorepo subtrees during inner-loop verification, scoping feedback strictly to the 20 affected tests (6 in validator.py, 9 in settlement.py, 5 E2E in statement.py) before running the full gate at final submit.

Key Result

Mode A dropped to 16 / 20 (80.0% — 4 failed cases) after 3 turns (52,146 cumulative input tokens)—missing extra key filtering (new_item = dict(item)), rejecting 1000 bps (0 <= bps < 1000), and rejecting $0 authorizations (cents <= 0). Mode B achieved a 20 / 20 (100%) pass in 1 turn, cutting LLM calls by -66.7% (3 ➔ 1), cumulative input tokens by -95.7% (52,146 ➔ 2,240 tokens—a 23.28x reduction), API cost by -84.4% ($0.0499 ➔ $0.0078), and running 2.11x faster (28.97s ➔ 13.76s, -52.5%).

Metric	Mode A: Unfiltered Full Suite (FullSuiteTestExecution)	Mode B: AST Dependency-Scoped (DependencyScopedIncremental)	Delta / Improvement
Hidden Grading Suite Score (labjudge30)	16 / 20 (80.0% — 4 failures)	20 / 20 (100% — Perfect)	+20.0% accuracy (+4 cases)
Full-Suite Noise Traps (gotcha_traps_hit)	2 traps (Turns 1 & 2)	0 traps	-100% (zero rework)
LLM Calls (model_calls)	3 calls	1 call (1-shot)	-66.7% (3 ➔ 1 turn)
Cumulative File Edits (file_edits)	5 edits	3 edits (1 per module)	-40.0% fewer edits
Cumulative Tool Output (tool_output_chars)	82,343 chars (168 suites)	104 chars (20 scoped tests)	-99.9% (791.8x noise reduction)
Turn 1 Tokens (In / Out)	1,657 in / 13 out	2,240 in / 1,624 out	+583 in for scoped contract matrix
Turn 2 Tokens (In / Out)	17,002 in / 1,256 out	— (completed on Turn 1)	Avoided 41,188-char Turn 1 suite log
Turn 3 Tokens (In / Out)	33,487 in / 1,608 out	— (completed on Turn 1)	Avoided 40,842-char Turn 2 suite log
Cumulative Input Tokens (promptTokenCount)	52,146 tokens	2,240 tokens	-95.7% (23.28x efficiency)
Cumulative Output Tokens (candidatesTokenCount)	2,877 tokens	1,624 tokens	-43.6% output reduction
Elapsed Execution Time (elapsed_seconds)	28.97 sec	13.76 sec	-52.5% (2.11x faster)
Estimated API Cost (estimated_micro_usd)	$0.0499 (49,900 micro-USD)	$0.0078 (7,770 micro-USD)	-84.4% (6.42x cheaper)

---

2. Why Full-Suite Noise Caused 4 Missed Boundary Cases (16/20)

Inspecting Mode A's final validator.py (full-suite-bundle.json) reveals how burying 3 target contracts inside 82,343 characters of unrelated monorepo test logs degraded code quality:

# ❌ Mode A (FullSuiteTestExecutionRunner — Turn 3 Final validator.py, 16/20)
        if cnt <= 0 or cents <= 0 or not (0 <= bps < 1000):  # <-- Fails validator_05 & e2e_03 ($0 auth rejected)
                                                             # <-- Fails validator_04 (1000 bps rejected by `< 1000`)
            raise ValueError("numeric bounds violated")
        new_item = dict(item)                                # <-- Fails validator_01 (leaks unknown "extra" keys)
        new_item["merchant_id"] = mid.strip().upper()
        new_item["currency"] = cur.strip().upper()
        out.append(new_item)

---

3. Drop-In Implementation Recipe: Reverse AST Import Graph + Transitive Test Selector

With zero third-party dependencies, you can build an exact reverse-import DAG in Python's standard ast module and select only the test files transitively impacted by git diff --name-only:

import ast
from collections import defaultdict
import os

def build_reverse_import_graph(src_files: list[str]) -> dict[str, set[str]]:
    """Builds a reverse dependency map `imported_module -> set(dependent_modules)` via Python AST."""
    rev_graph: dict[str, set[str]] = defaultdict(set)
    for path in src_files:
        mod_name = os.path.splitext(os.path.basename(path))[0]
        with open(path, "r", encoding="utf-8") as f:
            tree = ast.parse(f.read(), filename=path)
        for node in ast.walk(tree):
            if isinstance(node, ast.Import):
                for alias in node.names:
                    rev_graph[alias.name.split(".")[0]].add(mod_name)
            elif isinstance(node, ast.ImportFrom) and node.module:
                rev_graph[node.module.split(".")[0]].add(mod_name)
    return rev_graph

def select_affected_modules(changed_modules: list[str], rev_graph: dict[str, set[str]]) -> set[str]:
    """Finds all modules and test suites transitively affected by `changed_modules` via BFS."""
    affected = set(changed_modules)
    queue = list(changed_modules)
    while queue:
        cur = queue.pop(0)
        for dep in rev_graph.get(cur, ()):
            if dep not in affected:
                affected.add(dep)
                queue.append(dep)
    return affected

---

4. Practical Takeaway: Two-Tier Test Execution (Scoped Inner Loop, Full Outer Gate)

1. Scope inner-loop test runs strictly to the transitive closure of modified files: During iterative editing turns, run only select_affected_modules(git_diff_files, rev_graph) so feedback arrives in milliseconds with 100% relevant assertions.
2. Run the full repository test suite only at the final SUBMIT gate (filtered via E16 + E29): This gives you fast, high-SNR inner-loop iteration without ever sacrificing full-repo regression safety at merge time.

---

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🎯 Previous Sequential Post (E29): Deduplicating Cascading Test Failures for Coding Agents: Raw Error Dumps vs. AST Root-Cause Error Clustering (-95.3% Input Tokens)
• 🗺️ Related AST Graph Deep Dive (E09): Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster)
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md