Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster)

💡 AI Labo Benchmark Series — Experiment E09 (Guides & Context Hygiene)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

---

Metadata & Experiment Identity

• Experiment ID: E09 (Backlog Candidate C005)
• Run ID: e09-20261028-vm213-01
• Sandbox Environment: Proxmox Isolated VM VM213 (Debian 13 Trixie, 0 NICs, virtio-serial QGA relay, POSIX UID separation)
• Model: gemini-2.5-flash (temperature=0.0, thinkingBudget=0)
• Pattern Under Test: "Context Engine / AST Dependency Graph Pre-Injection vs. Monolithic Grep Waste" — Free-Form Grep Exploration vs. 0-Token Deterministic AST Symbol & Call-Graph Mini-Map

---

1. Executive Summary (TL;DR)

In any realistic production repository, core domain terms—like volume_discount, error_pct, rebate, base_cost, or tier names (STARTER, GROWTH, SCALE)—don't just live in the active execution pipeline. They also appear in legacy migration adapters, Prometheus metrics exporters, compliance archivers, and webhook formatters.

When you hand an AI coding agent grep and file-read tools and tell it to explore on its own (Mode A: RawGrepExploration), keyword searches light up every file that shares domain vocabulary. The agent ends up reading unreferenced auxiliary files, polluting its context window, and burning ~60% of its tokens before finishing the actual fix.

To measure how much exploration waste can be eliminated deterministically, we benchmarked two harness modes on an 8-module multi-tenant API quota & usage billing repository (3 active pipeline modules containing 7 coupled bugs + 5 unreferenced auxiliary/legacy decoy modules) inside isolated VM VM213:

• Mode A (RawGrepExploration): Standard repository-wide grep and iterative file inspection across matching modules.
• Mode B (ASTDependencyMap): Before calling the LLM, a millisecond Python ast.parse() pre-scan (0 tokens) extracts function signatures, constants, and the live call chain (ACTIVE_PIPELINE vs. AUXILIARY_UNREFERENCED) into an 11-line AST mini-map injected into Turn 1.

Key Result

Both modes achieved a perfect 20 / 20 (100%) pass on the hidden grading suite. However, pre-injecting the 11-line AST dependency map eliminated 100% of decoy file reads (5 ➔ 0 files), cut LLM calls by -66.7% (3 ➔ 1 single-shot pass), reduced input tokens by -59.1% (9,508 ➔ 3,893), cut output tokens by -64.5% (4,992 ➔ 1,774), lowered API cost by -63.0% ($0.0259 ➔ $0.0096), and ran 3.15x faster (42.30s ➔ 13.45s, -68.2%).

Metric	Mode A: Free-Form Grep (RawGrepExploration)	Mode B: 0-Token AST Map (ASTDependencyMap)	Delta / Improvement
Hidden Grading Suite Score (20 Cases)	20 / 20 (100%)	20 / 20 (100%)	Identical 100% accuracy
Total Files Inspected (files_inspected_count)	8 / 8 files	3 / 8 files	-62.5% (-5 files opened)
Unreferenced Decoy Files Read (decoy_files_inspected)	5 / 5 decoys	0 / 5 decoys	-100.0% (zero decoy waste)
LLM Calls (model_calls)	3 calls	1 call (1-shot)	-66.7% (-2 exploration turns)
Cumulative File Edits (file_edits)	8 edits	3 edits	-62.5% (-5 redundant edits)
Cumulative Input Tokens (promptTokenCount)	9,508 tokens	3,893 tokens	-59.1% (-5,615 tokens)
Cumulative Output Tokens (candidatesTokenCount)	4,992 tokens	1,774 tokens	-64.5% (-3,218 tokens)
Elapsed Execution Time (elapsed_seconds)	42.30 sec	13.45 sec	-68.2% (3.15x faster)
Estimated API Cost (estimated_micro_usd)	$0.0259 (25,852 micro-USD)	$0.0096 (9,573 micro-USD)	-63.0% cost reduction

---

2. Experiment Architecture & Repository Layout

+-------------------------------------------------------------------------+
| Proxmox Isolated VM213 (0 NICs, virtio-serial QGA, offline Debian 13)   |
|  Target Repo: 8-Module Multi-Tenant API Quota & Billing Codebase        |
|   - 3 Active Pipeline Files: ingest.py ➔ quota_engine.py ➔ receipt.py   |
|   - 5 Unreferenced Decoys:   legacy_v1_adapter, audit_exporter,         |
|                              webhook_notifier, metrics_collector,       |
|                              csv_archiver                               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: RawGrepExploration (Keyword Grep + Iterative Inspection)  |  |
|  |  - Turn 1: Repo grep hits domain terms ➔ reads 5 files (3 decoys) |  |
|  |  - Turn 2: Reads remaining 3 files (2 decoys + receipt.py)        |  |
|  |  - Turn 3: Final boundary & non-mutation patch pass               |  |
|  |  --> 3 calls | 8/8 files read (5 decoys) | 9,508 in | 42.30s      |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ASTDependencyMap (0-Token ast.parse() Call-Graph Map)     |  |
|  |  - Pre-Scan (0 tokens, <5ms): build_ast_dependency_map()          |  |
|  |    Marks 3 files ACTIVE_PIPELINE & 5 files AUXILIARY_UNREFERENCED |  |
|  |  - Turn 1: Opens 0 decoys ➔ patches all 7 bugs in 3 active files  |  |
|  |  --> 1 call  | 3/8 files read (0 decoys) | 3,893 in | 13.45s      |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

Why Keyword grep Gets Trapped: Active Modules vs. Vocabulary Decoys

Module	AST Classification	Key Symbols	Bug / Why grep Gets Baited
ingest.py	ACTIVE_PIPELINE	parse_events(raw_events) [L6-L37], ALLOWED_ENDPOINTS	Bugs 1–3: Mutates input dict & leaks extra keys; accepts bool via isinstance(x, int); rejects free-tier base_cost == 0
quota_engine.py	ACTIVE_PIPELINE	settle_quota(events, tier, prepay_credit) [L6-L67], ALLOWED_TIERS	Bugs 4–6: Penalizes error_pct >= 5 instead of > 5; checks subtotal > 10000 instead of >= 10000; ignores conflicting rates on duplicate keys
receipt.py	ACTIVE_PIPELINE	format_receipt(summary) [L4-L20]	Bug 7: Outputs 0 instead of mandatory signed zero -0 when discount/credit is zero
legacy_v1_adapter.py	AUXILIARY_UNREFERENCED	convert_legacy_v1_record, estimate_legacy_rebate	Contains rebate, error_pct, base_cost, STARTER_V1, GROWTH_V1
audit_exporter.py	AUXILIARY_UNREFERENCED	compute_audit_digest, build_compliance_envelope	Contains volume_discount, credit_applied, STARTER (offline SOX/SOC2 exporter)
webhook_notifier.py	AUXILIARY_UNREFERENCED	should_trigger_overage_webhook, format_webhook_payload	Contains STARTER, GROWTH, SCALE, volume_discount, error_pct_max
metrics_collector.py	AUXILIARY_UNREFERENCED	render_prometheus_metrics, aggregate_endpoint_error_rates	Contains error_pct, STARTER, subtotal
csv_archiver.py	AUXILIARY_UNREFERENCED	settlement_lines_to_csv, parse_archived_csv_line	Contains base_cost, error_pct, line_total

The Actual 11-Line 0-Token AST Mini-Map Injected in Mode B

Using only Python's standard ast library (build_ast_dependency_map()), the harness generated this compact structural map in milliseconds before making a single API call:

# AST Symbol & Pipeline Dependency Map (0-Token Deterministic Pre-Scan)
Active Pipeline Chain: ingest.parse_events(raw_events) -> quota_engine.settle_quota(events, tier, prepay_credit) -> receipt.format_receipt(summary)

- ingest.py [ACTIVE_PIPELINE]: funcs=[parse_events(raw_events) [L6-L37]] consts=[ALLOWED_ENDPOINTS]
- quota_engine.py [ACTIVE_PIPELINE]: funcs=[settle_quota(events, tier, prepay_credit) [L6-L67]] consts=[ALLOWED_TIERS]
- receipt.py [ACTIVE_PIPELINE]: funcs=[format_receipt(summary) [L4-L20]] consts=[]
- legacy_v1_adapter.py [AUXILIARY_UNREFERENCED]: funcs=[convert_legacy_v1_record(v1_row) [L23-L39]; estimate_legacy_rebate(gross_amount, error_pct, legacy_tier) [L42-L47]; summarize_legacy_migration_batch(rows) [L50-L58]] consts=[LEGACY_ENDPOINT_MAP, LEGACY_TIER_LIMITS]
- audit_exporter.py [AUXILIARY_UNREFERENCED]: funcs=[compute_audit_digest(tenant_id, endpoint, calls, line_total) [L12-L15]; build_compliance_envelope(settlement_dict, region, auditor_id) [L18-L46]; serialize_compliance_ndjson(envelopes) [L49-L51]] consts=[AUDIT_SCHEMA_VERSION, COMPLIANCE_REGIONS]
- webhook_notifier.py [AUXILIARY_UNREFERENCED]: funcs=[should_trigger_overage_webhook(tier, grand_total, error_pct_max) [L10-L17]; format_webhook_payload(tenant_id, tier, subtotal, volume_discount, grand_total) [L20-L35]] consts=[ALERT_THRESHOLDS]
- metrics_collector.py [AUXILIARY_UNREFERENCED]: funcs=[render_prometheus_metrics(settlement_list) [L4-L18]; aggregate_endpoint_error_rates(raw_events) [L21-L34]] consts=[]
- csv_archiver.py [AUXILIARY_UNREFERENCED]: funcs=[settlement_lines_to_csv(summary) [L14-L20]; parse_archived_csv_line(line_text) [L23-L35]] consts=[CSV_COLUMNS]

---

3. Turn-by-Turn Token & Cost Breakdown (usage.json)

Inspecting all 4 recorded model invocations (seq 0–2 for Mode A, seq 3 for Mode B) shows exactly how blind grep multiplies both input and output tokens:

Seq	Mode	Node ID	Input Tokens	Output Tokens	Cost (micro-USD)	Execution Behavior
0	Mode A (RawGrep)	raw_grep_turn_1_explore_and_patch	5,161	1,477	9,410 ($0.0094)	Runs repo-wide grep, loads 5 matching files (including 3 decoys: legacy_v1_adapter, audit_exporter, webhook_notifier), and makes partial edits to ingest.py & quota_engine.py.
1	Mode A (RawGrep)	raw_grep_turn_2_refine_receipt_and_bounds	2,630	1,775	8,629 ($0.0086)	Loads the remaining 3 files (including 2 decoys: metrics_collector, csv_archiver) plus receipt.py; patches -0 formatting and discount bounds.
2	Mode A (RawGrep)	raw_grep_turn_3_final_patch	1,717	1,740	7,813 ($0.0078)	Re-edits the 3 active files to finalize bool rejection, input non-mutation, and first-seen ordering (8 cumulative edits).
Mode A Total	—	3 turns (5 decoys read)	9,508	4,992	25,852 ($0.0259)	20 / 20 Pass in 42.30s
3	Mode B (ASTDepMap)	ast_dep_map_single_shot_fix	3,893	1,774	9,573 ($0.0096)	Sees the 11-line AST mini-map upfront, ignores all 5 unreferenced decoys, loads only the 3 active modules, and fixes all 7 bugs in a single shot.
Mode B Total	—	1 turn (0 decoys read)	3,893	1,774	9,573 ($0.0096)	20 / 20 Pass in 13.45s (3.15x faster)

Notice that Mode B's entire run ($0.00957) cost almost the exact same as Turn 1 alone of Mode A ($0.00941), because Mode B never wasted turns reading legacy adapters or re-patching partially understood pipelines.

---

4. Practical Takeaway: Use Static Analysis for Navigation, LLMs for Reasoning

1. Never pay frontier LLM token prices to reconstruct an import graph: Python's ast, TypeScript's compiler API, or tree-sitter can build an exact symbol and reachability map in < 10 ms for $0.00.
2. Label reachable vs. unreferenced modules explicitly: Tagging modules as [ACTIVE_PIPELINE] vs. [AUXILIARY_UNREFERENCED] (and pairing this map with Tree-sitter Interface Skeletons (E21) for large dependencies) prevents legacy and observability code from hijacking the agent's attention.

---

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 📐 Previous Post (E08): Vibe Coding vs. Spec-First SDD: How Locking an Executable Spec Eliminates Rework and Cuts Input Tokens by 32.9%
• ⚔️ Next Companion Post (E10): Why Self-Review Fails in AI Coding Agents: Single-Agent Self-Review vs. an Adversarial Critic (0% vs. 100% Mutant Kill Rate)
• 🥈 Rank #2 Deep Dive (E21): Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons