Stop Letting Coding Agents Do Manual Multi-Spot String Replacements: Manual Text Edits vs. Parameterized AST Codemod Tools (-94.8% Input Tokens)

💡 AI Labo Benchmark Series — Experiment E31 (Edit Primitives & AST Tooling)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

---

Metadata & Experiment Identity

• Experiment ID: E31 (Backlog Candidate C027)
• Run ID: e31-20261029-vm235-01
• Sandbox Environment: Proxmox Isolated VM VM235 (ai-e31-235, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp31 / labjudge31)
• Model: gemini-3.8-flash (thinkingLevel: LOW, temperature=0.0)
• Pattern Under Test: "AST Codemod Primitives in Coding Agent Harnesses (LibCST / ast.NodeTransformer / ast-grep Tooling vs. Raw String Edits)" — Manual Multi-Spot Text Replacement (ManualMultiSpotTextEditRunner) vs. Parameterized AST Codemod Tool (ParameterizedAstCodemodRunner)

---

1. Executive Summary (TL;DR)

When a cross-module refactoring task requires updating 10+ call sites, comparison operators (>= 300 ➔ > 300, > 15000 ➔ >= 15000), and dictionary construction patterns across multiple files, asking an LLM to execute manual text replacements (replace_string / edit_file) one spot at a time triggers Partial Edit Omission: the model updates the first few occurrences, forgets a downstream consumer or boundary check, and burns multiple turns scanning raw error logs to find what it missed.

Providing a deterministic Parameterized AST Codemod Tool (ast.NodeTransformer / ast-grep structural primitive) allows the agent to express structural syntax-tree transformations once and apply them across all target modules atomically.

We benchmarked both approaches inside isolated VM VM235 on the 3-module enterprise payment settlement pipeline (validator.py ➔ settlement.py ➔ statement.py):

• Mode A (ManualMultiSpotTextEditRunner): Performs manual multi-spot text replacements across turns, suffering partial edit omissions (statement.py left unpatched on Turn 2) and ingesting 70,362 chars of drift diagnostics across 3 turns.
• Mode B (ParameterizedAstCodemodRunner): Uses a structural AST transformation specification (PARAMETERIZED_AST_CODEMOD_MD + apply_ast_codemod_bundle) to transform all 3 modules deterministically in a single turn.

Key Result

While both modes ultimately reached 20 / 20 on the hidden grading suite, Mode A required 3 turns, 5 file edits, 2 rework traps, and 41,680 cumulative input tokens. Mode B solved 20 / 20 (100%) in 1 turn on the first shot, cutting LLM calls by -66.7% (3 ➔ 1), cumulative input tokens by -94.8% (41,680 ➔ 2,164 tokens—a 19.26x reduction), API cost by -82.0% ($0.0421 ➔ $0.0076), and running 2.22x faster (29.04s ➔ 13.07s, -55.0%).

Metric	Mode A: Manual Multi-Spot Edit (ManualMultiSpotTextEdit)	Mode B: Parameterized AST Codemod (ParameterizedAstCodemod)	Delta / Improvement
Hidden Grading Suite Score (labjudge31)	20 / 20 (100% — Turn 3)	20 / 20 (100% — Turn 1 One-Shot)	1-shot convergence (3 ➔ 1 turn)
Partial-Omission Traps (gotcha_traps_hit)	2 traps (Turns 1 & 2)	0 traps	-100% (zero omissions)
LLM Calls (model_calls)	3 calls	1 call (1-shot)	-66.7% (3.00x faster convergence)
Cumulative File Edits (file_edits)	5 edits	3 edits (1 per module)	-40.0% fewer edits
Cumulative Tool Output (tool_output_chars)	70,362 chars	97 chars	-99.9% (725.4x noise reduction)
Turn 1 Tokens (In / Out)	1,660 in / 17 out	2,164 in / 1,595 out	+504 in for AST codemod spec
Turn 2 Tokens (In / Out)	13,525 in / 1,283 out	— (completed on Turn 1)	Avoided partial-edit drift logs
Turn 3 Tokens (In / Out)	26,495 in / 1,603 out	— (completed on Turn 1)	Avoided secondary rework bloat
Cumulative Input Tokens (promptTokenCount)	41,680 tokens	2,164 tokens	-94.8% (19.26x efficiency)
Cumulative Output Tokens (candidatesTokenCount)	2,903 tokens	1,595 tokens	-45.1% output reduction
Elapsed Execution Time (elapsed_seconds)	29.04 sec	13.07 sec	-55.0% (2.22x faster)
Estimated API Cost (estimated_micro_usd)	$0.0421 (42,147 micro-USD)	$0.0076 (7,605 micro-USD)	-82.0% (5.54x cheaper)

---

2. Experiment Architecture: Manual Multi-Spot Edits vs. AST Codemod Primitives

+-------------------------------------------------------------------------+
| Proxmox Isolated VM235 (0 NICs, InaccessiblePaths, grader labjudge31)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py ➔ settlement.py ➔ statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: ManualMultiSpotTextEditRunner (Spot-by-Spot Text Edits)   |  |
|  |  - Turn 1 (1,660 in):  Calls `manual_spot_text_replace` ➔ misses  |  |
|  |                        structural AST nodes across files (35,391c)|  |
|  |  - Turn 2 (13,525 in): Calls `fix_spot_edit_drift` on only 2 files|  |
|  |                        (`validator.py`, `settlement.py`), leaving |  |
|  |                        `statement.py` unpatched (34,874c)         |  |
|  |  - Turn 3 (26,495 in): Final 3-file submission reaches 20/20      |  |
|  |  --> 3 calls | 41,680 in | $0.0421 | 29.04s | 20/20 (2 Traps Hit) |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ParameterizedAstCodemodRunner (AST Structural Transform)  |  |
|  |  - Injects `PARAMETERIZED_AST_CODEMOD_MD` (+504 tokens):          |  |
|  |    1. `ast_transform_validator`: Tuple("USD","EUR","JPY"), bool   |  |
|  |       guard, `0<=bps<=1000`, `cents>=0`, fresh 5-key Dict literal |  |
|  |    2. `ast_transform_settlement`: `sorted(order)` ➔ `order`,      |  |
|  |       `bps > 300`, `subtotal >= 15000`, `min(fee_rebate, ...)`    |  |
|  |    3. `ast_transform_statement`: `#` separator & `-0` format      |  |
|  |  - Turn 1 (2,164 in): Calls `apply_ast_codemod_bundle` once       |  |
|  |  --> 1 call  | 2,164 in  | $0.0076 | 13.07s | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

---

3. Drop-In Implementation Recipe: SettlementBoundaryCodemod (ast.NodeTransformer)

Using Python's built-in ast module, you can expose deterministic structural codemod primitives to your coding agent in under 35 lines:

import ast


class SettlementBoundaryCodemod(ast.NodeTransformer):
    """Deterministically transforms `sorted(order)` and boundary operators (`>= 300` -> `> 300`, `> 15000` -> `>= 15000`)."""

    def visit_For(self, node: ast.For) -> ast.AST:
        self.generic_visit(node)
        if (
            isinstance(node.iter, ast.Call)
            and isinstance(node.iter.func, ast.Name)
            and node.iter.func.id == "sorted"
            and len(node.iter.args) == 1
        ):
            node.iter = node.iter.args[0]
        return node

    def visit_Compare(self, node: ast.Compare) -> ast.AST:
        self.generic_visit(node)
        if len(node.ops) == 1 and len(node.comparators) == 1:
            comp = node.comparators[0]
            if isinstance(comp, ast.Constant) and comp.value == 300 and isinstance(node.ops[0], ast.GtE):
                node.ops[0] = ast.Gt()
            elif isinstance(comp, ast.Constant) and comp.value == 15000 and isinstance(node.ops[0], ast.Gt):
                node.ops[0] = ast.GtE()
        return node


def apply_codemod(source_code: str) -> str:
    tree = ast.parse(source_code)
    transformed = SettlementBoundaryCodemod().visit(tree)
    ast.fix_missing_locations(transformed)
    return ast.unparse(transformed)

---

4. Practical Takeaway: Use Structural AST Tools for Multi-Site Refactoring

1. Manual text replacement does not scale past 3–4 call sites: LLMs are probabilistic token generators, not regex engines. Asking an agent to manually edit 12 occurrences across 3 files almost guarantees at least one missed site and an expensive retry turn.
2. Pair AST Codemod Tools (E31) with SEARCH/REPLACE Blocks (E15): Use SEARCH/REPLACE blocks for localized logic edits and expose ast-grep or ast.NodeTransformer primitives for repository-wide signature, operator, and call-site migrations.

---

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• ⏩ Previous Sequential Post (E30): Stop Running Unrelated Monorepo Test Suites on Every Agent Edit: Full-Suite Runs vs. AST Dependency-Scoped Incremental Tests (-95.7% Input Tokens)
• 🔍 Next Sequential Post (E32): Why Coding Agents Need a Pre-Submit Diff Audit Gate: Blind Unchecked Submits vs. Blast-Radius & Entropy Auditing (-95.0% Input Tokens)
• ✂️ Related Edit Format Deep Dive (E15): Why LLMs Fail at Unified Diffs: Standard Diff Hunks vs. Line-Number-Free SEARCH/REPLACE Blocks (2.41x Faster)