# Stop Letting Coding Agents Do Manual Multi-Spot String Replacements: Manual Text Edits vs. Parameterized AST Codemod Tools (-94.8% Input Tokens)

> 💡 **AI Labo Benchmark Series — Experiment `E31` (Edit Primitives & AST Tooling)**
>
> This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
> * 🚀 **[View All 41 Benchmark Results & Master Ranking](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)**
> * 🛡️ **[Read How Our Proxmox & systemd Anti-Cheating Sandbox Works](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)**

---

## Metadata & Experiment Identity
* **Experiment ID:** `E31` (Backlog Candidate `C027`)
* **Run ID:** `e31-20261029-vm235-01`
* **Sandbox Environment:** Proxmox Isolated VM `VM235` (`ai-e31-235`, 0 NICs, `PrivateNetwork=yes`, `NoNewPrivileges=yes`, `InaccessiblePaths`, UID split `labexp31` / `labjudge31`)
* **Model:** `gemini-3.8-flash` (`thinkingLevel: LOW`, `temperature=0.0`)
* **Pattern Under Test:** *"AST Codemod Primitives in Coding Agent Harnesses (LibCST / `ast.NodeTransformer` / ast-grep Tooling vs. Raw String Edits)"* — **Manual Multi-Spot Text Replacement (`ManualMultiSpotTextEditRunner`) vs. Parameterized AST Codemod Tool (`ParameterizedAstCodemodRunner`)**

---

## 1. Executive Summary (TL;DR)

When a cross-module refactoring task requires updating 10+ call sites, comparison operators (`>= 300` ➔ `> 300`, `> 15000` ➔ `>= 15000`), and dictionary construction patterns across multiple files, asking an LLM to execute manual text replacements (`replace_string` / `edit_file`) one spot at a time triggers **Partial Edit Omission**: the model updates the first few occurrences, forgets a downstream consumer or boundary check, and burns multiple turns scanning raw error logs to find what it missed.

Providing a deterministic **Parameterized AST Codemod Tool** (`ast.NodeTransformer` / `ast-grep` structural primitive) allows the agent to express structural syntax-tree transformations once and apply them across all target modules atomically.

We benchmarked both approaches inside isolated VM `VM235` on the 3-module enterprise payment settlement pipeline (`validator.py` ➔ `settlement.py` ➔ `statement.py`):

* **Mode A (`ManualMultiSpotTextEditRunner`):** Performs manual multi-spot text replacements across turns, suffering partial edit omissions (`statement.py` left unpatched on Turn 2) and ingesting `70,362` chars of drift diagnostics across 3 turns.
* **Mode B (`ParameterizedAstCodemodRunner`):** Uses a structural AST transformation specification (`PARAMETERIZED_AST_CODEMOD_MD` + `apply_ast_codemod_bundle`) to transform all 3 modules deterministically in a single turn.

### Key Result
While both modes ultimately reached `20 / 20` on the hidden grading suite, Mode A required **3 turns, 5 file edits, 2 rework traps, and `41,680` cumulative input tokens**. **Mode B solved `20 / 20` (100%) in 1 turn on the first shot**, cutting **LLM calls by `-66.7%` (`3 ➔ 1`)**, **cumulative input tokens by `-94.8%` (`41,680 ➔ 2,164` tokens—a `19.26x` reduction)**, **API cost by `-82.0%` (`$0.0421 ➔ $0.0076`)**, and running **`2.22x` faster (`29.04s ➔ 13.07s`, `-55.0%`)**.

| Metric | Mode A: Manual Multi-Spot Edit (`ManualMultiSpotTextEdit`) | Mode B: Parameterized AST Codemod (`ParameterizedAstCodemod`) | Delta / Improvement |
| :--- | :---: | :---: | :---: |
| **Hidden Grading Suite Score (`labjudge31`)** | `20 / 20` (100% — Turn 3) | **`20 / 20` (100% — Turn 1 One-Shot)** | **1-shot convergence (`3 ➔ 1` turn)** |
| **Partial-Omission Traps (`gotcha_traps_hit`)** | `2 traps` (Turns 1 & 2) | **`0 traps`** | **-100% (zero omissions)** |
| **LLM Calls (`model_calls`)** | `3 calls` | **`1 call` (1-shot)** | **-66.7% (`3.00x` faster convergence)** |
| **Cumulative File Edits (`file_edits`)** | `5 edits` | **`3 edits` (1 per module)** | **-40.0% fewer edits** |
| **Cumulative Tool Output (`tool_output_chars`)** | `70,362 chars` | **`97 chars`** | **-99.9% (`725.4x` noise reduction)** |
| **Turn 1 Tokens (In / Out)** | `1,660 in` / `17 out` | **`2,164 in` / `1,595 out`** | `+504` in for AST codemod spec |
| **Turn 2 Tokens (In / Out)** | `13,525 in` / `1,283 out` | *— (completed on Turn 1)* | Avoided partial-edit drift logs |
| **Turn 3 Tokens (In / Out)** | `26,495 in` / `1,603 out` | *— (completed on Turn 1)* | Avoided secondary rework bloat |
| **Cumulative Input Tokens (`promptTokenCount`)** | `41,680 tokens` | **`2,164 tokens`** | **-94.8% (`19.26x` efficiency)** |
| **Cumulative Output Tokens (`candidatesTokenCount`)** | `2,903 tokens` | **`1,595 tokens`** | **-45.1% output reduction** |
| **Elapsed Execution Time (`elapsed_seconds`)** | `29.04 sec` | **`13.07 sec`** | **-55.0% (`2.22x` faster)** |
| **Estimated API Cost (`estimated_micro_usd`)** | `$0.0421` (`42,147 micro-USD`) | **`$0.0076` (`7,605 micro-USD`)** | **-82.0% (`5.54x` cheaper)** |

---

## 2. Experiment Architecture: Manual Multi-Spot Edits vs. AST Codemod Primitives

```text
+-------------------------------------------------------------------------+
| Proxmox Isolated VM235 (0 NICs, InaccessiblePaths, grader labjudge31)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py ➔ settlement.py ➔ statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: ManualMultiSpotTextEditRunner (Spot-by-Spot Text Edits)   |  |
|  |  - Turn 1 (1,660 in):  Calls `manual_spot_text_replace` ➔ misses  |  |
|  |                        structural AST nodes across files (35,391c)|  |
|  |  - Turn 2 (13,525 in): Calls `fix_spot_edit_drift` on only 2 files|  |
|  |                        (`validator.py`, `settlement.py`), leaving |  |
|  |                        `statement.py` unpatched (34,874c)         |  |
|  |  - Turn 3 (26,495 in): Final 3-file submission reaches 20/20      |  |
|  |  --> 3 calls | 41,680 in | $0.0421 | 29.04s | 20/20 (2 Traps Hit) |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ParameterizedAstCodemodRunner (AST Structural Transform)  |  |
|  |  - Injects `PARAMETERIZED_AST_CODEMOD_MD` (+504 tokens):          |  |
|  |    1. `ast_transform_validator`: Tuple("USD","EUR","JPY"), bool   |  |
|  |       guard, `0<=bps<=1000`, `cents>=0`, fresh 5-key Dict literal |  |
|  |    2. `ast_transform_settlement`: `sorted(order)` ➔ `order`,      |  |
|  |       `bps > 300`, `subtotal >= 15000`, `min(fee_rebate, ...)`    |  |
|  |    3. `ast_transform_statement`: `#` separator & `-0` format      |  |
|  |  - Turn 1 (2,164 in): Calls `apply_ast_codemod_bundle` once       |  |
|  |  --> 1 call  | 2,164 in  | $0.0076 | 13.07s | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
```

---

## 3. Drop-In Implementation Recipe: `SettlementBoundaryCodemod` (`ast.NodeTransformer`)

Using Python's built-in `ast` module, you can expose deterministic structural codemod primitives to your coding agent in under 35 lines:

```python
import ast


class SettlementBoundaryCodemod(ast.NodeTransformer):
    """Deterministically transforms `sorted(order)` and boundary operators (`>= 300` -> `> 300`, `> 15000` -> `>= 15000`)."""

    def visit_For(self, node: ast.For) -> ast.AST:
        self.generic_visit(node)
        if (
            isinstance(node.iter, ast.Call)
            and isinstance(node.iter.func, ast.Name)
            and node.iter.func.id == "sorted"
            and len(node.iter.args) == 1
        ):
            node.iter = node.iter.args[0]
        return node

    def visit_Compare(self, node: ast.Compare) -> ast.AST:
        self.generic_visit(node)
        if len(node.ops) == 1 and len(node.comparators) == 1:
            comp = node.comparators[0]
            if isinstance(comp, ast.Constant) and comp.value == 300 and isinstance(node.ops[0], ast.GtE):
                node.ops[0] = ast.Gt()
            elif isinstance(comp, ast.Constant) and comp.value == 15000 and isinstance(node.ops[0], ast.Gt):
                node.ops[0] = ast.GtE()
        return node


def apply_codemod(source_code: str) -> str:
    tree = ast.parse(source_code)
    transformed = SettlementBoundaryCodemod().visit(tree)
    ast.fix_missing_locations(transformed)
    return ast.unparse(transformed)
```

---

## 4. Practical Takeaway: Use Structural AST Tools for Multi-Site Refactoring

1. **Manual text replacement does not scale past 3–4 call sites:** LLMs are probabilistic token generators, not regex engines. Asking an agent to manually edit 12 occurrences across 3 files almost guarantees at least one missed site and an expensive retry turn.
2. **Pair AST Codemod Tools (`E31`) with `SEARCH/REPLACE` Blocks (`E15`):** Use `SEARCH/REPLACE` blocks for localized logic edits and expose `ast-grep` or `ast.NodeTransformer` primitives for repository-wide signature, operator, and call-site migrations.

---

### 🔗 Continue Reading the Series
* 🚀 **Master Benchmark Hub (All 41 Patterns Ranked):** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)
* 🛡️ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)
* ⏩ **Previous Sequential Post (`E30`):** [Stop Running Unrelated Monorepo Test Suites on Every Agent Edit: Full-Suite Runs vs. AST Dependency-Scoped Incremental Tests (-95.7% Input Tokens)](https://tappers.hashnode.dev/e30-dependency-scoped-incremental-test)
* 🔍 **Next Sequential Post (`E32`):** [Why Coding Agents Need a Pre-Submit Diff Audit Gate: Blind Unchecked Submits vs. Blast-Radius & Entropy Auditing (-95.0% Input Tokens)](https://tappers.hashnode.dev/e32-pre-submit-blast-radius-diff-audit)
* ✂️ **Related Edit Format Deep Dive (`E15`):** [Why LLMs Fail at Unified Diffs: Standard Diff Hunks vs. Line-Number-Free SEARCH/REPLACE Blocks (2.41x Faster)](https://tappers.hashnode.dev/e15-search-replace-blocks)