Eliminating Goal Drift in Multi-File Coding Agents: In-Chat Loops vs. Code-Mode Dynamic Workflows

💡 AI Labo Benchmark Series — Experiment E07 (Dynamic Code-Mode Orchestration)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

🚀 View All 41 Benchmark Results & Master Ranking

🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

Experiment ID: E07 (Backlog Candidate C003b)
Run ID: e07-20261028-vm211-01
Sandbox Environment: Proxmox Isolated VM VM211 (ai-e07-211, 0 NICs, virtio-serial QGA relay, systemd UID separation)
Model: gemini-3.8-flash (thinkingLevel: LOW)
Pattern Under Test: Anthropic Claude Code Dynamic Workflows ("Stop Babysitting Your Agents / Code-Mode Orchestration", Sid Bidasaria & Thariq Shihipar) — In-Chat Iterative Loop vs. Dynamic Python Workflow Generation

1. Executive Summary (TL;DR)

When an AI coding agent audits and refactors multiple files turn-by-turn inside a single growing chat thread (In-Chat Iterative Loop), two failure modes reliably appear in later turns:

Quadratic Prompt Bloat: Every file patch emitted on Turn 1, Turn 2, and Turn 3 stays in chat history and gets re-billed on every subsequent turn.
Goal Drift & Late-Turn Laziness: By Turn 4, accumulated code diffs crowd out fine-grained specification edge cases from Turn 1, causing the agent to skip boundary checks and write a superficial verification check just to finish.

Anthropic's Code-Mode Dynamic Workflows proposes flipping the control model: instead of iterating step-by-step in chat, have the agent first synthesize a deterministic Python audit & orchestration script (workflow.py) encoding the full contract, run it in Python at zero token cost, and pass the structured audit output to a fresh worker context to fix all modules in a single batch.

We tested both architectures head-to-head on a 3-module financial settlement & risk-scoring pipeline (validator.py, settlement.py, manifest.py, plus workflow.py) inside isolated Proxmox VM VM211:

Mode A (ChatLoop): Patches validator.py (Turn 1), settlement.py (Turn 2), manifest.py (Turn 3), and workflow.py (Turn 4) sequentially inside one persistent chat thread.
Mode B (CodeModeWorkflow):
1. role_1_workflow_architect (Turn 1): Writes a comprehensive 39-line programmatic audit script (workflow.py) directly from TASK.md, which the Python runtime executes immediately to catch initial bugs (audit_caught_initial_bugs: true).
2. role_2_code_mode_fixer (Turn 2, fresh context): Receives the locked workflow.py and fixes all 3 modules in one pass until workflow.py passes 100%.

Key Result

Mode A suffered late-turn Goal Drift, dropping two validation boundaries (18 / 20 on the hidden suite) and writing a lazy 1-case smoke check on Turn 4 (0 / 6 mutant kill rate). Mode B achieved 20 / 20 (100%) accuracy and a 6 / 6 (100%) mutant kill rate, while cutting LLM calls by -50.0% (4 ➔ 2), cumulative input tokens by -48.5% (12,509 ➔ 6,445), wall-clock time by -9.5%, and API cost by -2.1%.

Metric	Mode A: In-Chat Iterative Loop (`ChatLoop`)	Mode B: Code-Mode Dynamic Workflow (`CodeModeWorkflow`)	Delta / Improvement
Hidden Grading Suite Score (20 Cases)	`18 / 20` (90.0% — ❌ 2 boundary bugs)	`20 / 20` (100% Perfect Pass)	+10.0 pt (Zero Goal Drift)
Workflow Mutant Kill Rate (6 Mutants)	`0 / 6` (0.0%)	`6 / 6` (100.0%)	+100.0 pt audit coverage
Initial Bugs Caught Deterministically	No (untested until Turn 4)	Yes (`audit_caught_initial_bugs: true`)	0-token Python audit gate
LLM Calls (`model_calls`)	`4 calls`	`2 calls`	-50.0% fewer calls
Cumulative Input Tokens (`promptTokenCount`)	`12,509 tokens`	`6,445 tokens`	-48.5% prompt reduction
Cumulative Output Tokens (`candidatesTokenCount`)	`1,880 tokens`	`3,003 tokens`	+59.7% (invested in rich audit code)
Elapsed Execution Time (`elapsed_seconds`)	`26.42 sec`	`23.92 sec`	-9.5% faster
Estimated API Cost (`estimated_micro_usd`)	`$0.0164` (`16,433 micro-USD`)	`$0.0161` (`16,096 micro-USD`)	-2.1% cheaper with 100% quality

2. Architecture & Turn-by-Turn Token Progression

+-------------------------------------------------------------------------+
| Proxmox Isolated VM211 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Financial Risk & Settlement Pipeline             |
|               (validator.py ➔ settlement.py ➔ manifest.py)              |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: ChatLoop (Sequential File-by-File Chat Iteration)         |  |
|  |  - Turn 1: patch validator.py   (2,248 in / 425 out)              |  |
|  |  - Turn 2: patch settlement.py  (2,720 in / 814 out)              |  |
|  |  - Turn 3: patch manifest.py    (3,591 in / 303 out)              |  |
|  |  - Turn 4: write workflow.py    (3,950 in / 338 out — lazy check!)|  |
|  |  --> 12,509 in | 18/20 Hidden Cases | 0/6 Mutants Killed          |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: CodeModeWorkflow (Dynamic Script Generation + Fresh Fixer)|  |
|  |  - Turn 1 [Workflow Architect]: Generates 39-line `workflow.py`   |  |
|  |            encoding all boundary & anomaly invariants (2,067 in)  |  |
|  |  - Python Runtime [0 tokens]:   Runs `workflow.py` & locks file   |  |
|  |  - Turn 2 [Code-Mode Fixer]:    Fresh context fixes all 3 modules |  |
|  |            against locked `workflow.py` in 1 batch (4,378 in)     |  |
|  |  --> 6,445 in  | 20/20 Hidden Cases | 6/6 Mutants Killed          |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Code-Level Postmortem: Late-Turn Laziness vs. Programmatic Rigor

3.1 Mode A (`ChatLoop`): Goal Drift & Turn-4 Laziness

By Turn 4 (3,950 input tokens of accumulated patches), Mode A had missed two critical edge cases in its implementation:

validator_02_empty_and_non_list_rejected: Checked if not isinstance(raw_batches, list): but forgot to reject empty lists [] with ValueError.
settlement_08_invalid_tier_or_bool_reserve_rejected: Silently defaulted unknown tiers like "GOLD" to rebate = 0 and omitted negative/boolean validation on reserve_cents.

And when asked on Turn 4 to write workflow.py to verify the pipeline, the fatigued chat context produced a minimal happy-path script that tested zero negative or boundary conditions:

# ❌ Mode A (`ChatLoop`, Turn 4 `workflow.py`): Superficial happy-path check (0/6 mutants killed)
def run_workflow():
    raw = [
        {"tenant_id": "  t-1 ", "region": "us", "tx_count": 10, "amount_cents": 100000, "risk_score": 75},
        {"tenant_id": "T-1", "region": "US", "tx_count": 5, "amount_cents": 50000, "risk_score": 80},
        {"tenant_id": "T-2", "region": "EU", "tx_count": 2, "amount_cents": 20000, "risk_score": 10},
    ]
    batches = validator.validate_batches(raw)
    st = settlement.compute_settlement(batches, tier="ENTERPRISE", reserve_cents=500)
    txt = manifest.render_manifest(st)
    assert "SETTLEMENT MANIFEST (ENTERPRISE)" in txt
    assert "REBATE: -" in txt
    assert "RESERVE: -500" in txt
    return True

3.2 Mode B (`CodeModeWorkflow`): Full Contract Encoded in Python on Turn 1

In Mode B, role_1_workflow_architect started with a clean 2,067-token prompt dedicated solely to turning TASK.md into executable Python assertions:

# ✅ Mode B (`CodeModeWorkflow`, Turn 1 `workflow.py` excerpt — 6/6 mutants killed)
def bad(fn, *a, **k):
    try: fn(*a, **k); return False
    except ValueError: return True

def run_workflow():
    orig = [{"tenant_id": " t1 ", "region": " us ", "tx_count": 5, "amount_cents": 1000, "risk_score": 70, "extra": 99}]
    orig_copy = [dict(orig[0])]
    v = validate_batches(orig)
    assert orig == orig_copy and "extra" not in v[0]
    assert bad(validate_batches, []) and bad(validate_batches, "not_list")
    for k, val in [("tenant_id", 1), ("tenant_id", " "), ("tenant_id", True), ("region", "XX"),
                   ("tx_count", 0), ("tx_count", True), ("amount_cents", -1), ("amount_cents", False),
                   ("risk_score", -1), ("risk_score", 101), ("risk_score", True)]:
        b = dict(orig_copy[0]); b[k] = val
        assert bad(validate_batches, [b])
    assert bad(compute_settlement, v, "INVALID") and bad(compute_settlement, v, "STANDARD", -1) and bad(compute_settlement, v, "STANDARD", True)
    # Exact boundary checks: risk_score 69 vs 70, anomaly |r1-r2| 50 vs 51, ENTERPRISE 99999 vs 100000, REBATE: -0
    ...

4. Mutation Audit (`M1`–`M6`) Summary (`labjudge07`)

Mutant ID	Boundary Mutation Injected	Mode A (`ChatLoop`)	Mode B (`CodeModeWorkflow`)
`M1`	Risk surcharge `>= 70` mutated to strict `> 70`	❌ Survived	✅ Killed
`M2`	`ENTERPRISE` rebate `>= 100000` mutated to `> 100000`	❌ Survived	✅ Killed
`M3`	`isinstance(tx, bool)` check removed from `tx_count`	❌ Survived	✅ Killed
`M4`	Zero-amount batch (`amount_cents == 0`) rejected	❌ Survived	✅ Killed
`M5`	Risk anomaly check (`abs(r1 - r2) > 50`) omitted	❌ Survived	✅ Killed
`M6`	Zero rebate formatted as `REBATE: 0` instead of `REBATE: -0`	❌ Survived	✅ Killed
Total	6 Boundary Mutants	`0 / 6` (0.0%)	`6 / 6` (100.0%)

5. Practical Takeaway: Offload Multi-Module Audits From Chat to Code

Whenever an agent task touches 3+ files or repeated cross-file checks:

Have the agent write a Python audit/orchestration script on Turn 1 instead of stepping through files manually inside chat history.
Pass the script's structured output—not the prior chat log—to a fresh fixer context, cutting input tokens nearly in half while eliminating late-turn Goal Drift.

🔗 Continue Reading the Series

🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
🧪 Previous (E06): Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)
📐 Next (E08): Vibe Coding vs. Spec-First SDD: How Locking an Executable Spec Eliminates Rework and Cuts Input Tokens by 32.9%

Eliminating Goal Drift in Multi-File Coding Agents: In-Chat Loops vs. Code-Mode Dynamic Workflows

Metadata & Experiment Identity

1. Executive Summary (TL;DR)

Key Result

2. Architecture & Turn-by-Turn Token Progression

3. Code-Level Postmortem: Late-Turn Laziness vs. Programmatic Rigor

3.1 Mode A (`ChatLoop`): Goal Drift & Turn-4 Laziness

3.2 Mode B (`CodeModeWorkflow`): Full Contract Encoded in Python on Turn 1

4. Mutation Audit (`M1`–`M6`) Summary (`labjudge07`)

5. Practical Takeaway: Offload Multi-Module Audits From Chat to Code

🔗 Continue Reading the Series

Comments

More from this blog

Why LLMs Fail at Unified Diffs: Standard Diff Hunks vs. Line-Number-Free SEARCH/REPLACE Blocks (2.41x Faster)

Escaping Sunk-Cost Retry Loops in Coding Agents: Single-Session Retries vs. a Parallel Best-of-N Tournament (-67.2% Input Tokens)

Why Long System Prompts Can't Enforce Code Architecture: Prompt Rules vs. Deterministic Pre-Commit AST Hooks

Why Self-Review Fails in AI Coding Agents: Single-Agent Self-Review vs. an Adversarial Critic (0% vs. 100% Mutant Kill Rate)

Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster)

Metadata & Experiment Identity

1. Executive Summary (TL;DR)

Key Result

2. Architecture & Turn-by-Turn Token Progression

3. Code-Level Postmortem: Late-Turn Laziness vs. Programmatic Rigor

3.1 Mode A (ChatLoop): Goal Drift & Turn-4 Laziness

3.2 Mode B (CodeModeWorkflow): Full Contract Encoded in Python on Turn 1

4. Mutation Audit (M1–M6) Summary (labjudge07)

5. Practical Takeaway: Offload Multi-Module Audits From Chat to Code

🔗 Continue Reading the Series

Comments

More from this blog

Why LLMs Fail at Unified Diffs: Standard Diff Hunks vs. Line-Number-Free SEARCH/REPLACE Blocks (2.41x Faster)

Escaping Sunk-Cost Retry Loops in Coding Agents: Single-Session Retries vs. a Parallel Best-of-N Tournament (-67.2% Input Tokens)

Why Long System Prompts Can't Enforce Code Architecture: Prompt Rules vs. Deterministic Pre-Commit AST Hooks

Why Self-Review Fails in AI Coding Agents: Single-Agent Self-Review vs. an Adversarial Critic (0% vs. 100% Mutant Kill Rate)

Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster)

3.1 Mode A (`ChatLoop`): Goal Drift & Turn-4 Laziness

3.2 Mode B (`CodeModeWorkflow`): Full Contract Encoded in Python on Turn 1

4. Mutation Audit (`M1`–`M6`) Summary (`labjudge07`)