Why Long System Prompts Can't Enforce Code Architecture: Prompt Rules vs. Deterministic Pre-Commit AST Hooks

💡 AI Labo Benchmark Series — Experiment E11 (Deterministic Guardrails & Sensors)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite and AST policy audits.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

---

Metadata & Experiment Identity

• Experiment ID: E11
• Run ID: e11-vm215-20261028-01
• Sandbox Environment: Proxmox Isolated VM VM215 (0 NICs, virtio-serial QGA relay, isolated grader labjudge11)
• Model: gemini-3.8-flash
• Pattern Under Test: Verbose System-Prompt Style Manuals vs. Upfront AST Violation Diagnostics + Deterministic Pre-Commit AST Hooks

---

1. Executive Summary (TL;DR)

To keep AI coding agents from eroding codebase architecture—violating layer boundaries, introducing mutable global state, skipping return type annotations, or bloating files past modular size limits—teams routinely stuff dozens of rules into CLAUDE.md, .cursorrules, or system prompts (Prompt-Only Guidelines).

Yet in production, prompt-only architectural rules suffer from a fundamental flaw: Silent Soft-Constraint Drift. While an LLM focuses on making functional unit tests pass, it probabilistically violates structural constraints (especially line-count budgets and cross-layer invariants)—and because functional smoke tests still pass, the agent submits the violation without ever realizing it broke a rule.

We tested this head-to-head inside isolated VM VM215 on a 3-module cross-border freight & customs tariff pipeline (intake.py ➔ tariff.py ➔ waybill.py) whose initial buggy fixtures contained 6 functional bugs plus 14 AST architecture violations across 5 strict policy rules:

• Mode A (BloatedPromptRunner): Places a long, emphatic architectural manual (RULE 1–RULE 5) in the system prompt, while SUBMIT runs only functional smoke checks.
• Mode B (DeterministicHookRunner): Runs a 0-token Python AST linter upfront to inject the exact list of 14 existing AST violations into the initial prompt, and enforces a hard Pre-Commit AST Hook gate on COMMIT_AND_SUBMIT that mechanically rejects any commit violating RULE 1–RULE 5.

Key Result

Both modes passed 20 / 20 (100%) on functional tests in a single turn. However, Mode A failed overall compliance (4 / 5 architecture rules, False): it expanded tariff.py to 74 lines (silently breaching RULE_5's 65-line cap) and submitted anyway. Mode B achieved 20 / 20 functional accuracy AND 5 / 5 (100%) architectural compliance (0 violations) on Turn 1 (hook_rejections = 0), writing -18.1% more compact code (113 vs. 138 total lines) while reducing output tokens by -10.8%, latency by -5.6%, and API cost by -6.6%.

Metric	Mode A: Prompt-Only Manual (BloatedPrompt)	Mode B: Pre-Commit AST Hook (DeterministicHook)	Delta / Improvement
Hidden Functional Suite Score (20 Cases)	20 / 20 (100%)	20 / 20 (100%)	Both pass functional tests
AST Architecture Policy Compliance (5 Rules)	4 / 5 (80% — FAILED)	5 / 5 (100% — PASSED)	Eliminated silent architectural drift
Remaining AST Violations at Submit	1 violation (tariff.py: 74 > 65 lines)	0 violations	100% policy enforcement
Overall Pass (Functional 20/20 AND Arch 5/5)	❌ False	✅ True	Only Hook mode met full spec
Total Generated Code Size (3 Modules)	138 lines (46 + 74 + 18)	113 lines (35 + 60 + 18)	-18.1% cleaner, tighter code
LLM Calls (model_calls)	1 call	1 call	Single-shot completion
Input Tokens (promptTokenCount)	2,535 tokens	2,753 tokens	+218 tokens (14-item AST audit)
Output Tokens (candidatesTokenCount)	1,849 tokens	1,650 tokens	-10.8% (-199 output tokens)
Elapsed Execution Time (elapsed_seconds)	15.10 sec	14.26 sec	-5.6% faster
Estimated API Cost (estimated_micro_usd)	$0.0088 (8,835 micro-USD)	$0.0083 (8,253 micro-USD)	-6.6% cost reduction

---

2. Experiment Architecture: Prompt Manual vs. Pre-Commit AST Hook

+-------------------------------------------------------------------------+
| Proxmox Isolated VM215 (0 NICs, virtio-serial QGA, grader labjudge11)   |
|  Target Task: Cross-Border Freight & Tariff Pipeline                    |
|               (intake.py ➔ tariff.py ➔ waybill.py)                      |
|               Initial State: Functional Bugs + 14 AST Rule Violations   |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: BloatedPromptRunner (Verbose Prompt Rules, Smoke-Only)    |  |
|  |  - Prompt: Long prose manual explaining RULE 1 through RULE 5     |  |
|  |  - Turn 1: Writes intake (46L), tariff (74L), waybill (18L)       |  |
|  |  - Gate:   Runs functional smoke test only (PASS) ➔ SUBMIT        |  |
|  |  --> Functional: 20/20 | Architecture: 4/5 FAIL (tariff.py = 74L) |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: DeterministicHookRunner (AST Diagnostic + Hard Gate)      |  |
|  |  - Pre-Scan (0 tokens): audit_architecture_guardrails() injects   |  |
|  |    structured list of all 14 initial AST violations + 65L cap     |  |
|  |  - Turn 1: Writes intake (35L), tariff (60L), waybill (18L)       |  |
|  |  - Gate:   COMMIT_AND_SUBMIT runs Pre-Commit AST Hook (5/5 PASS)  |  |
|  |  --> Functional: 20/20 | Architecture: 5/5 PASS (0 violations)    |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

---

3. Audit of the 5 AST Architecture Policy Rules (labjudge11)

The isolated labjudge11 grader ran deterministic Python ast static analysis across the initial buggy fixtures, Mode A's submission, and Mode B's submission:

Policy Rule ID	Architectural Constraint (AST Check)	Initial Fixture	Mode A (BloatedPrompt)	Mode B (DeterministicHook)
RULE_1_ZERO_IMPORTS	Self-contained pure domain core: zero import or from ... import statements allowed	❌ 3 violations (math, decimal, os)	✅ 0 violations	✅ 0 violations
RULE_2_EXPLICIT_RETURN_TYPE	Explicit type contracts: every def function must declare an AST returns annotation (-> list, etc.)	❌ 3 violations (missing on all 3 functions)	✅ 0 violations	✅ 0 violations
RULE_3_NO_MUTABLE_GLOBAL_STATE	Immutable module globals: top-level list, dict, and set literals prohibited (must use tuple)	❌ 4 violations (ALLOWED_HS_CLASSES = [...], etc.)	✅ 0 violations	✅ 0 violations
RULE_4_DOMAIN_BOUNDARY	Strict layer separation: intake.py cannot reference contract tiers; waybill.py (view layer) cannot perform arithmetic (*, //)	❌ 1 violation (recalculation inside waybill.py)	✅ 0 violations	✅ 0 violations
RULE_5_DOCSTRING_AND_SIZE	Every module & function must have a docstring, AND every file must stay <= 65 lines	❌ 3 violations (missing docstrings)	❌ 1 violation (tariff.py hit 74 lines > 65)	✅ 0 violations (tariff.py = 60 lines)

---

4. Why LLMs Violate Prompt Rules—and Why Hooks Fix Both Sides

Why Mode A Silently Broke RULE_5

Inspecting prompt-bundle.json shows why prose instructions fail on structural budgets:

• Token-predicting models have poor real-time line-counting awareness while generating code token-by-token. Mode A followed local syntax rules (tuple globals, -> dict return types, docstrings), but spread function signatures and dictionary literals across extra lines in tariff.py, pushing it to 74 lines (+9 over the 65-line limit).
• Because SUBMIT only executed functional tests, the harness replied "Smoke check passed" and accepted the commit. Without a deterministic sensor, any rule that doesn't crash unit tests will eventually decay.

Why Mode B Succeeded in 1 Shot With Fewer Tokens

Mode B combined a Feed-Forward Guide with a Feedback Sensor (a preview of the pattern we explore further in E39):

1. Upfront 0-Token AST Diagnostic: Running audit_architecture_guardrails() before Turn 1 gave the model exact line numbers for all 14 existing violations and highlighted that COMMIT_AND_SUBMIT mechanically blocks any file exceeding MAX_RULE5_LINES = 65.
2. Tighter Code Generation: Knowing a mechanical AST gate stood at commit time, the model formatted intake.py in 35 lines (-11 vs. Mode A) and tariff.py in 60 lines (-14 vs. Mode A)—clearing the hook on the first call (hook_rejections = 0) and saving 10.8% in output tokens.

---

5. Practical Takeaway: Move Rules Out of CLAUDE.md and Into Pre-Commit Hooks

1. If a rule can be checked by an AST parser or linter in 5ms, delete the paragraph from your system prompt and make it a Pre-Commit Hook: Layer imports (intake vs. waybill), mutable globals, missing type annotations, and file-length caps belong in ruff, eslint, semgrep, or a 40-line ast script—not a 500-line prompt manual.
2. Pair commit hooks with upfront diagnostics and concrete fix hints: As we also demonstrate in E17 (Actionable Fix-Suggestion Linter) and E24 (Positive Few-Shot vs. Negative Rules), deterministic hooks work best when violations return an exact line:col plus a 1-line remediation recipe.

---

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• ⚔️ Previous Post (E10): Why Self-Review Fails in AI Coding Agents: Single-Agent Self-Review vs. an Adversarial Critic (0% vs. 100% Mutant Kill Rate)
• 🗜️ Rank #6 Companion Post (E12): How Save-Game Context Compaction Fixes Spec Drift in Multi-Step Coding Agents (-91.9% Input Tokens)
• 🏆 Next Sequential Post (E13): Escaping Sunk-Cost Retry Loops in Coding Agents: Single-Session Retries vs. a Parallel Best-of-N Tournament (-67.2% Input Tokens)