Vibe Coding vs. Spec-First SDD: How Locking an Executable Spec Eliminates Rework and Cuts Input Tokens by 32.9%

💡 AI Labo Benchmark Series — Experiment E08 (Spec-Driven Development)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

🚀 View All 41 Benchmark Results & Master Ranking

🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

Experiment ID: E08 (Backlog Candidate C004)
Run ID: e08-20261028-vm212-01
Sandbox Environment: Proxmox Isolated VM VM212 (ai-e08-212, 0 NICs, virtio-serial QGA relay, systemd UID separation)
Model: gemini-2.5-flash (temperature: 0.0, thinkingBudget: 0)
Pattern Under Test: Spec-Driven Development (SDD) — Iterative Vibe Coding vs. Read-Only Executable Spec Lock (spec_plan.py) + Single-Pass Implementation

1. Executive Summary (TL;DR)

Vibe Coding—prompting an LLM with a high-level feature request, eyeballing the generated code, and sending follow-up prompts to patch edge cases—feels fast on Turn 1. By Turn 3, however, iterative prompt rework inflates input tokens and routinely introduces regressions or missed edge cases.

Spec-First Software Driven Development (SDD) enforces a strict two-step harness gate: before a single line of production code can be edited, the agent must formalize all boundary conditions into an executable contract (spec_plan.py), which the harness locks read-only (0444 + SHA-256 verification) before unlocking production file edits.

We tested both approaches inside isolated Proxmox VM VM212 on a 3-module cloud usage & SLA billing pipeline (parser.py, calculator.py, statement.py) containing 7 initial bugs:

Mode A (VibeCoding): Jumps straight into coding from a high-level prompt on Turn 1, followed by two iterative rework prompts (Turn 2: SLA & conflict rules; Turn 3: bool rejection & credit caps).
Mode B (SpecFirstSDD):
1. role_1_spec_architect (Turn 1): Translates TASK.md into an executable contract spec_plan.py covering all 10 domain invariants. The harness verifies it catches the initial bugs (spec_caught_initial_gaps: true) and locks spec_plan.py read-only (0444).
2. role_2_plan_implementer (Turn 2): Implements all 3 production modules in a single pass against the locked contract (rework_turns = 0).

Key Result

Despite 3 turns and 6 file edits, Mode A (VibeCoding) still missed empty-list/non-list input rejection (19 / 20 on the hidden grading suite) and left spec_plan.py at 0 / 6 (0.0%) mutant kill rate. Mode B (SpecFirstSDD) achieved 20 / 20 (100%) accuracy and 6 / 6 (100%) mutant kill rate with zero rework turns, while cutting cumulative input tokens by -32.9% (9,375 ➔ 6,284), LLM calls by -33.3% (3 ➔ 2), API cost by -6.9%, and execution time by -6.0%.

Metric	Mode A: Iterative Vibe Coding (`VibeCoding`)	Mode B: Read-Only Spec-First SDD (`SpecFirstSDD`)	Delta / Improvement
Hidden Grading Suite Score (20 Cases)	`19 / 20` (95.0% — ❌ failed `parser_02`)	`20 / 20` (100% Perfect Pass)	+5.0 pt (100% compliance)
Spec Mutant Kill Rate (6 Mutants)	`0 / 6` (0.0%)	`6 / 6` (100.0%)	+100.0 pt contract rigor
Rework Turns (`rework_turns`)	`2 rework turns` (Turn 2 & Turn 3)	`0 rework turns` (1-shot pass)	-100% rework eliminated
Total File Edits (`file_edits`)	`6 edits`	`4 edits` (`spec_plan.py` + 3 modules)	-33.3% fewer edits
LLM Calls (`model_calls`)	`3 calls`	`2 calls`	-33.3% fewer calls
Cumulative Input Tokens (`promptTokenCount`)	`9,375 tokens`	`6,284 tokens`	-32.9% (`-3,091` tokens)
Cumulative Output Tokens (`candidatesTokenCount`)	`2,879 tokens`	`3,169 tokens`	+10.1% (executable spec investment)
Elapsed Execution Time (`elapsed_seconds`)	`28.12 sec`	`26.43 sec`	-6.0% faster
Estimated API Cost (`estimated_micro_usd`)	`$0.0178` (`17,829 micro-USD`)	`$0.0166` (`16,598 micro-USD`)	-6.9% cheaper

2. Experiment Architecture & Turn-by-Turn Token Breakdown

+-------------------------------------------------------------------------+
| Proxmox Isolated VM212 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Cloud SLA & Tiered Billing Pipeline (7 Bugs)     |
|               (parser.py ➔ calculator.py ➔ statement.py)                |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: VibeCoding (Code First, Fix Edge Cases via Follow-Ups)    |  |
|  |  - Turn 1: Initial 3-file vibe patch    (1,759 in / 1,448 out)    |  |
|  |  - Turn 2: Rework #1 (SLA < 99 & tiers) (3,382 in /   718 out)    |  |
|  |  - Turn 3: Rework #2 (bool & credit)    (4,234 in /   713 out)    |  |
|  |  --> 9,375 in | 6 file edits | 19/20 Pass (missed [] check!)      |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: SpecFirstSDD (Lock Executable Contract ➔ Implement Once)  |  |
|  |  - Turn 1 [Spec Architect]: Writes `spec_plan.py` (1,936 in)      |  |
|  |  - Harness Gate: Verifies red phase & locks `spec_plan.py` (0444) |  |
|  |  - Turn 2 [Plan Implementer]: Patches 3 files in 1 pass (4,348 in)|  |
|  |  --> 6,284 in | 0 rework     | 20/20 Pass | 6/6 Mutants Killed    |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Code-Level Postmortem: Why Follow-Up Vibe Prompts Leak Edge Cases

Inspecting vibe-bundle.json (Mode A) vs. sdd-bundle.json (Mode B) shows why reactive prompting struggles with completeness.

On Turn 3 of Mode A (VibeCoding), when prompted to "reject None, invalid inputs, and bool types", the model patched parser.py narrowly around None—forgetting to reject empty lists [] or non-list iterables like strings:

# ❌ Mode A (`VibeCoding`, Turn 3 final `parser.py`): Narrow patch missed `[]` and strings!
def parse_usage(raw_records):
    if raw_records is None:
        raise ValueError("invalid_records")
    out = []
    for item in raw_records:
        ...

Because Mode A never upgraded spec_plan.py beyond its initial smoke check, parse_usage([]) silently returned [] and Mode A declared victory—failing parser_02_empty_and_non_list_rejected in the hidden grader.

In Mode B (SpecFirstSDD), role_1_spec_architect wrote and locked explicit boundary assertions before any production file was touched:

# ✅ Mode B (`SpecFirstSDD`, Turn 1 locked `spec_plan.py` excerpt):
def verify_spec_plan():
    orig = [{"project_id": " p1 ", "service": " compute ", "units": 10, "unit_rate": 5, "SLA_pct": 99, "extra": 1}]
    res = parse_usage(orig)
    assert orig[0] == {"project_id": " p1 ", "service": " compute ", "units": 10, "unit_rate": 5, "SLA_pct": 99, "extra": 1}
    assert res == [{"project_id": "P1", "service": "COMPUTE", "units": 10, "unit_rate": 5, "SLA_pct": 99}]
    assert bad(parse_usage, []) and bad(parse_usage, None)
    assert bad(parse_usage, [{"project_id": "P", "service": "COMPUTE", "units": True, "unit_rate": 1, "SLA_pct": 99}])
    # Exact off-by-one gates: SLA 99 (no penalty) vs 98 (10% penalty), PRO 4999 (0%) vs 5000 (10%)
    assert calculate_charges([{"project_id": "P", "service": "GPU", "units": 5000, "unit_rate": 1, "SLA_pct": 100}], "PRO")["tier_discount"] == 500
    assert calculate_charges([{"project_id": "P", "service": "GPU", "units": 4999, "unit_rate": 1, "SLA_pct": 100}], "PRO")["tier_discount"] == 0

With spec_plan.py locked at chmod 0444 (verified in inode-manifest.json), role_2_plan_implementer had an unambiguous, machine-checked target and nailed all 20 hidden cases on Turn 2.

4. Mutation Audit (`M1`–`M6`) Summary (`labjudge08`)

Mutant ID	Injected Off-by-One / Type Defect	Mode A (`VibeCoding`)	Mode B (`SpecFirstSDD`)
`M1`	SLA penalty mutated from `< 99` to `<= 99` (penalizes 99% SLA)	❌ Survived	✅ Killed
`M2`	`PRO` tier threshold mutated from `>= 5000` to `> 5000`	❌ Survived	✅ Killed
`M3`	Boolean `units=True` accepted as integer `1`	❌ Survived	✅ Killed
`M4`	Promotional `unit_rate == 0` erroneously rejected	❌ Survived	✅ Killed
`M5`	Conflicting `unit_rate` / `SLA_pct` for same key ignored	❌ Survived	✅ Killed
`M6`	Zero discount formatted as `0` instead of `-0`	❌ Survived	✅ Killed
Total	6 Contract Mutants	`0 / 6` (0.0%)	`6 / 6` (100.0%)

5. Practical Takeaway: Lock the Spec Read-Only Before Unlocking Edits

Even on a fast, lightweight model (gemini-2.5-flash), 2 turns of Spec-First SDD beat 3 turns of Vibe Coding on accuracy (100% vs. 95%), mutation kill rate (100% vs. 0%), speed, and token cost (-32.9% input tokens).

In your own agent workflows:

Require a Phase 1 Read-Only / Spec-Lock step that turns vague requirements into executable assertions (types, empty inputs, exact < vs. <= boundaries).
Lock that contract file (chmod 0444 or pre-commit hash check, see E38) so the implementation pass cannot weaken the spec to make tests green.

🔗 Continue Reading the Series

🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
⚙️ Previous (E07): Eliminating Goal Drift in Multi-File Coding Agents: In-Chat Loops vs. Code-Mode Dynamic Workflows
🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md

Vibe Coding vs. Spec-First SDD: How Locking an Executable Spec Eliminates Rework and Cuts Input Tokens by 32.9%

Metadata & Experiment Identity

1. Executive Summary (TL;DR)

Key Result

2. Experiment Architecture & Turn-by-Turn Token Breakdown

3. Code-Level Postmortem: Why Follow-Up Vibe Prompts Leak Edge Cases

4. Mutation Audit (`M1`–`M6`) Summary (`labjudge08`)

5. Practical Takeaway: Lock the Spec Read-Only Before Unlocking Edits

🔗 Continue Reading the Series

Comments

More from this blog

Why LLMs Fail at Unified Diffs: Standard Diff Hunks vs. Line-Number-Free SEARCH/REPLACE Blocks (2.41x Faster)

Escaping Sunk-Cost Retry Loops in Coding Agents: Single-Session Retries vs. a Parallel Best-of-N Tournament (-67.2% Input Tokens)

Why Long System Prompts Can't Enforce Code Architecture: Prompt Rules vs. Deterministic Pre-Commit AST Hooks

Why Self-Review Fails in AI Coding Agents: Single-Agent Self-Review vs. an Adversarial Critic (0% vs. 100% Mutant Kill Rate)

Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster)

Metadata & Experiment Identity

1. Executive Summary (TL;DR)

Key Result

2. Experiment Architecture & Turn-by-Turn Token Breakdown

3. Code-Level Postmortem: Why Follow-Up Vibe Prompts Leak Edge Cases

4. Mutation Audit (M1–M6) Summary (labjudge08)

5. Practical Takeaway: Lock the Spec Read-Only Before Unlocking Edits

🔗 Continue Reading the Series

Comments

More from this blog

Why LLMs Fail at Unified Diffs: Standard Diff Hunks vs. Line-Number-Free SEARCH/REPLACE Blocks (2.41x Faster)

Escaping Sunk-Cost Retry Loops in Coding Agents: Single-Session Retries vs. a Parallel Best-of-N Tournament (-67.2% Input Tokens)

Why Long System Prompts Can't Enforce Code Architecture: Prompt Rules vs. Deterministic Pre-Commit AST Hooks

Why Self-Review Fails in AI Coding Agents: Single-Agent Self-Review vs. an Adversarial Critic (0% vs. 100% Mutant Kill Rate)

Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster)

4. Mutation Audit (`M1`–`M6`) Summary (`labjudge08`)