# Vibe Coding vs. Spec-First SDD: How Locking an Executable Spec Eliminates Rework and Cuts Input Tokens by 32.9%

> 💡 **AI Labo Benchmark Series — Experiment `E08` (Spec-Driven Development)**
>
> This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
> * 🚀 **[View All 41 Benchmark Results & Master Ranking](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)**
> * 🛡️ **[Read How Our Proxmox & systemd Anti-Cheating Sandbox Works](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)**

---

## Metadata & Experiment Identity
* **Experiment ID:** `E08` (Backlog Candidate `C004`)
* **Run ID:** `e08-20261028-vm212-01`
* **Sandbox Environment:** Proxmox Isolated VM `VM212` (`ai-e08-212`, 0 NICs, `virtio-serial` QGA relay, `systemd` UID separation)
* **Model:** `gemini-2.5-flash` (`temperature: 0.0`, `thinkingBudget: 0`)
* **Pattern Under Test:** Spec-Driven Development (SDD) — **Iterative Vibe Coding vs. Read-Only Executable Spec Lock (`spec_plan.py`) + Single-Pass Implementation**

---

## 1. Executive Summary (TL;DR)

**Vibe Coding**—prompting an LLM with a high-level feature request, eyeballing the generated code, and sending follow-up prompts to patch edge cases—feels fast on Turn 1. By Turn 3, however, iterative prompt rework inflates input tokens and routinely introduces regressions or missed edge cases.

**Spec-First Software Driven Development (SDD)** enforces a strict two-step harness gate: before a single line of production code can be edited, the agent must formalize all boundary conditions into an **executable contract (`spec_plan.py`)**, which the harness locks **read-only (`0444` + SHA-256 verification)** before unlocking production file edits.

We tested both approaches inside isolated Proxmox VM `VM212` on a 3-module cloud usage & SLA billing pipeline (`parser.py`, `calculator.py`, `statement.py`) containing 7 initial bugs:

* **Mode A (`VibeCoding`):** Jumps straight into coding from a high-level prompt on Turn 1, followed by two iterative rework prompts (Turn 2: SLA & conflict rules; Turn 3: `bool` rejection & credit caps).
* **Mode B (`SpecFirstSDD`):**
  1. **`role_1_spec_architect` (Turn 1):** Translates `TASK.md` into an executable contract `spec_plan.py` covering all 10 domain invariants. The harness verifies it catches the initial bugs (`spec_caught_initial_gaps: true`) and locks `spec_plan.py` read-only (`0444`).
  2. **`role_2_plan_implementer` (Turn 2):** Implements all 3 production modules in a single pass against the locked contract (`rework_turns = 0`).

### Key Result
Despite 3 turns and 6 file edits, **Mode A (`VibeCoding`)** still missed empty-list/non-list input rejection (`19 / 20` on the hidden grading suite) and left `spec_plan.py` at **`0 / 6` (0.0%) mutant kill rate**. **Mode B (`SpecFirstSDD`)** achieved **`20 / 20` (100%) accuracy** and **`6 / 6` (100%) mutant kill rate** with **zero rework turns**, while cutting **cumulative input tokens by `-32.9%` (`9,375 ➔ 6,284`)**, **LLM calls by `-33.3%` (`3 ➔ 2`)**, **API cost by `-6.9%`**, and **execution time by `-6.0%`**.

| Metric | Mode A: Iterative Vibe Coding (`VibeCoding`) | Mode B: Read-Only Spec-First SDD (`SpecFirstSDD`) | Delta / Improvement |
| :--- | :---: | :---: | :---: |
| **Hidden Grading Suite Score (20 Cases)** | `19 / 20` (95.0% — **❌ failed `parser_02`**) | **`20 / 20` (100% Perfect Pass)** | **+5.0 pt (100% compliance)** |
| **Spec Mutant Kill Rate (6 Mutants)** | **`0 / 6` (0.0%)** | **`6 / 6` (100.0%)** | **+100.0 pt contract rigor** |
| **Rework Turns (`rework_turns`)** | `2 rework turns` (Turn 2 & Turn 3) | **`0 rework turns` (1-shot pass)** | **-100% rework eliminated** |
| **Total File Edits (`file_edits`)** | `6 edits` | **`4 edits` (`spec_plan.py` + 3 modules)** | **-33.3% fewer edits** |
| **LLM Calls (`model_calls`)** | `3 calls` | **`2 calls`** | **-33.3% fewer calls** |
| **Cumulative Input Tokens (`promptTokenCount`)** | `9,375 tokens` | **`6,284 tokens`** | **-32.9% (`-3,091` tokens)** |
| **Cumulative Output Tokens (`candidatesTokenCount`)** | `2,879 tokens` | **`3,169 tokens`** | +10.1% (executable spec investment) |
| **Elapsed Execution Time (`elapsed_seconds`)** | `28.12 sec` | **`26.43 sec`** | **-6.0% faster** |
| **Estimated API Cost (`estimated_micro_usd`)** | `$0.0178` (`17,829 micro-USD`) | **`$0.0166` (`16,598 micro-USD`)** | **-6.9% cheaper** |

---

## 2. Experiment Architecture & Turn-by-Turn Token Breakdown

```text
+-------------------------------------------------------------------------+
| Proxmox Isolated VM212 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Cloud SLA & Tiered Billing Pipeline (7 Bugs)     |
|               (parser.py ➔ calculator.py ➔ statement.py)                |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: VibeCoding (Code First, Fix Edge Cases via Follow-Ups)    |  |
|  |  - Turn 1: Initial 3-file vibe patch    (1,759 in / 1,448 out)    |  |
|  |  - Turn 2: Rework #1 (SLA < 99 & tiers) (3,382 in /   718 out)    |  |
|  |  - Turn 3: Rework #2 (bool & credit)    (4,234 in /   713 out)    |  |
|  |  --> 9,375 in | 6 file edits | 19/20 Pass (missed [] check!)      |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: SpecFirstSDD (Lock Executable Contract ➔ Implement Once)  |  |
|  |  - Turn 1 [Spec Architect]: Writes `spec_plan.py` (1,936 in)      |  |
|  |  - Harness Gate: Verifies red phase & locks `spec_plan.py` (0444) |  |
|  |  - Turn 2 [Plan Implementer]: Patches 3 files in 1 pass (4,348 in)|  |
|  |  --> 6,284 in | 0 rework     | 20/20 Pass | 6/6 Mutants Killed    |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
```

---

## 3. Code-Level Postmortem: Why Follow-Up Vibe Prompts Leak Edge Cases

Inspecting `vibe-bundle.json` (Mode A) vs. `sdd-bundle.json` (Mode B) shows why reactive prompting struggles with completeness.

On Turn 3 of **Mode A (`VibeCoding`)**, when prompted to *"reject `None`, invalid inputs, and `bool` types"*, the model patched `parser.py` narrowly around `None`—forgetting to reject empty lists `[]` or non-list iterables like strings:

```python
# ❌ Mode A (`VibeCoding`, Turn 3 final `parser.py`): Narrow patch missed `[]` and strings!
def parse_usage(raw_records):
    if raw_records is None:
        raise ValueError("invalid_records")
    out = []
    for item in raw_records:
        ...
```

Because Mode A never upgraded `spec_plan.py` beyond its initial smoke check, `parse_usage([])` silently returned `[]` and Mode A declared victory—failing `parser_02_empty_and_non_list_rejected` in the hidden grader.

In **Mode B (`SpecFirstSDD`)**, `role_1_spec_architect` wrote and locked explicit boundary assertions *before* any production file was touched:

```python
# ✅ Mode B (`SpecFirstSDD`, Turn 1 locked `spec_plan.py` excerpt):
def verify_spec_plan():
    orig = [{"project_id": " p1 ", "service": " compute ", "units": 10, "unit_rate": 5, "SLA_pct": 99, "extra": 1}]
    res = parse_usage(orig)
    assert orig[0] == {"project_id": " p1 ", "service": " compute ", "units": 10, "unit_rate": 5, "SLA_pct": 99, "extra": 1}
    assert res == [{"project_id": "P1", "service": "COMPUTE", "units": 10, "unit_rate": 5, "SLA_pct": 99}]
    assert bad(parse_usage, []) and bad(parse_usage, None)
    assert bad(parse_usage, [{"project_id": "P", "service": "COMPUTE", "units": True, "unit_rate": 1, "SLA_pct": 99}])
    # Exact off-by-one gates: SLA 99 (no penalty) vs 98 (10% penalty), PRO 4999 (0%) vs 5000 (10%)
    assert calculate_charges([{"project_id": "P", "service": "GPU", "units": 5000, "unit_rate": 1, "SLA_pct": 100}], "PRO")["tier_discount"] == 500
    assert calculate_charges([{"project_id": "P", "service": "GPU", "units": 4999, "unit_rate": 1, "SLA_pct": 100}], "PRO")["tier_discount"] == 0
```

With `spec_plan.py` locked at `chmod 0444` (verified in `inode-manifest.json`), `role_2_plan_implementer` had an unambiguous, machine-checked target and nailed all 20 hidden cases on Turn 2.

---

## 4. Mutation Audit (`M1`–`M6`) Summary (`labjudge08`)

| Mutant ID | Injected Off-by-One / Type Defect | Mode A (`VibeCoding`) | Mode B (`SpecFirstSDD`) |
| :--- | :--- | :---: | :---: |
| **`M1`** | SLA penalty mutated from `< 99` to `<= 99` (penalizes 99% SLA) | ❌ Survived | ✅ **Killed** |
| **`M2`** | `PRO` tier threshold mutated from `>= 5000` to `> 5000` | ❌ Survived | ✅ **Killed** |
| **`M3`** | Boolean `units=True` accepted as integer `1` | ❌ Survived | ✅ **Killed** |
| **`M4`** | Promotional `unit_rate == 0` erroneously rejected | ❌ Survived | ✅ **Killed** |
| **`M5`** | Conflicting `unit_rate` / `SLA_pct` for same key ignored | ❌ Survived | ✅ **Killed** |
| **`M6`** | Zero discount formatted as `0` instead of `-0` | ❌ Survived | ✅ **Killed** |
| **Total** | **6 Contract Mutants** | **`0 / 6` (0.0%)** | **`6 / 6` (100.0%)** |

---

## 5. Practical Takeaway: Lock the Spec Read-Only Before Unlocking Edits

Even on a fast, lightweight model (`gemini-2.5-flash`), **2 turns of Spec-First SDD beat 3 turns of Vibe Coding on accuracy (`100%` vs. `95%`), mutation kill rate (`100%` vs. `0%`), speed, and token cost (`-32.9%` input tokens)**.

In your own agent workflows:
1. Require a **Phase 1 Read-Only / Spec-Lock step** that turns vague requirements into executable assertions (types, empty inputs, exact `<` vs. `<=` boundaries).
2. **Lock that contract file (`chmod 0444` or pre-commit hash check, see `E38`)** so the implementation pass cannot weaken the spec to make tests green.

---

### 🔗 Continue Reading the Series
* 🚀 **Master Benchmark Hub (All 41 Patterns Ranked):** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)
* 🛡️ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)
* ⚙️ **Previous (`E07`):** [Eliminating Goal Drift in Multi-File Coding Agents: In-Chat Loops vs. Code-Mode Dynamic Workflows](https://tappers.hashnode.dev/e07-code-mode-workflows)
* 🥇 **Rank #1 Deep Dive (`E20`):** [Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item `MEMORY.md`](https://tappers.hashnode.dev/e20-episodic-gotchas-memory)