# Stop Accepting 'Done' Without Proof: Unsupported Completion Claims vs. a Risk-Tiered Change Receipt Gate (-95.1% Tokens)

> 💡 **AI Labo Benchmark Series — Experiment `E45` (Change Receipts & Risk-Tiered Escalation)**
>
> This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
> * 🚀 **[View All Benchmark Results & Master Ranking](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)**
> * 🛡️ **[Read How Our Proxmox & systemd Anti-Cheating Sandbox Works](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)**

---

## Metadata & Experiment Identity
* **Experiment ID:** `E45` (Backlog Candidate `C041`)
* **Run ID:** `e45-20261101-vm249-01`
* **Sandbox Environment:** Proxmox Isolated VM `VM249` (`ai-e45-249`, 0 NICs, `PrivateNetwork=yes`, `NoNewPrivileges=yes`, `InaccessiblePaths`, UID split `labexp45` / `labjudge45`)
* **Model:** `gemini-3.8-flash` (`thinkingLevel: LOW`, `temperature=0.0`)
* **Pattern Under Test:** *"`@LunarResearcher`: Harness Engineering (Ch.9 Risk-Tiered Policy, Ch.13 Change Receipts, Ch.18 Unsupported Completion Claims)"* — **Unsupported Completion Claims (`UnsupportedClaimBlindSubmitRunner`) vs. Change Receipt & Risk-Tiered Escalation Gate (`ChangeReceiptRiskEscalationRunner`)**

---

## 1. Executive Summary (TL;DR)

*"An agent saying 'done' is not evidence."* As highlighted in `@LunarResearcher`'s 19-chapter treatise *Harness Engineering: The Complete Guide to Building AI Agents That Don't Fall Apart*, one of the most pervasive failure modes in autonomous coding agents is **Unsupported Completion Claims**: the agent declares a task finished without proving which local invariants were mechanically verified (`VERIFIED`) versus which external, irreversible side effects could not be verified locally and require human sign-off (`NOT_VERIFIED` / `APPROVAL_NEEDED`).

By combining a **4-Tier Risk Policy** (Tier 1 Read-Only, Tier 2 Reversible Workspace Contracts, Tier 3 External Side Effects, Tier 4 Irreversible Production Wire Operations) with a mandatory **Structured Change Receipt Gate**, the harness rejects unverified "done" claims and forces 100% mechanical verification of Tier-2 contracts while cleanly escalating Tier-3/4 boundaries.

We tested this head-to-head inside isolated VM `VM249` on the 3-module enterprise payment settlement pipeline (`validator.py` ➔ `settlement.py` ➔ `statement.py`):

* **Mode A (`UnsupportedClaimBlindSubmitRunner`):** Allows unverified completion claims (`claim_unverified_completion`) and partial unreceipted fixes (`stage_partial_unreceipted_fix`), accumulating `93,588` chars of audit rejection logs across Turns 1–2 and failing `validator_01_normalize_and_non_mutation` (`copied = dict(item)` leaks `"extra"` keys) for a final score of **`19 / 20` (95.0%)**.
* **Mode B (`ChangeReceiptRiskEscalationRunner`):** Enforces the **Risk-Tiered Policy & Change Receipt Gate** (`CHANGE_RECEIPT_RISK_ESCALATION_MD` + `submit_verified_change_receipt_bundle`), achieving **`20 / 20` (100%) in 1 turn**.

### Key Result
Mode A stalled at **`19 / 20` (95.0%)** after 3 turns (`49,675` cumulative input tokens and `27.42s`). **Mode B achieved a `20 / 20` (100%) pass in 1 turn**, cutting **LLM calls by `-66.7%` (`3 ➔ 1`)**, **cumulative input tokens by `-95.1%` (`49,675 ➔ 2,423` tokens—a `20.50x` reduction)**, **API cost by `-83.7%` (`$0.0479 ➔ $0.0078`)**, and running **`2.58x` faster (`27.42s ➔ 10.64s`, `-61.2%`)**.

| Metric | Mode A: Unsupported Claim (`UnsupportedClaimBlindSubmit`) | Mode B: Change Receipt Gate (`ChangeReceiptRiskEscalation`) | Delta / Improvement |
| :--- | :---: | :---: | :---: |
| **Hidden Grading Suite Score (`labjudge45`)** | `19 / 20` (95.0% — 1 failure) | **`20 / 20` (100% — Perfect)** | **+5.0% accuracy (`19/20 ➔ 20/20`)** |
| **Unverified-Claim Traps (`gotcha_traps_hit`)** | `2 traps` (Turns 1 & 2) | **`0 traps`** | **-100% (zero false claims)** |
| **LLM Calls (`model_calls`)** | `3 calls` | **`1 call` (1-shot)** | **-66.7% (`3.00x` faster convergence)** |
| **Cumulative File Edits (`file_edits`)** | `5 edits` | **`3 edits` (1 per module)** | **-40.0% fewer edits** |
| **Cumulative Tool Output (`tool_output_chars`)** | `93,588 chars` | **`100 chars`** | **-99.9% (`935.9x` noise reduction)** |
| **Turn 1 Tokens (In / Out)** | `1,642 in` / `11 out` | **`2,423 in` / `1,598 out`** | `+781` in for Change Receipt contract |
| **Turn 2 Tokens (In / Out)** | `16,181 in` / `1,248 out` | *— (completed on Turn 1)* | Avoided 46K-char audit rejection dump |
| **Turn 3 Tokens (In / Out)** | `31,852 in` / `1,581 out` | *— (completed on Turn 1)* | Avoided secondary rework bloat |
| **Cumulative Input Tokens (`promptTokenCount`)** | `49,675 tokens` | **`2,423 tokens`** | **-95.1% (`20.50x` efficiency)** |
| **Cumulative Output Tokens (`candidatesTokenCount`)** | `2,840 tokens` | **`1,598 tokens`** | **-43.7% output reduction** |
| **Elapsed Execution Time (`elapsed_seconds`)** | `27.42 sec` | **`10.64 sec`** | **-61.2% (`2.58x` faster)** |
| **Estimated API Cost (`estimated_micro_usd`)** | `$0.0479` (`47,907 micro-USD`) | **`$0.0078` (`7,810 micro-USD`)** | **-83.7% (`6.13x` cheaper)** |

---

## 2. Experiment Architecture: Unsupported Completion Claims vs. Change Receipt Gate

```text
+-------------------------------------------------------------------------+
| Proxmox Isolated VM249 (0 NICs, InaccessiblePaths, grader labjudge45)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py ➔ settlement.py ➔ statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: UnsupportedClaimBlindSubmitRunner (No Change Receipt)     |  |
|  |  - Turn 1 (1,642 in):  Calls `claim_unverified_completion`        |  |
|  |                        ➔ 46,807 chars of audit rejection logs     |  |
|  |  - Turn 2 (16,181 in): Calls `stage_partial_unreceipted_fix`      |  |
|  |                        ➔ partial 2-file edit (46,461 chars)       |  |
|  |  - Turn 3 (31,852 in): Submits `copied = dict(item)` (retains     |  |
|  |                        `"extra"` key ➔ fails `validator_01`)      |  |
|  |  --> 3 calls | 49,675 in | $0.0479 | 27.42s | 19/20 (1 FAILED)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ChangeReceiptRiskEscalationRunner (Risk-Tiered Receipt)   |  |
|  |  - Enforces `CHANGE_RECEIPT_RISK_ESCALATION_MD` (+781 tokens):    |  |
|  |    * Tier 2 (`REVERSIBLE WORKSPACE CONTRACT`): Mechanically       |  |
|  |      verifies `validator.py`, `settlement.py`, `statement.py`     |  |
|  |    * Tier 3/4 (`CHANGE RECEIPT & ESCALATION GATE`): Separates     |  |
|  |      `VERIFIED` local checks from `NOT_VERIFIED` / `APPROVAL`     |  |
|  |  - Turn 1 (2,423 in): Calls `submit_verified_change_receipt_bundle`| |
|  |  --> 1 call  | 2,423 in  | $0.0078 | 10.64s | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
```

---

## 3. Drop-In Implementation Recipe: `RiskTieredChangeReceiptGate`

Replace free-form "task complete" tool calls with a deterministic `RiskTieredChangeReceiptGate` that validates all Tier-2 checks and requires explicit escalation of Tier-3/4 external boundaries:

```python
from dataclasses import dataclass
from typing import Callable


@dataclass
class ChangeReceipt:
    changed_files: list[str]
    verified_checks: list[str]
    not_verified_external: list[str]
    approval_needed_tier3_4: list[str]


class RiskTieredChangeReceiptGate:
    """Blocks unsupported completion claims and enforces a structured Change Receipt."""

    def __init__(self, tier2_verifiers: dict[str, Callable[[dict[str, str]], bool]]):
        self.tier2_verifiers = tier2_verifiers

    def submit_with_receipt(
        self,
        bundle: dict[str, str],
        not_verified_external: list[str],
        approval_needed_tier3_4: list[str],
    ) -> ChangeReceipt:
        failed = [name for name, verify_fn in self.tier2_verifiers.items() if not verify_fn(bundle)]
        if failed:
            raise ValueError(f"REJECTED_UNSUPPORTED_COMPLETION_CLAIM: failed Tier-2 checks={failed}")
        if not not_verified_external or not approval_needed_tier3_4:
            raise ValueError("REJECTED_MISSING_ESCALATION_BOUNDARY: must declare NOT_VERIFIED and APPROVAL_NEEDED")
        return ChangeReceipt(
            changed_files=sorted(bundle.keys()),
            verified_checks=list(self.tier2_verifiers.keys()),
            not_verified_external=not_verified_external,
            approval_needed_tier3_4=approval_needed_tier3_4,
        )
```

---

## 4. Practical Takeaway: Make Every Agent Earn "Done" With a Structured Change Receipt

1. **Separate what the harness verified locally (`VERIFIED`) from what cannot be verified locally (`NOT_VERIFIED`):** When an agent is forced to populate both fields in a Change Receipt, it stops hallucinating that external side effects were tested and stops skipping local edge-case checks.
2. **Pair Change Receipts (`E45`) with Pre-Submit Blast-Radius Audits (`E32`) and Shift-Handover Artifacts (`E36`):** Attach the generated `ChangeReceipt` directly to your pull request or multi-session handover log.

---

### 🔗 Continue Reading the Series
* 🚀 **Master Benchmark Hub:** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)
* 🛡️ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)
* 🧬 **Previous Sequential Post (`E44`):** [Which Harnesses Survive Model Scaling? Syntax Scaffolding vs. the 4 Irreplaceable Core Harnesses (-95.5% Input Tokens)](https://tappers.hashnode.dev/e44-model-evolution-harness-boundary)
* 🔍 **Related Pre-Submit Diff Audit Deep Dive (`E32`):** [Why Coding Agents Need a Pre-Submit Diff Audit Gate: Blind Unchecked Submits vs. Blast-Radius & Entropy Auditing (-95.0% Input Tokens)](https://tappers.hashnode.dev/e32-pre-submit-blast-radius-diff-audit)
* 🤝 **Related Shift-Handover Harness Deep Dive (`E36`):** [Preventing Multi-Session Amnesia in Coding Agents: Prose Summaries vs. a 3-Artifact Shift-Handover Harness (-95.2% Input Tokens)](https://tappers.hashnode.dev/e36-shift-handover-harness)
