Skip to main content

Stop Accepting 'Done' Without Proof: Unsupported Completion Claims vs. a Risk-Tiered Change Receipt Gate (-95.1% Tokens)

Updated 7 min readMarkdown

πŸ’‘ AI Labo Benchmark Series β€” Experiment E45 (Change Receipts & Risk-Tiered Escalation)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.


Metadata & Experiment Identity

  • Experiment ID: E45 (Backlog Candidate C041)
  • Run ID: e45-20261101-vm249-01
  • Sandbox Environment: Proxmox Isolated VM VM249 (ai-e45-249, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp45 / labjudge45)
  • Model: gemini-3.8-flash (thinkingLevel: LOW, temperature=0.0)
  • Pattern Under Test: "@LunarResearcher: Harness Engineering (Ch.9 Risk-Tiered Policy, Ch.13 Change Receipts, Ch.18 Unsupported Completion Claims)" β€” Unsupported Completion Claims (UnsupportedClaimBlindSubmitRunner) vs. Change Receipt & Risk-Tiered Escalation Gate (ChangeReceiptRiskEscalationRunner)

1. Executive Summary (TL;DR)

"An agent saying 'done' is not evidence." As highlighted in @LunarResearcher's 19-chapter treatise Harness Engineering: The Complete Guide to Building AI Agents That Don't Fall Apart, one of the most pervasive failure modes in autonomous coding agents is Unsupported Completion Claims: the agent declares a task finished without proving which local invariants were mechanically verified (VERIFIED) versus which external, irreversible side effects could not be verified locally and require human sign-off (NOT_VERIFIED / APPROVAL_NEEDED).

By combining a 4-Tier Risk Policy (Tier 1 Read-Only, Tier 2 Reversible Workspace Contracts, Tier 3 External Side Effects, Tier 4 Irreversible Production Wire Operations) with a mandatory Structured Change Receipt Gate, the harness rejects unverified "done" claims and forces 100% mechanical verification of Tier-2 contracts while cleanly escalating Tier-3/4 boundaries.

We tested this head-to-head inside isolated VM VM249 on the 3-module enterprise payment settlement pipeline (validator.py βž” settlement.py βž” statement.py):

  • Mode A (UnsupportedClaimBlindSubmitRunner): Allows unverified completion claims (claim_unverified_completion) and partial unreceipted fixes (stage_partial_unreceipted_fix), accumulating 93,588 chars of audit rejection logs across Turns 1–2 and failing validator_01_normalize_and_non_mutation (copied = dict(item) leaks "extra" keys) for a final score of 19 / 20 (95.0%).
  • Mode B (ChangeReceiptRiskEscalationRunner): Enforces the Risk-Tiered Policy & Change Receipt Gate (CHANGE_RECEIPT_RISK_ESCALATION_MD + submit_verified_change_receipt_bundle), achieving 20 / 20 (100%) in 1 turn.

Key Result

Mode A stalled at 19 / 20 (95.0%) after 3 turns (49,675 cumulative input tokens and 27.42s). Mode B achieved a 20 / 20 (100%) pass in 1 turn, cutting LLM calls by -66.7% (3 βž” 1), cumulative input tokens by -95.1% (49,675 βž” 2,423 tokensβ€”a 20.50x reduction), API cost by -83.7% ($0.0479 βž” $0.0078), and running 2.58x faster (27.42s βž” 10.64s, -61.2%).

Metric Mode A: Unsupported Claim (UnsupportedClaimBlindSubmit) Mode B: Change Receipt Gate (ChangeReceiptRiskEscalation) Delta / Improvement
Hidden Grading Suite Score (labjudge45) 19 / 20 (95.0% β€” 1 failure) 20 / 20 (100% β€” Perfect) +5.0% accuracy (19/20 βž” 20/20)
Unverified-Claim Traps (gotcha_traps_hit) 2 traps (Turns 1 & 2) 0 traps -100% (zero false claims)
LLM Calls (model_calls) 3 calls 1 call (1-shot) -66.7% (3.00x faster convergence)
Cumulative File Edits (file_edits) 5 edits 3 edits (1 per module) -40.0% fewer edits
Cumulative Tool Output (tool_output_chars) 93,588 chars 100 chars -99.9% (935.9x noise reduction)
Turn 1 Tokens (In / Out) 1,642 in / 11 out 2,423 in / 1,598 out +781 in for Change Receipt contract
Turn 2 Tokens (In / Out) 16,181 in / 1,248 out β€” (completed on Turn 1) Avoided 46K-char audit rejection dump
Turn 3 Tokens (In / Out) 31,852 in / 1,581 out β€” (completed on Turn 1) Avoided secondary rework bloat
Cumulative Input Tokens (promptTokenCount) 49,675 tokens 2,423 tokens -95.1% (20.50x efficiency)
Cumulative Output Tokens (candidatesTokenCount) 2,840 tokens 1,598 tokens -43.7% output reduction
Elapsed Execution Time (elapsed_seconds) 27.42 sec 10.64 sec -61.2% (2.58x faster)
Estimated API Cost (estimated_micro_usd) $0.0479 (47,907 micro-USD) $0.0078 (7,810 micro-USD) -83.7% (6.13x cheaper)

2. Experiment Architecture: Unsupported Completion Claims vs. Change Receipt Gate

+-------------------------------------------------------------------------+
| Proxmox Isolated VM249 (0 NICs, InaccessiblePaths, grader labjudge45)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py βž” settlement.py βž” statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: UnsupportedClaimBlindSubmitRunner (No Change Receipt)     |  |
|  |  - Turn 1 (1,642 in):  Calls `claim_unverified_completion`        |  |
|  |                        βž” 46,807 chars of audit rejection logs     |  |
|  |  - Turn 2 (16,181 in): Calls `stage_partial_unreceipted_fix`      |  |
|  |                        βž” partial 2-file edit (46,461 chars)       |  |
|  |  - Turn 3 (31,852 in): Submits `copied = dict(item)` (retains     |  |
|  |                        `"extra"` key βž” fails `validator_01`)      |  |
|  |  --> 3 calls | 49,675 in | $0.0479 | 27.42s | 19/20 (1 FAILED)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ChangeReceiptRiskEscalationRunner (Risk-Tiered Receipt)   |  |
|  |  - Enforces `CHANGE_RECEIPT_RISK_ESCALATION_MD` (+781 tokens):    |  |
|  |    * Tier 2 (`REVERSIBLE WORKSPACE CONTRACT`): Mechanically       |  |
|  |      verifies `validator.py`, `settlement.py`, `statement.py`     |  |
|  |    * Tier 3/4 (`CHANGE RECEIPT & ESCALATION GATE`): Separates     |  |
|  |      `VERIFIED` local checks from `NOT_VERIFIED` / `APPROVAL`     |  |
|  |  - Turn 1 (2,423 in): Calls `submit_verified_change_receipt_bundle`| |
|  |  --> 1 call  | 2,423 in  | $0.0078 | 10.64s | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Drop-In Implementation Recipe: RiskTieredChangeReceiptGate

Replace free-form "task complete" tool calls with a deterministic RiskTieredChangeReceiptGate that validates all Tier-2 checks and requires explicit escalation of Tier-3/4 external boundaries:

from dataclasses import dataclass
from typing import Callable


@dataclass
class ChangeReceipt:
    changed_files: list[str]
    verified_checks: list[str]
    not_verified_external: list[str]
    approval_needed_tier3_4: list[str]


class RiskTieredChangeReceiptGate:
    """Blocks unsupported completion claims and enforces a structured Change Receipt."""

    def __init__(self, tier2_verifiers: dict[str, Callable[[dict[str, str]], bool]]):
        self.tier2_verifiers = tier2_verifiers

    def submit_with_receipt(
        self,
        bundle: dict[str, str],
        not_verified_external: list[str],
        approval_needed_tier3_4: list[str],
    ) -> ChangeReceipt:
        failed = [name for name, verify_fn in self.tier2_verifiers.items() if not verify_fn(bundle)]
        if failed:
            raise ValueError(f"REJECTED_UNSUPPORTED_COMPLETION_CLAIM: failed Tier-2 checks={failed}")
        if not not_verified_external or not approval_needed_tier3_4:
            raise ValueError("REJECTED_MISSING_ESCALATION_BOUNDARY: must declare NOT_VERIFIED and APPROVAL_NEEDED")
        return ChangeReceipt(
            changed_files=sorted(bundle.keys()),
            verified_checks=list(self.tier2_verifiers.keys()),
            not_verified_external=not_verified_external,
            approval_needed_tier3_4=approval_needed_tier3_4,
        )

4. Practical Takeaway: Make Every Agent Earn "Done" With a Structured Change Receipt

  1. Separate what the harness verified locally (VERIFIED) from what cannot be verified locally (NOT_VERIFIED): When an agent is forced to populate both fields in a Change Receipt, it stops hallucinating that external side effects were tested and stops skipping local edge-case checks.
  2. Pair Change Receipts (E45) with Pre-Submit Blast-Radius Audits (E32) and Shift-Handover Artifacts (E36): Attach the generated ChangeReceipt directly to your pull request or multi-session handover log.

πŸ”— Continue Reading the Series

No comments yet. Be the first to comment.

More from this blog