π‘ AI Labo Benchmark Series β Experiment
E45(Change Receipts & Risk-Tiered Escalation)This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
Metadata & Experiment Identity
- Experiment ID:
E45(Backlog CandidateC041) - Run ID:
e45-20261101-vm249-01 - Sandbox Environment: Proxmox Isolated VM
VM249(ai-e45-249, 0 NICs,PrivateNetwork=yes,NoNewPrivileges=yes,InaccessiblePaths, UID splitlabexp45/labjudge45) - Model:
gemini-3.8-flash(thinkingLevel: LOW,temperature=0.0) - Pattern Under Test: "
@LunarResearcher: Harness Engineering (Ch.9 Risk-Tiered Policy, Ch.13 Change Receipts, Ch.18 Unsupported Completion Claims)" β Unsupported Completion Claims (UnsupportedClaimBlindSubmitRunner) vs. Change Receipt & Risk-Tiered Escalation Gate (ChangeReceiptRiskEscalationRunner)
1. Executive Summary (TL;DR)
"An agent saying 'done' is not evidence." As highlighted in @LunarResearcher's 19-chapter treatise Harness Engineering: The Complete Guide to Building AI Agents That Don't Fall Apart, one of the most pervasive failure modes in autonomous coding agents is Unsupported Completion Claims: the agent declares a task finished without proving which local invariants were mechanically verified (VERIFIED) versus which external, irreversible side effects could not be verified locally and require human sign-off (NOT_VERIFIED / APPROVAL_NEEDED).
By combining a 4-Tier Risk Policy (Tier 1 Read-Only, Tier 2 Reversible Workspace Contracts, Tier 3 External Side Effects, Tier 4 Irreversible Production Wire Operations) with a mandatory Structured Change Receipt Gate, the harness rejects unverified "done" claims and forces 100% mechanical verification of Tier-2 contracts while cleanly escalating Tier-3/4 boundaries.
We tested this head-to-head inside isolated VM VM249 on the 3-module enterprise payment settlement pipeline (validator.py β settlement.py β statement.py):
- Mode A (
UnsupportedClaimBlindSubmitRunner): Allows unverified completion claims (claim_unverified_completion) and partial unreceipted fixes (stage_partial_unreceipted_fix), accumulating93,588chars of audit rejection logs across Turns 1β2 and failingvalidator_01_normalize_and_non_mutation(copied = dict(item)leaks"extra"keys) for a final score of19 / 20(95.0%). - Mode B (
ChangeReceiptRiskEscalationRunner): Enforces the Risk-Tiered Policy & Change Receipt Gate (CHANGE_RECEIPT_RISK_ESCALATION_MD+submit_verified_change_receipt_bundle), achieving20 / 20(100%) in 1 turn.
Key Result
Mode A stalled at 19 / 20 (95.0%) after 3 turns (49,675 cumulative input tokens and 27.42s). Mode B achieved a 20 / 20 (100%) pass in 1 turn, cutting LLM calls by -66.7% (3 β 1), cumulative input tokens by -95.1% (49,675 β 2,423 tokensβa 20.50x reduction), API cost by -83.7% ($0.0479 β $0.0078), and running 2.58x faster (27.42s β 10.64s, -61.2%).
| Metric | Mode A: Unsupported Claim (UnsupportedClaimBlindSubmit) |
Mode B: Change Receipt Gate (ChangeReceiptRiskEscalation) |
Delta / Improvement |
|---|---|---|---|
Hidden Grading Suite Score (labjudge45) |
19 / 20 (95.0% β 1 failure) |
20 / 20 (100% β Perfect) |
+5.0% accuracy (19/20 β 20/20) |
Unverified-Claim Traps (gotcha_traps_hit) |
2 traps (Turns 1 & 2) |
0 traps |
-100% (zero false claims) |
LLM Calls (model_calls) |
3 calls |
1 call (1-shot) |
-66.7% (3.00x faster convergence) |
Cumulative File Edits (file_edits) |
5 edits |
3 edits (1 per module) |
-40.0% fewer edits |
Cumulative Tool Output (tool_output_chars) |
93,588 chars |
100 chars |
-99.9% (935.9x noise reduction) |
| Turn 1 Tokens (In / Out) | 1,642 in / 11 out |
2,423 in / 1,598 out |
+781 in for Change Receipt contract |
| Turn 2 Tokens (In / Out) | 16,181 in / 1,248 out |
β (completed on Turn 1) | Avoided 46K-char audit rejection dump |
| Turn 3 Tokens (In / Out) | 31,852 in / 1,581 out |
β (completed on Turn 1) | Avoided secondary rework bloat |
Cumulative Input Tokens (promptTokenCount) |
49,675 tokens |
2,423 tokens |
-95.1% (20.50x efficiency) |
Cumulative Output Tokens (candidatesTokenCount) |
2,840 tokens |
1,598 tokens |
-43.7% output reduction |
Elapsed Execution Time (elapsed_seconds) |
27.42 sec |
10.64 sec |
-61.2% (2.58x faster) |
Estimated API Cost (estimated_micro_usd) |
$0.0479 (47,907 micro-USD) |
$0.0078 (7,810 micro-USD) |
-83.7% (6.13x cheaper) |
2. Experiment Architecture: Unsupported Completion Claims vs. Change Receipt Gate
+-------------------------------------------------------------------------+
| Proxmox Isolated VM249 (0 NICs, InaccessiblePaths, grader labjudge45) |
| Target Task: 3-Module Enterprise Payment Settlement Pipeline |
| validator.py β settlement.py β statement.py |
| |
| +-------------------------------------------------------------------+ |
| | Mode A: UnsupportedClaimBlindSubmitRunner (No Change Receipt) | |
| | - Turn 1 (1,642 in): Calls `claim_unverified_completion` | |
| | β 46,807 chars of audit rejection logs | |
| | - Turn 2 (16,181 in): Calls `stage_partial_unreceipted_fix` | |
| | β partial 2-file edit (46,461 chars) | |
| | - Turn 3 (31,852 in): Submits `copied = dict(item)` (retains | |
| | `"extra"` key β fails `validator_01`) | |
| | --> 3 calls | 49,675 in | $0.0479 | 27.42s | 19/20 (1 FAILED) | |
| +-------------------------------------------------------------------+ |
| |
| +-------------------------------------------------------------------+ |
| | Mode B: ChangeReceiptRiskEscalationRunner (Risk-Tiered Receipt) | |
| | - Enforces `CHANGE_RECEIPT_RISK_ESCALATION_MD` (+781 tokens): | |
| | * Tier 2 (`REVERSIBLE WORKSPACE CONTRACT`): Mechanically | |
| | verifies `validator.py`, `settlement.py`, `statement.py` | |
| | * Tier 3/4 (`CHANGE RECEIPT & ESCALATION GATE`): Separates | |
| | `VERIFIED` local checks from `NOT_VERIFIED` / `APPROVAL` | |
| | - Turn 1 (2,423 in): Calls `submit_verified_change_receipt_bundle`| |
| | --> 1 call | 2,423 in | $0.0078 | 10.64s | 20/20 (100% PASS) | |
| +-------------------------------------------------------------------+ |
+-------------------------------------------------------------------------+
3. Drop-In Implementation Recipe: RiskTieredChangeReceiptGate
Replace free-form "task complete" tool calls with a deterministic RiskTieredChangeReceiptGate that validates all Tier-2 checks and requires explicit escalation of Tier-3/4 external boundaries:
from dataclasses import dataclass
from typing import Callable
@dataclass
class ChangeReceipt:
changed_files: list[str]
verified_checks: list[str]
not_verified_external: list[str]
approval_needed_tier3_4: list[str]
class RiskTieredChangeReceiptGate:
"""Blocks unsupported completion claims and enforces a structured Change Receipt."""
def __init__(self, tier2_verifiers: dict[str, Callable[[dict[str, str]], bool]]):
self.tier2_verifiers = tier2_verifiers
def submit_with_receipt(
self,
bundle: dict[str, str],
not_verified_external: list[str],
approval_needed_tier3_4: list[str],
) -> ChangeReceipt:
failed = [name for name, verify_fn in self.tier2_verifiers.items() if not verify_fn(bundle)]
if failed:
raise ValueError(f"REJECTED_UNSUPPORTED_COMPLETION_CLAIM: failed Tier-2 checks={failed}")
if not not_verified_external or not approval_needed_tier3_4:
raise ValueError("REJECTED_MISSING_ESCALATION_BOUNDARY: must declare NOT_VERIFIED and APPROVAL_NEEDED")
return ChangeReceipt(
changed_files=sorted(bundle.keys()),
verified_checks=list(self.tier2_verifiers.keys()),
not_verified_external=not_verified_external,
approval_needed_tier3_4=approval_needed_tier3_4,
)
4. Practical Takeaway: Make Every Agent Earn "Done" With a Structured Change Receipt
- Separate what the harness verified locally (
VERIFIED) from what cannot be verified locally (NOT_VERIFIED): When an agent is forced to populate both fields in a Change Receipt, it stops hallucinating that external side effects were tested and stops skipping local edge-case checks. - Pair Change Receipts (
E45) with Pre-Submit Blast-Radius Audits (E32) and Shift-Handover Artifacts (E36): Attach the generatedChangeReceiptdirectly to your pull request or multi-session handover log.
π Continue Reading the Series
- π Master Benchmark Hub: I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
- π‘οΈ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
- 𧬠Previous Sequential Post (
E44): Which Harnesses Survive Model Scaling? Syntax Scaffolding vs. the 4 Irreplaceable Core Harnesses (-95.5% Input Tokens) - π Related Pre-Submit Diff Audit Deep Dive (
E32): Why Coding Agents Need a Pre-Submit Diff Audit Gate: Blind Unchecked Submits vs. Blast-Radius & Entropy Auditing (-95.0% Input Tokens) - π€ Related Shift-Handover Harness Deep Dive (
E36): Preventing Multi-Session Amnesia in Coding Agents: Prose Summaries vs. a 3-Artifact Shift-Handover Harness (-95.2% Input Tokens)
Comments
Join the discussionNo comments yet. Be the first to comment.