# Why Outcome-Only Evals Let Coding Agents Guess Blindly: Outcome Evals vs. a Behavioral Action-Sequence Gate (-95.2% Tokens) > πŸ’‘ **AI Labo Benchmark Series β€” Experiment `E40` (Behavioral Trajectory Gates)** > > This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite. > * πŸš€ **[View All 41 Benchmark Results & Master Ranking](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)** > * πŸ›‘οΈ **[Read How Our Proxmox & systemd Anti-Cheating Sandbox Works](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)** --- ## Metadata & Experiment Identity * **Experiment ID:** `E40` (Backlog Candidate `C036`) * **Run ID:** `e40-20261029-vm244-01` * **Sandbox Environment:** Proxmox Isolated VM `VM244` (`ai-e40-244`, 0 NICs, `PrivateNetwork=yes`, `NoNewPrivileges=yes`, `InaccessiblePaths`, UID split `labexp40` / `labjudge40`) * **Model:** `gemini-3.8-flash` (`thinkingLevel: LOW`, `temperature=0.0`) * **Pattern Under Test:** *"Google Developers Blog (Taylor Mullen & Christian Gunderman): The Anatomy of Harness Engineering β€” Behavioral Trajectory Evaluations vs. Outcome-Only Evals"* β€” **End-to-End Outcome-Only Eval (`OutcomeOnlyBlindGuessingRunner`) vs. Behavioral Action-Sequence Gate (`BehavioralActionSequenceGateRunner`)** --- ## 1. Executive Summary (TL;DR) When an agent harness evaluates only the **final end-to-end outcome** (whether tests pass at the end), it permits **Blind Guessing Edits** during intermediate turns: the agent modifies symbols without first checking their callers, references, or boundary contracts, triggering avoidable breakage cascades. As described in Google Developers Blog's *The Anatomy of Harness Engineering: Behavioral Evaluations*, enforcing a **Behavioral Action-Sequence Gate** on the agent's intermediate tool trajectory (`contract_inspected` βž” `boundaries_verified` βž” `edit_production_files`) blocks unverified blind-guessing edits before they happen. We tested this head-to-head inside isolated VM `VM244` on the 3-module enterprise payment settlement pipeline (`validator.py` βž” `settlement.py` βž” `statement.py`): * **Mode A (`OutcomeOnlyBlindGuessingRunner`):** Evaluates only end-to-end outcomes without gating intermediate actions, allowing `blind_edit_without_search` on Turn 1 and `stage_unsearched_fix` on Turn 2 (`73,202` chars of blind-guess breakage logs across 3 turns). * **Mode B (`BehavioralActionSequenceGateRunner`):** Enforces the **Reference-Check Before Edit** behavioral sequence (`BEHAVIORAL_ACTION_SEQUENCE_GATE_MD` + `verify_action_sequence_and_submit`), completing all 3 modules in 1 shot. ### Key Result While both modes reached `20 / 20` on the hidden grading suite, Mode A burned **3 turns, 5 file edits, 2 rework traps, and `46,436` cumulative input tokens**. **Mode B solved `20 / 20` (100%) in 1 turn on the first shot**, cutting **LLM calls by `-66.7%` (`3 βž” 1`)**, **cumulative input tokens by `-95.2%` (`46,436 βž” 2,218` tokensβ€”a `20.94x` reduction)**, **API cost by `-83.2%` (`$0.0456 βž” $0.0076`)**, and running **`2.01x` faster (`28.04s βž” 13.97s`, `-50.2%`)**. | Metric | Mode A: Outcome-Only Eval (`OutcomeOnlyBlindGuessing`) | Mode B: Behavioral Sequence Gate (`BehavioralActionSequenceGate`) | Delta / Improvement | | :--- | :---: | :---: | :---: | | **Hidden Grading Suite Score (`labjudge40`)** | `20 / 20` (100% β€” Turn 3) | **`20 / 20` (100% β€” Turn 1 One-Shot)** | **1-shot convergence (`3 βž” 1` turn)** | | **Blind-Guessing Traps (`gotcha_traps_hit`)** | `2 traps` (Turns 1 & 2) | **`0 traps`** | **-100% (zero blind edits)** | | **LLM Calls (`model_calls`)** | `3 calls` | **`1 call` (1-shot)** | **-66.7% (`3.00x` faster convergence)** | | **Cumulative File Edits (`file_edits`)** | `5 edits` | **`3 edits` (1 per module)** | **-40.0% fewer edits** | | **Cumulative Tool Output (`tool_output_chars`)** | `73,202 chars` | **`101 chars`** | **-99.9% (`724.8x` noise reduction)** | | **Turn 1 Tokens (In / Out)** | `1,660 in` / `12 out` | **`2,218 in` / `1,595 out`** | `+558` in for behavioral sequence gate | | **Turn 2 Tokens (In / Out)** | `15,097 in` / `1,260 out` | *β€” (completed on Turn 1)* | Avoided blind-guess breakage logs | | **Turn 3 Tokens (In / Out)** | `29,679 in` / `1,602 out` | *β€” (completed on Turn 1)* | Avoided secondary rework bloat | | **Cumulative Input Tokens (`promptTokenCount`)** | `46,436 tokens` | **`2,218 tokens`** | **-95.2% (`20.94x` efficiency)** | | **Cumulative Output Tokens (`candidatesTokenCount`)** | `2,874 tokens` | **`1,595 tokens`** | **-44.5% output reduction** | | **Elapsed Execution Time (`elapsed_seconds`)** | `28.04 sec` | **`13.97 sec`** | **-50.2% (`2.01x` faster)** | | **Estimated API Cost (`estimated_micro_usd`)** | `$0.0456` (`45,605 micro-USD`) | **`$0.0076` (`7,645 micro-USD`)** | **-83.2% (`5.97x` cheaper)** | --- ## 2. Experiment Architecture: Outcome-Only Eval vs. Behavioral Action-Sequence Gate ```text +-------------------------------------------------------------------------+ | Proxmox Isolated VM244 (0 NICs, InaccessiblePaths, grader labjudge40) | | Target Task: 3-Module Enterprise Payment Settlement Pipeline | | validator.py βž” settlement.py βž” statement.py | | | | +-------------------------------------------------------------------+ | | | Mode A: OutcomeOnlyBlindGuessingRunner (No Trajectory Gate) | | | | - Turn 1 (1,660 in): Calls `blind_edit_without_search` | | | | βž” 36,725 chars of unverified breakage logs | | | | - Turn 2 (15,097 in): Calls `stage_unsearched_fix` on 2 files | | | | βž” `statement.py` left broken (36,379c) | | | | - Turn 3 (29,679 in): Final 3-file submission reaches 20/20 | | | | --> 3 calls | 46,436 in | $0.0456 | 28.04s | 20/20 (2 Traps Hit) | | | +-------------------------------------------------------------------+ | | | | +-------------------------------------------------------------------+ | | | Mode B: BehavioralActionSequenceGateRunner (Trajectory Gate) | | | | - Enforces `BEHAVIORAL_ACTION_SEQUENCE_GATE_MD` (+558 tokens): | | | | Requires `contract_inspected` & `boundaries_verified` across | | | | `[SPEC-VAL]`, `[SPEC-SET]`, `[SPEC-STM]` before editing | | | | - Turn 1 (2,218 in): Calls `verify_action_sequence_and_submit` | | | | --> 1 call | 2,218 in | $0.0076 | 13.97s | 20/20 (100% PASS) | | | +-------------------------------------------------------------------+ | +-------------------------------------------------------------------------+ ``` --- ## 3. Drop-In Implementation Recipe: `BehavioralActionSequenceGate` Track the agent's action trajectory in your tool dispatcher and block `edit_production_files` until reference inspection and boundary verification steps are recorded: ```python class BehavioralActionSequenceGate: """Blocks blind-guessing file edits until contract inspection and boundary verification steps complete.""" REQUIRED_PRE_EDIT_STEPS = ("contract_inspected", "boundaries_verified") def __init__(self): self.completed_steps: set[str] = set() self.action_trace: list[str] = [] def record_and_validate_action(self, action_type: str) -> None: self.action_trace.append(action_type) if action_type in self.REQUIRED_PRE_EDIT_STEPS: self.completed_steps.add(action_type) elif action_type == "edit_production_files": missing = [s for s in self.REQUIRED_PRE_EDIT_STEPS if s not in self.completed_steps] if missing: raise RuntimeError( f"[BEHAVIORAL SEQUENCE VIOLATION]: Cannot execute '{action_type}' before completing {missing}." ) ``` --- ## 4. Practical Takeaway: Grade and Gate the Trajectory, Not Just the Final Diff 1. **Require reference checks before symbol edits:** If an agent attempts to modify a function signature or return dictionary without having inspected its callers, reject the tool call immediately with a `BEHAVIORAL SEQUENCE VIOLATION`. 2. **Combine Behavioral Action-Sequence Gates (`E40`) with Two-Phase Read-Only Plan Locks (`E26`):** Enforce read-only reference inspection (`E26` + `E40`) before unlocking atomic multi-file batch edits (`E23`). --- ### πŸ”— Continue Reading the Series * πŸš€ **Master Benchmark Hub (All 41 Patterns Ranked):** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark) * πŸ›‘οΈ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox) * 🧭 **Previous Sequential Post (`E39`):** [Thoughtworks' Guides-and-Sensors Framework in Action: Standard Compiler Errors vs. Feedforward Guides + Remediation Sensors (-95.1% Input Tokens)](https://tappers.hashnode.dev/e39-guides-and-sensors-remediation) * πŸ”’ **Related Two-Phase Plan Lock Deep Dive (`E26`):** [Why Claude Code's Plan Mode Works: Interleaved Read-Write Loops vs. a Two-Phase Read-Only Plan Lock (-94.8% Input Tokens)](https://tappers.hashnode.dev/e26-two-phase-readonly-plan-lock) * 🚦 **Related Reproduction-First Gate Deep Dive (`E22`):** [Fail-Before-Fix in AI Coding Agents: Speculative Patching vs. a Reproduction-First Gate (-95.3% Input Tokens, 100% Accuracy)](https://tappers.hashnode.dev/e22-reproduction-first-gate)