Skip to main content
Hashnode
Tappers Tech Blog

Why Outcome-Only Evals Let Coding Agents Guess Blindly: Outcome Evals vs. a Behavioral Action-Sequence Gate (-95.2% Tokens)

Updated 6 min readMarkdown
T
tappers

💡 AI Labo Benchmark Series — Experiment E40 (Behavioral Trajectory Gates)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

Metadata & Experiment Identity

  • Experiment ID: E40 (Backlog Candidate C036)
  • Run ID: e40-20261029-vm244-01
  • Sandbox Environment: Proxmox Isolated VM VM244 (ai-e40-244, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp40 / labjudge40)
  • Model: gemini-3.8-flash (thinkingLevel: LOW, temperature=0.0)
  • Pattern Under Test: "Google Developers Blog (Taylor Mullen & Christian Gunderman): The Anatomy of Harness Engineering — Behavioral Trajectory Evaluations vs. Outcome-Only Evals" — End-to-End Outcome-Only Eval (OutcomeOnlyBlindGuessingRunner) vs. Behavioral Action-Sequence Gate (BehavioralActionSequenceGateRunner)

1. Executive Summary (TL;DR)

When an agent harness evaluates only the final end-to-end outcome (whether tests pass at the end), it permits Blind Guessing Edits during intermediate turns: the agent modifies symbols without first checking their callers, references, or boundary contracts, triggering avoidable breakage cascades.

As described in Google Developers Blog's The Anatomy of Harness Engineering: Behavioral Evaluations, enforcing a Behavioral Action-Sequence Gate on the agent's intermediate tool trajectory (contract_inspected ➔ boundaries_verified ➔ edit_production_files) blocks unverified blind-guessing edits before they happen.

We tested this head-to-head inside isolated VM VM244 on the 3-module enterprise payment settlement pipeline (validator.py ➔ settlement.py ➔ statement.py):

  • Mode A (OutcomeOnlyBlindGuessingRunner): Evaluates only end-to-end outcomes without gating intermediate actions, allowing blind_edit_without_search on Turn 1 and stage_unsearched_fix on Turn 2 (73,202 chars of blind-guess breakage logs across 3 turns).
  • Mode B (BehavioralActionSequenceGateRunner): Enforces the Reference-Check Before Edit behavioral sequence (BEHAVIORAL_ACTION_SEQUENCE_GATE_MD + verify_action_sequence_and_submit), completing all 3 modules in 1 shot.

Key Result

While both modes reached 20 / 20 on the hidden grading suite, Mode A burned 3 turns, 5 file edits, 2 rework traps, and 46,436 cumulative input tokens. Mode B solved 20 / 20 (100%) in 1 turn on the first shot, cutting LLM calls by -66.7% (3 ➔ 1), cumulative input tokens by -95.2% (46,436 ➔ 2,218 tokens—a 20.94x reduction), API cost by -83.2% ($0.0456 ➔ $0.0076), and running 2.01x faster (28.04s ➔ 13.97s, -50.2%).

Metric Mode A: Outcome-Only Eval (OutcomeOnlyBlindGuessing) Mode B: Behavioral Sequence Gate (BehavioralActionSequenceGate) Delta / Improvement
Hidden Grading Suite Score (labjudge40) 20 / 20 (100% — Turn 3) 20 / 20 (100% — Turn 1 One-Shot) 1-shot convergence (3 ➔ 1 turn)
Blind-Guessing Traps (gotcha_traps_hit) 2 traps (Turns 1 & 2) 0 traps -100% (zero blind edits)
LLM Calls (model_calls) 3 calls 1 call (1-shot) -66.7% (3.00x faster convergence)
Cumulative File Edits (file_edits) 5 edits 3 edits (1 per module) -40.0% fewer edits
Cumulative Tool Output (tool_output_chars) 73,202 chars 101 chars -99.9% (724.8x noise reduction)
Turn 1 Tokens (In / Out) 1,660 in / 12 out 2,218 in / 1,595 out +558 in for behavioral sequence gate
Turn 2 Tokens (In / Out) 15,097 in / 1,260 out — (completed on Turn 1) Avoided blind-guess breakage logs
Turn 3 Tokens (In / Out) 29,679 in / 1,602 out — (completed on Turn 1) Avoided secondary rework bloat
Cumulative Input Tokens (promptTokenCount) 46,436 tokens 2,218 tokens -95.2% (20.94x efficiency)
Cumulative Output Tokens (candidatesTokenCount) 2,874 tokens 1,595 tokens -44.5% output reduction
Elapsed Execution Time (elapsed_seconds) 28.04 sec 13.97 sec -50.2% (2.01x faster)
Estimated API Cost (estimated_micro_usd) $0.0456 (45,605 micro-USD) $0.0076 (7,645 micro-USD) -83.2% (5.97x cheaper)

2. Experiment Architecture: Outcome-Only Eval vs. Behavioral Action-Sequence Gate

+-------------------------------------------------------------------------+
| Proxmox Isolated VM244 (0 NICs, InaccessiblePaths, grader labjudge40)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py ➔ settlement.py ➔ statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: OutcomeOnlyBlindGuessingRunner (No Trajectory Gate)       |  |
|  |  - Turn 1 (1,660 in):  Calls `blind_edit_without_search`          |  |
|  |                        ➔ 36,725 chars of unverified breakage logs |  |
|  |  - Turn 2 (15,097 in): Calls `stage_unsearched_fix` on 2 files    |  |
|  |                        ➔ `statement.py` left broken (36,379c)     |  |
|  |  - Turn 3 (29,679 in): Final 3-file submission reaches 20/20      |  |
|  |  --> 3 calls | 46,436 in | $0.0456 | 28.04s | 20/20 (2 Traps Hit) |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: BehavioralActionSequenceGateRunner (Trajectory Gate)      |  |
|  |  - Enforces `BEHAVIORAL_ACTION_SEQUENCE_GATE_MD` (+558 tokens):   |  |
|  |    Requires `contract_inspected` & `boundaries_verified` across   |  |
|  |    `[SPEC-VAL]`, `[SPEC-SET]`, `[SPEC-STM]` before editing        |  |
|  |  - Turn 1 (2,218 in): Calls `verify_action_sequence_and_submit`   |  |
|  |  --> 1 call  | 2,218 in  | $0.0076 | 13.97s | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Drop-In Implementation Recipe: BehavioralActionSequenceGate

Track the agent's action trajectory in your tool dispatcher and block edit_production_files until reference inspection and boundary verification steps are recorded:

class BehavioralActionSequenceGate:
    """Blocks blind-guessing file edits until contract inspection and boundary verification steps complete."""

    REQUIRED_PRE_EDIT_STEPS = ("contract_inspected", "boundaries_verified")

    def __init__(self):
        self.completed_steps: set[str] = set()
        self.action_trace: list[str] = []

    def record_and_validate_action(self, action_type: str) -> None:
        self.action_trace.append(action_type)
        if action_type in self.REQUIRED_PRE_EDIT_STEPS:
            self.completed_steps.add(action_type)
        elif action_type == "edit_production_files":
            missing = [s for s in self.REQUIRED_PRE_EDIT_STEPS if s not in self.completed_steps]
            if missing:
                raise RuntimeError(
                    f"[BEHAVIORAL SEQUENCE VIOLATION]: Cannot execute '{action_type}' before completing {missing}."
                )

4. Practical Takeaway: Grade and Gate the Trajectory, Not Just the Final Diff

  1. Require reference checks before symbol edits: If an agent attempts to modify a function signature or return dictionary without having inspected its callers, reject the tool call immediately with a BEHAVIORAL SEQUENCE VIOLATION.
  2. Combine Behavioral Action-Sequence Gates (E40) with Two-Phase Read-Only Plan Locks (E26): Enforce read-only reference inspection (E26 + E40) before unlocking atomic multi-file batch edits (E23).

🔗 Continue Reading the Series

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog