Skip to main content
Hashnode
Tappers Tech Blog

Preventing Multi-Session Amnesia in Coding Agents: Prose Summaries vs. a 3-Artifact Shift-Handover Harness (-95.2% Input Tokens)

Updated 6 min readMarkdown
T
tappers

💡 AI Labo Benchmark Series — Experiment E36 (Multi-Session Continuity & State Handover)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

Metadata & Experiment Identity

  • Experiment ID: E36 (Backlog Candidate C032)
  • Run ID: e36-20261029-vm240-01
  • Sandbox Environment: Proxmox Isolated VM VM240 (ai-e36-240, 0 NICs, PrivateNetwork=yes, NoNewPrivileges=yes, InaccessiblePaths, UID split labexp36 / labjudge36)
  • Model: gemini-3.8-flash (thinkingLevel: LOW, temperature=0.0)
  • Pattern Under Test: "Anthropic: Effective Harnesses for Long-Running Agents (feature_list.json + init.sh + progress.txt Shift-Handover Protocol vs. Unstructured Prose Summary)" — Unstructured Summary Handoff (UnstructuredSummaryHandoffRunner) vs. 3-Artifact Shift-Handover Harness (ShiftHandoverProtocolRunner)

1. Executive Summary (TL;DR)

When a long-running coding task spans multiple context resets ("shifts"), handing the next agent session an unstructured prose chat summary ("Most of the payment pipeline refactoring is done; just finish up the remaining checks") triggers two failure modes documented in Anthropic's Effective Harnesses for Long-Running Agents: Feature Amnesia (forgetting subtle edge-case contracts) and Premature Victory (declaring the task complete while boundary tests are still failing).

Replacing prose handoffs with a deterministic 3-Artifact Shift-Handover Harness—combining a machine-readable contract checklist (feature_list.json), an executable verification gate (init.sh), and a structured progress log (progress.txt)—gives the incoming session exact ground truth from Turn 1.

We tested this head-to-head inside isolated VM VM240 on the 3-module enterprise payment settlement pipeline (validator.py ➔ settlement.py ➔ statement.py):

  • Mode A (UnstructuredSummaryHandoffRunner): Receives an unstructured prose handoff summary across 74 unverified checklist paragraphs (73,841 chars of amnesia rework logs across Turns 1–2), overlooking the 5-key dictionary stripping invariant (clean_item = dict(item) leaks "extra") and finishing at 19 / 20 (95.0%).
  • Mode B (ShiftHandoverProtocolRunner): Enforces the 3-Artifact Shift-Handover Protocol (SHIFT_HANDOVER_HARNESS_MD + submit_handover_verified_bundle), solving 20 / 20 (100%) in 1 turn.

Key Result

Mode A failed validator_01_normalize_and_non_mutation (19 / 20) after 3 turns (46,227 cumulative input tokens). Mode B achieved a 20 / 20 (100%) pass in 1 turn, cutting LLM calls by -66.7% (3 ➔ 1), cumulative input tokens by -95.2% (46,227 ➔ 2,237 tokens—a 20.66x reduction), API cost by -83.1% ($0.0454 ➔ $0.0077), and running 2.27x faster (28.95s ➔ 12.75s, -55.9%).

Metric Mode A: Prose Summary Handoff (UnstructuredSummaryHandoff) Mode B: 3-Artifact Shift Handover (ShiftHandoverProtocol) Delta / Improvement
Hidden Grading Suite Score (labjudge36) 19 / 20 (95.0% — 1 failure) 20 / 20 (100% — Perfect) +5.0% accuracy (19/20 ➔ 20/20)
Feature-Amnesia Traps (gotcha_traps_hit) 2 traps (Turns 1 & 2) 0 traps -100% (zero amnesia)
LLM Calls (model_calls) 3 calls 1 call (1-shot) -66.7% (3.00x faster convergence)
Cumulative File Edits (file_edits) 5 edits 3 edits (1 per module) -40.0% fewer edits
Cumulative Tool Output (tool_output_chars) 73,841 chars 100 chars -99.9% (738.4x noise reduction)
Turn 1 Tokens (In / Out) 1,665 in / 13 out 2,237 in / 1,597 out +572 in for 3-artifact handover
Turn 2 Tokens (In / Out) 15,029 in / 1,258 out — (completed on Turn 1) Avoided prose handoff drift logs
Turn 3 Tokens (In / Out) 29,533 in / 1,591 out — (completed on Turn 1) Avoided secondary rework bloat
Cumulative Input Tokens (promptTokenCount) 46,227 tokens 2,237 tokens -95.2% (20.66x efficiency)
Cumulative Output Tokens (candidatesTokenCount) 2,862 tokens 1,597 tokens -44.2% output reduction
Elapsed Execution Time (elapsed_seconds) 28.95 sec 12.75 sec -55.9% (2.27x faster)
Estimated API Cost (estimated_micro_usd) $0.0454 (45,404 micro-USD) $0.0077 (7,667 micro-USD) -83.1% (5.92x cheaper)

2. Experiment Architecture: Prose Summary vs. 3-Artifact Shift-Handover Harness

+-------------------------------------------------------------------------+
| Proxmox Isolated VM240 (0 NICs, InaccessiblePaths, grader labjudge36)   |
|  Target Task: 3-Module Enterprise Payment Settlement Pipeline           |
|               validator.py ➔ settlement.py ➔ statement.py               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: UnstructuredSummaryHandoffRunner (Prose Chat Summary)     |  |
|  |  - Turn 1 (1,665 in):  Calls `read_unstructured_shift_summary`    |  |
|  |                        ➔ 36,932 chars of unverified prose notes   |  |
|  |  - Turn 2 (15,029 in): Calls `stage_amnesia_partial_patch`        |  |
|  |                        ➔ partial 2-file patch (36,586 chars)      |  |
|  |  - Turn 3 (29,533 in): Overlooks 5-key dict stripping (`clean_item|  |
|  |                        = dict(item)` leaks `"extra"` key)         |  |
|  |  --> 3 calls | 46,227 in | $0.0454 | 28.95s | 19/20 (1 FAILED)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ShiftHandoverProtocolRunner (3-Artifact Mechanical State) |  |
|  |  - Injects `SHIFT_HANDOVER_HARNESS_MD` (+572 tokens):             |  |
|  |    * `feature_list.json`: Boolean verification status per contract|  |
|  |    * `init.sh`: Deterministic diagnostic error keys               |  |
|  |    * `progress.txt`: Scoped module pointers (`[SPEC-VAL/SET/STM]`)|  |
|  |  - Turn 1 (2,237 in): Calls `submit_handover_verified_bundle`     |  |
|  |  --> 1 call  | 2,237 in  | $0.0077 | 12.75s | 20/20 (100% PASS)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Drop-In Implementation Recipe: build_shift_handover_payload

Never ask an outgoing LLM session to write a free-form essay for the next session. Generate a deterministic handover payload by running verify_fn() (init.sh) against feature_list.json:

import json
from typing import Callable


def build_shift_handover_payload(features: list[dict], verify_fn: Callable[[], list[str]]) -> str:
    """Builds a mechanical shift-handover payload combining feature_list.json and init.sh diagnostics."""
    active_errors = verify_fn()
    handover_state = {
        "protocol": "SHIFT_HANDOVER_V1",
        "init_sh_status": "PASS" if not active_errors else "FAIL",
        "active_diagnostic_errors": active_errors,
        "feature_list": [
            {
                "id": f["id"],
                "module": f["module"],
                "contract": f["contract"],
                "verified": f["error_key"] not in active_errors,
            }
            for f in features
        ],
    }
    return (
        "# SHIFT-HANDOVER HARNESS STATE (feature_list.json + init.sh)\n```json\n"
        + json.dumps(handover_state, indent=2, ensure_ascii=False)
        + "\n```"
    )

4. Practical Takeaway: Ground Every New Session in Executable State, Not Prose

  1. Compute "verified": true/false mechanically via init.sh at session boot: Never let an agent mark a feature as done in prose without an automated check flipping the flag in feature_list.json.
  2. Pair the Shift-Handover Harness (E36) with Milestone Context Compaction (E12): Compact completed milestones into a clean git commit and boot the next stage with the 3-artifact handover state.

🔗 Continue Reading the Series

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog