# Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md

> 💡 **AI Labo Benchmark Series — Experiment** `E20` **(Ranked #1 out of 41 Patterns)**
>
> This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
>
> - 🚀 [**View All 41 Benchmark Results & Master Ranking**](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)
> - 🛡️ [**Read How Our Proxmox & systemd Anti-Cheating Sandbox Works**](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)

---

## Metadata & Experiment Identity

- **Experiment ID:** `E20` (Backlog Candidate `C016`)
- **Run ID:** `e20-20261028-vm224-01`
- **Sandbox Environment:** Proxmox Isolated VM `VM224` (`ai-e20-224`, 0 NICs, `virtio-serial` QGA relay, `systemd` UID separation)
- **Model:** `gemini-3.8-flash` (`thinkingLevel: LOW`)
- **Pattern Under Test:** Letta / MemGPT & Claude Code `CLAUDE.md` / `MEMORY.md` Pattern — *"Episodic Gotchas Memory vs. Cold-Start Exploration"*

---

## 1\. Executive Summary (TL;DR)

When running multiple tasks sequentially on the same repository, starting every agent session from scratch (**Cold-Start Exploration**) forces the model to rediscover non-obvious domain invariants and repository quirks ("gotchas") through repeated test failures—burning tens of thousands of tokens on stack traces and diluting model attention.

To test whether a compact **Episodic Gotchas Memory (**`MEMORY.md`**)** solves this, we benchmarked a 3-module payment settlement pipeline (`validator.py`, `settlement.py`, `statement.py`) containing **5 subtle repository gotchas (7 domain invariants)** across two modes:

- **Mode A (**`Cold-Start Exploration`**):** No `MEMORY.md` injected; the agent explores the codebase and diagnoses failures via standard test/audit feedback.
- **Mode B (**`Episodic Gotchas Memory`**):** A **5-item** `MEMORY.md` **(**`+442 tokens`**)** distilled from prior session failures is injected into the initial prompt.

### Key Result

Injecting just `+442 tokens` **of** `MEMORY.md` reduced repository trap hits from `2 ➔ 0`, achieving a **1-turn** `20 / 20` **(100%) pass** on the hidden grading suite. Compared to Mode A (`17 / 20` pass across 3 turns), Mode B cut cumulative input tokens by `-95.5%` **(**`46,560` **➔** `2,078` **tokens)**, API cost by `-83.7%` **(**`$0.0454` **➔** `$0.0074`**)**, and wall-clock time by `-54.8%` **(**`2.21x` **faster)**.

| Metric | Mode A: Cold-Start Exploration (No `MEMORY.md`) | Mode B: Episodic Gotchas Memory (`MEMORY.md` Injected) | Delta / Improvement |
| :-- | :-: | :-: | :-: |
| **Hidden Grading Suite Score (20 Cases)** | `17 / 20` (85.0% — missed 3 edge cases) | `20 / 20` **(100% Perfect Pass)** | **+15.0 pt (Zero defects)** |
| **Repository Gotcha Traps Triggered** | `2 times` (Turn 1, Turn 2) | `0 times` **(Avoided on Turn 1)** | **\-100.0% (Zero traps)** |
| **LLM Calls (**`model_calls`**)** | `3 calls` | `1 call` **(1-shot completion)** | **\-66.7% (3.00x faster convergence)** |
| **Total File Edits (**`file_edits`**)** | `5 edits` | `3 edits` **(1 per module)** | **\-40.0% reduction** |
| **Tool Output Volume (**`tool_output_chars`**)** | `56,605 chars` | `101 chars` | **\-99.8% (560x less noise)** |
| **Turn 1 Input / Output Tokens** | `1,636` in / `11` out | `2,078` **in /** `1,560` **out** | `MEMORY.md` overhead is only `+442` **in** |
| **Turn 2 Input / Output Tokens** | `15,153` in / `1,220` out | — *(Completed on Turn 1)* | Avoided Turn 1 failure log bloat |
| **Turn 3 Input / Output Tokens** | `29,771` in / `1,569` out | — *(Completed on Turn 1)* | Avoided Turn 2 failure log bloat |
| **Cumulative Input Tokens (**`promptTokenCount`**)** | `46,560 tokens` | `2,078 tokens` | **\-95.5% (22.41x token efficiency)** |
| **Cumulative Output Tokens (**`candidatesTokenCount`**)** | `2,800 tokens` | `1,560 tokens` | **\-44.3% reduction** |
| **Elapsed Execution Time (**`elapsed_seconds`**)** | `30.16 sec` | `13.64 sec` | **\-54.8% (2.21x faster)** |
| **Estimated API Cost (**`estimated_micro_usd`**)** | `$0.0454` (`45,421 micro-USD`) | `$0.0074` **(**`7,409 micro-USD`**)** | **\-83.7% (6.13x cheaper)** |

---

## 2\. Experiment Architecture & Repository Gotchas Under Test

```text
+-------------------------------------------------------------------------+
| Proxmox Isolated VM224 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Bugfix                       |
|               (validator.py / settlement.py / statement.py)             |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: ColdStartRunner (No MEMORY.md — Cold-Start Exploration)   |  |
|  |  - Turn 1: probe_repo_gotchas   (1,636 in / 28,317 chars trap log)|  |
|  |  - Turn 2: stage_partial_fix    (15,153 in / 27,975 chars log)    |  |
|  |  - Turn 3: submit_settlement    (29,771 in --> 17 / 20 passed)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: EpisodicGotchasMemoryRunner (5-Item MEMORY.md [+442 tok]) |  |
|  |  - Turn 1: Reads 5-item MEMORY.md & submits all 3 modules         |  |
|  |            (2,078 in / 1,560 out --> 20 / 20 passed on 1st try!)  |  |
|  +-------------------------------------------------------------------+  |
|                                    |                                    |
|                                    v                                    |
|  +-------------------------------------------------------------------+  |
|  | Independent Grader (labjudge20): 20 Hidden Unit & E2E Test Cases  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
```

### The 5 Non-Obvious Repository Gotchas in the Initial Codebase

The initial workspace inside `VM224` contains three payment modules with subtle domain bugs and Python traps that are easy to miss on a cold read:

1. `validator.py` **(Transaction Validation & Normalization):**
   - `ALLOWED_CURRENCIES = ("USD", "EUR")` is missing `"JPY"`.
   - Empty list `[]` rejection is missing, and because `bool` **is a subclass of** `int` **in Python (**`isinstance(True, int) == True`**)**, `isinstance(v, bool)` must be explicitly rejected before checking `isinstance(v, int)`.
   - The bounds check `if cnt <= 0 or cents <= 0 or not (0 <= bps < 1000):` erroneously rejects `unit_cents == 0` **(**`$0` **card authorization probes)** and `chargeback_bps == 1000` **(inclusive upper bound)**.
   - It mutates caller dictionaries in-place (`item["merchant_id"] = ...; out.append(item)`) and leaks extra unvalidated keys (`extra`) into the output.
2. `settlement.py` **(Batch Aggregation, Reserve & Tier Discounts):**
   - Iterates with `for k in sorted(order):`, destroying **first-seen** `(merchant_id, currency)` **ordering**.
   - Applies the 10% chargeback reserve at `chargeback_bps >= 300` instead of strictly `> 300` (at exactly `300 bps`, reserve must be `0`).
   - Uses `subtotal > 15000` instead of inclusive `>= 15000`, omits the `10%` discount for `ENTERPRISE` below `15000`, and forgets to cap `rebate_applied = min(fee_rebate, after_discount)`.
3. `statement.py` **(Canonical Statement Formatter):**
   - Uses `:` instead of `#` between merchant ID and currency (`{merchant_id}#{currency}:`).
   - Formats zero discounts as `"0"` instead of preserving the mandatory leading minus sign (`TIER_DISCOUNT: -0`, `REBATE_APPLIED: -0`).

---

## 3\. Inside the Prompts: What Mode B Saw vs. What Mode A Endured

### 3.1 The Exact 5-Item `MEMORY.md` Injected into Mode B (`+442 tokens`)

In Mode B (`input-03.json`), the harness prepended this exact 5-bullet `# MEMORY.md` section to the initial prompt:

```markdown
# MEMORY.md — Repository Episodic Gotchas & Non-Obvious Invariants (Distilled from Session #1)
1. [GOTCHA-01 / validator.py] `ALLOWED_CURRENCIES` MUST include `"JPY"` (`("USD", "EUR", "JPY")`), empty `records` list MUST raise `ValueError`, and `isinstance(v, bool)` MUST be rejected before `isinstance(v, int)` because `bool` is a subclass of `int` in Python.
2. [GOTCHA-02 / validator.py] Zero `unit_cents == 0` and boundary `chargeback_bps == 1000` are VALID (`tx_count > 0`, `unit_cents >= 0`, `0 <= chargeback_bps <= 1000`). Always return a brand-new 5-key dict without mutating caller dicts.
3. [GOTCHA-03 / settlement.py] Preserve first-seen `(merchant_id, currency)` order (`for k in order:`, NEVER `sorted(order)`). The 10% reserve applies STRICTLY when `chargeback_bps > 300` (at `300` bps, `reserve_cents == 0`).
4. [GOTCHA-04 / settlement.py] Tier discount threshold `subtotal >= 15000` is INCLUSIVE: `GROWTH` gets 10% (`>= 15000`), `ENTERPRISE` gets 20% (`>= 15000`) or 10% (`< 15000`), and `rebate_applied = min(fee_rebate, after_discount)`.
5. [GOTCHA-05 / statement.py] Header delimiter between merchant and currency is `#` (`{merchant_id}#{currency}: tx=...`) and discount/rebate lines MUST always carry `-` even when zero (`TIER_DISCOUNT: -0`, `REBATE_APPLIED: -0`).
```

### 3.2 The 56,605-Character Failure Trace That Flooded Mode A (`Cold-Start`)

Without `MEMORY.md`, Mode A called `probe_repo_gotchas` on Turn 1 and `stage_partial_fix` on Turn 2, accumulating `28,317` **characters** on Turn 1 and `27,975` **characters** on Turn 2 (totaling `56,605` **characters** of verbose audit stack traces) inside its conversation history:

```text
=== COLD-START REPOSITORY GOTCHA TRAP LOG (STAGE #1) ===
Status: FAILED (4 repository gotchas triggered due to missing cross-session MEMORY.md)
[COLD_START_GOTCHA_AUDIT_001] stage=1 invariant_probe=GOTCHA_02 trace_digest=0000000000000000000000000000000000000000000000000000000000000001
  File '/var/lib/ai-lab-e20-vm224-01/work_cold_start/settlement.py', line 21, in verify_repo_invariants
    AssertionError: Cold-start session rediscovered repository gotcha #2 via trial-and-error failure.
... (28,000+ chars of repetitive invariant audit traces) ...
TRIGGERED_GOTCHA [validator.py] (validator_contract_violation): Failed JPY currency, empty list check, bool rejection, zero unit_cents, 1000 bps, or mutated input dict.
TRIGGERED_GOTCHA [settlement.py] (settlement_order_or_reserve_boundary): Lines reordered by sorted(order) or 10% reserve charged at chargeback_bps == 300.
TRIGGERED_GOTCHA [settlement.py] (settlement_tier_discount_or_rebate_cap): Missed >=15000 inclusive discount boundary, ENTERPRISE 10% below 15000, or min(fee_rebate, after_discount) cap.
TRIGGERED_GOTCHA [statement.py] (statement_delimiter_or_signed_zero): Used ':' instead of '#' between merchant and currency or omitted '-' sign when discount/rebate is 0.
```

---

## 4\. Code-Level Postmortem: Why Mode A Failed 3 out of 20 Test Cases

Comparing the final submitted code bundles (`cold-start-bundle.json` vs. `episodic-memory-bundle.json`) reveals the exact mechanism of **Attention Dilution**:

```python
# ❌ Mode A (Cold-Start, Turn 3 Final Submission — from cold-start-bundle.json)
        if cnt <= 0 or cents <= 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")
        out.append({
            **item,
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
        })
```

```python
# ✅ Mode B (Episodic Gotchas Memory, Turn 1 Final Submission — from episodic-memory-bundle.json)
        if cnt <= 0 or cents < 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")

        out.append({
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
            "tx_count": cnt,
            "unit_cents": cents,
            "chargeback_bps": bps,
        })
```

By Turn 3, Mode A was carrying **29,771 input tokens** of noisy logs. While it managed to fix `settlement.py`, `statement.py`, `"JPY"`, `bool` rejection, and `chargeback_bps <= 1000`, its diluted attention missed two fine-grained details in `validator.py`:

1. **Left** `cents <= 0` **instead of** `cents < 0`**:** Rejecting `unit_cents == 0` (`$0` auth probes) caused both unit test `validator_05_zero_unit_cents_allowed_negative_rejected` and E2E test `e2e_03_zero_amount_auth_statement` to fail.
2. **Used** `{**item, ...}` **spread instead of constructing a clean 5-key dict:** Extra unvalidated keys from the input dict leaked into the normalized output, failing `validator_01_normalize_and_non_mutation`.

Mode B, reading `[GOTCHA-02 / validator.py]` inside a clean **2,078-token** Turn 1 prompt, nailed every single invariant on the first try.

### All 20 Hidden Test Case Results (Mode A vs. Mode B)

| # | Test Case ID | Invariant / Boundary Verified | Mode A (`Cold-Start`) | Mode B (`MEMORY.md`) |
| :-: | :-- | :-- | :-: | :-: |
| 1 | `validator_01_normalize_and_non_mutation` | Trim, uppercase, **strip extra keys (strict 5-key dict)**, no caller mutation | **❌ FAIL** (`**item` leaked keys) | **✅ PASS** |
| 2 | `validator_02_empty_and_non_list_rejected` | Reject `[]`, `None`, and non-dict elements with `ValueError` | ✅ PASS | **✅ PASS** |
| 3 | `validator_03_blank_or_non_str_merchant_and_unknown_currency_rejected` | Reject blank merchant ID / unknown currency; allow `JPY` | ✅ PASS | **✅ PASS** |
| 4 | `validator_04_tx_count_and_chargeback_bps_bounds_0_to_1000` | Allow `chargeback_bps` `0` and `1000` (inclusive); reject out-of-range | ✅ PASS | **✅ PASS** |
| 5 | `validator_05_zero_unit_cents_allowed_negative_rejected` | **Allow** `unit_cents == 0` **(**`$0` **auth)**; reject negative amounts | **❌ FAIL** (`cents <= 0`) | **✅ PASS** |
| 6 | `validator_06_bool_rejected_on_all_int_fields` | Reject Python `bool` (`True`/`False`) on all integer fields | ✅ PASS | **✅ PASS** |
| 7 | `settlement_01_first_seen_order_and_aggregation` | Preserve first-seen `(merchant_id, currency)` order and sum `tx_count` | ✅ PASS | **✅ PASS** |
| 8 | `settlement_02_conflicting_unit_cents_or_bps_rejected` | Reject conflicting `unit_cents` or `chargeback_bps` for same key | ✅ PASS | **✅ PASS** |
| 9 | `settlement_03_exact_chargeback_bps_300_vs_301_boundary` | Exact boundary: `300 bps` (0 reserve) vs. `301 bps` (10% reserve) | ✅ PASS | **✅ PASS** |
| 10 | `settlement_04_exact_15000_growth_discount_boundary` | `GROWTH` tier boundary: `14999` (0%) vs. `15000` (10% discount) | ✅ PASS | **✅ PASS** |
| 11 | `settlement_05_enterprise_tiered_10_vs_20_discount` | `ENTERPRISE` tier: `< 15000` (10%) vs. `>= 15000` (20%) | ✅ PASS | **✅ PASS** |
| 12 | `settlement_06_floor_truncation_reserve_and_fee` | Integer floor division (`//`) for reserve and wire fee | ✅ PASS | **✅ PASS** |
| 13 | `settlement_07_rebate_capped_at_after_discount` | Cap rebate via `min(fee_rebate, after_discount)` | ✅ PASS | **✅ PASS** |
| 14 | `settlement_08_invalid_tier_or_bool_rebate_rejected` | Reject unknown tier names and boolean `fee_rebate` | ✅ PASS | **✅ PASS** |
| 15 | `settlement_09_lowercase_tier_and_multi_currency_grouping` | Normalize lowercase tier names and group multi-currency lines | ✅ PASS | **✅ PASS** |
| 16 | `e2e_01_growth_below_15000_minus_zero_and_hash_separator` | E2E: `#` delimiter and `TIER_DISCOUNT: -0` / `REBATE_APPLIED: -0` | ✅ PASS | **✅ PASS** |
| 17 | `e2e_02_enterprise_reserve_and_rebate_statement` | E2E: Full `ENTERPRISE` reserve + 20% discount + rebate statement | ✅ PASS | **✅ PASS** |
| 18 | `e2e_03_zero_amount_auth_statement` | **E2E: End-to-end processing of** `unit_cents == 0` **(**`$0` **auth)** | **❌ FAIL** (Rejected in Stage 1) | **✅ PASS** |
| 19 | `e2e_04_line_order_preserved_in_statement` | E2E: Preserve first-seen line order in rendered statement | ✅ PASS | **✅ PASS** |
| 20 | `e2e_05_error_propagation_across_stages` | E2E: Propagate Stage 2 conflicting unit price `ValueError` | ✅ PASS | **✅ PASS** |
| **Total** | **20 Hidden Cases** | — | `17 / 20` **(85.0%)** | `20 / 20` **(100%)** |

---

## 5\. Practical Takeaway: How to Structure Your Own `MEMORY.md`

A **442-token investment** saved **44,482 input tokens** (a **100x+ token ROI**) while eliminating 3 subtle edge-case bugs.

To get the same result in your own repositories with Claude Code, Cursor, or custom agents:

- **Don't turn** `CLAUDE.md` **or** `MEMORY.md` **into a 500-line textbook.** (As we proved in `E35`, monolithic instruction files hurt performance.)
- Instead, maintain a **10-to-20 line** `MEMORY.md` containing *only* `[GOTCHA-ID / filename] 1-line invariant` entries distilled from actual test failures or PR review catches:

```markdown
# MEMORY.md (Repository Gotchas — Keep under 20 lines)
- [GOTCHA-01 / validator.py] `isinstance(x, bool)` is True for `int` in Python; always reject `bool` before checking `isinstance(x, int)`.
- [GOTCHA-02 / validator.py] `unit_cents == 0` ($0 auth) and `chargeback_bps == 1000` are valid; construct a new 5-key dict (never `**item` or mutate in-place).
- [GOTCHA-03 / settlement.py] Keep first-seen order (`for k in order:`); apply 10% reserve strictly when `chargeback_bps > 300`.
- [GOTCHA-04 / statement.py] Use `#` between merchant and currency (`{mid}#{cur}:`), and format zero discount/rebate with a leading minus (`-0`).
```

---

### 🔗 Continue Reading the Series

- 🚀 **Master Benchmark Hub (All 41 Patterns Ranked):** [I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy](https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark)
- 🛡️ **Methodology & Sandbox Architecture:** [How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox](https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox)