# Isolation Strategies The eval runner offers two strategies. The intent is identical in both: every eval starts from a clean slate so the result reflects the skill itself, not the host's accumulated state. ## What we are isolating from - The user's global `~/.claude/CLAUDE.md` (private global instructions) - Any ancestor `CLAUDE.md` in the project tree above the skill - Auto-memory at `~/.claude/projects/.../memory/MEMORY.md` - Cached settings, MCP configurations, IDE integrations - Prior conversation context bleeding via the shell ## Authentication The isolated `claude -p` subprocess needs to authenticate, but cannot read the host's `~/.claude/` (HOME is overridden) or the macOS Keychain (Keychain ACLs are scoped to the process that wrote the entry). The runner solves this in the parent process: 1. On macOS, read the OAuth credential JSON from the Keychain entry `Claude Code-credentials` via `security find-generic-password -s "Claude Code-credentials" -w`. This succeeds because the parent runs as the same user that wrote the entry. 2. Stage that JSON as `/.home/.claude/.credentials.json` (local mode) or copy it into `/home/evaluator/.claude/.credentials.json` inside the container (Docker mode). 3. The subprocess reads `.credentials.json` exactly the way Claude Code normally does, with no other host config bleed. If the parent has `ANTHROPIC_API_KEY` set, that env var is also forwarded — and it takes precedence over the Keychain credential. On non-macOS hosts, the Keychain step is skipped and `ANTHROPIC_API_KEY` is the only auth path. ## Docker (preferred) A single image, `bmad-eval-runner:latest`, is built once per machine. It contains Node 20, Claude Code (via `npm install -g @anthropic-ai/claude-code`), Python 3, and standard tools. The image is intentionally minimal — every eval starts from this baseline. ### Image build `scripts/docker_setup.py --build` builds the image from `assets/Dockerfile`. This runs once. Re-runs are a no-op unless `--rebuild` is passed. ### Per-eval container Each eval gets a fresh container: ``` docker run --rm \ -v ":/project:ro" \ -v "/:/output" \ -v ":/fixtures:ro" \ -e ANTHROPIC_API_KEY \ -e EVAL_PROMPT \ -e EVAL_ID \ -e SKILL_PATH \ bmad-eval-runner:latest \ /bin/bash -c "/scripts/run_one_eval.sh" ``` Inside the container: 1. The project is copied from `/project` (read-only) to `/workspace` (writable, container-local). Copy is fast because the underlying layer is shared. 2. Fixtures are copied into `/workspace/fixtures/`. 3. `HOME` is `/home/evaluator`, an empty directory created by the image — no global `CLAUDE.md`, no memory. 4. `claude -p "$EVAL_PROMPT" --output-format stream-json --verbose` runs at `/workspace`. 5. The stream-json transcript is captured to `/output/transcript.jsonl`. Any files the skill writes under `/workspace` are rsynced to `/output/artifacts/` after the run completes. 6. The container exits and is removed automatically. The host then has `//transcript.jsonl`, `//artifacts/`, and timing data. Nothing on the host is touched. ### Why Docker is preferred - The image is reproducible — every run starts from byte-identical state. - `HOME` is genuinely empty, not just overridden. - Filesystem isolation is real, not just convention. - Network can be locked down (`--network=none` for trigger evals; full network for artifact evals that may need it). ## Local fallback When Docker is unavailable, the runner falls back to per-eval temp directories under `~/bmad-evals///`. Layout: ``` ~/bmad-evals/// workspace/ # the eval's working directory .home/ # HOME override — empty .claude/ inside project/ # rsync'd copy of fixtures/ # staged fixture files transcript.jsonl # claude -p stream output artifacts/ # files Claude wrote under workspace/ metrics.json ``` Per-eval invocation roughly: ``` HOME="$WORKSPACE/.home" \ CLAUDE_CONFIG_DIR="$WORKSPACE/.home/.claude" \ ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \ claude -p "$EVAL_PROMPT" \ --output-format stream-json --verbose \ > transcript.jsonl ``` ### Limitations of local mode - `HOME` override prevents global `CLAUDE.md` and memory loading, but ancestor discovery still happens from the workspace's cwd. If the workspace is created inside a directory tree that contains a `.claude/skills/` further up, the subprocess may discover those skills regardless of `HOME`. This matters most for trigger evals, where stray host skills can fire instead of the synthetic skill we're testing — **prefer Docker for trigger evals**, where filesystem isolation is real. - Filesystem isolation is by convention only — the skill could write outside its workspace if it tries. We don't sandbox syscalls. - Network is unrestricted. Tell the user clearly when local mode is in use and that it is best-effort. ## Why a real skill, not a slash command, for trigger evals The trigger runner stages a synthetic skill at `/.claude/skills//SKILL.md` — not at `.claude/commands/.md`. Slash commands are user-invoked (`/`); they do not surface as `Skill` tool calls and so a description placed there can never be observed firing the way a real skill would. Anthropic's reference `run_eval.py` uses the commands path and is known to report 0% trigger rates as a result. Placing the synthetic at `.claude/skills/` matches how real skills load and lets the detector observe genuine `Skill` (or `Read` of the synthetic SKILL.md) tool calls. ## Why not `--add-dir` only? `claude -p --add-dir ` would let Claude see the skill but would still inherit the user's `CLAUDE.md` and memory from the cwd's ancestors. The whole point of this runner is to test the skill, not the host's accumulated state. So we always either Docker-isolate or temp-dir-isolate. ## Artifact retention Run folders are never deleted by this skill. Disk management is the user's responsibility. The runner emits the run folder path on completion; users who want to clean up old runs can delete `~/bmad-evals//` directly.