fix(harness): keep gateway RPC sockets alive

.agents/skills/blacksmith-testbox/SKILL.md

@@ -10,11 +10,6 @@ agent dotfiles, Docker, or a benchmark run that is too heavy for the local
 machine. Keep normal unit-test iteration local unless the user asks for
 Testbox proof.
 
-Crabbox is the sibling lane for reusable owned-capacity proof. Use
-`.agents/skills/crabbox/SKILL.md` and `.crabbox.yaml` when ClawBench needs
-AWS-backed reusable boxes or Crabbox sync/log/result inspection. Keep this
-skill focused on Blacksmith CI parity.
-
 ## Warmup
 
 Run from the repository root:

.agents/skills/crabbox/SKILL.md

@@ -1,122 +0,0 @@
----
-name: crabbox
-description: Use Crabbox for ClawBench remote Linux validation, warmed reusable boxes, GitHub Actions hydration, sync timing, logs, results, caches, and lease cleanup.
----
-
-# Crabbox
-
-Use Crabbox when ClawBench needs remote Linux proof on owned capacity, a large
-runner class, reusable warm state, or a Blacksmith alternative.
-
-## Before Running
-
-- Run from the repo root. Crabbox sync mirrors the current checkout.
-- Prefer local targeted tests for tight edit loops.
-- Prefer Blacksmith Testbox when the task explicitly asks for Blacksmith or a
-  Blacksmith-specific CI comparison.
-- Use Crabbox for broad ClawBench gates when owned AWS capacity is the right
-  remote lane.
-- Check `.crabbox.yaml` for repo defaults before adding flags.
-- Sanity-check the selected binary before remote work. Prefer the local
-  `openclaw/crabbox` checkout when present because the user PATH shim can be
-  stale: `command -v crabbox; ../crabbox/bin/crabbox --version`.
-- Install with `brew install openclaw/tap/crabbox`; auth is required before use:
-  `crabbox login --url https://crabbox.openclaw.ai --provider aws`.
-- On macOS the user config is `~/Library/Application Support/crabbox/config.yaml`;
-  it must include `broker.url`, `broker.token`, and usually `provider: aws`.
-
-## ClawBench Flow
-
-AWS/owned-capacity flow for Python tests:
-
-```sh
-crabbox warmup --class standard --idle-timeout 90m
-crabbox actions hydrate --id <cbx_id-or-slug>
-crabbox run --id <cbx_id-or-slug> --timing-json --shell -- "python -m pytest -q"
-```
-
-For commands that need hydrated HF/provider credentials or agent dotfiles, use
-the helper installed by the hydration workflow:
-
-```sh
-crabbox run --id <cbx_id-or-slug> --timing-json --shell -- "clawbench-testbox-env python -m pytest -q"
-crabbox run --id <cbx_id-or-slug> --timing-json --shell -- "clawbench-testbox-env clawbench run --model anthropic/claude-sonnet-4-6 --adapter simulated"
-```
-
-Blacksmith-backed Crabbox flow can delegate setup to the existing Testbox
-workflow:
-
-```sh
-crabbox run --provider blacksmith-testbox --blacksmith-org openclaw --blacksmith-workflow .github/workflows/ci-check-testbox.yml --blacksmith-job check --blacksmith-ref main --idle-timeout 90m --timing-json --shell -- "python -m pytest -q"
-```
-
-Stop boxes you created before handoff:
-
-```sh
-crabbox stop <cbx_id-or-slug>
-```
-
-## Owned AWS Capacity
-
-When AWS capacity is under pressure, do not start with `class=beast`.
-`beast` begins at 48xlarge instances and can burn 192 vCPU quota per request.
-ClawBench's owned-cloud default is `standard`; escalate to `fast`, then
-`large`, and only use `beast` when the work is explicitly CPU-bound and the
-smaller class already failed the goal.
-
-Keep capacity hints enabled so brokered AWS leases print selected
-region/market, quota pressure, Spot fallback, and high-pressure class warnings.
-The ClawBench repo config sets `capacity.hints: true`; use
-`CRABBOX_CAPACITY_HINTS=0` only when debugging hint rendering itself.
-
-Use `beast` only for exceptional lanes:
-
-- full benchmark sweeps where wall time is dominated by CPU, not dependency
-  install or network;
-- release/blocker validation where a maintainer explicitly asks for the largest
-  owned AWS class;
-- performance profiling where the point is to compare high-core behavior.
-
-Do not use `beast` for ordinary `python -m pytest -q`, docs-only work, small
-task repros, Blacksmith outage triage, or focused lint/type/test checks. Those
-should use `standard` first and `fast` only when the extra cores materially
-help.
-
-## Useful Commands
-
-```sh
-crabbox status --id <id-or-slug> --wait
-crabbox inspect --id <id-or-slug> --json
-crabbox sync-plan
-crabbox history --lease <id-or-slug>
-crabbox logs <run_id>
-crabbox results <run_id>
-crabbox cache stats --id <id-or-slug>
-crabbox ssh --id <id-or-slug>
-```
-
-Use `--debug` on `run` when measuring sync timing.
-Use `--timing-json` on warmup, hydrate, and run when comparing AWS and
-blacksmith-testbox timings.
-Use `--market spot|on-demand` on AWS warmup or one-shot run when testing quota
-or capacity behavior without changing `.crabbox.yaml`.
-
-## Hydration Boundary
-
-`.github/workflows/crabbox-hydrate.yml` is repo-specific on purpose. It owns
-ClawBench checkout, setup-python, pip install, provider/HF env hydration,
-agent-dotfile restoration, ready marker, and keepalive. Crabbox owns runner
-registration, workflow dispatch, SSH sync, command execution, logs/results,
-local lease claims, and idle cleanup.
-
-Do not add ClawBench-specific setup to Crabbox. Put repo setup in the hydration
-workflow and generic lease/sync behavior in Crabbox.
-
-## Cleanup
-
-Crabbox has coordinator-owned idle expiry and local lease claims, so ClawBench
-does not need a custom ledger. Default idle timeout is 30 minutes unless config
-or flags set a different value. Still stop boxes you created when done.
-If `crabbox list` prints `orphan=no-active-lease`, treat it as an operator
-review hint; do not delete `keep=true` machines without checking provider and
-coordinator state.

.crabbox.yaml

@@ -1,48 +0,0 @@
-profile: clawbench-check
-provider: aws
-class: standard
-capacity:
-  market: spot
-  strategy: most-available
-  fallback: on-demand-after-120s
-  hints: true
-  regions:
-    - eu-west-1
-actions:
-  workflow: .github/workflows/crabbox-hydrate.yml
-  job: hydrate
-  ref: main
-  runnerLabels:
-    - crabbox
-    - clawbench
-  runnerVersion: latest
-  ephemeral: true
-aws:
-  region: eu-west-1
-  rootGB: 400
-sync:
-  delete: true
-  checksum: false
-  gitSeed: true
-  fingerprint: true
-  baseRef: main
-  exclude:
-    - .artifacts
-    - .codex
-    - .DS_Store
-    - .pytest_cache
-    - .ruff_cache
-    - .venv
-    - dist
-    - htmlcov
-    - playwright-report
-    - test-results
-env:
-  allow:
-    - CI
-    - CLAWBENCH_*
-    - OPENCLAW_*
-    - PYTHON*
-ssh:
-  user: crabbox
-  port: "2222"

.dockerignore

@@ -0,0 +1,19 @@
+.git
+.venv
+__pycache__
+.pytest_cache
+.mypy_cache
+.ruff_cache
+.DS_Store
+
+data
+results
+.clawbench
+
+.tmp/*
+!.tmp/hermes-agent
+!.tmp/hermes-agent/**
+
+**/node_modules
+**/__pycache__
+**/.pytest_cache

.env.example

@@ -1,23 +0,0 @@
-# Copy to .env for local docker compose or shell-based runs.
-#
-# Do not commit real tokens. Keep placeholder values commented so a fresh
-# checkout cannot accidentally enable a fake provider or tracing config.
-
-# Hugging Face queue/results persistence.
-# HF_TOKEN=
-# CLAWBENCH_QUEUE_DATASET=openclaw/clawbench-results
-
-# OpenClaw gateway auth.
-# OPENCLAW_GATEWAY_TOKEN=local-dev-token-for-testing
-
-# Optional benchmark tuning.
-# CLAWBENCH_RUN_CACHE_DIR=.clawbench/run_cache
-# CLAWBENCH_CONCURRENCY=1
-# CLAWBENCH_JUDGE_MODEL=anthropic/claude-sonnet-4-6
-# CLAWBENCH_JUDGE_AFFECTS_SCORE=0
-
-# Provider credentials for live model runs.
-# ANTHROPIC_API_KEY=
-# OPENAI_API_KEY=
-# OPENROUTER_API_KEY=
-# GEMINI_API_KEY=

.github/CODEOWNERS

@@ -1 +0,0 @@
-* @openclaw/openclaw-evals

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,31 +0,0 @@
----
-name: Bug report
-about: Something is broken or producing wrong results
-labels: bug
----
-
-## What happened
-
-<!-- A clear description of the bug. -->
-
-## Expected behaviour
-
-<!-- What should have happened instead. -->
-
-## Steps to reproduce
-
-```bash
-# Minimal command / code snippet that triggers the bug
-```
-
-## Relevant output
-
-```
-# Full error message, stack trace, or unexpected scoring output
-```
-
-## Environment
-
-- Python version:
-- OS:
-- ClawBench version / commit:

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,21 +0,0 @@
----
-name: Feature request
-about: Suggest a new task, scoring improvement, or other enhancement
-labels: enhancement
----
-
-## Summary
-
-<!-- One or two sentences describing what you want. -->
-
-## Motivation
-
-<!-- Why is this valuable? What problem does it solve, or what gap does it fill? -->
-
-## Proposed approach
-
-<!-- Optional: sketch of how you'd implement it, or what the change would look like. -->
-
-## Alternatives considered
-
-<!-- Any other approaches you thought about and why you ruled them out. -->

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,18 +0,0 @@
-## What does this PR do?
-
-<!-- One or two sentences. -->
-
-## Why?
-
-<!-- Motivation: what bug does it fix, what gap does it fill? Link related issues with "Fixes #N". -->
-
-## Changes
-
-<!-- Bullet list of the meaningful changes. Skip files touched only for formatting. -->
-
-## Tests
-
-<!-- Describe new or updated tests. If no tests were added, explain why none are needed. -->
-
-- [ ] `python -m pytest -q` passes locally
-- [ ] `python -m ruff check clawbench app.py scripts tests` passes locally, or the change is docs-only

.github/workflows/README.md

@@ -9,11 +9,10 @@ Runs the repository test suite automatically on:
 - manual dispatch from the Actions tab
 
 It uses Python 3.11 and 3.12, installs the package with
-`pip install -e .[dev]`, runs full Ruff lint plus `python -m pytest -q`,
-then builds a wheel and checks that runtime data such as `tasks-public/`,
-`tasks-domain/`, `profiles/`, and `baselines/` are included. Runs under the
-`openclaw` organization use the Blacksmith Ubuntu runner; forks fall back to
-GitHub-hosted `ubuntu-latest`.
+`pip install -e .`, runs `python -m pytest -q`, then builds a wheel and
+checks that runtime data such as `tasks-public/`, `profiles/`, and
+`baselines/` are included. Runs under the `openclaw` organization use the
+Blacksmith Ubuntu runner; forks fall back to GitHub-hosted `ubuntu-latest`.
 
 ## `ci-check-testbox.yml` — Blacksmith Testbox warmup
 
@@ -29,22 +28,6 @@ It installs ClawBench, hydrates provider/HF secrets into
 dotfiles from repo or org secrets, and installs
 `~/.local/bin/clawbench-testbox-env` for commands that need that live auth.
 
-## `crabbox-hydrate.yml` — Crabbox Actions hydration
-
-This workflow exists for the Crabbox CLI from `openclaw/crabbox`:
-
-```bash
-crabbox warmup --idle-timeout 90m
-crabbox actions hydrate --id <cbx_id-or-slug>
-crabbox run --id <cbx_id-or-slug> --shell -- "python -m pytest -q"
-```
-
-It runs on the dynamic self-hosted runner label registered by Crabbox, installs
-ClawBench, hydrates the same provider/HF secrets and agent dotfiles as the
-Blacksmith Testbox workflow, writes the Crabbox ready marker under
-`~/.crabbox/actions/`, and keeps the job alive for follow-up SSH sync/run
-commands.
-
 ## `sync-to-hf-space.yml` — auto-mirror main to the HF Space
 
 Mirrors every push to `main` into the HF Space git remote so

.github/workflows/ci.yml

@@ -34,13 +34,7 @@ jobs:
       - name: Install project
         run: |
           python -m pip install --upgrade pip
-          python -m pip install -e .[dev]
-
-      - name: Run static lint
-        run: python -m ruff check clawbench app.py scripts tests
-
-      - name: Run runtime contract smoke tests
-        run: python -m pytest -q tests/test_runtime_contracts.py
+          python -m pip install -e .
 
       - name: Run test suite
         run: python -m pytest -q
@@ -57,7 +51,6 @@ jobs:
               names = set(archive.namelist())
           required = [
               "tasks-public/MANIFEST.yaml",
-              "tasks-domain/MANIFEST.yaml",
               "profiles/example_research_stack.yaml",
               "baselines/BASELINE_SOURCES.md",
           ]

.github/workflows/crabbox-hydrate.yml

@@ -1,166 +0,0 @@
-name: Crabbox Hydrate
-
-on:
-  workflow_dispatch:
-    inputs:
-      crabbox_id:
-        description: "Crabbox lease ID"
-        required: true
-        type: string
-      ref:
-        description: "Git ref to hydrate"
-        required: false
-        type: string
-      crabbox_runner_label:
-        description: "Dynamic Crabbox runner label"
-        required: true
-        type: string
-      crabbox_job:
-        description: "Hydration job identifier expected by Crabbox"
-        required: false
-        default: "hydrate"
-        type: string
-      crabbox_keep_alive_minutes:
-        description: "Minutes to keep the hydrated job alive"
-        required: false
-        default: "90"
-        type: string
-
-permissions:
-  contents: read
-
-jobs:
-  hydrate:
-    name: hydrate
-    runs-on: [self-hosted, "${{ inputs.crabbox_runner_label }}"]
-    timeout-minutes: 120
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-        with:
-          ref: ${{ inputs.ref || github.ref }}
-
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: "3.12"
-          cache: pip
-
-      - name: Install project
-        run: |
-          python -m pip install --upgrade pip
-          python -m pip install -e .
-
-      - name: Prepare Crabbox shell
-        shell: bash
-        run: |
-          set -euo pipefail
-          git fetch --no-tags --depth=50 origin "+refs/heads/main:refs/remotes/origin/main"
-          python_dir="$(dirname "$(python -c 'import sys; print(sys.executable)')")"
-          sudo ln -sf "$python_dir/python" /usr/local/bin/python
-          sudo ln -sf "$python_dir/python" /usr/local/bin/python3
-          sudo ln -sf "$python_dir/pip" /usr/local/bin/pip
-          sudo ln -sf "$python_dir/pip" /usr/local/bin/pip3
-          sudo ln -sf "$python_dir/pytest" /usr/local/bin/pytest
-
-      - name: Hydrate Crabbox env helper
-        shell: bash
-        env:
-          HF_TOKEN: ${{ secrets.HF_TOKEN }}
-          HF_USERNAME: ${{ secrets.HF_USERNAME }}
-          CLAWBENCH_QUEUE_DATASET: ${{ vars.CLAWBENCH_QUEUE_DATASET || 'openclaw/clawbench-results' }}
-          CLAWBENCH_JUDGE_MODEL: ${{ vars.CLAWBENCH_JUDGE_MODEL }}
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          ANTHROPIC_API_KEY_OLD: ${{ secrets.ANTHROPIC_API_KEY_OLD }}
-          ANTHROPIC_API_TOKEN: ${{ secrets.ANTHROPIC_API_TOKEN }}
-          CEREBRAS_API_KEY: ${{ secrets.CEREBRAS_API_KEY }}
-          DEEPINFRA_API_KEY: ${{ secrets.DEEPINFRA_API_KEY }}
-          FIREWORKS_API_KEY: ${{ secrets.FIREWORKS_API_KEY }}
-          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
-          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
-          GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
-          KIMI_API_KEY: ${{ secrets.KIMI_API_KEY }}
-          MINIMAX_API_KEY: ${{ secrets.MINIMAX_API_KEY }}
-          MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
-          MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }}
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          OPENAI_BASE_URL: ${{ secrets.OPENAI_BASE_URL }}
-          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
-          QWEN_API_KEY: ${{ secrets.QWEN_API_KEY }}
-          TOGETHER_API_KEY: ${{ secrets.TOGETHER_API_KEY }}
-          XAI_API_KEY: ${{ secrets.XAI_API_KEY }}
-          ZAI_API_KEY: ${{ secrets.ZAI_API_KEY }}
-          Z_AI_API_KEY: ${{ secrets.Z_AI_API_KEY }}
-          OPENCLAW_CODEX_AUTH_JSON: ${{ secrets.OPENCLAW_CODEX_AUTH_JSON }}
-          OPENCLAW_CODEX_CONFIG_TOML: ${{ secrets.OPENCLAW_CODEX_CONFIG_TOML }}
-          OPENCLAW_CLAUDE_JSON: ${{ secrets.OPENCLAW_CLAUDE_JSON }}
-          OPENCLAW_CLAUDE_CREDENTIALS_JSON: ${{ secrets.OPENCLAW_CLAUDE_CREDENTIALS_JSON }}
-          OPENCLAW_CLAUDE_SETTINGS_JSON: ${{ secrets.OPENCLAW_CLAUDE_SETTINGS_JSON }}
-          OPENCLAW_CLAUDE_SETTINGS_LOCAL_JSON: ${{ secrets.OPENCLAW_CLAUDE_SETTINGS_LOCAL_JSON }}
-          OPENCLAW_GEMINI_SETTINGS_JSON: ${{ secrets.OPENCLAW_GEMINI_SETTINGS_JSON }}
-          CLAWBENCH_CODEX_AUTH_JSON: ${{ secrets.CLAWBENCH_CODEX_AUTH_JSON }}
-          CLAWBENCH_CODEX_CONFIG_TOML: ${{ secrets.CLAWBENCH_CODEX_CONFIG_TOML }}
-          CLAWBENCH_CLAUDE_JSON: ${{ secrets.CLAWBENCH_CLAUDE_JSON }}
-          CLAWBENCH_CLAUDE_CREDENTIALS_JSON: ${{ secrets.CLAWBENCH_CLAUDE_CREDENTIALS_JSON }}
-          CLAWBENCH_CLAUDE_SETTINGS_JSON: ${{ secrets.CLAWBENCH_CLAUDE_SETTINGS_JSON }}
-          CLAWBENCH_CLAUDE_SETTINGS_LOCAL_JSON: ${{ secrets.CLAWBENCH_CLAUDE_SETTINGS_LOCAL_JSON }}
-          CLAWBENCH_GEMINI_SETTINGS_JSON: ${{ secrets.CLAWBENCH_GEMINI_SETTINGS_JSON }}
-        run: |
-          bash scripts/ci-hydrate-testbox-env.sh
-          sudo ln -sf "$HOME/.local/bin/clawbench-testbox-env" /usr/local/bin/clawbench-testbox-env
-
-      - name: Mark Crabbox ready
-        shell: bash
-        run: |
-          set -euo pipefail
-          job="${{ inputs.crabbox_job }}"
-          if [ -z "$job" ]; then job=hydrate; fi
-          mkdir -p "$HOME/.crabbox/actions"
-          state="$HOME/.crabbox/actions/${{ inputs.crabbox_id }}.env"
-          env_file="$HOME/.crabbox/actions/${{ inputs.crabbox_id }}.env.sh"
-          services_file="$HOME/.crabbox/actions/${{ inputs.crabbox_id }}.services"
-          write_export() {
-            key="$1"
-            value="${!key-}"
-            if [ -n "$value" ]; then
-              printf 'export %s=%q\n' "$key" "$value"
-            fi
-          }
-          {
-            for key in CI GITHUB_ACTIONS GITHUB_WORKSPACE GITHUB_REPOSITORY GITHUB_RUN_ID GITHUB_RUN_NUMBER GITHUB_RUN_ATTEMPT GITHUB_REF GITHUB_REF_NAME GITHUB_SHA GITHUB_EVENT_NAME GITHUB_ACTOR RUNNER_OS RUNNER_ARCH RUNNER_TEMP RUNNER_TOOL_CACHE; do
-              write_export "$key"
-            done
-          } > "${env_file}.tmp"
-          mv "${env_file}.tmp" "$env_file"
-          {
-            echo "# Docker containers visible from the hydrated runner"
-            docker ps --format '{{.Names}}\t{{.Image}}\t{{.Ports}}' 2>/dev/null || true
-          } > "${services_file}.tmp"
-          mv "${services_file}.tmp" "$services_file"
-          tmp="${state}.tmp"
-          {
-            echo "WORKSPACE=${GITHUB_WORKSPACE}"
-            echo "RUN_ID=${GITHUB_RUN_ID}"
-            echo "JOB=${job}"
-            echo "ENV_FILE=${env_file}"
-            echo "SERVICES_FILE=${services_file}"
-            echo "READY_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
-          } > "$tmp"
-          mv "$tmp" "$state"
-
-      - name: Keep Crabbox job alive
-        shell: bash
-        run: |
-          set -euo pipefail
-          minutes="${{ inputs.crabbox_keep_alive_minutes }}"
-          case "$minutes" in
-            ''|*[!0-9]*) minutes=90 ;;
-          esac
-          stop="$HOME/.crabbox/actions/${{ inputs.crabbox_id }}.stop"
-          deadline=$(( $(date +%s) + minutes * 60 ))
-          while [ "$(date +%s)" -lt "$deadline" ]; do
-            if [ -f "$stop" ]; then
-              exit 0
-            fi
-            sleep 15
-          done

.pre-commit-config.yaml

@@ -1,16 +0,0 @@
-repos:
-  - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.14.14
-    hooks:
-      - id: ruff
-
-  - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v6.0.0
-    hooks:
-      - id: check-added-large-files
-      - id: check-case-conflict
-      - id: check-merge-conflict
-      - id: check-toml
-      - id: check-yaml
-      - id: end-of-file-fixer
-      - id: trailing-whitespace

CLAWBENCH_V0_4_SPEC.md

@@ -104,11 +104,24 @@ Each task will declare:
 - `family`
 - `surface`
 - `capabilities`
+- `category`
+- `domain`
+- `functionality`
+- `trace_distribution`
+- `tool_surface`
+- `risk_tags`
 - `pool`
 - `variant_group`
 - `official`
 - `semantic_judge`
 
+The added dimensions are flat, orthogonal leaderboard axes. They are not
+sublevels of tier or scenario, and they must not encode a specific agent
+product. The result schema aggregates scores by each axis so OpenClaw,
+Hermes, plugin-backed runs, and other third-party harnesses can compare
+the same verifier set by task mix without rewarding a harness-specific
+setup.
+
 Recommended capability tags:
 
 - `bugfix`

CONTRIBUTING.md

@@ -1,127 +0,0 @@
-# Contributing to ClawBench
-
-Thank you for your interest in contributing. This document explains how to get
-set up, what kinds of contributions are welcome, and how the review process
-works.
-
----
-
-## Getting started
-
-**Requirements:** Python 3.11+, Docker (for full end-to-end runs).
-
-```bash
-git clone https://github.com/openclaw/clawbench.git
-cd clawbench
-python -m venv .venv && source .venv/bin/activate
-python -m pip install -e ".[dev]"
-```
-
-Run the test suite to confirm everything is working:
-
-```bash
-python -m pytest -q
-python -m ruff check clawbench app.py scripts tests
-```
-
-The full local suite should pass before you make any changes.
-
----
-
-## What we welcome
-
-| Type | Notes |
-|------|-------|
-| **Bug fixes** | Include a test that reproduces the bug before the fix |
-| **New tasks** | See [Adding tasks](#adding-tasks) below |
-| **Scoring improvements** | Changes to `trajectory.py`, `scorer.py`, or `judge.py` must include updated tests and a clear rationale |
-| **Documentation** | Fixes to README, spec docs, or inline comments |
-| **Tooling / CI** | Workflow improvements, linting, dependency updates |
-
-We are unlikely to merge:
-- Large architectural rewrites without prior discussion in an issue
-- New dependencies without justification
-- Changes that reduce test coverage
-
----
-
-## Making a change
-
-1. **Open an issue first** for anything non-trivial. This lets us align on
-   approach before you invest time writing code.
-
-2. **Create a branch** from `main`:
-   ```bash
-   git checkout -b fix/short-description
-   ```
-   Branch names: `fix/`, `feat/`, `docs/`, `chore/` prefixes.
-
-3. **Write tests.** Bug fixes must include a test that fails before the fix
-   and passes after. New features must include tests covering the new
-   behaviour.
-
-4. **Run the test suite:**
-   ```bash
-   python -m pytest -q
-   ```
-
-5. **Open a pull request** against `main`. Fill in the PR template.
-
----
-
-## Adding tasks
-
-Public tasks live in `tasks-public/tier{1-5}/` as YAML files. Domain and
-partner tasks live under `tasks-domain/`. Each task needs:
-
-- A unique `id` and descriptive `name`
-- The correct `tier` (1 = simple single-tool, 5 = adversarial/multi-step)
-- `completion` checks — at least one deterministic verifier (`execution_checks`,
-  `file_equality`, or a gateway assertion)
-- `trajectory` expectations that reflect how a competent agent should approach
-  the task
-- A `judge` rubric for semantic tasks
-
-Before submitting a new task, run it against at least one agent to verify the
-completion checks fire correctly.
-
----
-
-## Commit style
-
-```
-type: short imperative summary (≤72 chars)
-
-Optional longer explanation. Wrap at 72 chars. Explain *why*, not what —
-the diff shows what changed.
-```
-
-Types: `fix`, `feat`, `docs`, `test`, `chore`, `refactor`.
-
----
-
-## Code style
-
-The project uses Ruff and pre-commit for local guardrails. Please follow the
-style of the surrounding code: 4-space indentation, descriptive variable names,
-and comments only where the logic is not self-evident.
-
-```bash
-python -m ruff check clawbench app.py scripts tests
-pre-commit run --files <changed files>
-```
-
----
-
-## Reporting bugs
-
-Use the [bug report template](.github/ISSUE_TEMPLATE/bug_report.md). Include:
-- The command you ran
-- The full error output or unexpected behaviour
-- The Python version and OS
-
----
-
-## Questions
-
-Open an issue for questions that are not bug reports or feature requests.

Dockerfile

@@ -1,8 +1,8 @@
 # ClawBench HF Docker Space
-# Layer the benchmark harness on top of a pinned OpenClaw image.
+# Layer the benchmark harness on top of the official OpenClaw image.
 
-ARG OPENCLAW_IMAGE=ghcr.io/openclaw/openclaw@sha256:2e32f4f2e4f653f12d5dc6e5c93cc71e60f49d1dfaf061b18e53c3e61a38fb48
-FROM ${OPENCLAW_IMAGE}
+ARG BASE=ghcr.io/openclaw/openclaw:latest
+FROM ${BASE}
 
 USER root
 
@@ -13,8 +13,10 @@ RUN apt-get update && \
 
 RUN ln -s /app /openclaw
 
-ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright
-RUN cd /tmp && npx -y playwright@1.59.1 install --with-deps chromium && \
+ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright \
+    NODE_PATH=/usr/local/lib/node_modules
+RUN npm install -g playwright@1.59.1 && \
+    playwright install --with-deps chromium && \
     CHROME_PATH="$(find /ms-playwright -path '*/chrome' -type f | sort | head -n 1)" && \
     test -x "$CHROME_PATH" && \
     ln -sf "$CHROME_PATH" /usr/bin/chromium
@@ -28,7 +30,6 @@ COPY --chown=node:node tasks-public/ tasks-public/
 COPY --chown=node:node tasks-domain/ tasks-domain/
 COPY --chown=node:node profiles/ profiles/
 COPY --chown=node:node baselines/ baselines/
-COPY --chown=node:node scripts/ scripts/
 COPY --chown=node:node app.py .
 
 RUN python3 -m pip install --break-system-packages --no-cache-dir .
@@ -39,7 +40,7 @@ RUN mkdir -p \
     /home/node/.openclaw/agents/dev \
     /home/node/.openclaw/agents/main/agent && \
     chown -R node:node /data /home/node/.openclaw && \
-    chmod -R 775 /data /home/node/.openclaw
+    chmod -R 777 /data /home/node/.openclaw
 
 USER node
 

Dockerfile.clawbench-426-agent-hotfix

@@ -0,0 +1,53 @@
+# ClawBench HF Docker Space with OpenClaw 2026.4.26 agent-create race hotfix.
+
+ARG BASE=openclaw-426-agent-hotfix:latest
+FROM ${BASE}
+
+USER root
+
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt-get update && \
+    apt-get install -y python3-pip python-is-python3 && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN ln -s /app /openclaw
+
+ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright \
+    NODE_PATH=/usr/local/lib/node_modules
+RUN npm install -g playwright@1.59.1 && \
+    playwright install --with-deps chromium && \
+    CHROME_PATH="$(find /ms-playwright -path '*/chrome' -type f | sort | head -n 1)" && \
+    test -x "$CHROME_PATH" && \
+    ln -sf "$CHROME_PATH" /usr/bin/chromium
+
+ENV HOME=/home/node PATH=/home/node/.local/bin:$PATH
+WORKDIR /home/node/app
+
+COPY --chown=node:node pyproject.toml README.md CLAWBENCH_V0_4_SPEC.md PARTNER_TRACE_SPEC.md ./
+COPY --chown=node:node clawbench/ clawbench/
+COPY --chown=node:node scripts/ scripts/
+COPY --chown=node:node profiles/ profiles/
+COPY --chown=node:node tasks/ tasks/
+COPY --chown=node:node tasks-public/ tasks-public/
+COPY --chown=node:node tasks-domain/ tasks-domain/
+COPY --chown=node:node baselines/ baselines/
+COPY --chown=node:node app.py .
+
+RUN python3 -m pip install --break-system-packages --no-cache-dir .
+
+RUN mkdir -p \
+    /data/results \
+    /data/queue \
+    /home/node/.openclaw/agents/dev \
+    /home/node/.openclaw/agents/main/agent && \
+    chown -R node:node /data /home/node/.openclaw && \
+    chmod -R 777 /data /home/node/.openclaw
+
+USER node
+
+ENV GATEWAY_PORT=18789
+ENV OPENCLAW_HOME=/home/node
+ENV OPENCLAW_STATE_DIR=/home/node/.openclaw
+
+EXPOSE 7860
+CMD ["python", "app.py"]

Dockerfile.gbrain

@@ -0,0 +1,113 @@
+# ClawBench + latest upstream GBrain for OpenClaw harness comparisons.
+#
+# Secrets are not baked into this image. Runtime API keys are read from the
+# mounted OpenClaw config/env by scripts/setup_gbrain_runtime.sh.
+
+ARG BASE=ghcr.io/openclaw/openclaw:latest
+FROM ${BASE}
+
+USER root
+
+ARG GBRAIN_REPO=https://github.com/garrytan/gbrain.git
+ARG GBRAIN_REF=be8fffad71ea36bc51c2d58564762b0fe271e8f4
+
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+      ca-certificates curl git jq python3-pip python-is-python3 unzip && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN ln -s /app /openclaw
+
+ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright \
+    NODE_PATH=/usr/local/lib/node_modules
+RUN npm install -g playwright@1.59.1 && \
+    playwright install --with-deps chromium && \
+    CHROME_PATH="$(find /ms-playwright -path '*/chrome' -type f | sort | head -n 1)" && \
+    test -x "$CHROME_PATH" && \
+    ln -sf "$CHROME_PATH" /usr/bin/chromium
+
+ENV BUN_INSTALL=/usr/local/bun
+RUN mkdir -p /usr/local/bun && \
+    curl -fsSL https://bun.sh/install | bash
+RUN git clone "${GBRAIN_REPO}" /opt/gbrain && \
+    cd /opt/gbrain && \
+    git checkout "${GBRAIN_REF}" && \
+    /usr/local/bun/bin/bun install --frozen-lockfile
+
+RUN mkdir -p /opt/gbrain/.codex-plugin /opt/gbrain/bin && \
+    printf '%s\n' \
+      '#!/usr/bin/env bash' \
+      'set -euo pipefail' \
+      'cd /opt/gbrain' \
+      'exec /usr/local/bun/bin/bun run src/cli.ts "$@"' \
+      > /opt/gbrain/bin/gbrain && \
+    printf '%s\n' \
+      '{' \
+      '  "id": "gbrain",' \
+      '  "name": "gbrain",' \
+      '  "description": "Personal knowledge brain with PGLite-backed CLI, skills, and MCP server",' \
+      '  "version": "0.22.6",' \
+      '  "skills": "skills",' \
+      '  "mcpServers": {' \
+      '    "gbrain": {' \
+      '      "command": "/opt/gbrain/bin/gbrain",' \
+      '      "args": ["serve"],' \
+      '      "cwd": "/opt/gbrain",' \
+      '      "connectionTimeoutMs": 120000,' \
+      '      "env": {' \
+      '        "PATH": "/opt/gbrain/bin:/usr/local/bun/bin:/usr/local/bin:/usr/bin:/bin"' \
+      '      }' \
+      '    }' \
+      '  },' \
+      '  "configSchema": {' \
+      '    "type": "object",' \
+      '    "additionalProperties": true,' \
+      '    "properties": {' \
+      '      "database_url": {"type": "string"},' \
+      '      "openai_api_key": {"type": "string"}' \
+      '    }' \
+      '  }' \
+      '}' \
+      > /opt/gbrain/.codex-plugin/plugin.json && \
+    chmod +x /opt/gbrain/bin/gbrain && \
+    ln -sf /opt/gbrain/bin/gbrain /usr/local/bin/gbrain && \
+    ln -sf /usr/local/bun/bin/bun /usr/local/bin/bun && \
+    chown -R node:node /opt/gbrain && \
+    git config --system --add safe.directory /opt/gbrain
+
+ENV PATH=/opt/gbrain/bin:/usr/local/bun/bin:/home/node/.local/bin:$PATH \
+    HOME=/home/node \
+    CLAWBENCH_ENABLE_GBRAIN=1 \
+    CLAWBENCH_LANE_PREPARE_CMD=/home/node/app/scripts/setup_gbrain_runtime.sh \
+    GBRAIN_ALLOW_SHELL_JOBS=1
+
+WORKDIR /home/node/app
+
+COPY --chown=node:node pyproject.toml README.md ./
+COPY --chown=node:node clawbench/ clawbench/
+COPY --chown=node:node tasks-public/ tasks-public/
+COPY --chown=node:node tasks-domain/ tasks-domain/
+COPY --chown=node:node baselines/ baselines/
+COPY --chown=node:node scripts/container_adapter_eval.sh scripts/container_lane_eval.sh scripts/setup_gbrain_runtime.sh scripts/
+COPY --chown=node:node app.py .
+
+RUN chmod +x scripts/container_adapter_eval.sh scripts/container_lane_eval.sh scripts/setup_gbrain_runtime.sh && \
+    python3 -m pip install --break-system-packages --no-cache-dir .
+
+RUN mkdir -p \
+    /data/results \
+    /data/queue \
+    /home/node/.openclaw/agents/dev \
+    /home/node/.openclaw/agents/main/agent && \
+    chown -R node:node /data /home/node/.openclaw && \
+    chmod -R 777 /data /home/node/.openclaw
+
+USER node
+
+ENV GATEWAY_PORT=18789
+ENV OPENCLAW_HOME=/home/node
+ENV OPENCLAW_STATE_DIR=/home/node/.openclaw
+
+EXPOSE 7860
+CMD ["python", "app.py"]

Dockerfile.main

@@ -16,8 +16,10 @@ RUN apt-get update && \
 
 RUN ln -s /app /openclaw
 
-ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright
-RUN npx -y playwright@1.59.1 install --with-deps chromium && \
+ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright \
+    NODE_PATH=/usr/local/lib/node_modules
+RUN npm install -g playwright@1.59.1 && \
+    playwright install --with-deps chromium && \
     CHROME_PATH="$(find /ms-playwright -path '*/chrome' -type f | sort | head -n 1)" && \
     test -x "$CHROME_PATH" && \
     ln -sf "$CHROME_PATH" /usr/bin/chromium

Dockerfile.openclaw-426-agent-hotfix

@@ -0,0 +1,8 @@
+FROM ghcr.io/openclaw/openclaw:2026.4.26
+
+USER root
+COPY patches/patch_openclaw_426_agent_create_queue.mjs /tmp/patch_openclaw_426_agent_create_queue.mjs
+RUN node /tmp/patch_openclaw_426_agent_create_queue.mjs && \
+    rm /tmp/patch_openclaw_426_agent_create_queue.mjs
+
+USER node

PARTNER_TRACE_SPEC.md

@@ -35,6 +35,7 @@ Each trace record should have this top-level structure:
   "plugins": [],
   "skills": [],
   "prompts": {},
+  "task_metadata": {},
   "transcript": {
     "messages": []
   },
@@ -58,6 +59,7 @@ These fields should always be present:
 - `config`: effective runtime configuration for the run
 - `plugins`: plugins or tool bundles available to the agent, even if empty
 - `prompts.user`: the user task or user-visible request
+- `task_metadata`: benchmark task axes, when the trace corresponds to a ClawBench task
 - `transcript.messages`: ordered message list for the run
 
 ## Strongly Recommended Fields
@@ -75,7 +77,28 @@ These materially improve trace quality and downstream usefulness:
 
 ## Metadata We Want
 
-### 1. Harness
+### 1. Task Metadata
+
+When a trace maps to a benchmark task, include the same flat task axes
+used by ClawBench result aggregation. These axes are intentionally
+orthogonal and harness-neutral; do not nest them under an agent product
+or plugin stack.
+
+Recommended fields:
+
+```json
+{
+  "task_id": "t4-browser-research-and-code",
+  "category": "software_engineering",
+  "domain": "devtools",
+  "functionality": ["browser_research", "api_contract_extraction", "code_repair"],
+  "trace_distribution": ["browser_heavy", "read_heavy", "edit_heavy", "execute_heavy"],
+  "tool_surface": ["browser", "filesystem", "shell", "local_service"],
+  "risk_tags": ["code_regression", "hallucination"]
+}
+```
+
+### 2. Harness
 
 Use `harness` to describe the execution framework itself.
 
@@ -95,7 +118,7 @@ Recommended fields:
 }
 ```
 
-### 2. Model
+### 3. Model
 
 Use `model` to identify the model under test.
 
@@ -111,7 +134,7 @@ Recommended fields:
 }
 ```
 
-### 3. Config
+### 4. Config
 
 Use `config` for the effective runtime settings that could change behavior.
 
@@ -134,7 +157,7 @@ Recommended fields:
 
 If a field is unavailable, omit it rather than inventing a value.
 
-### 4. Plugins
+### 5. Plugins
 
 Use `plugins` for tools, plugin bundles, MCP servers, extensions, or other agent capabilities exposed by the harness.
 
@@ -162,7 +185,7 @@ Recommended entry shape:
 }
 ```
 
-### 5. Skills
+### 6. Skills
 
 Use `skills` for reusable instruction bundles, templates, internal playbooks, or any named capability layer available to the agent.
 
@@ -186,7 +209,7 @@ Recommended entry shape:
 }
 ```
 
-### 6. Prompts
+### 7. Prompts
 
 Use `prompts` for the prompt stack that shaped agent behavior.
 
@@ -217,7 +240,7 @@ Example:
 }
 ```
 
-### 7. Transcript
+### 8. Transcript
 
 `transcript.messages` is the core behavioral record.
 

README.md

@@ -13,592 +13,197 @@ license: mit
 
 # ClawBench
 
-**Rigorous agent evaluation. Signal-curated tasks. Dynamical-systems diagnostics.**
+**Trace-scored agent evaluation for OpenClaw.**
 
 [![Python 3.11+](https://img.shields.io/badge/python-3.11+-3776AB.svg?style=flat-square)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg?style=flat-square)](LICENSE)
 [![Core v1: 19 tasks](https://img.shields.io/badge/Core%20v1-19%20tasks-blue.svg?style=flat-square)](tasks-public/)
-[![Diagnostics](https://img.shields.io/badge/diagnostics-dynamical-blueviolet.svg?style=flat-square)](#3-dynamical-systems-diagnostics-how-agents-fail-not-just-whether)
 [![HF Dataset](https://img.shields.io/badge/HF-dataset-yellow.svg?style=flat-square)](https://huggingface.co/datasets/openclaw/clawbench-results)
 
 </div>
 
 ---
 
-## What's new in Core v1 (2026-04-20)
+## What This Repo Contains
 
-A reproducibility-first public release of the benchmark, informed by a full 8-model, 1,080-run sweep audit and five new methodology layers that most agent benchmarks simply don't have:
+ClawBench evaluates AI agents by running real local tasks, capturing the
+execution trace, and scoring both the final state and the process used to get
+there.
 
-| Innovation | What it means | Why it matters |
-|---|---|---|
-| **Signal-curated task set** | 19 tasks selected from 40-task dev pool by greedy SNR-preserving elimination | Drops tasks where seed noise exceeds capability signal (21 such tasks exist in the raw 40) |
-| **Variance decomposition** | Measures and reports seed-noise vs capability-signal ratio per task | **47% of 40-task variance is seed noise** — we quantify it; most benchmarks hide it |
-| **Dynamical-systems diagnostics** | Per-run regime classification (trapped / limit-cycle / diffusive / mixed) | Reveals *how* agents fail, not just whether. Inspired by Markov-kernel / attractor-basin framework |
-| **Constraint Index C(q)** | Principled task-weighting via participation ratio + entropy + Bayes prediction | Distinguishes "everyone converges" from "everyone diverges" tasks — enables honest weighted ranking |
-| **Reproducibility-first infrastructure** | Per-container state isolation, judge-infra rejudge pipeline, documented OpenRouter-routing caveats | Eliminates the cascading-failure / silent-judge-error patterns that bias most agent benchmarks |
+The public repository contains:
 
-All of it lives in `scripts/` and `tasks-public/` — auditable code, not opaque numbers.
+- `tasks-public/`: Core v1, a 19-task public reproducibility suite.
+- `clawbench/`: the benchmark harness, adapters, canonical task conversion,
+  scoring, statistics, and diagnostics.
+- `profiles/`: example model/profile definitions.
+- `scripts/`: reusable analysis and container runner utilities.
+- `tests/`: unit and integration coverage for the public harness.
 
----
+The private holdout is intentionally not included:
 
-## The problem with every agent benchmark
+- private task YAML files,
+- private task assets and verifier scripts,
+- private expected outputs,
+- private run traces, logs, and per-task reports.
 
-You run a benchmark. Model A scores 73%. Model B scores 71%. You pick Model A.
+Internal hidden-suite runs can restore a private `tasks/` directory locally.
+The public code is designed to run without that directory by falling back to
+`tasks-public/`.
 
-Then Model A deletes your test fixtures, hallucinates that it ran `pytest` (it didn't), and confidently reports "all tests pass" while your CI is on fire. Model B would have taken 10 seconds longer but actually verified its work.
+## Core v1
 
-**The benchmark told you Model A was better. Your users would disagree.**
+Core v1 is a signal-curated 19-task public release selected from the internal
+development pool. It preserves tier and family coverage while avoiding tasks
+whose public release would leak holdout material or add mostly run-to-run
+noise.
 
-Beyond that, most benchmarks don't tell you:
-- Whether the gap is signal or noise
-- Which tasks actually discriminate models and which are coin-flips
-- How the agent *dynamically* fails — attractor, limit-cycle, goal drift
-- Whether re-running gives the same ranking (spoiler: on most benchmarks, no)
-- What's driving your score — the model, the plugin stack, or the harness version
+| Dimension | Breakdown |
+|---|---|
+| Tasks | 19 |
+| Runs per official comparison | 3 per task |
+| Total runs per model | 57 |
+| Tiers | T1=2, T2=6, T3=5, T4=5, T5=1 |
+| Families | tools=8, coding=2, repo=3, browser=2, multi_tool=3, adversarial=1 |
 
-ClawBench addresses all of this. Below is how.
-
----
-
-## What makes ClawBench different
-
-### 1. We score from execution traces, not just final output
-
-Every agent run produces a full execution trace: every tool call, every file read, every `pytest` invocation, every retry after failure. Most benchmarks throw this away and check the final state. ClawBench scores *from the trace itself*.
-
-| Axis | Weight | What it measures | Where it comes from |
-|------|--------|-----------------|-------------------|
-| **Completion** | 40% | Did the work actually get done? | Deterministic verifiers: `pytest`, exit codes, file equality, DOM assertions, memory state |
-| **Trajectory** | 30% | Did the agent work well? | Trace analysis: read-before-write ratio, self-verification, recovery after failure, tool-family fit |
-| **Behavior** | 20% | Was the agent safe and communicative? | Pattern detection: planning, progress updates, destructive command avoidance |
-| **Judge** | Advisory | Is the semantic quality good? | LLM evaluation sidecar; opt-in experimental judge-weighted scoring is gated |
-
-**The key invariant**: the LLM judge can never rescue a failed deterministic check. Official scoring keeps judge results as a sidecar signal. Experimental judge-weighted scoring must be explicitly enabled and still gates judge contribution behind deterministic completion.
-
-### 2. We measure reliability AND quantify noise
-
-A model that scores 90% on one run and 20% on the next is not a 55% model. It's an unreliable model. Users experience the worst run, not the average.
-
-ClawBench runs every task 3 times and reports:
-
-- **pass^k** — did ALL runs pass? (not just "did any run pass?")
-- **Taguchi Signal-to-Noise** — asymmetrically penalizes the worst runs, because that's what matters in production
-- **Bootstrap confidence intervals** — 10,000 resamples per task, so you know when a score difference is real vs. noise
-- **Worst-of-n** — the score that actually determines user trust
-- **13 failure modes** — `hallucinated_completion`, `tool_misuse`, `verification_skipped`, `state_regression`, `graceful_refusal`, and 8 more (not just "pass/fail")
-
-Beyond per-run reliability, we decompose **benchmark-wide variance** into seed-noise vs capability signal:
-
-```
-SNR(task) = capability_variance(across models) / mean_seed_variance(per model)
-```
-
-Findings from the v4-19-full sweep audit:
-- **Only 52.7% of run_score variance is real capability signal**; 47.3% is seed noise
-- **2 tasks have SNR ≥ 5** (reliably discriminate models)
-- **21 tasks have SNR < 1** (seed noise ≥ capability signal; rankings on these tasks are essentially random)
-
-Core v1 drops the noisy tasks and reports variance decomposition alongside rankings. This is the level of rigor most benchmarks don't attempt.
-
-### 3. Dynamical-systems diagnostics: how agents fail, not just whether
-
-Inspired by *"When LLMs Are Dreaming, Where Do They Go?"* — we treat each agent run as a stochastic trajectory in semantic state space and extract signal that flat `run_score` averages away.
-
-Current code-path formulas:
-
-```text
-Per assistant step t:
-x_t = [tool_family_proportions(6), error_flag, normalized_tokens, normalized_text_len, progress]
-drift_t = cosine_distance(x_0, x_t)
-step_t = cosine_distance(x_{t-1}, x_t)
-
-Task-level Constraint Index:
-PR(q) = tr(Σ_q)^2 / tr(Σ_q^2)
-H(q) = -Σ_i p_i log2 p_i,   p_i = λ_i / Σ_j λ_j,   λ = eigvals(Σ_q)
-BOPS(q) = mean_m mean_{i<j} cos(v_{q,m,i}, v_{q,m,j})
-C(q) = -z(PR(q)) - z(H(q)) + z(BOPS(q))
-
-Per-run constraint index used inside the regime classifier:
-PR_run = 1 / Σ_i p_i^2
-constraint_index_run = 1 - (PR_run - 1) / (d - 1)
-
-Variance decomposition:
-seed_var(q) = mean_m Var(run_score_{q,m,*})
-cap_var(q) = Var_m Mean(run_score_{q,m,*})
-SNR(q) = cap_var(q) / (seed_var(q) + 1e-9)
-capability_fraction = mean_q cap_var(q) / (mean_q cap_var(q) + mean_q seed_var(q))
-
-Survival:
-T_F = first assistant turn with empty text and no tool calls,
-      else final assistant turn if run_score < 0.7 and delivery_outcome in {fail, partial}
-S(t) = P(T_F > t)
-h(t) = P(T_F = t | T_F >= t)
-```
-
-Implemented regime classifier in `clawbench/dynamics.py`:
-
-```text
-trapped      if H_tools < 0.5 or (error_rate > 0.6 and std(drift) < 0.05)
-convergent   if std(drift_last_quartile) < 0.1 and mean(step_last_quartile) < 0.15 and error_rate < 0.2
-diffusive    if H_tools > 1.5 and error_rate < 0.15 and constraint_index_run < 0.8
-chaotic      if H_tools > 2.0 and var(step[1:]) > 0.02
-limit_cycle  if max autocorr(centered step[1:], lags 2..5) > 0.3
-unknown      otherwise, or <3 assistant turns
-```
-
-The task-level `C(q)` uses a normalized bag-of-words response vector built from the full assistant trajectory text plus tool-call names and compacted inputs, not just the last assistant turn.
-
-From the v4-19 sweep data:
-- **Gemini 3.1 Pro** exhibits `trapped` regime on 42/120 runs — commits early, doesn't iterate
-- **GPT 5.4** has the most `limit_cycle` runs (20) — tool-use loops, productive or stuck
-- **Kimi K2.5** dies at median turn 3 (worst survival); **GPT 5.4** survives to turn 8 at 60% rate (best)
-
-All scripts under `scripts/` run on cached per-run JSONs with plain numpy-based tooling; no torch or sentence-transformers required.
-
-### 4. We ablate configurations, not just models
-
-On realistic tasks, **swapping the plugin configuration produces score swings 10x larger than swapping the model**. The same Claude Sonnet can beat Claude Opus when wrapped in better tooling.
-
-If the configuration drives 10x more variance than the model, the benchmark should measure it. ClawBench's Configuration Diagnostic:
-
-1. **Fingerprint** your plugin configuration into a typed feature vector (hooks, tools, capabilities, slots)
-2. **Predict** your score before you spend a dollar on compute (k-NN over historical submissions)
-3. **Run** the benchmark and detect surprises (actual vs. predicted deltas)
-4. **Explain** which plugins are actually driving your score (fANOVA factor importance)
-5. **Recommend** specific, evidence-backed configuration changes with estimated impact
-
-No other benchmark can do this — no other benchmark has access to typed plugin manifests. OpenClaw's plugin-native architecture makes the configuration transparent, not a black box.
-
-### 5. Reproducibility-first infrastructure
-
-The v4-19-full sweep exposed multiple failure modes that silently bias numbers in other benchmarks:
-
-- **Shared state dir contamination** — accumulated `agents/` cruft across sequential sweeps caused `RPC agents.create timed out` cascades. Fixed via per-container `OPENCLAW_STATE_DIR` isolation (`scripts/container_sweep_single.sh`).
-- **Gateway judge failures** — the in-process judge returned "Gateway is restarting" / empty scores on infrastructure hiccups. Fixed via direct-API rejudge pipeline (`scripts/rejudge_all.py`).
-- **OpenRouter provider routing** — slug `z-ai/glm-5.1` canonically routes to different backing models over time. GLM 5.1 scored 0.79 at 14:00 PST, became untestable by 17:00 PST when OpenRouter repointed the slug to a reasoning-enabled variant with insufficient token budget. Numbers measured against OpenRouter-hosted models are explicitly flagged.
-- **Platform version drift** — OpenClaw 4.9 → 4.15-beta.1 shifted scores by +0.13 to +0.29 across all models. When comparing two model runs, build both against the same OpenClaw release.
-
-All of these are documented in code + commit messages. The state-isolation patch + rejudge pipeline + provider caveats turn a flaky harness into one whose drift sources are at least visible.
-
----
-
-## How trace-based scoring works
-
-Traditional benchmarks check the output: "does `output.json` match `expected.json`?" ClawBench checks the output *and* the process that produced it.
-
-### The execution trace
-
-Every tool call the agent makes is recorded with:
-- **Family classification** — `read`, `edit`, `search`, `execute`, `browser`, `memory`, `delegate`, `cron`, `plan`
-- **Mutation flag** — did this call change state?
-- **Success/failure** — and if failed, the error
-- **Output** — what the tool returned
-- **Timing** — when it happened, how long it took
-
-### What we grade from the trace
-
-**Read-before-write ratio**: Before editing a file, did the agent read it first? Agents that blind-patch without reading produce correct output ~40% of the time but break things the other 60%. The trace catches this.
-
-**Self-verification**: After making changes, did the agent run tests? A model that edits code and immediately says "done" without running `pytest` might get lucky once. It won't get lucky 3 times in a row. The trajectory score penalizes skipping verification.
-
-**Recovery patterns**: When a tool call fails, does the agent retry intelligently or loop on the same broken command? The trace reveals whether the agent actually *reasoned* about the failure.
-
-**Safety violations**: Did the agent run `rm -rf`, `git reset --hard`, `sudo`, or other destructive commands when not appropriate? These get caught and penalized, even if the final output looks fine.
-
-### Why this matters for users
-
-A user doesn't see a pass/fail. They see an agent that reads their code carefully, makes targeted changes, runs the tests, fixes what broke, and communicates what it did. Or they see an agent that blindly rewrites files and claims success. **Both might produce the same final output.** Only trace-based scoring tells them apart.
-
----
-
-## The 13 failure modes
-
-When an agent fails, "fail" is not useful information. ClawBench classifies every failure into one of 13 deterministic modes:
-
-| Mode | What happened | Example |
-|------|--------------|---------|
-| `hallucinated_completion` | Agent fabricated work it didn't do | "Tests pass!" (no tests were run) |
-| `tool_misuse` | Wrong tool or wrong arguments | Using `edit` on a file that doesn't exist |
-| `verification_skipped` | Never ran verification after changes | Edited code, skipped `pytest` |
-| `state_regression` | Environment changed unexpectedly | Background service crashed mid-run |
-| `graceful_refusal` | Correctly refused an impossible task | "This encryption cannot be reversed" |
-| `browser_navigation_failure` | Failed to reach the target page | Form server URL unreachable |
-| `memory_miss` | Failed to read/write required memory | Forgot to store context for continuation |
-| `repeated_error_loop` | Stuck retrying the same failure | Same command failed 5 times |
-| `delegation_failed` | Sub-agent spawning failed | Agent-to-agent handoff broken |
-| `unsafe_mutation` | Dangerous command executed | `rm -rf` on production directory |
-| `environment_unavailable` | Service not ready or timed out | Database not started yet |
-| `timeout` | Exceeded wall-clock budget | 600s hard limit |
-| `reward_hack_suspected` | Agent gamed the verifier | Echoed expected output instead of computing it |
-
-These are surfaced per-run in the result, not hidden in logs. They make failures *actionable*.
-
----
-
-## Core v1 task suite: 19 tasks
-
-Core v1 is a signal-curated public release of 19 tasks from the internal 40-task dev pool. Selected for:
-- **0 ranking inversions** — the mean reproduces the reference 8-model order exactly
-- **Preserved coverage** — all 5 tiers and 6 families represented
-- **Dropped noise** — excludes tasks where cross-model SNR < 0.5
-
-| Tier | Core v1 count | What it tests | Examples |
-|------|:---:|---|---|
-| **Tier 1** | 2 | Single-tool basics | Bugfix discount calc, quick file note |
-| **Tier 2** | 6 | Multi-step, 2-3 tools | Config loader repair, browser form fix, priv redaction |
-| **Tier 3** | 5 | Complex orchestration | SQL query analysis, inbox triage, data pipeline report |
-| **Tier 4** | 5 | Cross-system reasoning | Cross-repo migration, delegation repair, memory continuation, browser research+code |
-| **Tier 5** | 1 | Adversarial | Hallucination-resistant evidence |
-
-Full manifest: [`tasks-public/MANIFEST.yaml`](tasks-public/MANIFEST.yaml).
-
-### Task design principles
-
-**Intentionally vague prompts.** Users don't write numbered step lists. They say "fix the bug and make sure the tests pass." The agent has to figure out what "fix the bug" means.
-
-**Real tool composition.** Tasks require reading files, editing code, running tests, navigating browsers, querying memory, scheduling cron jobs — in combination, not isolation.
-
-**Deterministic verification.** Every task has execution-based verification: `pytest` pass, exit code check, file content match, DOM state assertion, network trace check. The LLM judge is optional and never overrides a deterministic failure.
-
-**Adversarial tier.** Tier 5 tasks are designed to test what most benchmarks can't: does the agent correctly identify when a task is impossible? Does it resist hallucinating evidence that doesn't exist? Does it handle contradictory instructions gracefully? These tasks separate models that are *capable* from models that are *trustworthy*.
-
-### Private holdout (21 tasks)
-
-The remaining 21 tasks from the internal pool stay private:
-- **9 ceiling tasks** — all frontier models score >0.85; don't discriminate at the frontier
-- **9 low-signal tasks** — SNR < 0.5; either broken verifiers or genuinely ambiguous prompts (scheduled for redesign)
-- **3 ranking-inconsistent tasks** — cross-model ordering conflicts with reference ranking (`t2-node-search-patch`, `t5-contradictory-requirements`, `t1-cal-quick-reminder`)
-
----
-
-## The scoring math
-
-### Per-run score
-```
-run_score = 0.4 * completion + 0.3 * trajectory + 0.2 * behavior + [0.1 * judge if completion >= 0.9999]
-```
-
-The judge term is gated: it only contributes when the deterministic completion score is near-perfect. You can't get a good score by producing output that *looks* right but doesn't pass execution checks.
-
-### Per-task score (across 3 runs)
-```
-task_score = 0.9 * bootstrap_mean(run_scores) + 0.1 * reliability_score
-reliability = 0.5 * pass^k + 0.3 * pass_rate + 0.2 * variance_score
-```
-
-`pass^k` is 1 only if ALL runs pass. Not any run — all runs.
-
-### Taguchi Signal-to-Noise (robustness)
-```
-S/N = -10 * log10( (1/n) * sum(1/y_i^2) )
-```
-
-The `1/y_i^2` term means the worst score dominates. A configuration scoring 0.85 average but 0.10 on adversarial tasks is **worse in production** than 0.78 average with a 0.65 floor.
-
-### SNR-weighted alternative (for ranking differentiation)
-
-Flat-mean compresses frontier model gaps. An alternative that weights tasks by their signal density:
-
-```
-w_q = max(0, SNR(q)) × |C(q)|
-w_q^wins = min(w_q, p95({w_q}))
-
-flat_score(model) = mean_q mean_run_score(model, q) over covered tasks
-weighted_score(model) = Σ_q w_q mean_run_score(model, q) / Σ_q w_q
-winsorized_score(model) = Σ_q w_q^wins mean_run_score(model, q) / Σ_q w_q^wins
-```
-
-Under SNR × |C(q)| winsorized on the same 1,080-run archive, **Opus 4.7 ranks #1** (instead of Opus 4.6 under flat mean) and **GPT 5.4 drops from #3 to #7** — its task-specific cliffs (0.16 on `t3-feature-export`) fall on the highest-signal tasks. This exposes what the flat mean averages away.
-
-Generate alternate rankings: `scripts/snr_weighted_ranking.py`.
-
----
-
-## Reproducibility caveats
-
-Being honest about what reproduces and what doesn't:
-
-### What reproduces deterministically
-
-- **Fair comparison audit** — given an archive dir, `scripts/audit_runs.py` produces identical numbers every time.
-- **Dynamical diagnostics** — C(q), regime classification, variance decomposition, survival curves: all deterministic functions of the archive.
-- **Rankings at the aggregate level** — top-cluster ranking stable across multiple sweeps when both runs use the same OpenClaw release + direct-API models.
-
-### What drifts
-
-- **Absolute scores** — seed noise is ~0.02 stddev per task per model. Expect run_score to drift within that envelope.
-- **OpenRouter-served models** — `openrouter/*` model slugs can silently re-route to different underlying providers. We observed GLM 5.1 at 0.79 then 0.33 within hours as OpenRouter flipped its backing provider. Pin to canonical versions (e.g., `z-ai/glm-5.1-20260406`) for stable measurement.
-- **OpenClaw platform drift** — 4.9 → 4.15-beta.1 shifted scores by +0.13 to +0.29 across all models. 60-70% reduction in `tool_misuse` and `verification_skipped` failure modes across that jump. Pin the base to reproduce published numbers.
-
-### Mitigating the drift
-
-Build both sides of any comparison from the same source state:
+The manifest is the source of truth:
 
 ```bash
-docker build -t clawbench .
-docker run --rm --entrypoint openclaw clawbench --version
-# -> records the OpenClaw version of THIS build
+python3 - <<'PY'
+import yaml
+manifest = yaml.safe_load(open("tasks-public/MANIFEST.yaml"))
+for task in manifest["tasks"]:
+    print(task["id"])
+PY
 ```
 
-When publishing scores, record the OpenClaw version your image
-resolved to and treat numbers from a different version as separate
-populations.
+## Scoring
 
----
+Each run is scored from four signals:
 
-## Quick start
+| Axis | Weight | What it measures |
+|---|---:|---|
+| Completion | 40% | Deterministic task checks such as tests, exact outputs, DOM assertions, and file verification |
+| Trajectory | 30% | Tool-use quality such as read-before-write, self-verification, recovery, and tool-family fit |
+| Behavior | 20% | Planning, progress updates, blocker handling, and destructive-command avoidance |
+| Judge | Up to 10% | Optional semantic quality, gated so it cannot rescue failed deterministic checks |
 
-### Build the image
+Reliability is first-class. Official comparisons run each task three times and
+report per-task variance, pass rate, pass^k, confidence intervals, and
+worst-of-n style robustness signals.
+
+## Quick Start
+
+Install locally:
 
 ```bash
-git clone git@github.com:openclaw/clawbench.git && cd clawbench
-cp .env.example .env  # optional: fill tokens for local Docker/HF uploads
-docker build -t clawbench .
-
-# Record the OpenClaw version baked in (for reproducibility):
-docker run --rm --entrypoint openclaw clawbench --version
+python3.11 -m venv .venv
+source .venv/bin/activate
+python -m pip install --upgrade pip
+python -m pip install -e .
 ```
 
-### Run Core v1 on a model
+List public tasks:
+
+```bash
+clawbench list-tasks --tasks-dir tasks-public
+```
+
+Run a small public smoke:
 
 ```bash
 export OPENCLAW_GATEWAY_TOKEN=<your-token>
 
-# Core v1 = 19 specific tasks. List them via the manifest:
-python3 -c "import yaml; m = yaml.safe_load(open('tasks-public/MANIFEST.yaml'));
-             print(' '.join(f'-t {t[\"id\"]}' for t in m['tasks']))"
+clawbench run \
+  --model anthropic/claude-opus-4-6 \
+  --runs 1 \
+  --task t1-bugfix-discount \
+  --task t1-fs-quick-note \
+  --output results/public_smoke.json
+```
+
+Run the full Core v1 task list:
+
+```bash
+TASK_ARGS=$(python3 - <<'PY'
+import yaml
+manifest = yaml.safe_load(open("tasks-public/MANIFEST.yaml"))
+print(" ".join(f"--task {task['id']}" for task in manifest["tasks"]))
+PY
+)
 
-# Then run:
 clawbench run \
   --model anthropic/claude-opus-4-6 \
   --runs 3 \
   --concurrency 4 \
-  --profile profiles/frontier_opus_4_6.yaml \
-  --judge-model anthropic/claude-sonnet-4-6 \
-  -t t1-bugfix-discount -t t1-fs-quick-note \
-  -t t2-add-tests-normalizer -t t2-browser-form-fix \
-  -t t2-config-loader -t t2-fs-find-that-thing \
-  -t t2-msg-summarize-thread -t t2-priv-redact-doc \
-  -t t3-data-pipeline-report -t t3-data-sql-query \
-  -t t3-feature-export -t t3-msg-inbox-triage \
-  -t t3-web-research-and-cite \
-  -t t4-browser-research-and-code -t t4-cross-repo-migration \
-  -t t4-delegation-repair -t t4-life-trip-plan \
-  -t t4-memory-recall-continuation \
-  -t t5-hallucination-resistant-evidence \
-  -o results/opus46_core_v1.json
+  $TASK_ARGS \
+  --output results/core_v1_opus46.json
 ```
 
-### Analyze a real archive
+Build the public Space image:
 
 ```bash
-# Fair-comparison audit
-python3 scripts/audit_runs.py
-python3 scripts/generate_fair_report.py --tag v2026-4-19-full
-
-# Posterior dynamics + ranking from cached per-run JSONs
-python3 scripts/run_posterior_dynamics_pipeline.py \
-  --archive-dir .clawbench/run_cache \
-  --reports-dir results/posterior_reports \
-  --include-dynamics-report \
-  --output-dir results/per_model_dynamics
-
-# Writes:
-#   results/posterior_reports/constraint_index.json
-#   results/posterior_reports/regimes.json
-#   results/posterior_reports/variance_decomposition.json
-#   results/posterior_reports/survival_analysis.json
-#   results/posterior_reports/snr_weighted_ranking.json
-#   results/posterior_reports/EVAL_REPORT_DYNAMICAL.md
-#   results/per_model_dynamics/<safe_model_name>/dynamics.json
-#   results/per_model_dynamics/<safe_model_name>/*.png
+docker build -t clawbench .
+docker run --rm --entrypoint openclaw clawbench --version
 ```
 
-If you only want one model's offline dynamics bundle:
+## Hidden-Suite Reproduction
+
+The hidden full-suite runner is public, but the task content is not. To rerun
+an internal hidden-suite comparison, restore the private task archive into
+`./tasks/` before building the hidden eval image. Do not commit that directory,
+its logs, or generated per-task traces.
 
 ```bash
-clawbench dynamics-report \
-  --archive-dir .clawbench/run_cache \
-  --model ollama/gpt-oss:20b \
-  --output-dir results/gptoss_dynamics
+docker build -f Dockerfile.openclaw-426-agent-hotfix \
+  -t openclaw-426-agent-hotfix:latest .
 
-# Quick CI path: skip plot rendering
-clawbench dynamics-report \
-  --archive-dir .clawbench/run_cache \
-  --model ollama/gpt-oss:20b \
-  --output-dir results/gptoss_dynamics \
-  --no-plots
-
-# Writes:
-#   results/gptoss_dynamics/dynamics.json
+docker build -f Dockerfile.clawbench-426-agent-hotfix \
+  -t clawbench-openclaw-426-agent-hotfix:latest .
 ```
 
-### Running locally with small models (Ollama)
+The public repo intentionally does not include exact private task IDs, prompts,
+assets, expected artifacts, or trace-derived private reports.
 
-A single consumer GPU running an open-weight model is enough to develop plugin profiles and validate algorithmic ideas — no API keys or cloud spend required.
+## Analysis Tools
 
-```bash
-ollama pull gpt-oss:20b
-export OPENCLAW_GATEWAY_TOKEN=<your-gateway-token>
-export CLAWBENCH_RUN_CACHE_DIR=$PWD/.clawbench/run_cache
+Reusable scripts that operate on public or private result archives:
 
-# Real benchmark run + immediate per-run dynamics bundle
-clawbench run \
-  --model ollama/gpt-oss:20b \
-  --task t1-fs-quick-note \
-  --runs 1 \
-  --dynamics \
-  -o results/ollama_smoke.json
+- `scripts/container_lane_eval.sh`: isolated OpenClaw lane runner.
+- `scripts/container_adapter_eval.sh`: adapter/model runner for fair adapter comparisons.
+- `scripts/run_posterior_dynamics_pipeline.py`: one-shot offline dynamics analysis.
+- `scripts/compute_constraint_index.py`: task-level constraint index.
+- `scripts/variance_decomp.py`: seed-noise vs capability-signal decomposition.
+- `scripts/survival_analysis.py`: per-turn failure survival curves.
+- `scripts/snr_weighted_ranking.py`: SNR-weighted ranking.
 
-# Optional second local model
-ollama pull qwen3.5:27b
+Generated data, traces, and reports are local artifacts and are ignored by Git.
 
-# Offline posterior analysis reads CLAWBENCH_RUN_CACHE_DIR
-python3 scripts/run_posterior_dynamics_pipeline.py \
-  --archive-dir .clawbench/run_cache \
-  --reports-dir results/posterior_reports
+## Repository Layout
 
-clawbench diagnose profiles/local_ollama_gpt_oss.yaml
-```
-
-### Running on Kubernetes
-
-See [`docs/kubernetes.md`](docs/kubernetes.md) for the full runbook. The short
-version:
-
-```bash
-export CLAWBENCH_NAMESPACE=clawbench-eval
-export OPENAI_API_KEY="sk-..."       # or ANTHROPIC_API_KEY, OPENROUTER_API_KEY, etc.
-export CLAWBENCH_MODEL="openai/gpt-5.5"
-# export MLFLOW_NAMESPACE="mlflow"   # MLflow deploys in a separate namespace (default: mlflow)
-
-./scripts/k8s/deploy.sh              # deploys OpenClaw + MLflow + starts eval
-./scripts/k8s/deploy.sh --logs       # follow progress
-./scripts/k8s/deploy.sh --teardown   # tear down openclaw & eval (does not delete MLflow)
-```
-
-API keys are stored in a Kubernetes Secret created by the deploy script.
-MLflow is deployed in its own namespace (default: `mlflow`, configurable via
-`MLFLOW_NAMESPACE`).
-
----
-
-## Partner Trace Spec
-
-ClawBench defines a [JSONL interchange format](PARTNER_TRACE_SPEC.md) for agent execution traces. If you're building an agent framework and want your runs scored by ClawBench, you don't need to integrate with OpenClaw — you just emit traces in this format.
-
-The trace captures:
-- **Harness provenance** — git SHA, container image digest, runtime version
-- **Full tool-call sequence** — family, arguments, output, success/failure, timing
-- **Token accounting** — input, output, reasoning, cache tokens per message
-- **Artifacts** — final files, test results, command outputs
-- **Redaction metadata** — what was removed for privacy, so scoring can account for it
-
-This means ClawBench scores are **reproducible** across different harness implementations, and **auditable** down to individual tool calls.
-
----
-
-## Repository layout
-
-```
+```text
 clawbench/
-├── clawbench/                      # Core package
-│   ├── scorer.py                   # 4-axis scoring with gated judge
-│   ├── trajectory.py               # Trace-based process quality grading
-│   ├── environment.py              # 5 deterministic verifier types
-│   ├── judge.py                    # LLM judge (gated, never rescues failures)
-│   ├── harness.py                  # Benchmark orchestration + parallel lanes
-│   ├── schemas.py                  # 13-mode failure taxonomy + result schemas
-│   ├── stats.py                    # Bootstrap CI + Taguchi S/N
-│   ├── profile.py                  # v0.5 plugin fingerprinting
-│   ├── diagnostic.py               # Configuration Diagnostic report
-│   ├── factor_analysis.py          # fANOVA factor importance
-│   ├── dynamics.py                 # Trajectory metrics + sensitivity analysis
-│   ├── dynamics_archive.py         # Cached-run loading + offline report assembly
-│   ├── dynamics_plots.py           # Offline dynamics visualizations
-│   └── cli.py                      # CLI entry points
-│
-├── tasks-public/                   # Core v1 PUBLIC release (19 tasks)
-│   ├── MANIFEST.yaml               # Task list + reference ranking + metadata
-│   ├── README.md                   # Rationale, build + run instructions
-│   ├── tier1/ ... tier5/           # 19 task YAMLs with verification specs
-│   └── assets/                     # 19 asset packs (verifiers + fixtures)
-│
-├── tasks-domain/                   # Planned domain coverage scaffold
-│
-├── tasks/                          # PRIVATE 40-task dev pool (gitignored)
-│
-├── scripts/                        # Reproducibility + analysis pipeline
-│   ├── container_sweep_single.sh   # Per-container OPENCLAW_STATE_DIR isolation
-│   ├── audit_runs.py               # Aggregate coverage + fair-comparison audit
-│   ├── audit_per_run.py            # Per-run cross-model audit
-│   ├── rejudge_all.py              # Direct-API rejudge for broken gateway judges
-│   ├── generate_fair_report.py     # Fair N-model comparison report
-│   ├── run_posterior_dynamics_pipeline.py # One-shot posterior analysis driver
-│   ├── compute_constraint_index.py # C(q) per task
-│   ├── classify_regimes.py         # Per-run dynamical regime classifier
-│   ├── variance_decomp.py          # Seed-noise vs capability-signal decomposition
-│   ├── survival_analysis.py        # Per-turn failure survival curves
-│   ├── snr_weighted_ranking.py     # SNR × |C(q)|-weighted ranking
-│   └── generate_dynamical_report.py # Combined dynamical-systems report
-│
-├── profiles/                       # v0.5 plugin profile YAMLs
-├── tests/                          # Test suite
-├── Dockerfile                      # Layered on a pinned ghcr.io/openclaw/openclaw image
-├── CLAWBENCH_V0_4_SPEC.md          # Full specification
-└── PARTNER_TRACE_SPEC.md           # Trace interchange format
+├── clawbench/                  # Harness, adapters, scoring, diagnostics
+├── tasks-public/               # Core v1 public task suite
+├── tasks-domain/               # Domain expansion scaffold
+├── profiles/                   # Model/profile definitions
+├── scripts/                    # Reusable runners and offline analysis
+├── tests/                      # Public test suite
+├── Dockerfile                  # Public HF Space image
+├── Dockerfile.main             # Main-variant public image
+├── Dockerfile.openclaw-426-agent-hotfix
+├── Dockerfile.clawbench-426-agent-hotfix
+├── CLAWBENCH_V0_4_SPEC.md
+└── PARTNER_TRACE_SPEC.md
 ```
 
----
-
-## How ClawBench compares
-
-|  | ClawBench | SWE-bench | HumanEval | LLM-judge leaderboards |
-|---|---|---|---|---|
-| **Scores process, not just output** | ✓ Trace-based trajectory + behavior | No | No | No |
-| **Reliability as first-class metric** | ✓ pass^k, Taguchi S/N, bootstrap CI | Single pass rate | pass@k | Best-of-n |
-| **Variance decomposition reported** | ✓ seed-noise vs capability-signal ratio | No | No | No |
-| **Per-run dynamical regime** | ✓ trapped / cycle / diffusive | No | No | No |
-| **SNR-weighted alternative ranking** | ✓ principled task weighting | No | No | No |
-| **Failure taxonomy** | ✓ 13 deterministic modes | Binary pass/fail | Binary | None |
-| **LLM judge role** | Capped 10%, gated on deterministic floor | Not used | Not used | Primary scorer |
-| **Configuration diagnostics** | ✓ Fingerprint, predict, explain, recommend | No | No | No |
-| **State-isolation per run** | ✓ per-container OPENCLAW_STATE_DIR | No | No | No |
-| **Multiple runs per task** | 3 runs mandatory, statistical tests | Usually 1 | Varies | Usually 1 |
-| **Provider-routing caveats** | ✓ documented (OpenRouter drift) | Not flagged | Not flagged | Not flagged |
-| **Real tool composition** | ✓ Browser + code + memory + cron + delegation | Code only | Code only | Varies |
-
----
-
 ## Testing
 
 ```bash
 python -m pytest -q
 ```
 
-Key test invariants:
-- Judge never rescues failed deterministic completion (`test_scorer.py`)
-- Parallel lanes are isolated (`test_parallel_harness.py`)
-- Bootstrap CIs are statistically valid (`test_e2e_significance.py`)
-- fANOVA factor importance converges (`test_v05_framework.py`)
-
----
-
-## Version log
-
-| Version | Date | Summary |
-|:---:|---|---|
-| **Core v1** | 2026-04-20 | 19-task signal-curated public release; dynamical-systems diagnostics (C(q), regimes, survival, SNR-weighted); per-container state isolation; rejudge pipeline |
-| v0.5 | earlier | Configuration Diagnostic (fingerprint, predict, fANOVA); plugin-native ablation |
-| v0.4 | earlier | 4-axis scoring with gated judge; 13-mode failure taxonomy; Partner Trace Spec |
-
-Planned for Core v2:
-- **Tier 6 long-horizon tasks** (100+ turn runs) — unlock real Lyapunov / attractor measurement
-- **Paraphrased prompt pairs** — enable perturbation-sensitivity ranking
-- **Creative-synthesis tasks** — currently absent from Core v1
-- **Human-performance baseline** on 10 tasks — calibrate difficulty
-
----
+The test suite includes public-surface checks to keep the README and Space
+description aligned with `tasks-public/MANIFEST.yaml`.
 
 ## License
 
@@ -614,13 +219,3 @@ MIT. See `LICENSE`.
   url    = {https://github.com/openclaw/clawbench}
 }
 ```
-
----
-
-<div align="center">
-
-**ClawBench** — Rigorous. Reproducible. Dynamical.
-
-[Dataset](https://huggingface.co/datasets/openclaw/clawbench-results) · [Space](https://huggingface.co/spaces/openclaw/clawbench) · [Core v1](tasks-public/) · [Spec](CLAWBENCH_V0_4_SPEC.md)
-
-</div>

SPACE_README.md

@@ -13,188 +13,70 @@ license: mit
 
 Execution-first benchmark for AI models acting as OpenClaw agents.
 
-This Space evaluates models on realistic local agent tasks and scores them with a deterministic pipeline that emphasizes:
-
-- **Completion**: did the work actually pass executable checks?
-- **Trajectory**: did the agent explore, recover, and use tools well?
-- **Behavior**: did the transcript show planning, progress updates, and safe handling?
-- **Reliability**: was performance stable across repeated runs?
-
-## Why this benchmark exists
-
-ClawBench is built to avoid three common benchmark failures:
-
-1. trusting what the agent said instead of running the work,
-2. rewarding one reference trajectory instead of rewarding good agent properties,
-3. hiding instability by reporting only one lucky run.
-
-## Benchmark shape
+## Benchmark Shape
 
 ```text
-tasks          : 20
+public suite   : Core v1
+tasks          : 19
+runs/model     : 57 for official Core v1 comparisons
 tiers          : 5
-prompt modes   : clear + ambiguous on every task
 browser tasks  : 2
-multi-phase    : 1
-judge-enabled  : 6 advisory tasks
-primary metric : pass^k
+primary metric : trace-scored task score plus reliability
 ```
 
-### Tier mix
-
-```text
-tier1 | ###   3
-tier2 | ##### 5
-tier3 | ##### 5
-tier4 | ####  4
-tier5 | ###   3
-```
-
-### Family mix
-
-```text
-repo        | ###### 6
-coding      | ####   4
-multi_tool  | ###    3
-adversarial | ###    3
-browser     | ##     2
-tools       | ##     2
-```
-
-## Official score stack
-
-Per-run score:
-
-```text
-normalize(0.4 * completion + 0.3 * trajectory + 0.2 * behavior)
-```
-
-Per-task score after repeated runs:
-
-```text
-0.9 * mean_run_score + 0.1 * reliability_score
-```
-
-Reliability:
-
-```text
-0.5 * pass_hat_k + 0.3 * pass_rate + 0.2 * variance_score
-```
-
-## What gets verified
+## What Gets Scored
 
 | Layer | Verification style |
-| --- | --- |
-| Completion | `pytest`, `node --test`, exact output checks, browser flow checks, cron checks, memory checks, gateway assertions |
-| Trajectory | read-before-write, self-verification, recovery quality, tool-family fit, safety rules |
-| Behavior | deterministic transcript rules for planning, progress, blocker handling, refusal quality, destructive-command avoidance |
+|---|---|
+| Completion | `pytest`, exact output checks, browser flow checks, file checks, and verifier scripts |
+| Trajectory | read-before-write, self-verification, recovery quality, tool-family fit, and safety rules |
+| Behavior | deterministic transcript checks for planning, progress, blockers, and safe handling |
 | Reliability | repeated runs with pass^k, pass rate, and score variance |
 
-The official score stays deterministic.
+The advisory judge is optional and cannot replace deterministic verification.
 
-Optional advisory judge results are reported separately and never replace executable verification.
-
-## Runtime flow
+## Runtime Flow
 
 ```text
 task yaml + assets
   -> isolated workspace
   -> optional local background services
-  -> OpenClaw agent session(s)
+  -> OpenClaw agent session
   -> transcript + tool-result capture
   -> completion / trajectory / behavior scoring
-  -> repeated runs
   -> reliability aggregation
-  -> leaderboard result
 ```
 
-## Browser policy
+## Public Task Inventory
 
-Browser tasks in this Space are deterministic and local:
+The Space uses `tasks-public/MANIFEST.yaml` as the source of truth. Current
+Core v1 tasks are:
 
-```text
-task-owned local app or docs
-  -> OpenClaw browser tool
-  -> real browser interaction
-  -> deterministic local verification
-```
+| Task | Tier | Family |
+|---|---|---|
+| `t1-bugfix-discount` | tier1 | coding |
+| `t1-fs-quick-note` | tier1 | tools |
+| `t2-add-tests-normalizer` | tier2 | coding |
+| `t2-browser-form-fix` | tier2 | browser |
+| `t2-config-loader` | tier2 | repo |
+| `t2-fs-find-that-thing` | tier2 | tools |
+| `t2-msg-summarize-thread` | tier2 | tools |
+| `t2-priv-redact-doc` | tier2 | tools |
+| `t3-data-pipeline-report` | tier3 | multi_tool |
+| `t3-data-sql-query` | tier3 | tools |
+| `t3-feature-export` | tier3 | repo |
+| `t3-msg-inbox-triage` | tier3 | tools |
+| `t3-web-research-and-cite` | tier3 | tools |
+| `t4-browser-research-and-code` | tier4 | browser |
+| `t4-cross-repo-migration` | tier4 | repo |
+| `t4-delegation-repair` | tier4 | multi_tool |
+| `t4-life-trip-plan` | tier4 | tools |
+| `t4-memory-recall-continuation` | tier4 | multi_tool |
+| `t5-hallucination-resistant-evidence` | tier5 | adversarial |
 
-No public websites are used for official browser tasks.
+## Holdout Policy
 
-## Parallel Space runtime
-
-On upgraded CPU Spaces, the worker can use conservative parallel lanes:
-
-```text
-submission
-  -> task partitioner
-  -> lane 1 gateway + lane-local state
-  -> lane 2 gateway + lane-local state
-  -> browser lane gateway + lane-local state
-  -> merged benchmark result
-```
-
-Important rule: browser tasks stay serialized on one dedicated lane to avoid Chromium and port-range collisions.
-
-## Submission presets
-
-The Submit tab now exposes two preset audiences so the Space can serve both general Claw users and lower-budget exploratory runs:
-
-- `Claw Users` keeps the full preset catalog, including provider-backed frontier models.
-- `Budget Researchers` narrows the list to local or lower-cost presets such as `ollama/gpt-oss:20b`, `ollama/qwen3.5:27b`, `huggingface/Qwen/Qwen3-32B`, and `huggingface/google/gemma-4-26B-A4B-it`.
-
-You can still enter any custom model ID directly; the preset audience only filters the shortcut catalog and the bulk-submit action.
-
-## Task inventory
-
-| Task | Tier | Family | Main verification |
-| --- | --- | --- | --- |
-| `t1-architecture-brief` | tier1 | tools | fact verifier + smoke command |
-| `t1-bugfix-discount` | tier1 | coding | `pytest` |
-| `t1-refactor-csv-loader` | tier1 | coding | `pytest` + verification script |
-| `t2-add-tests-normalizer` | tier2 | coding | `pytest` + added-test checks |
-| `t2-browser-form-fix` | tier2 | browser | local browser flow verification |
-| `t2-config-loader` | tier2 | repo | `pytest` |
-| `t2-log-analyzer-cli` | tier2 | coding | exact JSON output |
-| `t2-node-search-patch` | tier2 | repo | `node --test` |
-| `t3-data-pipeline-report` | tier3 | multi_tool | exact report output |
-| `t3-debug-timezone-regression` | tier3 | repo | `pytest` |
-| `t3-feature-export` | tier3 | repo | `pytest` + CLI smoke |
-| `t3-monitoring-automation` | tier3 | tools | script output + cron state |
-| `t3-node-multifile-refactor` | tier3 | repo | `node --test` |
-| `t4-browser-research-and-code` | tier4 | browser | browser evidence + tests |
-| `t4-cross-repo-migration` | tier4 | repo | both test suites pass |
-| `t4-delegation-repair` | tier4 | multi_tool | final suite + delegation transcript evidence |
-| `t4-memory-recall-continuation` | tier4 | multi_tool | tests + memory assertions |
-| `t5-contradictory-requirements` | tier5 | adversarial | latest-instruction artifact checks |
-| `t5-hallucination-resistant-evidence` | tier5 | adversarial | exact answer + evidence-first checks |
-| `t5-impossible-graceful-fail` | tier5 | adversarial | no harmful mutation + clear refusal |
-
-## Query coverage layer
-
-The benchmark also carries dataset-backed metadata from a spreadsheet-derived query corpus:
-
-- scenario-domain mapping,
-- clear vs ambiguous prompt slices,
-- pass / partial / fail delivery buckets,
-- weighted query-score reporting.
-
-This lets the benchmark report both:
-
-- how strong a model is,
-- and what parts of the user-query landscape the suite is actually stressing.
-
-## What makes ClawBench meaningful now
-
-- execution-based completion checks instead of file-exists-only scoring
-- property-based trajectory scoring instead of reference-trace matching
-- deterministic local browser tasks instead of internet targets
-- repeated-run reliability instead of one-shot success stories
-- tiered tasks with delegation, memory, browser, repo, and adversarial surfaces
-- advisory judge support without making the official score depend on a second model
-
-## Auth model
-
-The benchmark does not require a separate scorer or user-simulation API key.
-
-It uses the model-under-test auth already configured for OpenClaw. If you enable the optional advisory judge, that model can reuse the same general auth path if available.
+Private task bodies, assets, expected outputs, verifier details, run traces,
+logs, and per-task private reports are not part of the public Space. Public
+Core v1 is intended for reproducibility and development; hidden-suite runs use
+the same harness with a private task directory restored locally.

app.py

@@ -17,8 +17,6 @@ import json
 import logging
 import os
 import threading
-import time
-from dataclasses import dataclass, field
 from pathlib import Path
 
 import gradio as gr
@@ -28,7 +26,6 @@ from clawbench.hub import (
     load_submission_rows_from_parquet,
     resolve_dataset_repo,
 )
-from clawbench.queue import JobQueue, SubmissionRequest
 from clawbench.submission_models import (
     build_preset_submission_specs,
     CUSTOM_PRESET_LABEL,
@@ -36,6 +33,7 @@ from clawbench.submission_models import (
     PRESET_AUDIENCE_CHOICES,
     PRESET_MODEL_MAP,
     preset_labels_for_audience,
+    preset_models_for_audience,
     resolve_model_selection,
 )
 
@@ -48,16 +46,6 @@ HF_DATASET_TOKEN = os.environ.get("HF_TOKEN", "")
 HF_DATASET_REPO = resolve_dataset_repo(HF_DATASET_TOKEN)
 
 
-@dataclass
-class _LeaderboardCache:
-    lock: threading.Lock = field(default_factory=threading.Lock)
-    loaded_at: float = 0.0
-    frame: pd.DataFrame | None = None
-
-
-_LEADERBOARD_CACHE = _LeaderboardCache()
-
-
 def _env_int(name: str, default: int, *, minimum: int, maximum: int) -> int:
     raw = os.environ.get(name, "").strip()
     if not raw:
@@ -74,12 +62,14 @@ MAX_RUNS_PER_SUBMISSION = _env_int("CLAWBENCH_MAX_RUNS_PER_SUBMISSION", 3, minim
 MAX_LANES_PER_SUBMISSION = _env_int("CLAWBENCH_MAX_LANES_PER_SUBMISSION", 4, minimum=1, maximum=8)
 DEFAULT_RUNS_PER_TASK = _env_int("CLAWBENCH_DEFAULT_RUNS_PER_TASK", 3, minimum=1, maximum=MAX_RUNS_PER_SUBMISSION)
 DEFAULT_PARALLEL_LANES = _env_int("CLAWBENCH_DEFAULT_PARALLEL_LANES", 1, minimum=1, maximum=MAX_LANES_PER_SUBMISSION)
-LEADERBOARD_CACHE_SECONDS = _env_int("CLAWBENCH_LEADERBOARD_CACHE_SECONDS", 60, minimum=0, maximum=3600)
 ENABLE_BULK_SUBMIT = os.environ.get("CLAWBENCH_ENABLE_BULK_SUBMIT", "").strip().lower() in {"1", "true", "yes", "on"}
-JUDGE_AFFECTS_SCORE = os.environ.get("CLAWBENCH_JUDGE_AFFECTS_SCORE", "").strip().lower() in {"1", "true", "yes", "on"}
 
 # ---------------------------------------------------------------------------
 # Background worker (starts in a thread)
+# ---------------------------------------------------------------------------
+
+from clawbench.queue import JobQueue, SubmissionRequest
+
 queue = JobQueue()
 
 
@@ -106,24 +96,6 @@ logger.info("Background eval worker started")
 
 
 def load_leaderboard() -> pd.DataFrame:
-    now = time.monotonic()
-    with _LEADERBOARD_CACHE.lock:
-        if (
-            _LEADERBOARD_CACHE.frame is not None
-            and LEADERBOARD_CACHE_SECONDS > 0
-            and now - _LEADERBOARD_CACHE.loaded_at < LEADERBOARD_CACHE_SECONDS
-        ):
-            return _LEADERBOARD_CACHE.frame.copy()
-
-    frame = _load_leaderboard_uncached()
-    if LEADERBOARD_CACHE_SECONDS > 0:
-        with _LEADERBOARD_CACHE.lock:
-            _LEADERBOARD_CACHE.loaded_at = time.monotonic()
-            _LEADERBOARD_CACHE.frame = frame.copy()
-    return frame.copy()
-
-
-def _load_leaderboard_uncached() -> pd.DataFrame:
     rows = []
 
     # Load from HF Dataset via direct parquet reads. This avoids
@@ -292,7 +264,6 @@ def submit_model(
         model=model_id,
         provider=provider_id,
         judge_model=judge_model.strip(),
-        judge_affects_score=JUDGE_AFFECTS_SCORE,
         runs_per_task=int(runs),
         max_parallel_lanes=int(max_parallel_lanes),
         tier=selected_tier,
@@ -342,7 +313,6 @@ def submit_all_presets(
     submitted = []
     blocked = []
     for preset, request_kwargs in preset_specs:
-        request_kwargs["judge_affects_score"] = JUDGE_AFFECTS_SCORE
         request = SubmissionRequest(**request_kwargs)
         try:
             job = asyncio.run(queue.submit(request))

clawbench/adapters/hermes.py

@@ -68,8 +68,7 @@ def _load_mini_swe_runner() -> tuple[Any, Exception | None]:
         from mini_swe_runner import MiniSWERunner as runner_cls  # type: ignore[import-not-found]
 
         return runner_cls, None
-    except Exception as exc:  # pragma: no cover - import-guard branch
-        import_error = exc
+    except Exception as import_exc:  # pragma: no cover - import-guard branch
         candidates: list[Path] = []
         explicit_file = os.environ.get("HERMES_MINI_SWE_RUNNER")
         if explicit_file:
@@ -99,9 +98,9 @@ def _load_mini_swe_runner() -> tuple[Any, Exception | None]:
                 spec.loader.exec_module(module)
                 return module.MiniSWERunner, None
             except Exception as path_exc:
-                import_error = path_exc
+                import_exc = path_exc
                 continue
-        return None, import_error
+        return None, import_exc
 
 
 MiniSWERunner, _HERMES_IMPORT_ERROR = _load_mini_swe_runner()
@@ -112,8 +111,7 @@ def _load_ai_agent() -> tuple[Any, Exception | None]:
         from run_agent import AIAgent as agent_cls  # type: ignore[import-not-found]
 
         return agent_cls, None
-    except Exception as exc:  # pragma: no cover - import-guard branch
-        import_error = exc
+    except Exception as import_exc:  # pragma: no cover - import-guard branch
         candidates: list[Path] = []
         for env_name in ("HERMES_AGENT_REPO", "HERMES_INSTALL_DIR"):
             value = os.environ.get(env_name)
@@ -140,9 +138,9 @@ def _load_ai_agent() -> tuple[Any, Exception | None]:
                 spec.loader.exec_module(module)
                 return module.AIAgent, None
             except Exception as path_exc:
-                import_error = path_exc
+                import_exc = path_exc
                 continue
-        return None, import_error
+        return None, import_exc
 
 
 AIAgent, _HERMES_AGENT_IMPORT_ERROR = _load_ai_agent()

clawbench/adapters/openclaw.py

@@ -8,9 +8,8 @@ simulated-user turns, and resolves `StateQuery` assertions against the
 gateway's `memory.search` / `sessions.resolve` / `cron.list` / arbitrary
 `_rpc(method)` surface.
 
-The legacy harness still owns the executable CLI path for now; this
-adapter is the canonical wrapper used by adapter-level tests and later
-harness wiring.
+The benchmark harness now routes OpenClaw through this adapter, matching
+the same canonical task/run lifecycle used by other harness adapters.
 """
 
 from __future__ import annotations
@@ -19,6 +18,8 @@ import json
 import logging
 import uuid
 from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
 
 from clawbench.adapters import register_adapter
 from clawbench.adapters.base import (
@@ -35,12 +36,16 @@ from clawbench.canonical import (
 )
 from clawbench.client import GatewayClient, GatewayConfig
 from clawbench.environment_files import (
+    memory_visible_in_transcript,
     resolve_json_path,
     verify_memory_fallback,
 )
 from clawbench.schemas import (
+    CronState,
     MemoryState,
     PromptVariant,
+    SessionState,
+    Transcript,
 )
 from clawbench.session_labels import unique_session_label
 from clawbench.simulated_user import UserSimulator

clawbench/cli.py

@@ -11,9 +11,21 @@ import click
 
 from clawbench.client import GatewayConfig
 from clawbench.harness import BenchmarkHarness, KNOWN_ADAPTERS
-from clawbench.schemas import ScenarioDomain
 
-SCENARIO_CHOICES = [scenario.value for scenario in ScenarioDomain]
+SCENARIO_CHOICES = [
+    "file_system_ops",
+    "web_info_ops",
+    "calendar_reminders",
+    "communication_messaging",
+    "data_processing_analysis",
+    "coding_dev_assist",
+    "personal_life_assistant",
+    "multi_step_compound",
+    "context_continuation",
+    "error_boundary_cases",
+    "skill_calling",
+    "system_capabilities",
+]
 
 
 @click.group()
@@ -34,22 +46,23 @@ def cli(verbose: bool) -> None:
     type=click.Choice(KNOWN_ADAPTERS),
     default="openclaw",
     show_default=True,
-    help="Agent harness adapter. OpenClaw is executable today; other adapters are tracked targets.",
+    help="Agent harness adapter. OpenClaw uses the gateway; Hermes runs hermes-agent locally.",
 )
 @click.option("--gateway-token", envvar="OPENCLAW_GATEWAY_TOKEN", default="", help="Gateway auth token")
+@click.option(
+    "--gateway-url",
+    envvar="OPENCLAW_GATEWAY_URL",
+    default="ws://localhost:18789",
+    show_default=True,
+    help="OpenClaw gateway websocket URL",
+)
 @click.option(
     "--judge-model",
     envvar="CLAWBENCH_JUDGE_MODEL",
     default="",
     help="Optional advisory LLM judge model (does not affect official score)",
 )
-@click.option(
-    "--judge-affects-score",
-    is_flag=True,
-    envvar="CLAWBENCH_JUDGE_AFFECTS_SCORE",
-    help="Opt in to experimental judge-weighted scoring. Official scoring keeps judge advisory.",
-)
-@click.option("--runs", "-n", default=3, show_default=True, help="Runs per task (reliability uses all runs)")
+@click.option("--runs", "-n", default=5, help="Runs per task (reliability uses all runs)")
 @click.option("--tier", type=click.Choice(["tier1", "tier2", "tier3", "tier4", "tier5"]), help="Filter tier")
 @click.option("--scenario", type=click.Choice(SCENARIO_CHOICES), help="Filter query scenario")
 @click.option("--artifact-type", type=click.Choice(["file", "information", "operation", "code", "external_action", "memory", "automation", "mixed"]), help="Filter expected artifact type")
@@ -110,6 +123,11 @@ def cli(verbose: bool) -> None:
          "completes the v0.5 Configuration Diagnostic Report is generated and "
          "the run is recorded in the historical profile database.",
 )
+@click.option(
+    "--tool-profile",
+    default=None,
+    help="Optional label for the tool/profile axis recorded in result metadata.",
+)
 @click.option(
     "--insights-dir",
     type=click.Path(path_type=Path),
@@ -126,8 +144,8 @@ def run(
     model: str,
     adapter: str,
     gateway_token: str,
+    gateway_url: str,
     judge_model: str,
-    judge_affects_score: bool,
     runs: int,
     tier: str | None,
     scenario: str | None,
@@ -144,16 +162,16 @@ def run(
     concurrency: int,
     browser_concurrency: int,
     profile: Path | None,
+    tool_profile: str | None,
     insights_dir: Path,
     dynamics: bool,
 ) -> None:
-    gateway_config = GatewayConfig(token=gateway_token)
+    gateway_config = GatewayConfig(url=gateway_url, token=gateway_token)
     harness = BenchmarkHarness(
         gateway_config=gateway_config,
         model=model,
         adapter=adapter,
         judge_model=judge_model,
-        judge_affects_score=judge_affects_score,
         runs_per_task=runs,
         tier=tier,
         scenario=scenario,
@@ -167,6 +185,7 @@ def run(
         randomize_order=not no_randomize,
         concurrency=concurrency,
         browser_concurrency=browser_concurrency,
+        tool_profile_name=tool_profile,
     )
 
     result = asyncio.run(harness.run())
@@ -194,6 +213,40 @@ def run(
         asyncio.run(upload_result(result))
 
 
+@cli.command("compare-results")
+@click.argument("results", nargs=-1, type=click.Path(exists=True, path_type=Path), required=True)
+@click.option("--json-out", is_flag=True, help="Print machine-readable comparison JSON.")
+def compare_results_cmd(results: tuple[Path, ...], json_out: bool) -> None:
+    """Compare BenchmarkResult JSON files with fairness checks."""
+    from clawbench.ablation import compare_results
+    from clawbench.schemas import BenchmarkResult
+
+    loaded: dict[str, BenchmarkResult] = {}
+    for path in results:
+        with path.open(encoding="utf-8") as handle:
+            loaded[path.stem] = BenchmarkResult(**json.load(handle))
+    comparison = compare_results(loaded)
+    if json_out:
+        click.echo(json.dumps(comparison, indent=2, default=str))
+        return
+
+    click.echo(f"Task/verifier fair: {comparison['task_verifier_fair']}")
+    click.echo(f"Controlled ablation: {comparison['controlled_ablation']}")
+    click.echo(f"Same model: {comparison['same_model']}")
+    click.echo(f"Same task set: {comparison['same_task_set']}")
+    click.echo(f"Same task snapshot: {comparison['same_task_snapshot']}")
+    click.echo(f"Same prompt variant: {comparison['same_prompt_variant']}")
+    for label, row in comparison["rows"].items():
+        click.echo(
+            f"{label}: model={row['model']} adapter={row['adapter']} "
+            f"tasks={row['task_count']} score={row['score']:.3f} "
+            f"C={row['completion']:.3f} T={row['trajectory']:.3f} "
+            f"B={row['behavior']:.3f} R={row['reliability']:.3f}"
+        )
+    for label, delta in comparison["deltas"].items():
+        click.echo(f"{label}: {delta:+.3f}")
+
+
 @cli.command("dynamics-report")
 @click.option(
     "--archive-dir",
@@ -793,6 +846,20 @@ def show(result_file: str) -> None:
     )
     console.print(f"  [bold]pass^k reliability: {result.overall_pass_hat_k:.0%}[/]\n")
 
+    for label, dimension_items in (
+        ("Category", result.category_results),
+        ("Domain", result.domain_results),
+    ):
+        if not dimension_items:
+            continue
+        summary = ", ".join(
+            f"{item.value}={item.weighted_score:.3f}"
+            for item in sorted(dimension_items, key=lambda item: item.value)
+        )
+        console.print(f"  [bold]{label}:[/] {summary}")
+    if result.category_results or result.domain_results:
+        console.print()
+
     for task in result.task_results:
         color = "green" if task.mean_task_score >= 0.7 else "yellow" if task.mean_task_score >= 0.4 else "red"
         top_failure = max(task.failure_mode_counts.items(), key=lambda item: item[1])[0] if task.failure_mode_counts else "-"

clawbench/client.py

@@ -232,6 +232,14 @@ class GatewayClient:
                     max_size=10 * 1024 * 1024,
                     open_timeout=attempt_timeout,
                     additional_headers={"Origin": host},
+                    # The benchmark uses loopback gateway sockets and can issue
+                    # long-lived RPCs (notably agent.wait while a provider call
+                    # is in flight). Python websockets' default keepalive can
+                    # close the connection before the gateway surfaces the
+                    # actual model/provider result, contaminating runs as infra
+                    # timeouts. The gateway already owns run-level timeouts.
+                    ping_interval=None,
+                    ping_timeout=None,
                 )
                 self._listen_task = asyncio.create_task(self._listener())
                 challenge = await self._wait_event(
@@ -578,17 +586,14 @@ class GatewayClient:
         effective_timeout = timeout if timeout is not None else self.config.request_timeout
         future: asyncio.Future[dict[str, Any]] = asyncio.get_running_loop().create_future()
         self._pending[request_id] = future
+        await self._ws.send(json.dumps(frame))
         try:
-            await self._ws.send(json.dumps(frame))
             response = await asyncio.wait_for(future, timeout=effective_timeout)
         except asyncio.TimeoutError:
             self._pending.pop(request_id, None)
             raise TimeoutError(
                 f"RPC {method} timed out after {effective_timeout:.1f}s"
             )
-        except Exception:
-            self._pending.pop(request_id, None)
-            raise
 
         if not response.get("ok", False):
             error = response.get("error", {})

clawbench/diagnostic.py

@@ -17,13 +17,16 @@ leaderboards.
 
 from __future__ import annotations
 
+import json
 from dataclasses import dataclass, field, asdict
+from pathlib import Path
 from typing import Any
 
 from clawbench.factor_analysis import FactorAnalysisReport, analyze
 from clawbench.prediction import (
     HistoricalDatabase,
     HistoricalRun,
+    PredictionReport,
     attribute_surprise,
     predict_profile,
 )

clawbench/environment.py

@@ -1,18 +1,44 @@
-"""Completion verification for ClawBench v0.3."""
+"""Completion verification — OpenClaw-aware entry point.
+
+Historically this module contained both agent-agnostic verification
+primitives (file states, execution checks, workspace memory scans, JSON
+path resolution) and OpenClaw-specific verifiers that reach into the
+gateway via RPCs (`memory.search`, `sessions.resolve`, `cron.list`,
+arbitrary `_rpc(method)`).
+
+Phase-4 splits them:
+
+- The agent-agnostic primitives now live in `clawbench.environment_files`
+  and are used by every adapter.
+- The OpenClaw-specific primitives stay here for now and will move into
+  `clawbench/adapters/openclaw.py` once the adapter wiring lands in a
+  later step.
+
+The public surface — `verify_completion`, `run_execution_check`, module-
+level helpers — stays unchanged so existing callers (harness, scorer,
+tests) keep working. Function bodies that used to do real work now
+delegate to `environment_files` to keep behavior identical.
+"""
 
 from __future__ import annotations
 
-import asyncio
 import json
 import logging
 import re
-import shlex
 from pathlib import Path
 from typing import Any
 
 from clawbench.client import GatewayClient
-from clawbench.paths import resolve_workspace_path
-from clawbench.render import render_template, render_value
+from clawbench.environment_files import (
+    MEMORY_FILE_CANDIDATES,
+    evaluate_execution_result as _evaluate_execution_result_impl,
+    memory_visible_in_transcript as _memory_visible_in_transcript_impl,
+    read_workspace_memory_text,
+    resolve_json_path,
+    run_execution_check as _run_execution_check_impl,
+    verify_file_state as _verify_file_state_impl,
+    verify_memory_fallback,
+)
 from clawbench.schemas import (
     CompletionResult,
     CompletionSpec,
@@ -53,7 +79,9 @@ async def verify_completion(
             failures.append(f"FILE {spec.path}: {reason}")
 
     for spec in completion.memory:
-        ok, reason = await _verify_memory(spec, client, session_key, agent_id=agent_id, transcript=transcript)
+        ok, reason = await _verify_memory(
+            spec, client, session_key, agent_id=agent_id, transcript=transcript, workspace=workspace
+        )
         total += 1
         if ok:
             passed += 1
@@ -103,95 +131,20 @@ async def verify_completion(
     )
 
 
+# ---------------------------------------------------------------------------
+# Agent-agnostic primitives — re-exported via delegates so historical
+# callers that import from `clawbench.environment` keep working.
+# ---------------------------------------------------------------------------
+
+
 async def run_execution_check(
     spec: ExecutionCheck,
     *,
     workspace: Path,
     runtime_values: dict[str, Any],
 ) -> ExecutionCheckResult:
-    rendered_command = render_template(spec.command, runtime_values)
-    try:
-        rendered_cwd = resolve_workspace_path(
-            workspace,
-            render_template(spec.cwd, runtime_values),
-            field=f"execution check cwd for {spec.name}",
-        )
-    except ValueError as exc:
-        return ExecutionCheckResult(
-            name=spec.name,
-            command=rendered_command,
-            exit_code=-1,
-            passed=False,
-            reason=str(exc),
-        )
-    rendered_env = render_value(spec.env, runtime_values)
-    import os
-    import sys
-
-    full_env = {
-        **os.environ,
-        **{key: str(value) for key, value in rendered_env.items()},
-        "PYTHONUNBUFFERED": "1",
-    }
-    python_bin_dir = str(Path(sys.executable).parent)
-    full_env["PATH"] = f"{python_bin_dir}:{full_env.get('PATH', '')}"
-    python_path_parts = [str(rendered_cwd), str(workspace)]
-    existing_pythonpath = full_env.get("PYTHONPATH")
-    if existing_pythonpath:
-        python_path_parts.append(existing_pythonpath)
-    full_env["PYTHONPATH"] = ":".join(python_path_parts)
-
-    try:
-        if spec.shell:
-            process = await asyncio.create_subprocess_shell(
-                rendered_command,
-                cwd=str(rendered_cwd),
-                env=full_env,
-                stdout=asyncio.subprocess.PIPE,
-                stderr=asyncio.subprocess.PIPE,
-            )
-        else:
-            process = await asyncio.create_subprocess_exec(
-                *shlex.split(rendered_command),
-                cwd=str(rendered_cwd),
-                env=full_env,
-                stdout=asyncio.subprocess.PIPE,
-                stderr=asyncio.subprocess.PIPE,
-            )
-        stdout_bytes, stderr_bytes = await asyncio.wait_for(
-            process.communicate(),
-            timeout=spec.timeout_seconds,
-        )
-    except asyncio.TimeoutError:
-        process.kill()
-        await process.communicate()
-        return ExecutionCheckResult(
-            name=spec.name,
-            command=rendered_command,
-            exit_code=-1,
-            passed=False,
-            reason=f"Timed out after {spec.timeout_seconds}s",
-        )
-    except Exception as exc:
-        return ExecutionCheckResult(
-            name=spec.name,
-            command=rendered_command,
-            exit_code=-1,
-            passed=False,
-            reason=str(exc),
-        )
-
-    stdout = stdout_bytes.decode("utf-8", errors="replace")
-    stderr = stderr_bytes.decode("utf-8", errors="replace")
-    passed, reason = _evaluate_execution_result(spec, workspace, runtime_values, process.returncode, stdout, stderr)
-    return ExecutionCheckResult(
-        name=spec.name,
-        command=rendered_command,
-        exit_code=process.returncode,
-        stdout=stdout,
-        stderr=stderr,
-        passed=passed,
-        reason=reason,
+    return await _run_execution_check_impl(
+        spec, workspace=workspace, runtime_values=runtime_values
     )
 
 
@@ -203,113 +156,27 @@ def _evaluate_execution_result(
     stdout: str,
     stderr: str,
 ) -> tuple[bool, str]:
-    if exit_code != spec.expected_exit_code:
-        return False, f"Exit code {exit_code} != expected {spec.expected_exit_code}"
-
-    for token in spec.stdout_contains:
-        rendered = render_template(token, runtime_values)
-        if rendered not in stdout:
-            return False, f"stdout missing '{rendered}'"
-
-    for token in spec.stdout_not_contains:
-        rendered = render_template(token, runtime_values)
-        if rendered in stdout:
-            return False, f"stdout unexpectedly contains '{rendered}'"
-
-    for token in spec.stderr_contains:
-        rendered = render_template(token, runtime_values)
-        if rendered not in stderr:
-            return False, f"stderr missing '{rendered}'"
-
-    if spec.stdout_matches and not re.search(render_template(spec.stdout_matches, runtime_values), stdout, re.MULTILINE | re.DOTALL):
-        return False, f"stdout does not match {spec.stdout_matches}"
-
-    if spec.stderr_matches and not re.search(render_template(spec.stderr_matches, runtime_values), stderr, re.MULTILINE | re.DOTALL):
-        return False, f"stderr does not match {spec.stderr_matches}"
-
-    if spec.expected_stdout is not None:
-        rendered = render_template(spec.expected_stdout, runtime_values).strip()
-        if stdout.strip() != rendered:
-            return False, "stdout did not match expected text"
-
-    if spec.expected_stdout_file:
-        try:
-            expected_path = resolve_workspace_path(
-                workspace,
-                render_template(spec.expected_stdout_file, runtime_values),
-                field=f"expected_stdout_file for {spec.name}",
-            )
-        except ValueError as exc:
-            return False, str(exc)
-        if stdout.strip() != expected_path.read_text(encoding="utf-8").strip():
-            return False, f"stdout did not match {spec.expected_stdout_file}"
-
-    if spec.expected_json is not None:
-        try:
-            parsed = json.loads(stdout)
-        except json.JSONDecodeError as exc:
-            return False, f"stdout was not valid JSON: {exc}"
-        if parsed != render_value(spec.expected_json, runtime_values):
-            return False, "stdout JSON did not match expected JSON"
-
-    if spec.expected_json_file:
-        try:
-            expected_path = resolve_workspace_path(
-                workspace,
-                render_template(spec.expected_json_file, runtime_values),
-                field=f"expected_json_file for {spec.name}",
-            )
-        except ValueError as exc:
-            return False, str(exc)
-        try:
-            parsed = json.loads(stdout)
-        except json.JSONDecodeError as exc:
-            return False, f"stdout was not valid JSON: {exc}"
-        expected_json = json.loads(expected_path.read_text(encoding="utf-8"))
-        if parsed != expected_json:
-            return False, f"stdout JSON did not match {spec.expected_json_file}"
-
-    return True, "OK"
+    return _evaluate_execution_result_impl(
+        spec, workspace, runtime_values, exit_code, stdout, stderr
+    )
 
 
 def _verify_file(spec: FileState, workspace: Path, runtime_values: dict[str, Any]) -> tuple[bool, str]:
-    try:
-        path = resolve_workspace_path(
-            workspace,
-            render_template(spec.path, runtime_values),
-            field=f"completion file {spec.path}",
-        )
-    except ValueError as exc:
-        return False, str(exc)
-    exists = path.exists() and path.is_file()
+    return _verify_file_state_impl(spec, workspace, runtime_values)
 
-    if not spec.exists:
-        return (not exists, "Correctly absent" if not exists else "File should not exist")
-    if not exists:
-        return False, "File does not exist"
 
-    content = path.read_text(encoding="utf-8", errors="replace")
-    if spec.min_size_bytes > 0 and path.stat().st_size < spec.min_size_bytes:
-        return False, f"File too small: {path.stat().st_size} < {spec.min_size_bytes}"
+def _memory_visible_in_transcript(spec: MemoryState, transcript: Transcript) -> bool:
+    return _memory_visible_in_transcript_impl(spec, transcript)
 
-    for token in spec.content_contains:
-        rendered = render_template(token, runtime_values)
-        if rendered not in content:
-            return False, f"Missing expected content '{rendered}'"
 
-    for token in spec.content_not_contains:
-        rendered = render_template(token, runtime_values)
-        if rendered in content:
-            return False, f"Contains forbidden content '{rendered}'"
+def _resolve_path(payload: Any, path: str) -> Any:
+    return resolve_json_path(payload, path)
 
-    if spec.content_matches and not re.search(
-        render_template(spec.content_matches, runtime_values),
-        content,
-        re.MULTILINE | re.DOTALL,
-    ):
-        return False, f"Content does not match {spec.content_matches}"
 
-    return True, "OK"
+# ---------------------------------------------------------------------------
+# OpenClaw-tied verifiers. These call `GatewayClient` RPCs; they will
+# migrate into `adapters/openclaw.py` once the adapter wiring lands.
+# ---------------------------------------------------------------------------
 
 
 async def _verify_memory(
@@ -319,6 +186,7 @@ async def _verify_memory(
     *,
     agent_id: str | None = None,
     transcript: Transcript | None = None,
+    workspace: Path | None = None,
 ) -> tuple[bool, str]:
     try:
         response = await client._rpc(
@@ -340,16 +208,42 @@ async def _verify_memory(
                 return False, f"Memory value missing '{token}'"
         return True, "OK"
     except Exception as exc:
-        logger.info("memory.search unavailable for verification, falling back to agent memory files: %s", exc)
+        logger.info(
+            "memory.search unavailable for verification, falling back to agent memory files: %s",
+            exc,
+        )
+
+    # Fallback path: pull the same set of memory files the agent would
+    # produce (MEMORY.md, memory/notes.md, …) via the gateway, then hand
+    # the resulting text to the shared filesystem-fallback resolver in
+    # `environment_files`. If no gateway is available (agent_id is None
+    # or the calls error) and a workspace was supplied, fall back further
+    # to scanning the workspace filesystem directly.
+
+    extra_memory_text = ""
+    if agent_id:
+        try:
+            extra_memory_text = await _read_agent_memory_text(client, agent_id)
+        except Exception:
+            extra_memory_text = ""
+
+    if workspace is not None:
+        return verify_memory_fallback(
+            spec,
+            workspace,
+            transcript=transcript,
+            extra_memory_text=extra_memory_text,
+        )
 
     if not agent_id:
         return False, "memory.search unavailable and no agent id was provided for fallback verification"
 
-    fallback_text = await _read_agent_memory_text(client, agent_id)
-    normalized = fallback_text.lower()
+    # Legacy pre-workspace path: agent_id is set but we don't have a
+    # workspace handle. Resolve using only the gateway-sourced text +
+    # transcript scan to preserve the exact prior behavior.
+    normalized = extra_memory_text.lower()
     needle = spec.key_pattern.lower()
     found = needle in normalized
-
     if not spec.exists:
         return (not found, "Correctly absent" if not found else "Memory entry exists")
     if found:
@@ -357,23 +251,17 @@ async def _verify_memory(
             if token.lower() not in normalized:
                 return False, f"Memory value missing '{token}'"
         return True, "OK"
-
     if transcript and _memory_visible_in_transcript(spec, transcript):
         return True, "Verified from transcript fallback"
-    return False, "No matching memory content found in persisted memory files or transcript fallback"
+    return (
+        False,
+        "No matching memory content found in persisted memory files or transcript fallback",
+    )
 
 
 async def _read_agent_memory_text(client: GatewayClient, agent_id: str) -> str:
     contents: list[str] = []
-    for file_name in (
-        "MEMORY.md",
-        "memory.md",
-        "memory/MEMORY.md",
-        "memory/memory.md",
-        "memory/notes.md",
-        "memory/NOTES.md",
-        "notes.md",
-    ):
+    for file_name in MEMORY_FILE_CANDIDATES:
         try:
             payload = await client.get_agent_file(agent_id, file_name)
         except Exception:
@@ -385,30 +273,6 @@ async def _read_agent_memory_text(client: GatewayClient, agent_id: str) -> str:
     return "\n".join(contents)
 
 
-def _memory_visible_in_transcript(spec: MemoryState, transcript: Transcript) -> bool:
-    needle = spec.key_pattern.lower()
-    for call in transcript.tool_call_sequence:
-        family = (call.family or "").lower()
-        name = call.name.lower()
-        path = str(call.input.get("path", "")).lower()
-        if family != "memory" and "memory" not in path:
-            continue
-        if family == "memory" and "search" in name and "write" not in name and "store" not in name and "save" not in name:
-            continue
-
-        serialized_bits = [call.output, call.error]
-        try:
-            serialized_bits.append(json.dumps(call.input, sort_keys=True))
-        except TypeError:
-            serialized_bits.append(str(call.input))
-        haystack = " ".join(bit for bit in serialized_bits if bit).lower()
-        if needle not in haystack:
-            continue
-        if all(token.lower() in haystack for token in spec.value_contains):
-            return True
-    return False
-
-
 async def _verify_session(
     spec: SessionState,
     client: GatewayClient,
@@ -439,8 +303,7 @@ async def _verify_cron(spec: CronState, client: GatewayClient) -> tuple[bool, st
         if not jobs:
             return False, "No cron jobs found"
         if spec.description_contains and not any(
-            spec.description_contains.lower() in json.dumps(job).lower()
-            for job in jobs
+            spec.description_contains.lower() in json.dumps(job).lower() for job in jobs
         ):
             return False, f"No cron job matched '{spec.description_contains}'"
         return True, "OK"
@@ -455,7 +318,7 @@ async def _verify_gateway_assertion(
     try:
         response = await client._rpc(spec.method, spec.params)
         payload = response.get("payload", {})
-        value = _resolve_path(payload, spec.assert_path)
+        value = resolve_json_path(payload, spec.assert_path)
         if not spec.assert_exists:
             return (value is None, "Correctly absent" if value is None else "Path exists")
         if value is None:
@@ -469,28 +332,13 @@ async def _verify_gateway_assertion(
         return False, str(exc)
 
 
-def _resolve_path(payload: Any, path: str) -> Any:
-    if path == "$":
-        return payload
-    current = payload
-    for part in path.lstrip("$").lstrip(".").split("."):
-        if not part:
-            continue
-        match = re.fullmatch(r"([^\[]+)\[(\d+)\]", part)
-        if match:
-            key, index = match.groups()
-            if not isinstance(current, dict) or key not in current:
-                return None
-            current = current[key]
-            if not isinstance(current, list):
-                return None
-            idx = int(index)
-            if idx >= len(current):
-                return None
-            current = current[idx]
-            continue
-        if isinstance(current, dict) and part in current:
-            current = current[part]
-            continue
-        return None
-    return current
+# Backward-compatible names for any external users that imported the
+# private delegates directly. The old symbols resolve to the new ones.
+_verify_file_state = _verify_file
+_verify_execution = _evaluate_execution_result_impl
+
+
+__all__ = [
+    "run_execution_check",
+    "verify_completion",
+]

clawbench/environment_files.py

@@ -24,7 +24,6 @@ import sys
 from pathlib import Path
 from typing import Any
 
-from clawbench.paths import resolve_workspace_path
 from clawbench.render import render_template, render_value
 from clawbench.schemas import (
     ExecutionCheck,
@@ -49,14 +48,7 @@ def verify_file_state(
 ) -> tuple[bool, str]:
     """Verify a single `FileState` against the workspace filesystem."""
 
-    try:
-        path = resolve_workspace_path(
-            workspace,
-            render_template(spec.path, runtime_values),
-            field=f"completion file {spec.path}",
-        )
-    except ValueError as exc:
-        return False, str(exc)
+    path = workspace / render_template(spec.path, runtime_values)
     exists = path.exists() and path.is_file()
 
     if not spec.exists:
@@ -102,20 +94,7 @@ async def run_execution_check(
     """Run a single `ExecutionCheck` subprocess and evaluate its output."""
 
     rendered_command = render_template(spec.command, runtime_values)
-    try:
-        rendered_cwd = resolve_workspace_path(
-            workspace,
-            render_template(spec.cwd, runtime_values),
-            field=f"execution check cwd for {spec.name}",
-        )
-    except ValueError as exc:
-        return ExecutionCheckResult(
-            name=spec.name,
-            command=rendered_command,
-            exit_code=-1,
-            passed=False,
-            reason=str(exc),
-        )
+    rendered_cwd = workspace / render_template(spec.cwd, runtime_values)
     rendered_env = render_value(spec.env, runtime_values)
 
     full_env = {
@@ -231,14 +210,7 @@ def evaluate_execution_result(
             return False, "stdout did not match expected text"
 
     if spec.expected_stdout_file:
-        try:
-            expected_path = resolve_workspace_path(
-                workspace,
-                render_template(spec.expected_stdout_file, runtime_values),
-                field=f"expected_stdout_file for {spec.name}",
-            )
-        except ValueError as exc:
-            return False, str(exc)
+        expected_path = workspace / render_template(spec.expected_stdout_file, runtime_values)
         if stdout.strip() != expected_path.read_text(encoding="utf-8").strip():
             return False, f"stdout did not match {spec.expected_stdout_file}"
 
@@ -251,14 +223,7 @@ def evaluate_execution_result(
             return False, "stdout JSON did not match expected JSON"
 
     if spec.expected_json_file:
-        try:
-            expected_path = resolve_workspace_path(
-                workspace,
-                render_template(spec.expected_json_file, runtime_values),
-                field=f"expected_json_file for {spec.name}",
-            )
-        except ValueError as exc:
-            return False, str(exc)
+        expected_path = workspace / render_template(spec.expected_json_file, runtime_values)
         try:
             parsed = json.loads(stdout)
         except json.JSONDecodeError as exc:

clawbench/factor_analysis.py

@@ -29,7 +29,7 @@ when data volume permits.
 
 from __future__ import annotations
 
-from dataclasses import dataclass, asdict
+from dataclasses import dataclass, field, asdict
 from itertools import combinations
 
 from clawbench.prediction import HistoricalDatabase
@@ -199,6 +199,7 @@ def _analyze_lite(
     main_effects.sort(key=lambda m: m.importance, reverse=True)
 
     # Pairwise interactions (only the top-k by absolute residual)
+    me_lookup = {m.feature: m for m in main_effects}
     candidates = [m.feature for m in main_effects[:20]]  # cap to prevent explosion
     interactions: list[InteractionImportance] = []
     for fa, fb in combinations(candidates, 2):
@@ -271,6 +272,7 @@ def _analyze_random_forest(
         for j, fname in enumerate(all_features):
             X[i, j] = 1.0 if feats.get(fname, False) else 0.0
 
+    grand_mean = float(y.mean())
     total_variance = float(y.var(ddof=1)) if n_samples > 1 else 0.0
     if total_variance < 1e-9:
         return FactorAnalysisReport(

clawbench/harness.py

@@ -5,26 +5,38 @@ from __future__ import annotations
 import asyncio
 import datetime
 import hashlib
-import json
 import logging
 import os
 import shutil
+import subprocess
 import time
 import uuid
 from collections.abc import Awaitable, Callable
 from pathlib import Path
 from typing import Any
+from urllib.parse import urlparse
 
 from rich.console import Console
 from rich.table import Table
 
 from clawbench import __version__
-from clawbench.ablation import build_ablation_profile
+from clawbench.ablation import build_ablation_profile, git_head
+from clawbench.adapters import get_adapter
+from clawbench.adapters.base import AdapterContext
+from clawbench.adapters.hermes import HermesAdapterConfig
+from clawbench.adapters.openclaw import OpenClawAdapterConfig
+from clawbench.canonical.convert import from_task_definition
 from clawbench.client import GatewayClient, GatewayConfig
+from clawbench.environment_files import run_execution_check, verify_file_state
+from clawbench.judge import judge_task_run
 from clawbench.releases import compute_task_snapshot_fingerprint, load_active_release
 from clawbench.schemas import (
     BenchmarkResult,
+    CompletionResult,
+    DimensionResult,
     DeliveryOutcome,
+    EfficiencyResult,
+    JudgeResult,
     ScenarioResult,
     TaskDefinition,
     TaskRunResult,
@@ -32,19 +44,38 @@ from clawbench.schemas import (
     TierResult,
     Transcript,
 )
-from clawbench.scorer import classify_error_failure_mode, score_task_run
-from clawbench.session_labels import unique_session_label
+from clawbench.scorer import (
+    classify_delivery_outcome,
+    classify_error_failure_mode,
+    classify_failure_mode,
+    combine_run_score,
+    evaluate_behavior,
+)
 from clawbench.services import build_runtime_values, start_background_services, stop_background_services
-from clawbench.simulated_user import UserSimulator
 from clawbench.stats import bootstrap_ci, summarize_task_runs
 from clawbench.tasks import get_assets_dir, load_all_tasks
+from clawbench.trajectory import annotate_transcript_tool_calls, evaluate_trajectory
 
 logger = logging.getLogger(__name__)
 console = Console()
 
 KNOWN_ADAPTERS = ("openclaw", "hermes", "codex", "claude-code")
-EXECUTABLE_ADAPTERS = {"openclaw"}
-RUN_CACHE_SCHEMA_VERSION = 2
+EXECUTABLE_ADAPTERS = {"openclaw", "hermes"}
+
+
+def _command_version(command: list[str]) -> str:
+    try:
+        result = subprocess.run(
+            command,
+            check=False,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT,
+            text=True,
+            timeout=5,
+        )
+    except Exception:
+        return ""
+    return (result.stdout or "").strip().splitlines()[0] if result.stdout else ""
 
 
 class _NullCtx:
@@ -86,10 +117,7 @@ class BenchmarkHarness:
         concurrency: int = 1,
         browser_concurrency: int = 1,
         adapter: str = "openclaw",
-        judge_affects_score: bool = False,
         tool_profile_name: str | None = None,
-        enabled_toolsets: list[str] | None = None,
-        disabled_toolsets: list[str] | None = None,
     ) -> None:
         self.gateway_config = gateway_config
         self.model = model
@@ -101,7 +129,6 @@ class BenchmarkHarness:
         self.artifact_type = artifact_type
         self.prompt_variant = prompt_variant
         self.judge_model = judge_model
-        self.judge_affects_score = judge_affects_score
         self.pool = pool
         self.subsets = subsets or []
         self.capabilities = capabilities or []
@@ -116,8 +143,6 @@ class BenchmarkHarness:
         self.browser_concurrency = max(1, int(browser_concurrency))
         self.adapter = adapter
         self.tool_profile_name = tool_profile_name
-        self.enabled_toolsets = enabled_toolsets or []
-        self.disabled_toolsets = disabled_toolsets or []
         self.repo_root = Path(__file__).parent.parent
         self.last_task_runs: dict[str, list[TaskRunResult]] = {}
 
@@ -147,6 +172,8 @@ class BenchmarkHarness:
         if not tasks:
             raise ValueError("No tasks to run")
 
+        tasks = self._filter_tasks_for_adapter(tasks)
+
         if self.randomize_order:
             import random
 
@@ -272,65 +299,168 @@ class BenchmarkHarness:
             console.print(f"    [red]! {failure}[/]")
 
     async def _run_single(self, task: TaskDefinition, run_index: int) -> TaskRunResult:
-        # Per-turn timeout cap: prevents a single send_and_wait from burning the entire task
-        # timeout (often 300-600s). Default 180s is enough for any reasonable single-turn
-        # response and fails fast on stuck models. Override with env var if needed.
-        per_turn_cap = float(os.environ.get("CLAWBENCH_PER_TURN_TIMEOUT_SECONDS", "180"))
-        # Per-run hard budget: total wall time a single (task, run) is allowed to consume.
-        # Default 300s (5 min) bounds the worst case to 5min * 120 = 10h/model if fully
-        # serial, and <3h/model at lanes=4. Env override available for longer slower models.
-        per_run_budget = float(os.environ.get("CLAWBENCH_PER_RUN_BUDGET_SECONDS", "300"))
+        return await self._run_single_with_agent_adapter(task, run_index)
 
-        # Per-run result cache: allows a failed job to resume from previously completed
-        # (task, run) pairs on resubmit. Keyed by model + task + run_index so the same
-        # model's runs are reused, but different models stay isolated. The cache is
-        # written AFTER successful score_task_run and read at the start of this method.
-        # Set CLAWBENCH_RUN_CACHE_DIR="" to disable.
+    def _filter_tasks_for_adapter(self, tasks: list[TaskDefinition]) -> list[TaskDefinition]:
+        """Drop tasks the selected adapter cannot execute."""
+
+        adapter_cls = get_adapter(self.adapter)
+        adapter_config = self._adapter_config()
+        compatible: list[TaskDefinition] = []
+        skipped: list[tuple[str, str]] = []
+        for task in tasks:
+            canonical = from_task_definition(task)
+            missing = adapter_cls.missing_capabilities_for(canonical, adapter_config)
+            if missing:
+                skipped.append((task.id, ", ".join(sorted(cap.value for cap in missing))))
+                continue
+            compatible.append(task)
+
+        if skipped and not self.quiet:
+            console.print(
+                f"[yellow]Adapter '{self.adapter}' skipped {len(skipped)} incompatible task(s).[/]"
+            )
+            for task_id, caps in skipped[:5]:
+                console.print(f"    [yellow]- {task_id}: missing {caps}[/]")
+            if len(skipped) > 5:
+                console.print(f"    [yellow]- ... {len(skipped) - 5} more[/]")
+
+        if not compatible:
+            raise ValueError(
+                f"No selected tasks are compatible with adapter '{self.adapter}'. "
+                "Try a files/execution task such as t1-bugfix-discount, or use adapter 'openclaw'."
+            )
+        return compatible
+
+    def _adapter_config(self) -> object:
+        if self.adapter == "openclaw":
+            per_turn_cap = float(os.environ.get("CLAWBENCH_PER_TURN_TIMEOUT_SECONDS", "180"))
+            return OpenClawAdapterConfig(
+                gateway=self.gateway_config,
+                prompt_variant=self.prompt_variant,
+                turn_timeout_seconds=per_turn_cap,
+            )
+        if self.adapter == "hermes":
+            provider = os.environ.get("HERMES_PROVIDER") or None
+            base_url = os.environ.get("HERMES_BASE_URL") or None
+            api_mode = os.environ.get("HERMES_API_MODE") or None
+            api_key = (
+                os.environ.get("HERMES_API_KEY")
+                or os.environ.get("OPENROUTER_API_KEY")
+                or os.environ.get("OPENAI_API_KEY")
+                or None
+            )
+            if provider:
+                base_url = None
+                api_key = None
+            elif provider is None and self.model.startswith("openai/"):
+                base_url = (
+                    base_url
+                    or os.environ.get("OPENAI_BASE_URL")
+                    or ("https://api.openai.com/v1" if os.environ.get("OPENAI_API_KEY") else None)
+                )
+                host = ""
+                try:
+                    host = urlparse(base_url or "").hostname or ""
+                except Exception:
+                    host = ""
+                if host == "api.openai.com":
+                    api_key = os.environ.get("OPENAI_API_KEY") or os.environ.get("HERMES_API_KEY") or None
+                    if api_mode is None and self.model.split("/", 1)[1].lower().startswith("gpt-5"):
+                        api_mode = "codex_responses"
+            elif provider is None and self.model.startswith("anthropic/"):
+                provider = "anthropic"
+                base_url = None
+                api_key = None
+            elif (
+                base_url is None
+                and os.environ.get("OPENAI_API_KEY")
+                and not os.environ.get("HERMES_API_KEY")
+                and not os.environ.get("OPENROUTER_API_KEY")
+            ):
+                base_url = "https://api.openai.com/v1"
+            enabled_toolsets = [
+                item.strip()
+                for item in os.environ.get("HERMES_TOOLSETS", "hermes-api-server").split(",")
+                if item.strip()
+            ]
+            disabled_toolsets = [
+                item.strip()
+                for item in os.environ.get("HERMES_DISABLED_TOOLSETS", "").split(",")
+                if item.strip()
+            ] or None
+            return HermesAdapterConfig(
+                model=self.model,
+                env_type=os.environ.get("HERMES_ENV_TYPE", "local"),
+                max_iterations=int(os.environ.get("HERMES_MAX_ITERATIONS", "15")),
+                timeout_seconds=int(os.environ.get("HERMES_STEP_TIMEOUT_SECONDS", "60")),
+                base_url=base_url,
+                api_key=api_key,
+                provider=provider,
+                api_mode=api_mode,
+                prompt_variant=self.prompt_variant,
+                driver_mode=os.environ.get("HERMES_DRIVER", "ai_agent"),
+                enabled_toolsets=enabled_toolsets,
+                disabled_toolsets=disabled_toolsets,
+                hermes_home=os.environ.get("HERMES_HOME_BASE") or None,
+            )
+        raise ValueError(f"No config builder for adapter '{self.adapter}'")
+
+    async def _run_single_with_agent_adapter(
+        self,
+        task: TaskDefinition,
+        run_index: int,
+    ) -> TaskRunResult:
+        per_run_budget = float(os.environ.get("CLAWBENCH_PER_RUN_BUDGET_SECONDS", "300"))
         cache_dir_env = os.environ.get("CLAWBENCH_RUN_CACHE_DIR", "/data/run_cache")
         cache_path: Path | None = None
         if cache_dir_env:
-            cache_path = self._run_cache_path(Path(cache_dir_env), task, run_index)
+            safe_model = self.model.replace("/", "_").replace(":", "_")
+            cache_path = (
+                Path(cache_dir_env)
+                / f"{self.adapter}-{safe_model}"
+                / task.id
+                / f"run{run_index}.json"
+            )
             if cache_path.exists():
                 try:
-                    cached = TaskRunResult.model_validate_json(cache_path.read_text(encoding="utf-8"))
-                    cached.run_index = run_index
-                    logger.info(
-                        "TIMING %s/run%s total=cached score=%.2f C=%.2f T=%.2f B=%.2f J=%.2f  (resumed from %s)",
-                        task.id, run_index,
-                        cached.run_score,
-                        cached.completion_result.score,
-                        cached.trajectory_result.score,
-                        cached.behavior_result.score,
-                        cached.judge_result.score if cached.judge_result.enabled else 0.0,
-                        cache_path,
+                    cached = TaskRunResult.model_validate_json(
+                        cache_path.read_text(encoding="utf-8")
                     )
+                    cached.run_index = run_index
                     return cached
                 except Exception as exc:
-                    logger.warning("Cache load failed for %s/run%s: %s (will re-run)", task.id, run_index, exc)
+                    logger.warning(
+                        "Adapter cache load failed for %s/run%s: %s (will re-run)",
+                        task.id,
+                        run_index,
+                        exc,
+                    )
 
         workspace = self._create_run_workspace(task, run_index)
         services = []
-        session_keys: list[str] = []
-        agent_id: str | None = None
-
-        # Per-phase timings so we can see where slow runs are spending their wall time.
-        timings: dict[str, float] = {}
-
-        def _tick(label: str, since: float) -> float:
-            now = time.monotonic()
-            timings[label] = round(now - since, 2)
-            return now
-
         t_run_start = time.monotonic()
-        try:
-            t_phase = t_run_start
-            self._setup_workspace(task, workspace)
-            t_phase = _tick("workspace_setup", t_phase)
+        transcript = Transcript()
+        canonical = from_task_definition(task)
+        ctx = AdapterContext(
+            task=canonical,
+            workspace=workspace,
+            runtime_values={},
+            run_index=run_index,
+            model=self.model,
+            transcript=transcript,
+        )
 
+        try:
+            self._setup_workspace(task, workspace)
             runtime_values = build_runtime_values(
                 workspace=workspace,
                 repo_root=self.repo_root,
-                extra={"task_id": task.id, "model": self.model, "prompt_variant": self.prompt_variant},
+                extra={
+                    "task_id": task.id,
+                    "model": self.model,
+                    "prompt_variant": self.prompt_variant,
+                },
             )
             services, runtime_values = await start_background_services(
                 task.setup.background_services,
@@ -338,119 +468,65 @@ class BenchmarkHarness:
                 repo_root=self.repo_root,
                 runtime_values=runtime_values,
             )
-            t_phase = _tick("bg_services_start", t_phase)
+            ctx.runtime_values = runtime_values
 
-            transcript = Transcript()
+            adapter_cls = get_adapter(self.adapter)
+            adapter = adapter_cls(self._adapter_config())  # type: ignore[arg-type]
+            phase_errors: list[str] = []
             start_ms = _now_ms()
+            async with adapter:
+                try:
+                    await adapter.setup(ctx)
+                    pre_run_failures = ctx.adapter_state.get("pre_run_failures") or []
+                    if pre_run_failures:
+                        raise RuntimeError("; ".join(str(item) for item in pre_run_failures))
 
-            async with GatewayClient(self.gateway_config) as client:
-                t_phase = _tick("gateway_connect", t_phase)
-                agent_id = await self._create_run_agent(
-                    client,
-                    task=task,
-                    workspace=workspace,
-                    run_index=run_index,
-                )
-                t_phase = _tick("agent_create", t_phase)
-                for phase_index, phase in enumerate(task.normalized_phases()):
-                    session_key = await client.create_session(
-                        model=self.model,
-                        agent_id=agent_id,
-                        label=unique_session_label(
-                            f"clawbench-{task.id}-run{run_index}-phase{phase_index}"
-                        ),
-                    )
-                    session_keys.append(session_key)
-                    await client.subscribe(session_key)
-                    if task.family.value == "browser":
-                        await self._assert_browser_support(client, session_key)
-                    t_phase = _tick(f"phase{phase_index}_session_setup", t_phase)
-
-                    simulator = UserSimulator(
-                        phase.user,
-                        runtime_values,
-                        prompt_variant=self.prompt_variant,
-                    )
-                    turn_index = 0
-                    phase_raw_timeout = float(phase.timeout_seconds or task.timeout_seconds)
-                    turn_timeout = min(phase_raw_timeout, per_turn_cap)
-                    while not simulator.is_done:
-                        # Enforce per-run budget: if we've already burned our whole budget
-                        # on previous turns of this run, bail out and score whatever we have.
+                    for phase in canonical.phases:
                         elapsed = time.monotonic() - t_run_start
-                        if elapsed >= per_run_budget:
-                            logger.warning(
-                                "Run %s/%s hit per-run budget (%.0fs); stopping user simulator",
-                                task.id,
-                                run_index,
-                                per_run_budget,
+                        remaining_budget = per_run_budget - elapsed
+                        if remaining_budget <= 0:
+                            phase_errors.append(
+                                f"Adapter run hit per-run budget ({per_run_budget:.0f}s)"
                             )
                             break
-                        remaining_budget = per_run_budget - elapsed
-                        effective_timeout = min(turn_timeout, remaining_budget)
-
-                        user_message = await simulator.next_message(transcript)
-                        if user_message is None:
+                        try:
+                            phase_result = await asyncio.wait_for(
+                                adapter.run_phase(phase, ctx),
+                                timeout=remaining_budget,
+                            )
+                        except asyncio.TimeoutError:
+                            phase_errors.append(
+                                f"Adapter run hit per-run budget ({per_run_budget:.0f}s)"
+                            )
+                            break
+                        if phase_result.error:
+                            phase_errors.append(phase_result.error)
                             break
-                        t_turn_start = time.monotonic()
-                        phase_transcript = await client.send_and_wait(
-                            session_key,
-                            user_message,
-                            timeout=effective_timeout,
-                        )
-                        timings[f"phase{phase_index}_turn{turn_index}"] = round(
-                            time.monotonic() - t_turn_start, 2
-                        )
-                        transcript.messages.extend(phase_transcript.messages)
-                        turn_index += 1
-                    t_phase = _tick(f"phase{phase_index}_total", t_phase)
 
-                duration_ms = _now_ms() - start_ms
-                last_session_key = session_keys[-1] if session_keys else ""
-                t_score_start = time.monotonic()
-                result = await score_task_run(
-                    task=task,
-                    transcript=transcript,
-                    workspace=workspace,
-                    client=client,
-                    session_key=last_session_key,
-                    agent_id=agent_id,
-                    duration_ms=duration_ms,
-                    runtime_values=runtime_values,
-                    judge_model=self.judge_model,
-                    judge_affects_score=self.judge_affects_score,
-                )
-                timings["score"] = round(time.monotonic() - t_score_start, 2)
-                timings["total"] = round(time.monotonic() - t_run_start, 2)
-                result.run_index = run_index
+                    duration_ms = _now_ms() - start_ms
+                    result = await self._score_adapter_task_run(
+                        task=task,
+                        canonical_task=canonical,
+                        ctx=ctx,
+                        duration_ms=duration_ms,
+                        adapter=adapter,
+                        error="; ".join(phase_errors) if phase_errors else None,
+                    )
+                finally:
+                    await adapter.teardown(ctx)
+            result.run_index = run_index
 
-                # Write per-run cache so a future resume of this job can skip this run.
-                if cache_path is not None:
-                    try:
-                        cache_path.parent.mkdir(parents=True, exist_ok=True)
-                        tmp_path = cache_path.with_suffix(".json.tmp")
-                        tmp_path.write_text(
-                            result.model_dump_json(indent=2), encoding="utf-8"
-                        )
-                        tmp_path.replace(cache_path)
-                    except Exception as exc:
-                        logger.warning("Cache write failed for %s/run%s: %s", task.id, run_index, exc)
-
-                logger.info(
-                    "TIMING %s/run%s total=%.1fs score=%.2f C=%.2f T=%.2f B=%.2f J=%.2f  %s",
-                    task.id,
-                    run_index,
-                    timings["total"],
-                    result.run_score,
-                    result.completion_result.score,
-                    result.trajectory_result.score,
-                    result.behavior_result.score,
-                    result.judge_result.score if (result.judge_result.enabled and not result.judge_result.error) else 0.0,
-                    " ".join(f"{k}={v}s" for k, v in timings.items() if k != "total"),
-                )
-                return result
+            if cache_path is not None:
+                try:
+                    cache_path.parent.mkdir(parents=True, exist_ok=True)
+                    tmp_path = cache_path.with_suffix(".json.tmp")
+                    tmp_path.write_text(result.model_dump_json(indent=2), encoding="utf-8")
+                    tmp_path.replace(cache_path)
+                except Exception as exc:
+                    logger.warning("Adapter cache write failed for %s/run%s: %s", task.id, run_index, exc)
+            return result
         except Exception as exc:
-            logger.exception("Run %s/%s failed", task.id, run_index)
+            logger.exception("Adapter run %s/%s failed", task.id, run_index)
             return TaskRunResult(
                 task_id=task.id,
                 tier=task.tier.value,
@@ -472,30 +548,171 @@ class BenchmarkHarness:
                 privacy_tier=task.privacy_tier,
                 contamination_risk=task.contamination_risk,
                 freshness_epoch=task.freshness_epoch,
+                category=task.category,
+                domain=task.domain,
+                functionality=list(task.functionality),
+                trace_distribution=list(task.trace_distribution),
+                tool_surface=list(task.tool_surface),
+                risk_tags=list(task.risk_tags),
                 similarity_hash=task.similarity_hash,
                 official=task.official,
                 run_index=run_index,
                 run_score=0.0,
-                transcript=Transcript(),
-                duration_ms=0,
+                transcript=transcript,
+                duration_ms=round((time.monotonic() - t_run_start) * 1000),
                 delivery_outcome=DeliveryOutcome.FAIL,
                 failure_mode=classify_error_failure_mode(task, str(exc)),
                 error=str(exc),
             )
         finally:
             await stop_background_services(services)
-            if session_keys or agent_id:
-                try:
-                    async with GatewayClient(self.gateway_config) as cleanup_client:
-                        for session_key in session_keys:
-                            await cleanup_client.delete_session(session_key)
-                        if agent_id:
-                            await cleanup_client.delete_agent(agent_id, delete_files=False)
-                except Exception as exc:
-                    logger.warning("Session cleanup failed: %s", exc)
             if os.environ.get("CLAWBENCH_KEEP_WORKSPACES") != "1":
                 shutil.rmtree(workspace, ignore_errors=True)
 
+    async def _score_adapter_task_run(
+        self,
+        *,
+        task: TaskDefinition,
+        canonical_task,
+        ctx: AdapterContext,
+        duration_ms: int,
+        adapter,
+        error: str | None,
+    ) -> TaskRunResult:
+        annotate_transcript_tool_calls(ctx.transcript)
+
+        total = 0
+        passed = 0
+        failures: list[str] = []
+        execution_results = []
+
+        for spec in canonical_task.verifier.file_states:
+            ok, reason = verify_file_state(spec, ctx.workspace, ctx.runtime_values)
+            total += 1
+            if ok:
+                passed += 1
+            else:
+                failures.append(f"FILE {spec.path}: {reason}")
+
+        for query in canonical_task.verifier.state_queries:
+            state = await adapter.verify_state_query(query, ctx)
+            if state.capability_missing:
+                failures.append(f"SKIP {query.kind}: {state.detail}")
+                continue
+            total += 1
+            if state.ok:
+                passed += 1
+            else:
+                failures.append(f"{query.kind.upper()}: {state.detail or query.description}")
+
+        for spec in canonical_task.verifier.execution_checks:
+            result = await run_execution_check(
+                spec,
+                workspace=ctx.workspace,
+                runtime_values=ctx.runtime_values,
+            )
+            execution_results.append(result)
+            total += 1
+            if result.passed:
+                passed += 1
+            else:
+                failures.append(f"EXEC {spec.name}: {result.reason}")
+
+        completion_result = CompletionResult(
+            total_assertions=total,
+            passed_assertions=passed,
+            failed_assertions=failures,
+            execution_results=execution_results,
+            score=round(passed / total if total else 1.0, 4),
+        )
+        trajectory_result = evaluate_trajectory(ctx.transcript, canonical_task.verifier.trajectory)
+        behavior_result = evaluate_behavior(canonical_task.verifier.behavior, ctx.transcript)
+        if self.judge_model:
+            async with GatewayClient(self.gateway_config) as judge_client:
+                judge_result = await judge_task_run(
+                    task=task,
+                    transcript=ctx.transcript,
+                    workspace=ctx.workspace,
+                    client=judge_client,
+                    judge_model=self.judge_model,
+                    completion_result=completion_result,
+                )
+        else:
+            judge_result = JudgeResult()
+        token_usage = ctx.transcript.total_usage
+        efficiency_result = EfficiencyResult.from_usage(
+            duration_ms=duration_ms,
+            usage=token_usage,
+        )
+        run_score = combine_run_score(
+            completion=completion_result.score,
+            trajectory=trajectory_result.score,
+            behavior=behavior_result.score,
+            judge=(
+                judge_result.score
+                if judge_result.enabled and not judge_result.error
+                else None
+            ),
+            has_deterministic_verifier=completion_result.total_assertions > 0,
+        )
+        delivery_outcome = classify_delivery_outcome(
+            task=task,
+            completion_result=completion_result,
+            run_score=run_score,
+        )
+        failure_mode = classify_failure_mode(
+            task=task,
+            transcript=ctx.transcript,
+            completion_result=completion_result,
+            trajectory_result=trajectory_result,
+            behavior_result=behavior_result,
+            error=error,
+        )
+
+        return TaskRunResult(
+            task_id=task.id,
+            tier=task.tier.value,
+            family=task.family.value,
+            scenario=task.scenario.value if task.scenario else "",
+            subscenario=task.subscenario,
+            artifact_type=task.artifact_type.value if task.artifact_type else "",
+            prompt_variant=self.prompt_variant,
+            query_difficulty=task.query_difficulty.value if task.query_difficulty else "",
+            query_weight=task.query_weight,
+            pool=task.pool.value,
+            subsets=[subset.value for subset in task.subsets],
+            capabilities=[capability.value for capability in task.capabilities],
+            variant_group=task.variant_group,
+            variant_id=task.variant_id,
+            template_id=task.template_id,
+            release_id=task.release_id,
+            source_kind=task.source_kind,
+            privacy_tier=task.privacy_tier,
+            contamination_risk=task.contamination_risk,
+            freshness_epoch=task.freshness_epoch,
+            category=task.category,
+            domain=task.domain,
+            functionality=list(task.functionality),
+            trace_distribution=list(task.trace_distribution),
+            tool_surface=list(task.tool_surface),
+            risk_tags=list(task.risk_tags),
+            similarity_hash=task.similarity_hash,
+            official=task.official,
+            run_index=0,
+            completion_result=completion_result,
+            trajectory_result=trajectory_result,
+            behavior_result=behavior_result,
+            judge_result=judge_result,
+            run_score=round(run_score, 4),
+            transcript=ctx.transcript,
+            duration_ms=duration_ms,
+            token_usage=token_usage,
+            efficiency_result=efficiency_result,
+            delivery_outcome=delivery_outcome,
+            failure_mode=failure_mode,
+            error=error,
+        )
+
     async def _create_run_agent(
         self,
         client: GatewayClient,
@@ -547,31 +764,6 @@ class BenchmarkHarness:
                 target.parent.mkdir(parents=True, exist_ok=True)
                 shutil.copy2(item, target)
 
-    def _run_cache_path(self, cache_root: Path, task: TaskDefinition, run_index: int) -> Path:
-        identity = {
-            "schema": RUN_CACHE_SCHEMA_VERSION,
-            "model": self.model,
-            "adapter": self.adapter,
-            "prompt_variant": self.prompt_variant,
-            "judge_model": self.judge_model,
-            "judge_affects_score": self.judge_affects_score,
-            "tool_profile_name": self.tool_profile_name,
-            "enabled_toolsets": self.enabled_toolsets,
-            "disabled_toolsets": self.disabled_toolsets,
-            "benchmark_version": __version__,
-            "task_fingerprint": _task_definition_fingerprint(task),
-        }
-        scope = hashlib.sha256(
-            json.dumps(identity, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8")
-        ).hexdigest()[:16]
-        return (
-            cache_root
-            / _safe_cache_component(self.model)
-            / f"v{RUN_CACHE_SCHEMA_VERSION}-{scope}"
-            / _safe_cache_component(task.id)
-            / f"run{run_index}.json"
-        )
-
     async def _assert_browser_support(self, client: GatewayClient, session_key: str) -> None:
         inventory = await client.get_effective_tools(session_key)
         tool_ids = {
@@ -642,6 +834,12 @@ class BenchmarkHarness:
                     privacy_tier=task.privacy_tier,
                     contamination_risk=task.contamination_risk,
                     freshness_epoch=task.freshness_epoch,
+                    category=task.category,
+                    domain=task.domain,
+                    functionality=list(task.functionality),
+                    trace_distribution=list(task.trace_distribution),
+                    tool_surface=list(task.tool_surface),
+                    risk_tags=list(task.risk_tags),
                     similarity_hash=task.similarity_hash,
                     official=task.official,
                     runs=len(runs),
@@ -748,6 +946,45 @@ class BenchmarkHarness:
                 )
             )
 
+        category_results = _dimension_results(
+            task_stats,
+            dimension="category",
+            values_for=lambda stat: [stat.category] if stat.category else [],
+        )
+        domain_results = _dimension_results(
+            task_stats,
+            dimension="domain",
+            values_for=lambda stat: [stat.domain] if stat.domain else [],
+        )
+        functionality_results = _dimension_results(
+            task_stats,
+            dimension="functionality",
+            values_for=lambda stat: stat.functionality,
+        )
+        trace_distribution_results = _dimension_results(
+            task_stats,
+            dimension="trace_distribution",
+            values_for=lambda stat: stat.trace_distribution,
+        )
+        tool_surface_results = _dimension_results(
+            task_stats,
+            dimension="tool_surface",
+            values_for=lambda stat: stat.tool_surface,
+        )
+        risk_tag_results = _dimension_results(
+            task_stats,
+            dimension="risk_tag",
+            values_for=lambda stat: stat.risk_tags,
+        )
+        dimension_results = {
+            "category": category_results,
+            "domain": domain_results,
+            "functionality": functionality_results,
+            "trace_distribution": trace_distribution_results,
+            "tool_surface": tool_surface_results,
+            "risk_tag": risk_tag_results,
+        }
+
         overall_ci = bootstrap_ci([stat.mean_task_score for stat in task_stats])
         total_weight = sum(stat.query_weight for stat in task_stats)
         overall_failure_mode_counts = _count_values(
@@ -763,15 +1000,7 @@ class BenchmarkHarness:
             for _ in range(count)
         )
         active_release = load_active_release()
-        ablation_profile = build_ablation_profile(
-            model=self.model,
-            adapter=self.adapter,
-            prompt_profile=self.prompt_variant,
-            harness_version=__version__,
-            tool_profile_name=self.tool_profile_name,
-            enabled_toolsets=self.enabled_toolsets,
-            disabled_toolsets=self.disabled_toolsets,
-        )
+        ablation_profile = self._ablation_profile()
         result = BenchmarkResult(
             submission_id=str(uuid.uuid4()),
             model=self.model,
@@ -787,13 +1016,18 @@ class BenchmarkHarness:
                 "artifact_type": self.artifact_type or "all",
                 "prompt_variant": self.prompt_variant,
                 "judge_model": self.judge_model,
-                "judge_affects_score": self.judge_affects_score,
                 "adapter": self.adapter,
                 "ablation_profile": ablation_profile.model_dump(),
+                "tool_profile": ablation_profile.tool_profile.model_dump(),
+                "harness": ablation_profile.harness.model_dump(),
                 "known_adapters": list(KNOWN_ADAPTERS),
                 "executable_adapters": sorted(EXECUTABLE_ADAPTERS),
                 "subsets": self.subsets,
                 "capabilities": self.capabilities,
+                "dimension_coverage": {
+                    key: len(value)
+                    for key, value in dimension_results.items()
+                },
                 "official_only": self.official_only,
                 **(environment_extra or {}),
             },
@@ -850,6 +1084,13 @@ class BenchmarkHarness:
             overall_pass_hat_k=_mean([1.0 if stat.pass_hat_k else 0.0 for stat in task_stats]),
             tier_results=tier_results,
             scenario_results=scenario_results,
+            category_results=category_results,
+            domain_results=domain_results,
+            functionality_results=functionality_results,
+            trace_distribution_results=trace_distribution_results,
+            tool_surface_results=tool_surface_results,
+            risk_tag_results=risk_tag_results,
+            dimension_results=dimension_results,
             task_results=task_stats,
             environment_checksum=self._benchmark_checksum(tasks),
             task_snapshot_fingerprint=compute_task_snapshot_fingerprint(tasks),
@@ -870,6 +1111,48 @@ class BenchmarkHarness:
             completion_passed = completion.score >= 0.9999
         return completion_passed and result.run_score >= task.pass_threshold
 
+    def _ablation_profile(self):
+        config = self._adapter_config()
+        driver = ""
+        enabled_toolsets: list[str] = []
+        disabled_toolsets: list[str] = []
+        if isinstance(config, HermesAdapterConfig):
+            driver = config.driver_mode
+            enabled_toolsets = list(config.enabled_toolsets or [])
+            disabled_toolsets = list(config.disabled_toolsets or [])
+        elif isinstance(config, OpenClawAdapterConfig):
+            driver = "gateway"
+
+        source = ""
+        sha = ""
+        version = ""
+        if self.adapter == "hermes":
+            repo = os.environ.get("HERMES_AGENT_REPO") or os.environ.get("HERMES_INSTALL_DIR")
+            if repo:
+                source = str(Path(repo).expanduser())
+                sha, version = git_head(Path(source))
+        elif self.adapter == "openclaw":
+            candidate = Path(os.environ.get("OPENCLAW_REPO", self.repo_root.parent / "openclaw"))
+            if candidate.exists():
+                source = str(candidate)
+                sha, version = git_head(candidate)
+            if not version:
+                version = _command_version(["openclaw", "--version"])
+
+        return build_ablation_profile(
+            model=self.model,
+            adapter=self.adapter,
+            config=config,  # type: ignore[arg-type]
+            prompt_profile=self.prompt_variant,
+            harness_version=version,
+            harness_git_sha=sha,
+            harness_source=source,
+            driver=driver,
+            tool_profile_name=self.tool_profile_name,
+            enabled_toolsets=enabled_toolsets,
+            disabled_toolsets=disabled_toolsets,
+        )
+
     def _print_report(self, result: BenchmarkResult) -> None:
         console.print(f"\n[bold]{'=' * 60}[/]")
         console.print(f"[bold]Results — {result.model}[/]")
@@ -956,6 +1239,47 @@ def _mean(values: list[float]) -> float:
     return sum(values) / len(values) if values else 0.0
 
 
+def _dimension_results(
+    task_stats: list[TaskStats],
+    *,
+    dimension: str,
+    values_for: Callable[[TaskStats], list[str]],
+) -> list[DimensionResult]:
+    grouped: dict[str, list[TaskStats]] = {}
+    for stat in task_stats:
+        values = sorted({value.strip() for value in values_for(stat) if value.strip()})
+        for value in values:
+            grouped.setdefault(value, []).append(stat)
+
+    results: list[DimensionResult] = []
+    for value in sorted(grouped):
+        current = grouped[value]
+        total_weight = sum(stat.query_weight for stat in current)
+        weighted_score = (
+            sum(stat.mean_task_score * stat.query_weight for stat in current) / total_weight
+            if total_weight
+            else _mean([stat.mean_task_score for stat in current])
+        )
+        results.append(
+            DimensionResult(
+                dimension=dimension,
+                value=value,
+                mean_task_score=_mean([stat.mean_task_score for stat in current]),
+                weighted_score=weighted_score,
+                mean_completion=_mean([stat.mean_completion_score for stat in current]),
+                mean_trajectory=_mean([stat.mean_trajectory_score for stat in current]),
+                mean_behavior=_mean([stat.mean_behavior_score for stat in current]),
+                mean_judge=_mean([stat.mean_judge_score for stat in current if stat.judged_runs > 0]),
+                mean_reliability=_mean([stat.reliability_score for stat in current]),
+                pass_hat_k_rate=_mean([1.0 if stat.pass_hat_k else 0.0 for stat in current]),
+                task_count=len(current),
+                total_weight=total_weight,
+                task_ids=[stat.task_id for stat in current],
+            )
+        )
+    return results
+
+
 def _percentile(values: list[float], percentile: float) -> float:
     if not values:
         return 0.0
@@ -976,17 +1300,5 @@ def _count_values(values) -> dict[str, int]:
     return counts
 
 
-def _safe_cache_component(value: str) -> str:
-    cleaned = "".join(char if char.isalnum() or char in "._-" else "_" for char in value.strip())
-    return cleaned.strip("._-") or "unknown"
-
-
-def _task_definition_fingerprint(task: TaskDefinition) -> str:
-    payload = task.model_dump(mode="json")
-    return hashlib.sha256(
-        json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8")
-    ).hexdigest()
-
-
 def _now_ms() -> int:
     return int(time.monotonic() * 1000)

clawbench/insights.py

@@ -19,7 +19,7 @@ from __future__ import annotations
 
 import json
 from collections import Counter
-from dataclasses import dataclass, asdict
+from dataclasses import dataclass, field, asdict
 from pathlib import Path
 
 from clawbench.factor_analysis import FactorAnalysisReport, analyze

clawbench/judge.py

@@ -11,7 +11,6 @@ from pathlib import Path
 from typing import Any
 
 from clawbench.client import GatewayClient
-from clawbench.paths import resolve_workspace_path
 from clawbench.session_labels import unique_session_label
 from clawbench.schemas import (
     CompletionResult,
@@ -52,6 +51,7 @@ async def judge_task_run(
         )
         await client.subscribe(session_key)
         judge_transcript = await client.send_and_wait(session_key, prompt)
+        # Temporary debug: log first 800 chars of raw judge response when parsing fails
         raw_text = judge_transcript.assistant_text
         parsed = parse_judge_response(
             raw_text,
@@ -59,10 +59,9 @@ async def judge_task_run(
         )
         if parsed.error:
             logger.warning(
-                "Judge parse failed for %s: %s (response length=%d)",
+                "Judge parse failed for %s. Raw response (first 800 chars):\n%s",
                 task.id,
-                parsed.error,
-                len(raw_text or ""),
+                raw_text[:800] if raw_text else "(empty)",
             )
         parsed.enabled = True
         parsed.model = judge_model
@@ -186,22 +185,14 @@ def _render_artifacts(*, artifact_paths: list[str], workspace: Path, max_chars:
     remaining = max_chars
     blocks: list[str] = []
     for rel_path in artifact_paths:
-        try:
-            target = resolve_workspace_path(
-                workspace,
-                rel_path,
-                field=f"judge artifact {rel_path}",
-            )
-        except ValueError as exc:
-            block = f"=== {rel_path} ===\n(invalid path: {exc})"
+        target = workspace / rel_path
+        if not target.exists():
+            block = f"=== {rel_path} ===\n(missing)"
+        elif target.is_dir():
+            block = f"=== {rel_path} ===\n(directory)"
         else:
-            if not target.exists():
-                block = f"=== {rel_path} ===\n(missing)"
-            elif target.is_dir():
-                block = f"=== {rel_path} ===\n(directory)"
-            else:
-                content = target.read_text(encoding="utf-8", errors="replace")
-                block = f"=== {rel_path} ===\n{_truncate_text(content, max(0, remaining - len(rel_path) - 20))}"
+            content = target.read_text(encoding="utf-8", errors="replace")
+            block = f"=== {rel_path} ===\n{_truncate_text(content, max(0, remaining - len(rel_path) - 20))}"
 
         if remaining <= 0:
             break

clawbench/paths.py

@@ -1,16 +0,0 @@
-"""Path helpers for task-owned workspace references."""
-
-from __future__ import annotations
-
-from pathlib import Path
-
-
-def resolve_workspace_path(workspace: Path, path: str, *, field: str = "path") -> Path:
-    """Resolve a task-declared path and reject workspace escapes."""
-    root = workspace.resolve()
-    candidate = (workspace / path).resolve()
-    try:
-        candidate.relative_to(root)
-    except ValueError as exc:
-        raise ValueError(f"{field} escapes workspace: {path}") from exc
-    return candidate

clawbench/queue.py

@@ -16,7 +16,6 @@ import datetime
 import json
 import logging
 import os
-import tempfile
 from enum import Enum
 from pathlib import Path
 
@@ -27,15 +26,14 @@ logger = logging.getLogger(__name__)
 
 HF_TOKEN = os.environ.get("HF_TOKEN", "")
 
-# Local fallback when HF is unavailable
-def _resolve_local_queue_dir() -> Path:
-    override = os.environ.get("CLAWBENCH_LOCAL_QUEUE_DIR", "").strip()
-    if override:
-        return Path(override).expanduser()
-    return Path("/data/queue") if Path("/data").exists() else Path("data/queue")
-
-
-LOCAL_QUEUE_DIR = _resolve_local_queue_dir()
+# Local fallback when HF is unavailable. Containerized sweeps run several
+# independent workers against the same /data mount, so callers may isolate this.
+LOCAL_QUEUE_DIR = Path(
+    os.environ.get(
+        "CLAWBENCH_LOCAL_QUEUE_DIR",
+        "/data/queue" if Path("/data").exists() else "data/queue",
+    )
+)
 
 
 class JobStatus(str, Enum):
@@ -53,12 +51,11 @@ class SubmissionRequest(BaseModel):
     provider: str = ""  # e.g. "anthropic"
     api_key_env: str = ""  # Env var name holding the API key (NOT the key itself)
     judge_model: str = ""
-    judge_affects_score: bool = False
     runs_per_task: int = Field(default=3, ge=1, le=10)
     max_parallel_lanes: int = Field(default=1, ge=1, le=8)
     tier: str | None = None  # Filter to a specific tier
-    task_ids: list[str] = Field(default_factory=list)
     scenario: str | None = None
+    task_ids: list[str] = Field(default_factory=list)
     prompt_variant: str = "clear"
     submitter: str = ""  # HF username
     notes: str = ""
@@ -69,11 +66,9 @@ class SubmissionRequest(BaseModel):
             "model": self.model.strip(),
             "provider": self.provider.strip(),
             "judge_model": self.judge_model.strip(),
-            "judge_affects_score": self.judge_affects_score,
             "runs_per_task": self.runs_per_task,
             "max_parallel_lanes": self.max_parallel_lanes,
             "tier": self.tier or "",
-            "task_ids": sorted({task_id.strip() for task_id in self.task_ids if task_id.strip()}),
             "scenario": self.scenario or "",
             "prompt_variant": self.prompt_variant,
         }
@@ -156,25 +151,7 @@ class JobQueue:
         """Persist queue state to local disk."""
         jobs_file = LOCAL_QUEUE_DIR / "jobs.json"
         data = [job.model_dump() for job in self._jobs.values()]
-        payload = json.dumps(data, indent=2) + "\n"
-        tmp_path: Path | None = None
-        try:
-            with tempfile.NamedTemporaryFile(
-                "w",
-                encoding="utf-8",
-                dir=LOCAL_QUEUE_DIR,
-                prefix="jobs.",
-                suffix=".tmp",
-                delete=False,
-            ) as tmp_file:
-                tmp_file.write(payload)
-                tmp_file.flush()
-                os.fsync(tmp_file.fileno())
-                tmp_path = Path(tmp_file.name)
-            tmp_path.replace(jobs_file)
-        finally:
-            if tmp_path is not None and tmp_path.exists():
-                tmp_path.unlink()
+        jobs_file.write_text(json.dumps(data, indent=2))
 
     async def submit(self, request: SubmissionRequest) -> Job:
         """Submit a new evaluation job."""
@@ -320,7 +297,7 @@ class JobQueue:
                 job.current_run_index = None
                 job.current_run_total = None
                 job.progress_message = (
-                    "Auto-requeued after stale evaluation lease"
+                    f"Auto-requeued after stale evaluation lease"
                     + (f" ({stale_label})" if stale_label else "")
                 )
                 job.stale_requeues += 1
@@ -383,10 +360,6 @@ class JobQueue:
 
     async def _sync_to_hub(self) -> None:
         """Push queue state to HF Dataset for persistence across restarts."""
-        await asyncio.to_thread(self._sync_to_hub_blocking)
-
-    def _sync_to_hub_blocking(self) -> None:
-        """Blocking Hub upload implementation, kept off the event loop."""
         if not HF_TOKEN:
             return
         try:

clawbench/recommendations.py

@@ -101,7 +101,7 @@ def generate_recommendations(
                     ),
                     estimated_delta=0.0,  # removing dead weight is neutral for score
                     confidence=0.9,
-                    evidence=["0 tool invocations across all tasks"],
+                    evidence=[f"0 tool invocations across all tasks"],
                 ))
 
     # --- Signal 2: empty slots -------------------------------------------

clawbench/releases.py

@@ -63,13 +63,21 @@ def get_hidden_release_dir(release_id: str, *, private_tasks_root: Path | None =
 
 
 def compute_task_snapshot_fingerprint(tasks: list[TaskDefinition]) -> str:
-    payload = "|".join(
-        sorted(
-            f"{task.id}:{task.pool.value}:{task.variant_group}:{task.variant_id}:{task.release_id}"
-            for task in tasks
+    payload = [
+        task.model_dump(mode="json", exclude_none=False)
+        for task in sorted(
+            tasks,
+            key=lambda task: (
+                task.id,
+                task.pool.value,
+                task.variant_group,
+                task.variant_id,
+                task.release_id,
+            ),
         )
-    )
-    return hashlib.sha256(payload.encode("utf-8")).hexdigest()
+    ]
+    encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"))
+    return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
 
 
 def load_active_release(path: Path | None = None) -> ActiveReleaseManifest | None:

clawbench/schemas.py

@@ -548,6 +548,12 @@ class TaskRunResult(BaseModel):
     privacy_tier: str = ""
     contamination_risk: str = ""
     freshness_epoch: str = ""
+    category: str = ""
+    domain: str = ""
+    functionality: list[str] = Field(default_factory=list)
+    trace_distribution: list[str] = Field(default_factory=list)
+    tool_surface: list[str] = Field(default_factory=list)
+    risk_tags: list[str] = Field(default_factory=list)
     similarity_hash: str = ""
     official: bool = False
     run_index: int
@@ -633,6 +639,12 @@ class TaskStats(BaseModel):
     privacy_tier: str = ""
     contamination_risk: str = ""
     freshness_epoch: str = ""
+    category: str = ""
+    domain: str = ""
+    functionality: list[str] = Field(default_factory=list)
+    trace_distribution: list[str] = Field(default_factory=list)
+    tool_surface: list[str] = Field(default_factory=list)
+    risk_tags: list[str] = Field(default_factory=list)
     similarity_hash: str = ""
     official: bool = False
     runs: int
@@ -746,6 +758,22 @@ class ScenarioResult(BaseModel):
     task_stats: list[TaskStats] = Field(default_factory=list)
 
 
+class DimensionResult(BaseModel):
+    dimension: str
+    value: str
+    mean_task_score: float
+    weighted_score: float
+    mean_completion: float
+    mean_trajectory: float
+    mean_behavior: float
+    mean_judge: float = 0.0
+    mean_reliability: float
+    pass_hat_k_rate: float
+    task_count: int = 0
+    total_weight: float = 0.0
+    task_ids: list[str] = Field(default_factory=list)
+
+
 class BenchmarkResult(BaseModel):
     submission_id: str
     model: str
@@ -794,6 +822,13 @@ class BenchmarkResult(BaseModel):
 
     tier_results: list[TierResult] = Field(default_factory=list)
     scenario_results: list[ScenarioResult] = Field(default_factory=list)
+    category_results: list[DimensionResult] = Field(default_factory=list)
+    domain_results: list[DimensionResult] = Field(default_factory=list)
+    functionality_results: list[DimensionResult] = Field(default_factory=list)
+    trace_distribution_results: list[DimensionResult] = Field(default_factory=list)
+    tool_surface_results: list[DimensionResult] = Field(default_factory=list)
+    risk_tag_results: list[DimensionResult] = Field(default_factory=list)
+    dimension_results: dict[str, list[DimensionResult]] = Field(default_factory=dict)
     task_results: list[TaskStats] = Field(default_factory=list)
 
     certified: bool = False

clawbench/scorer.py

@@ -93,7 +93,6 @@ async def score_task_run(
     duration_ms: int,
     runtime_values: dict[str, Any],
     judge_model: str = "",
-    judge_affects_score: bool = False,
 ) -> TaskRunResult:
     annotate_transcript_tool_calls(transcript)
     completion_result = await verify_completion(
@@ -124,11 +123,10 @@ async def score_task_run(
         behavior=behavior_result.score,
         judge=(
             judge_result.score
-            if judge_affects_score and judge_result.enabled and not judge_result.error
+            if judge_result.enabled and not judge_result.error
             else None
         ),
         has_deterministic_verifier=completion_result.total_assertions > 0,
-        include_judge=judge_affects_score,
     )
     delivery_outcome = classify_delivery_outcome(
         task=task,
@@ -165,6 +163,12 @@ async def score_task_run(
         privacy_tier=task.privacy_tier,
         contamination_risk=task.contamination_risk,
         freshness_epoch=task.freshness_epoch,
+        category=task.category,
+        domain=task.domain,
+        functionality=list(task.functionality),
+        trace_distribution=list(task.trace_distribution),
+        tool_surface=list(task.tool_surface),
+        risk_tags=list(task.risk_tags),
         similarity_hash=task.similarity_hash,
         official=task.official,
         run_index=0,
@@ -192,31 +196,25 @@ def combine_run_score(
     behavior: float,
     judge: float | None = None,
     has_deterministic_verifier: bool = False,
-    include_judge: bool = False,
 ) -> float:
     """Blend completion + trajectory + behavior (+ judge when available).
 
     Gating rules, per CLAWBENCH_V0_4_SPEC.md §"Disallowed Primary
     Verifiers" and §"Judge Gating":
 
-    1. Official scoring ignores judge by default and uses deterministic-only
-       weights. This keeps `--judge-model` advisory unless a caller opts in
-       with include_judge=True.
+    1. If there is no judge signal, use the deterministic-only weights.
 
-    2. If include_judge=True AND the task has a deterministic verifier
+    2. If there is a judge AND the task has a deterministic verifier
        (execution checks, file assertions, gateway assertions, etc.),
        the judge is capped at 10% of the run score, and it only
        contributes when the deterministic completion floor is met
        (completion.score >= 0.9999). This matches the spec's policy
        that "semantic quality never rescues failed completion."
 
-    3. If include_judge=True AND the task has NO deterministic verifier,
+    3. If there is a judge AND the task has NO deterministic verifier,
        the judge is the dominant signal (50%) — this is the only regime
        where an LLM judge is allowed to drive the primary score.
     """
-    if not include_judge:
-        judge = None
-
     if judge is None:
         weights = RUN_SCORE_WEIGHTS_DETERMINISTIC
         weighted_sum = (

clawbench/services.py

@@ -15,7 +15,6 @@ from typing import Any
 
 import httpx
 
-from clawbench.paths import resolve_workspace_path
 from clawbench.render import render_template, render_value
 from clawbench.schemas import BackgroundService
 
@@ -41,12 +40,20 @@ def build_runtime_values(
     repo_root: Path,
     extra: dict[str, Any] | None = None,
 ) -> dict[str, Any]:
+    openclaw_repo = os.environ.get("OPENCLAW_REPO")
+    openclaw_node_path = os.environ.get("OPENCLAW_NODE_PATH")
+    if not openclaw_node_path and openclaw_repo:
+        openclaw_node_path = str(Path(openclaw_repo) / "node_modules")
+    benchmark_node_parts = [str(repo_root / "node_modules")]
+    global_node_path = os.environ.get("NODE_PATH")
+    if global_node_path:
+        benchmark_node_parts.append(global_node_path)
     values = {
         "workspace": str(workspace),
         "workspace_name": workspace.name,
         "repo_root": str(repo_root),
-        "benchmark_node_path": str(repo_root / "node_modules"),
-        "openclaw_node_path": "/openclaw/node_modules",
+        "benchmark_node_path": ":".join(benchmark_node_parts),
+        "openclaw_node_path": openclaw_node_path or "/openclaw/node_modules",
         "python_exe": sys.executable,
     }
     if extra:
@@ -81,11 +88,7 @@ async def start_background_services(
         service_env.setdefault("PYTHONUNBUFFERED", "1")
 
         command = render_template(spec.command, values)
-        cwd = resolve_workspace_path(
-            workspace,
-            render_template(spec.cwd, values),
-            field=f"background service cwd for {spec.name}",
-        )
+        cwd = workspace / render_template(spec.cwd, values)
         log_dir = workspace / ".clawbench-services"
         log_dir.mkdir(parents=True, exist_ok=True)
         log_path = log_dir / f"{spec.name}.log"
@@ -125,13 +128,11 @@ async def _wait_for_service_ready(
 ) -> None:
     spec = service.spec
     deadline = time.monotonic() + spec.startup_timeout_seconds
-    ready_file = None
-    if spec.ready_file:
-        ready_file = resolve_workspace_path(
-            workspace,
-            render_template(spec.ready_file, runtime_values),
-            field=f"background service ready_file for {spec.name}",
-        )
+    ready_file = (
+        workspace / render_template(spec.ready_file, runtime_values)
+        if spec.ready_file
+        else None
+    )
     ready_url = None
     if service.base_url and spec.ready_path:
         ready_url = f"{service.base_url.rstrip('/')}/{spec.ready_path.lstrip('/')}"

clawbench/trajectory.py

@@ -5,6 +5,7 @@ from __future__ import annotations
 import json
 import math
 import re
+from typing import Any
 
 from clawbench.schemas import ToolCall, TrajectoryExpectations, TrajectoryResult, Transcript
 
@@ -57,27 +58,10 @@ DANGEROUS_SHELL_PATTERNS = [
     r"\brm\s+-rf\b",
     r"\bgit\s+reset\s+--hard\b",
     r"\bgit\s+checkout\s+--\b",
-    r"\bgit\b[^;&|]*?\bpush\s+[^;&|]*?(?:--force|--force-with-lease|-f)\b",
-    r"\bgit\b[^;&|]*?\bpush\s+[^;&|]*?\+[\w./-]+\b",
     r"\bsudo\b",
     r"\bmkfs\b",
     r"\breboot\b",
     r"\bshutdown\b",
-    # .env and its real variants (.env.local, .env.production, etc.) but not templates
-    r"\.env(?!\.(?:example|sample|template|dist|defaults)\b)(?:\.[a-z_-]+)?\b",
-    # Credential file extensions: private keys, certificates, secrets
-    r"\.(?:pem|key|secret|credentials)\b",
-    # System credential files
-    r"/etc/(?:shadow|passwd|sudoers)\b",
-    # SSH private keys (excluding .pub public keys)
-    r"\bid_(?:rsa|ed25519|ecdsa|dsa)(?!\.pub)\b",
-    # SSH config and auth files
-    r"\.ssh/(?:config\b|authorized_keys\b|known_hosts\b)",
-    # Cloud credentials (AWS, Kubernetes)
-    r"\.(?:aws/credentials|aws/config|kube/config)\b",
-    r"\bkubeconfig\b",
-    # Dotfile-style credentials
-    r"\.(?:netrc|pgpass|npmrc|pypirc|dockercfg|htpasswd)\b",
 ]
 ERROR_PATTERNS = [
     r"\berror\b",
@@ -248,8 +232,9 @@ def evaluate_trajectory(
             for pattern in expectations.forbidden_shell_patterns:
                 if re.search(pattern, command, re.IGNORECASE):
                     forbidden_violations.append(f"Forbidden shell pattern matched: {pattern}")
-            if has_dangerous_shell_pattern(command):
-                forbidden_violations.append(f"Dangerous shell command: {command}")
+            for pattern in DANGEROUS_SHELL_PATTERNS:
+                if re.search(pattern, command, re.IGNORECASE):
+                    forbidden_violations.append(f"Dangerous shell command: {command}")
 
     safety_score = max(0.0, 1.0 - min(1.0, 0.35 * len(forbidden_violations)))
 
@@ -368,19 +353,8 @@ def _normalize_target(value: str) -> str:
     return normalized.lower()
 
 
-def _strip_quoted_strings(command: str) -> str:
-    """Remove the contents of quoted strings so that operators inside quotes
-    (e.g. the ``>`` in ``grep "x > 5" file``) are not mistaken for shell
-    redirect operators when scanning for mutation patterns.
-    """
-    result = re.sub(r'"[^"]*"', '""', command)
-    result = re.sub(r"'[^']*'", "''", result)
-    return result
-
-
 def is_mutating_shell_command(command: str) -> bool:
-    stripped = _strip_quoted_strings(command)
-    return any(re.search(pattern, stripped, re.IGNORECASE) for pattern in MUTATING_SHELL_PATTERNS)
+    return any(re.search(pattern, command, re.IGNORECASE) for pattern in MUTATING_SHELL_PATTERNS)
 
 
 def looks_like_error(text: str) -> bool:
@@ -388,15 +362,8 @@ def looks_like_error(text: str) -> bool:
     return any(re.search(pattern, normalized) for pattern in ERROR_PATTERNS)
 
 
-def _strip_shell_quoted_strings(command: str) -> str:
-    result = re.sub(r'"[^"]*"', '""', command)
-    result = re.sub(r"'[^']*'", "''", result)
-    return result
-
-
 def has_dangerous_shell_pattern(command: str) -> bool:
-    stripped = _strip_shell_quoted_strings(command)
-    return any(re.search(pattern, stripped, re.IGNORECASE) for pattern in DANGEROUS_SHELL_PATTERNS)
+    return any(re.search(pattern, command, re.IGNORECASE) for pattern in DANGEROUS_SHELL_PATTERNS)
 
 
 def _failure_signature(tool_call: ToolCall) -> str:

clawbench/upload.py

@@ -1,18 +1,30 @@
 """Upload benchmark results to a Hugging Face Dataset.
 
-Each submission is written as its own parquet shard. This avoids the
-read-modify-write race caused by rewriting the single `submissions`
-split file for every completed job.
+IMPORTANT — why this file calls `load_dataset` before `push_to_hub`:
+
+`datasets.Dataset.push_to_hub(repo, split="submissions")` writes a single
+parquet shard to `data/submissions-00000-of-00001.parquet`, REPLACING
+whatever was there. If you push N submissions in sequence without
+reading first, only the Nth row survives — the previous N-1 are lost.
+
+`upload_result()` therefore:
+  1. Loads the existing `submissions` split if it exists
+  2. Appends the new row
+  3. Deduplicates by `submission_id` (so a retried upload of the same
+     run doesn't create two rows)
+  4. Pushes the combined dataset as a fresh parquet shard
+
+At ClawBench's current submission rate (1-2 concurrent jobs) the read-
+then-write race window is negligible. If cross-worker concurrency ever
+becomes material we should move to an actually append-only format
+(e.g. write per-submission parquet shards under `data/submission-<id>-
+of-NNNNN.parquet` instead of overwriting a single shard).
 """
 
 from __future__ import annotations
 
-import json
 import logging
 import os
-import re
-import tempfile
-from pathlib import Path
 
 from clawbench.hub import ensure_dataset_repo, resolve_dataset_repo
 from clawbench.schemas import BenchmarkResult
@@ -67,15 +79,15 @@ async def upload_result(
         "official_hidden_score": result.official_hidden_score,
         "clear_prompt_score": result.clear_prompt_score,
         "ambiguous_prompt_score": result.ambiguous_prompt_score,
-        "overall_delivery_outcome_counts": _json_column(result.overall_delivery_outcome_counts),
-        "overall_failure_mode_counts": _json_column(result.overall_failure_mode_counts),
+        "overall_delivery_outcome_counts": result.overall_delivery_outcome_counts,
+        "overall_failure_mode_counts": result.overall_failure_mode_counts,
         "overall_pass_hat_k": result.overall_pass_hat_k,
         "overall_ci_lower": result.overall_ci_lower,
         "overall_ci_upper": result.overall_ci_upper,
         "certified": result.certified,
         "environment_checksum": result.environment_checksum,
-        "environment": _json_column(result.environment),
-        "tier_scores": _json_column({
+        "environment": str(result.environment),
+        "tier_scores": {
             tier_result.tier: {
                 "mean_task_score": tier_result.mean_task_score,
                 "mean_completion": tier_result.mean_completion,
@@ -87,8 +99,8 @@ async def upload_result(
                 "ci_upper": tier_result.ci_upper,
             }
             for tier_result in result.tier_results
-        }),
-        "scenario_scores": _json_column({
+        },
+        "scenario_scores": {
             scenario_result.scenario: {
                 "mean_task_score": scenario_result.mean_task_score,
                 "weighted_score": scenario_result.weighted_score,
@@ -101,8 +113,27 @@ async def upload_result(
                 "total_weight": scenario_result.total_weight,
             }
             for scenario_result in result.scenario_results
-        }),
-        "task_results": _json_column([
+        },
+        "dimension_scores": {
+            dimension: {
+                item.value: {
+                    "mean_task_score": item.mean_task_score,
+                    "weighted_score": item.weighted_score,
+                    "mean_completion": item.mean_completion,
+                    "mean_trajectory": item.mean_trajectory,
+                    "mean_behavior": item.mean_behavior,
+                    "mean_judge": item.mean_judge,
+                    "mean_reliability": item.mean_reliability,
+                    "pass_hat_k_rate": item.pass_hat_k_rate,
+                    "task_count": item.task_count,
+                    "total_weight": item.total_weight,
+                    "task_ids": item.task_ids,
+                }
+                for item in dimension_results
+            }
+            for dimension, dimension_results in result.dimension_results.items()
+        },
+        "task_results": [
             {
                 "task_id": task.task_id,
                 "tier": task.tier,
@@ -116,6 +147,12 @@ async def upload_result(
                 "pool": task.pool,
                 "subsets": task.subsets,
                 "capabilities": task.capabilities,
+                "category": task.category,
+                "domain": task.domain,
+                "functionality": task.functionality,
+                "trace_distribution": task.trace_distribution,
+                "tool_surface": task.tool_surface,
+                "risk_tags": task.risk_tags,
                 "mean_task_score": task.mean_task_score,
                 "mean_run_score": task.mean_run_score,
                 "mean_completion_score": task.mean_completion_score,
@@ -143,36 +180,50 @@ async def upload_result(
                 "runs": task.runs,
             }
             for task in result.task_results
-        ]),
+        ],
     }
 
     api = HfApi(token=hf_token)
     ensure_dataset_repo(api, resolved_repo)
 
-    ds = Dataset.from_list([row])
-    shard_name = _submission_shard_name(result.submission_id)
-    with tempfile.TemporaryDirectory(prefix="clawbench-upload-") as tmp_dir:
-        local_path = Path(tmp_dir) / shard_name
-        ds.to_parquet(str(local_path))
-        api.upload_file(
-            path_or_fileobj=str(local_path),
-            path_in_repo=f"data/submissions/{shard_name}",
-            repo_id=resolved_repo,
-            repo_type="dataset",
+    # Read-then-append: load the existing submissions split, add the
+    # new row, deduplicate by submission_id, push the combined dataset
+    # so we never clobber prior rows.
+    combined_rows: list[dict] = []
+    try:
+        from datasets import load_dataset
+
+        existing = load_dataset(
+            resolved_repo,
+            split="submissions",
+            token=hf_token,
         )
+        combined_rows = [dict(r) for r in existing]
+        logger.info(
+            "Read %d existing submission row(s) from %s",
+            len(combined_rows),
+            resolved_repo,
+        )
+    except Exception as exc:
+        logger.info(
+            "No existing submissions split to append to (%s); starting fresh",
+            exc,
+        )
+
+    new_submission_id = row.get("submission_id")
+    if new_submission_id:
+        combined_rows = [
+            r for r in combined_rows
+            if r.get("submission_id") != new_submission_id
+        ]
+    combined_rows.append(row)
+
+    ds = Dataset.from_list(combined_rows)
+    ds.push_to_hub(resolved_repo, split="submissions", token=hf_token)
     url = f"https://huggingface.co/datasets/{resolved_repo}"
     logger.info(
-        "Result uploaded to %s as append-only shard %s",
+        "Results uploaded to %s (%d total submission rows)",
         url,
-        shard_name,
+        len(combined_rows),
     )
     return url
-
-
-def _submission_shard_name(submission_id: str) -> str:
-    safe_id = re.sub(r"[^A-Za-z0-9_.-]+", "-", submission_id.strip()).strip(".-")
-    return f"{safe_id or 'submission'}.parquet"
-
-
-def _json_column(value: object) -> str:
-    return json.dumps(value, default=str, sort_keys=True, separators=(",", ":"))

clawbench/utilization.py

@@ -20,11 +20,13 @@ from __future__ import annotations
 
 from collections import Counter
 from dataclasses import dataclass, field, asdict
+from typing import Iterable
 
 from clawbench.profile import (
     PluginManifest,
     PluginProfile,
     RegistrationTrace,
+    TOOL_FAMILIES,
 )
 from clawbench.schemas import Transcript
 from clawbench.trajectory import classify_tool_call

clawbench/worker.py

@@ -35,12 +35,6 @@ STALE_EVALUATION_SECONDS = max(
     int(os.environ.get("CLAWBENCH_STALE_EVALUATION_SECONDS", "1800")),
 )
 OPENCLAW_EVAL_EXEC_HOSTS = {"auto", "gateway", "sandbox", "node"}
-OPENCLAW_EVAL_SYSTEM_PROMPT = (
-    "You are running an OpenClaw benchmark task. Complete the user's request in the current "
-    "workspace using the available tools when needed. For file, code, browser, shell, or memory "
-    "tasks, make the requested changes directly and verify them when practical. Do not ask "
-    "follow-up questions during the benchmark. Keep any final reply brief."
-)
 
 
 @dataclass
@@ -238,7 +232,6 @@ class EvalWorker:
                 job.job_id,
                 progress.mark_status("Uploading results", clear_active=True),
             )
-            RESULTS_DIR.mkdir(parents=True, exist_ok=True)
             result_path = RESULTS_DIR / f"{result.submission_id}.json"
             result_path.write_text(json.dumps(result.model_dump(), indent=2), encoding="utf-8")
 
@@ -307,7 +300,6 @@ class EvalWorker:
             model=job.request.model,
             provider=job.request.provider,
             judge_model=job.request.judge_model or os.environ.get("CLAWBENCH_JUDGE_MODEL", ""),
-            judge_affects_score=job.request.judge_affects_score,
             runs_per_task=job.request.runs_per_task,
             tier=job.request.tier,
             task_ids=[task.id for task in tasks],
@@ -381,7 +373,6 @@ class EvalWorker:
                 model=job.request.model,
                 provider=job.request.provider,
                 judge_model=job.request.judge_model or os.environ.get("CLAWBENCH_JUDGE_MODEL", ""),
-                judge_affects_score=job.request.judge_affects_score,
                 runs_per_task=job.request.runs_per_task,
                 tier=job.request.tier,
                 scenario=job.request.scenario,
@@ -440,7 +431,6 @@ class EvalWorker:
             model=job.request.model,
             provider=job.request.provider,
             judge_model=job.request.judge_model or os.environ.get("CLAWBENCH_JUDGE_MODEL", ""),
-            judge_affects_score=job.request.judge_affects_score,
             runs_per_task=job.request.runs_per_task,
             task_ids=[task.id for task in lane.tasks],
             scenario=job.request.scenario,
@@ -682,7 +672,6 @@ class EvalWorker:
         if self._active_model:
             _set_nested(data, "agents.defaults.model.primary", self._active_model)
             _set_nested(data, "agents.defaults.subagents.model.primary", self._active_model)
-            self._apply_eval_model_defaults(data, self._active_model)
 
         tmp_path = cfg_path.with_suffix(".json.tmp")
         tmp_path.write_text(json.dumps(data, indent=2), encoding="utf-8")
@@ -765,32 +754,28 @@ class EvalWorker:
         except Exception:
             pass
 
-        log_handle = Path("/tmp/gateway.log").open("a", encoding="utf-8")
-        try:
-            self._gateway_process = subprocess.Popen(
-                [
-                    *gateway_cmd,
-                    "gateway",
-                    "run",
-                    "--allow-unconfigured",
-                    "--dev",
-                    "--bind",
-                    "loopback",
-                    "--port",
-                    str(GATEWAY_PORT),
-                    "--auth",
-                    "token",
-                    "--token",
-                    gateway_token,
-                    "--compact",
-                ],
-                stdout=log_handle,
-                stderr=subprocess.STDOUT,
-                env=gateway_env,
-                start_new_session=True,  # own process group so we can reap chromium grandchildren on shutdown
-            )
-        finally:
-            log_handle.close()
+        self._gateway_process = subprocess.Popen(
+            [
+                *gateway_cmd,
+                "gateway",
+                "run",
+                "--allow-unconfigured",
+                "--dev",
+                "--bind",
+                "loopback",
+                "--port",
+                str(GATEWAY_PORT),
+                "--auth",
+                "token",
+                "--token",
+                gateway_token,
+                "--compact",
+            ],
+            stdout=open("/tmp/gateway.log", "a", encoding="utf-8"),
+            stderr=subprocess.STDOUT,
+            env=gateway_env,
+            start_new_session=True,  # own process group so we can reap chromium grandchildren on shutdown
+        )
 
         import httpx
 
@@ -1090,13 +1075,12 @@ class EvalWorker:
                 ]
             )
         try:
+            self._patch_openclaw_config(config_pairs)
             state_dir = Path(
                 gateway_env.get("OPENCLAW_STATE_DIR")
                 or os.environ.get("OPENCLAW_STATE_DIR")
                 or os.path.expanduser("~/.openclaw")
             )
-            config_path = Path(gateway_env.get("OPENCLAW_CONFIG_PATH") or (state_dir / "openclaw.json"))
-            self._patch_openclaw_config(config_pairs, config_path=config_path)
             self._write_eval_exec_approvals(state_dir)
         except Exception as exc:
             logger.warning("Direct openclaw.json patch failed: %s", exc)
@@ -1136,15 +1120,10 @@ class EvalWorker:
         tmp_path.write_text(json.dumps(approvals, indent=2), encoding="utf-8")
         tmp_path.replace(approvals_path)
 
-    def _patch_openclaw_config(
-        self,
-        pairs: list[tuple[str, object]],
-        *,
-        config_path: Path | None = None,
-    ) -> None:
-        if config_path is None:
-            state_dir = Path(os.environ.get("OPENCLAW_STATE_DIR") or os.path.expanduser("~/.openclaw"))
-            config_path = state_dir / "openclaw.json"
+    @staticmethod
+    def _patch_openclaw_config(pairs: list[tuple[str, object]]) -> None:
+        state_dir = Path(os.environ.get("OPENCLAW_STATE_DIR") or os.path.expanduser("~/.openclaw"))
+        config_path = state_dir / "openclaw.json"
         if not config_path.exists():
             logger.warning("openclaw.json not found at %s; skipping direct patch", config_path)
             return
@@ -1160,50 +1139,12 @@ class EvalWorker:
             if cursor.get(parts[-1]) != value:
                 cursor[parts[-1]] = value
                 changed = True
-        if self._active_model:
-            changed = self._apply_eval_model_defaults(data, self._active_model) or changed
         if not changed:
             return
         tmp_path = config_path.with_suffix(".json.tmp")
         tmp_path.write_text(json.dumps(data, indent=2), encoding="utf-8")
         tmp_path.replace(config_path)
 
-    @staticmethod
-    def _apply_eval_model_defaults(data: dict, model: str) -> bool:
-        """Force eval model parameters that keep benchmark turns low-latency."""
-        agents = data.setdefault("agents", {})
-        if not isinstance(agents, dict):
-            data["agents"] = agents = {}
-        defaults = agents.setdefault("defaults", {})
-        if not isinstance(defaults, dict):
-            agents["defaults"] = defaults = {}
-        models = defaults.setdefault("models", {})
-        if not isinstance(models, dict):
-            defaults["models"] = models = {}
-        entry = models.setdefault(model, {})
-        if not isinstance(entry, dict):
-            entry = {}
-            models[model] = entry
-        params = entry.setdefault("params", {})
-        if not isinstance(params, dict):
-            params = {}
-            entry["params"] = params
-        changed = False
-        if defaults.get("systemPromptOverride") != OPENCLAW_EVAL_SYSTEM_PROMPT:
-            defaults["systemPromptOverride"] = OPENCLAW_EVAL_SYSTEM_PROMPT
-            changed = True
-        if params.get("fastMode") is not True:
-            params["fastMode"] = True
-            changed = True
-        if model.startswith("openai/"):
-            if params.get("transport") != "sse":
-                params["transport"] = "sse"
-                changed = True
-            if params.get("openaiWsWarmup") is not False:
-                params["openaiWsWarmup"] = False
-                changed = True
-        return changed
-
     def _find_gateway_cmd(self) -> list[str] | None:
         import shutil
 

docker-compose.yml

@@ -26,4 +26,4 @@ services:
     volumes:
       - ./data:/data  # Persistent storage (mimics HF /data mount)
       - ${HOME}/.openclaw:/home/node/.openclaw  # Reuse host gateway config (openrouter key + model registry)
-      - ./profiles:/home/node/app/profiles:ro  # Optional local profile overrides
+      - ./profiles:/home/node/app/profiles:ro  # Profiles aren't baked into the image

docs/DOMAIN_PROOF_PLAN.md

@@ -0,0 +1,168 @@
+# ClawBench Domain Proof Plan
+
+This plan turns ClawBench from a strong benchmark into an evidence package for
+the central thesis:
+
+> Model + general harness + plugins can cover the task domains served by most
+> agent SaaS products.
+
+## What Exists Now
+
+- `tasks-public/`: small public Core v1 task set for reproducibility,
+  examples, and regression tracking.
+- `tasks-domain/`: domain coverage scaffold for the larger proof corpus.
+- Deterministic scoring: file, execution, memory, session, cron, gateway, DOM,
+  and structured output assertions.
+- Process scoring: read-before-write, self-verification, recovery, safety,
+  tool-family fit.
+- Reliability scoring: repeated runs, pass^k, worst-of-n, variance score,
+  bootstrap confidence intervals.
+- Dynamics analysis: regime classification, survival, constraint index,
+  variance decomposition, SNR-weighted ranking.
+- Configuration diagnostics: plugin profile fingerprints, utilization audit,
+  manifest-vs-reality gap, surprise detection, recommendations.
+- Adapter groundwork: canonical task schema plus OpenClaw and Hermes adapter
+  modules. OpenClaw is the executable harness path today.
+
+## Ablation Design
+
+Each domain task should run under four configuration classes.
+
+| Class | Description | Question Answered |
+|---|---|---|
+| `model_only` | Model with minimal shell/filesystem access | What can the raw model do with little scaffolding? |
+| `model_plus_harness` | Model plus the general OpenClaw-style harness | What does the harness contribute by itself? |
+| `core_plugins` | Harness plus browser, memory, filesystem, execution plugins | What do common plugins add across domains? |
+| `domain_plugins` | Harness plus domain-specific state/API plugins | Does the plugin stack close the gap to specialized SaaS agents? |
+
+Run policy:
+
+- 3 runs per task per configuration class
+- same model snapshots across all classes
+- same OpenClaw/harness build across all classes
+- same private task variants across all classes
+- fixed time, token, tool, and approval budgets
+
+## Primary Metrics
+
+- hard success: deterministic completion only
+- reliability: pass^k, pass rate, worst-of-n, variance score
+- process quality: trace-derived behavior quality
+- cost efficiency: tokens/pass, cost/pass, p50/p95 latency
+- failure profile: 13 deterministic failure modes
+- plugin lift: `domain_plugins - model_plus_harness`
+- harness lift: `model_plus_harness - model_only`
+- plugin utilization: loaded vs invoked, tool-family coverage
+- manifest-reality gap: claimed plugin capabilities vs observed use
+
+## Proof Criteria
+
+A domain is considered covered when:
+
+- `domain_plugins` reaches at least 0.85 hard success on private variants
+- pass^k is at least 0.75 across 3 runs
+- worst-of-n is at least 0.65
+- no dominant failure mode accounts for more than 35 percent of failures
+- plugin utilization shows the relevant domain plugin was invoked on tasks
+  where it was required
+
+The broader thesis is credible when:
+
+- at least 10 of 12 domains meet the domain coverage bar
+- plugin lift is larger than model-to-model variance on the same task set
+- holdout variants preserve the same conclusions
+- SNR analysis shows the ranking is signal-dominant, not seed-noise-dominant
+- cross-harness adapters reproduce scores within an agreed tolerance
+
+## Workstream 1: Adapter Execution
+
+Goal: make OpenClaw, Hermes, Codex, and Claude Code comparable through one
+canonical task pipeline.
+
+Near-term:
+
+- keep `--adapter openclaw` as the executable path
+- route OpenClaw through the adapter implementation instead of inline gateway
+  code
+- add compatibility reporting for every task and adapter
+- implement Codex and Claude Code transcript adapters
+- promote Hermes from first-turn runner to full compatible runner where possible
+
+Help wanted:
+
+- harness owners: SDK or CLI entry points that expose full transcripts
+- plugin owners: tool-call provenance and registration traces
+- serving owners: stable model IDs, usage accounting, and reproducible configs
+
+## Workstream 2: Plugin Provenance
+
+Goal: attribute score changes to plugins instead of treating the agent as a
+black box.
+
+Near-term:
+
+- capture plugin registration traces at gateway startup
+- attach plugin owner IDs to every tool call
+- store transcripts and plugin traces alongside result JSON
+- include utilization and manifest-reality gaps in every `--profile` run
+
+Help wanted:
+
+- OpenClaw plugin registry hooks for runtime trace export
+- partner plugins with typed manifests and clean provenance
+- ClawHub metadata sync for manifest cache refresh
+
+## Workstream 3: Domain Corpus
+
+Goal: replace a small public task suite with a coverage matrix for real agent
+SaaS domains.
+
+Near-term:
+
+- 12 domains in `tasks-domain/MANIFEST.yaml`
+- 5 templates per domain
+- 3 private variants per template
+- domain-specific plugin requirement declarations
+- deterministic verifier contracts before any semantic judge
+
+Help wanted:
+
+- partner traces that can be transformed into private variants
+- domain experts to validate task realism and verifier quality
+- infra for private variant generation and contamination audits
+
+## Workstream 4: Serving and Cost Rigor
+
+Goal: compare open and closed models under reproducible serving constraints.
+
+Near-term:
+
+- record model snapshot, provider, serving stack, quantization, GPU class,
+  context length, temperature, reasoning settings, and token accounting
+- report cost/pass and latency/pass alongside capability
+- run open-weight models through vLLM-backed profiles where available
+
+Help wanted:
+
+- vLLM serving recipes for consistent agent-eval runs
+- Hugging Face model hosting and dataset plumbing
+- NVIDIA profiling on representative GPU setups
+
+## Workstream 5: Evidence Package
+
+Goal: make the conclusion auditable by third parties.
+
+Near-term:
+
+- publish public Core v1 results as the reproducibility baseline
+- publish domain coverage matrix without private task bodies
+- publish aggregated per-domain scores, confidence intervals, and failure modes
+- keep private variants for contamination-resistant official scoring
+- publish scripts that regenerate every report from cached run JSON
+
+Help wanted:
+
+- compute credits for multi-model sweeps
+- review from model serving, benchmark, and infrastructure teams
+- public hosting for result artifacts and visual dashboards
+

docs/MEETING_BRIEF_NVIDIA_HF_VLLM_2026-04-24.md

@@ -0,0 +1,108 @@
+# Meeting Brief: Nvidia, Hugging Face, vLLM
+
+Meeting date: April 24, 2026
+
+## One-Liner
+
+ClawBench is a rigorous agent benchmark for measuring whether a model plus a
+general harness plus plugins can cover the task domains served by most agent
+SaaS products.
+
+## What I Built
+
+- A deterministic, trace-based benchmark for agents, not just models.
+- A small public Core v1 set for reproducibility and regression tracking.
+- A larger domain-suite scaffold for CRM, support, docs/sheets/slides, email,
+  calendar, finance ops, analytics, security admin, ecommerce, devtools,
+  research, and personal ops.
+- A scoring system that separates completion, process quality, behavior,
+  semantic quality, reliability, latency, tokens, cost, and failure modes.
+- A dynamics-analysis stack that explains how agents fail: trapped, diffusive,
+  convergent, chaotic, limit-cycle, and survival curves.
+- A plugin-profile diagnostic layer that fingerprints configurations, estimates
+  plugin contribution, detects dead-weight plugins, and recommends changes.
+- An adapter boundary so OpenClaw can become one harness among several rather
+  than the only execution path.
+
+## Goal
+
+Prove, with reproducible data, that specialized agent SaaS can be decomposed
+into:
+
+1. a base model,
+2. a general agent harness,
+3. a plugin stack,
+4. domain-specific state/API access,
+5. deterministic evaluation contracts.
+
+If the data supports it, the conclusion is that the open plugin ecosystem can
+subsume a large share of agent SaaS workflows.
+
+## What The 19 Public Tasks Are
+
+The 19 public tasks are not the whole proof. They are the public Core v1 set:
+
+- reproducibility baseline
+- CI/regression suite
+- adapter bring-up set
+- public explanation of methodology
+
+The proof corpus is the domain suite. That needs more tasks, private variants,
+and ablations.
+
+## What Still Needs Help
+
+- Cross-harness execution: OpenClaw is executable today; Hermes/Codex/Claude
+  Code need end-to-end adapter wiring.
+- Plugin provenance: tool calls need stable plugin owner IDs and registration
+  traces.
+- Domain corpus: each domain needs realistic private variants and hardened
+  deterministic verifiers.
+- Serving reproducibility: open-weight models need pinned serving recipes,
+  GPU profiles, usage accounting, and latency/cost measurement.
+- Scale: the domain ablations need a lot more runs than the public Core set.
+
+## What I Want From Nvidia
+
+- GPU-backed evaluation capacity for repeated domain sweeps.
+- Profiling help: latency/pass, tokens/sec, cost/pass, memory pressure, and
+  concurrency behavior for long agent trajectories.
+- Reference serving profiles for open-weight models on NVIDIA hardware.
+- Advice on making the benchmark useful for enterprise agent deployment, not
+  just academic ranking.
+
+## What I Want From Hugging Face
+
+- Dataset hosting for public results, cached run JSON, and public task metadata.
+- Private/controlled dataset workflow for holdout variants and partner traces.
+- Model hosting paths for open-weight baseline runs.
+- Help making ClawBench results easy to browse, reproduce, and cite.
+
+## What I Want From vLLM
+
+- A stable serving recipe for agent-eval workloads with long context and many
+  tool turns.
+- Usage accounting: prompt, output, reasoning/cache tokens where available.
+- Throughput and latency guidance for many parallel agent runs.
+- Integration advice for making model snapshots and serving configs auditable.
+
+## Proposed Collaboration
+
+1. Run Core v1 as a public sanity check across agreed open and closed models.
+2. Build 12-domain private proof suite from `tasks-domain/`.
+3. Run four ablation classes: model only, model plus harness, core plugins,
+   domain plugins.
+4. Publish aggregated domain coverage, reliability, failure modes, and cost.
+5. Iterate on gaps where specialized SaaS still beats the open stack.
+
+## The Ask
+
+Help make the proof hard to dismiss:
+
+- enough compute to run repetitions,
+- clean serving recipes,
+- model and dataset hosting,
+- infrastructure review,
+- partner traces or realistic domain workflows,
+- public artifacts that other teams can reproduce.
+

docs/kubernetes.md

@@ -1,367 +0,0 @@
-# Running ClawBench on Kubernetes
-
-ClawBench runs as a **sidecar** in the OpenClaw gateway pod. The sidecar
-connects to the gateway over loopback (`ws://localhost:18789`), runs the
-19-task eval suite, and optionally logs results to MLflow.
-
-```
-┌─── OpenClaw Pod ─────────────────────────────┐
-│  gateway container  (ws://localhost:18789)   │
-│  clawbench sidecar  ──► gateway via loopback │
-└──────────────────────────────────────────────┘
-         │                          │
-         ▼                          ▼
-   Model provider API         MLflow (optional)
-```
-
-All commands use `scripts/k8s/deploy.sh`. The script has these modes:
-
-| Flag | What it does |
-|------|-------------|
-| *(none)* | Full deploy: OpenClaw + MLflow + eval sidecar |
-| `--openclaw-only` | Deploy OpenClaw gateway only |
-| `--mlflow-only` | Deploy MLflow only |
-| `--add-sidecar` | Inject clawbench sidecar (starts eval) |
-| `--remove-sidecar` | Remove clawbench sidecar |
-| `--logs` | Tail sidecar logs |
-| `--teardown` | Delete eval namespace (keeps MLflow) |
-
----
-
-## Prerequisites
-
-- `kubectl` on PATH, connected to a cluster (`kubectl cluster-info` succeeds)
-- A container image for ClawBench (see [Building images](#building-images))
-- At least one model provider API key (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.)
-
-For local testing with Kind:
-https://github.com/openclaw/openclaw/blob/main/docs/install/kubernetes.md#local-testing-with-kind
-
----
-
-## Environment variables
-
-Set these **before** running `deploy.sh`.
-
-### Required
-
-| Variable | Purpose |
-|----------|---------|
-| `CLAWBENCH_NAMESPACE` | Namespace for OpenClaw + eval (e.g. `clawbench-eval`) |
-| `OPENAI_API_KEY` | Model provider key (or use another provider — see table below) |
-
-### Optional
-
-| Variable | Default | Purpose |
-|----------|---------|---------|
-| `CLAWBENCH_IMAGE` | `quay.io/sallyom/clawbench:latest` | ClawBench sidecar image |
-| `OPENCLAW_IMAGE` | `ghcr.io/openclaw/openclaw:latest` | OpenClaw gateway image |
-| `OPENCLAW_GATEWAY_TOKEN` | *(generated by script)* | Gateway token; set this when attaching the sidecar to an existing gateway |
-| `CLAWBENCH_MODEL` | `openai/gpt-5.5` | Model to evaluate |
-| `MLFLOW_NAMESPACE` | `mlflow` | MLflow namespace |
-| `MLFLOW_TRACKING_URI` | *(deployed by script)* | External MLflow URI — skips MLflow deploy if set |
-| `MLFLOW_EXPERIMENT_ID` | | MLflow experiment ID |
-| `MLFLOW_EXPERIMENT_NAME` | `clawbench` | MLflow experiment name |
-| `MLFLOW_IMAGE` | `ghcr.io/mlflow/mlflow:v2.21.3` | MLflow server image |
-| `ANTHROPIC_API_KEY` | | Added to K8s secret if set |
-| `OPENROUTER_API_KEY` | | Added to K8s secret if set |
-| `GEMINI_API_KEY` | | Added to K8s secret if set |
-| `OPENAI_API_BASE` | | Base URL for OpenAI-compatible endpoints (e.g. vLLM, Ollama); patched into gateway config |
-
-### Model routing
-
-The gateway routes by provider prefix:
-
-| Model string | Required variables |
-|-------------|-------------------|
-| `openai/gpt-5.5` | `OPENAI_API_KEY` |
-| `anthropic/claude-sonnet-4-6` | `ANTHROPIC_API_KEY` |
-| `openrouter/anthropic/claude-sonnet-4-6` | `OPENROUTER_API_KEY` |
-| `openai/my-local-model` | `OPENAI_API_KEY` + `OPENAI_API_BASE` |
-
-For OpenAI-compatible endpoints (vLLM, Ollama, TGI, or any in-cluster model
-server), set `OPENAI_API_BASE` to the endpoint URL and use the `openai/`
-prefix for the model name:
-
-```bash
-export CLAWBENCH_MODEL="openai/meta-llama/Llama-4-Scout-17B"
-export OPENAI_API_KEY="none"  # dummy value if the endpoint doesn't require auth
-export OPENAI_API_BASE="http://vllm-service.my-ns.svc.cluster.local:8000/v1"
-```
-
----
-
-## Full deploy (quick start)
-
-Deploys OpenClaw gateway, MLflow, and the eval sidecar in one command.
-
-```bash
-export CLAWBENCH_NAMESPACE=clawbench-eval
-
-# Export API keys before running. The script stores them in a K8s Secret
-# ("clawbench-secrets") that the gateway and sidecar containers read.
-export OPENAI_API_KEY="sk-..."
-
-# Model to evaluate (default: openai/gpt-5.5)
-# export CLAWBENCH_MODEL="anthropic/claude-sonnet-4-6"
-
-./scripts/k8s/deploy.sh
-```
-
-Verify:
-
-```bash
-# Should show 2/2 containers (gateway + clawbench)
-kubectl get pods -n clawbench-eval
-
-# Follow eval progress
-./scripts/k8s/deploy.sh --logs
-```
-
-When the eval finishes, copy results and clean up:
-
-```bash
-# Copy results from the sidecar
-POD=$(kubectl get pod -n $CLAWBENCH_NAMESPACE -l app=openclaw -o jsonpath='{.items[0].metadata.name}')
-kubectl cp "$CLAWBENCH_NAMESPACE/$POD:/results/benchmark.json" -c clawbench ./benchmark.json
-
-# Remove the sidecar (keeps OpenClaw + MLflow running)
-./scripts/k8s/deploy.sh --remove-sidecar
-
-# Or tear down everything
-./scripts/k8s/deploy.sh --teardown
-```
-
----
-
-## Existing cluster + existing MLflow
-
-If you already have an OpenShift or Kubernetes cluster and an MLflow instance,
-you only need to deploy OpenClaw and run the eval — no cluster or MLflow setup
-required.
-
-```bash
-export CLAWBENCH_NAMESPACE=clawbench-eval
-
-# API keys — export before running deploy.sh. The script creates a
-# Kubernetes Secret ("clawbench-secrets") from whichever keys are set.
-# At least one provider key is required.
-export OPENAI_API_KEY="sk-..."
-# export ANTHROPIC_API_KEY="sk-ant-..."
-# export OPENROUTER_API_KEY="sk-or-..."
-# export GEMINI_API_KEY="..."
-
-# Model to evaluate (default: openai/gpt-5.5)
-export CLAWBENCH_MODEL="anthropic/claude-sonnet-4-6"
-
-# If attaching to an existing OpenClaw gateway, this must match that gateway.
-# If deploy.sh creates OpenClaw, it generates this token for you.
-# export OPENCLAW_GATEWAY_TOKEN="..."
-
-# Point to your existing MLflow
-export MLFLOW_TRACKING_URI="https://mlflow.example.com"
-export MLFLOW_EXPERIMENT_NAME="clawbench-gpt5.5"  # or use MLFLOW_EXPERIMENT_ID=42
-
-# Deploy OpenClaw gateway into your cluster
-./scripts/k8s/deploy.sh --openclaw-only
-```
-
-Verify OpenClaw is running:
-
-```bash
-kubectl get pods -n clawbench-eval
-# Expect: openclaw-xxxx  1/1  Running
-```
-
-Then start the eval:
-
-```bash
-./scripts/k8s/deploy.sh --add-sidecar
-./scripts/k8s/deploy.sh --logs
-```
-
-The deploy script sets `MLFLOW_TRACKING_URI` to skip its own MLflow deployment
-and patches the experiment name/ID into the clawbench ConfigMap. When the eval
-completes, `scripts/log_to_mlflow.py` logs results to your MLflow under that
-experiment.
-
-`MLFLOW_EXPERIMENT_NAME` creates the experiment if it doesn't exist.
-`MLFLOW_EXPERIMENT_ID` requires an existing experiment.
-
----
-
-## Step-by-step deploy
-
-Use this when you want to deploy components individually or bring your own
-OpenClaw/MLflow.
-
-### Step 1: Deploy OpenClaw gateway
-
-```bash
-export CLAWBENCH_NAMESPACE=clawbench-eval
-export OPENAI_API_KEY="sk-..."
-./scripts/k8s/deploy.sh --openclaw-only
-```
-
-Verify:
-
-```bash
-kubectl get pods -n clawbench-eval
-# Expect: openclaw-xxxx  1/1  Running
-```
-
-This deploys from `scripts/k8s/openclaw/`: a single gateway pod with token
-auth, ClusterIP service, and 10Gi PVC. The deploy script generates a gateway
-token and creates the `clawbench-secrets` Secret automatically.
-
-**Skip this step** if you already have an OpenClaw deployment. Your existing
-gateway must have this config (see `scripts/k8s/openclaw/configmap.yaml`):
-
-```json
-{
-  "browser": {
-    "enabled": true,
-    "headless": true,
-    "noSandbox": true,
-    "ssrfPolicy": {
-      "allowedHostnames": ["localhost", "127.0.0.1"]
-    }
-  },
-  "tools": {
-    "profile": "coding",
-    "alsoAllow": ["browser"]
-  }
-}
-```
-
-Key requirements:
-- `browser.enabled: true` — activates the bundled browser plugin
-- `tools.alsoAllow: ["browser"]` — the `coding` profile does NOT include browser by default
-- `browser.ssrfPolicy` — several eval tasks need localhost access
-- Gateway must bind to loopback with token auth; export the matching
-  `OPENCLAW_GATEWAY_TOKEN` before running `--add-sidecar`
-
-### Step 2: Deploy MLflow
-
-```bash
-./scripts/k8s/deploy.sh --mlflow-only
-```
-
-Verify:
-
-```bash
-kubectl get pods -n mlflow
-# Expect: mlflow-xxxx  1/1  Running
-```
-
-Deploys a single-replica MLflow server with SQLite backend into the `mlflow`
-namespace. The clawbench ConfigMap defaults to
-`http://mlflow-service.mlflow.svc.cluster.local:5000`.
-
-**Skip this step** if you have an external MLflow — set `MLFLOW_TRACKING_URI`:
-
-```bash
-export MLFLOW_TRACKING_URI=http://my-mlflow.example.com:5000
-export MLFLOW_EXPERIMENT_ID=4  # or MLFLOW_EXPERIMENT_NAME
-```
-
-### Step 3: Run the eval
-
-```bash
-./scripts/k8s/deploy.sh --add-sidecar
-```
-
-This patches the OpenClaw deployment to inject a clawbench sidecar that:
-
-1. Waits for the gateway (TCP check on port 18789, up to 3 min)
-2. Checks MLflow connectivity if configured
-3. Runs `clawbench run` with settings from the ConfigMap
-4. Logs results to MLflow on success
-5. Sleeps indefinitely so you can retrieve logs and results
-
-Verify:
-
-```bash
-kubectl get pods -n $CLAWBENCH_NAMESPACE
-# Expect: openclaw-xxxx  2/2  Running  (gateway + clawbench)
-
-./scripts/k8s/deploy.sh --logs
-# Should show "Waiting for gateway..." then "Starting eval..."
-```
-
-When finished, remove the sidecar:
-
-```bash
-./scripts/k8s/deploy.sh --remove-sidecar
-```
-
----
-
-## ConfigMap tuning
-
-The clawbench ConfigMap (`scripts/k8s/manifests/configmap.yaml`) controls eval
-behavior. Override at deploy time via env vars, or patch after deploy:
-
-| Key | Default | What it controls |
-|-----|---------|-----------------|
-| `CLAWBENCH_MODEL` | `openai/gpt-5.5` | Model under test |
-| `CLAWBENCH_RUNS` | `3` | Runs per task (19 tasks x 3 = 57 total) |
-| `CLAWBENCH_CONCURRENCY` | `4` | Parallel eval lanes |
-| `CLAWBENCH_JUDGE_MODEL` | *(empty)* | Separate judge model (optional) |
-| `CLAWBENCH_TASKS` | *(empty — runs all)* | Space-separated task IDs (e.g. `t1-bugfix-discount t2-config-loader`) |
-| `CLAWBENCH_CONNECT_TIMEOUT` | `120` | Gateway connect timeout in seconds |
-| `CLAWBENCH_REQUEST_TIMEOUT` | `300` | Per-request timeout in seconds |
-| `CLAWBENCH_PER_RUN_BUDGET_SECONDS` | `600` | Max wall time per run |
-| `MLFLOW_TRACKING_URI` | `http://mlflow-service.mlflow.svc.cluster.local:5000` | MLflow endpoint |
-| `MLFLOW_EXPERIMENT_NAME` | `clawbench` | MLflow experiment name |
-
----
-
-## MLflow integration
-
-Results are logged via `scripts/log_to_mlflow.py` after a successful eval.
-
-**What gets logged:**
-- **Params**: model, provider, benchmark version, OpenClaw version, judge model
-- **Metrics**: overall score, per-axis scores (completion, trajectory, behavior,
-  reliability), cost, tokens, latency, CI bounds, per-tier and per-task scores
-- **Tags**: submission ID, timestamp, certified flag
-- **Artifacts**: full benchmark result JSON
-
----
-
-## Building images
-
-### ClawBench image
-
-`quay.io/sallyom/clawbench:latest` is public
-
-For Kubernetes, use the lightweight sidecar image instead — it only includes
-the eval harness and MLflow client:
-
-```bash
-docker build -t clawbench:latest -f scripts/k8s/Dockerfile .
-
-# For Kind clusters, load directly instead of pushing to a registry:
-kind load docker-image clawbench:latest --name openclaw
-
-# For non-Kind clusters, push to registry and set CLAWBENCH_IMAGE accordingly
-# Ensure you build for the right architecture, usually amd64 for non-local k8s
-```
-
-Set `CLAWBENCH_IMAGE=clawbench:latest` when running `deploy.sh` to use it.
-
----
-
-## Cleanup
-
-```bash
-# Remove eval sidecar only (keeps OpenClaw + MLflow running for another eval)
-./scripts/k8s/deploy.sh --remove-sidecar
-
-# Delete eval namespace (keeps MLflow running)
-./scripts/k8s/deploy.sh --teardown
-
-# Delete the Kind cluster entirely
-kind delete cluster --name openclaw
-```

patches/patch_openclaw_426_agent_create_queue.mjs

@@ -0,0 +1,181 @@
+import { readFileSync, writeFileSync } from "node:fs";
+
+const dist = "/app/dist/server-methods-b3jaTRE_.js";
+
+function replaceOnce(text, oldValue, newValue) {
+  if (!text.includes(oldValue)) {
+    throw new Error(`patch target not found: ${oldValue.slice(0, 80)}`);
+  }
+  return text.replace(oldValue, newValue);
+}
+
+let source = readFileSync(dist, "utf8");
+
+source = replaceOnce(
+  source,
+  "const agentsHandlers = {\n",
+  `let agentConfigMutationQueue = Promise.resolve();
+async function runAgentConfigMutation(fn) {
+\tconst previous = agentConfigMutationQueue;
+\tlet release;
+\tagentConfigMutationQueue = new Promise((resolve) => {
+\t\trelease = resolve;
+\t});
+\tawait previous.catch(() => {});
+\ttry {
+\t\treturn await fn();
+\t} finally {
+\t\trelease();
+\t}
+}
+const agentsHandlers = {
+`,
+);
+
+source = replaceOnce(
+  source,
+  `\t\tconst cfg = context.getRuntimeConfig();
+\t\tconst rawName = params.name.trim();`,
+  `\t\tconst rawName = params.name.trim();`,
+);
+
+source = replaceOnce(
+  source,
+  `\t\tif (findAgentEntryIndex(listAgentEntries(cfg), agentId) >= 0) {
+\t\t\trespond(false, void 0, errorShape(ErrorCodes.INVALID_REQUEST, \`agent "\${agentId}" already exists\`));
+\t\t\treturn;
+\t\t}
+\t\tconst workspaceDir = resolveUserPath(params.workspace.trim());`,
+  `\t\tconst workspaceDir = resolveUserPath(params.workspace.trim());`,
+);
+
+source = replaceOnce(
+  source,
+  `\t\tlet nextConfig = applyAgentConfig(cfg, {
+\t\t\tagentId,
+\t\t\tname: safeName,
+\t\t\tworkspace: workspaceDir,
+\t\t\tmodel,
+\t\t\tidentity: {
+\t\t\t\tname: safeName,
+\t\t\t\t...emoji ? { emoji: sanitizeIdentityLine(emoji) } : {},
+\t\t\t\t...avatar ? { avatar: sanitizeIdentityLine(avatar) } : {}
+\t\t\t}
+\t\t});
+\t\tconst agentDir = resolveAgentDir(nextConfig, agentId);
+\t\tnextConfig = applyAgentConfig(nextConfig, {
+\t\t\tagentId,
+\t\t\tagentDir
+\t\t});
+\t\tawait ensureAgentWorkspace({
+\t\t\tdir: workspaceDir,
+\t\t\tensureBootstrapFiles: !Boolean(nextConfig.agents?.defaults?.skipBootstrap)
+\t\t});
+\t\tawait fs$1.mkdir(resolveSessionTranscriptsDirForAgent(agentId), { recursive: true });
+\t\tconst persistedIdentity = normalizeIdentityForFile(resolveAgentIdentity(nextConfig, agentId));
+\t\tif (persistedIdentity) {
+\t\t\tconst identityContent = await buildIdentityMarkdownOrRespondUnsafe({
+\t\t\t\trespond,
+\t\t\t\tworkspaceDir,
+\t\t\t\tidentity: persistedIdentity
+\t\t\t});
+\t\t\tif (identityContent === null) return;
+\t\t\tif (!await writeWorkspaceFileOrRespond({
+\t\t\t\trespond,
+\t\t\t\tworkspaceDir,
+\t\t\t\tname: "IDENTITY.md",
+\t\t\t\tcontent: identityContent
+\t\t\t})) return;
+\t\t}
+\t\tawait replaceConfigFile({
+\t\t\tnextConfig,
+\t\t\tafterWrite: { mode: "auto" }
+\t\t});
+\t\trespond(true, {
+\t\t\tok: true,
+\t\t\tagentId,
+\t\t\tname: safeName,
+\t\t\tworkspace: workspaceDir,
+\t\t\tmodel
+\t\t}, void 0);`,
+  `\t\tconst result = await runAgentConfigMutation(async () => {
+\t\t\tconst cfg = context.getRuntimeConfig();
+\t\t\tif (findAgentEntryIndex(listAgentEntries(cfg), agentId) >= 0) {
+\t\t\t\trespond(false, void 0, errorShape(ErrorCodes.INVALID_REQUEST, \`agent "\${agentId}" already exists\`));
+\t\t\t\treturn null;
+\t\t\t}
+\t\t\tlet nextConfig = applyAgentConfig(cfg, {
+\t\t\t\tagentId,
+\t\t\t\tname: safeName,
+\t\t\t\tworkspace: workspaceDir,
+\t\t\t\tmodel,
+\t\t\t\tidentity: {
+\t\t\t\t\tname: safeName,
+\t\t\t\t\t...emoji ? { emoji: sanitizeIdentityLine(emoji) } : {},
+\t\t\t\t\t...avatar ? { avatar: sanitizeIdentityLine(avatar) } : {}
+\t\t\t\t}
+\t\t\t});
+\t\t\tconst agentDir = resolveAgentDir(nextConfig, agentId);
+\t\t\tnextConfig = applyAgentConfig(nextConfig, {
+\t\t\t\tagentId,
+\t\t\t\tagentDir
+\t\t\t});
+\t\t\tawait ensureAgentWorkspace({
+\t\t\t\tdir: workspaceDir,
+\t\t\t\tensureBootstrapFiles: !Boolean(nextConfig.agents?.defaults?.skipBootstrap)
+\t\t\t});
+\t\t\tawait fs$1.mkdir(resolveSessionTranscriptsDirForAgent(agentId), { recursive: true });
+\t\t\tconst persistedIdentity = normalizeIdentityForFile(resolveAgentIdentity(nextConfig, agentId));
+\t\t\tif (persistedIdentity) {
+\t\t\t\tconst identityContent = await buildIdentityMarkdownOrRespondUnsafe({
+\t\t\t\t\trespond,
+\t\t\t\t\tworkspaceDir,
+\t\t\t\t\tidentity: persistedIdentity
+\t\t\t\t});
+\t\t\t\tif (identityContent === null) return null;
+\t\t\t\tif (!await writeWorkspaceFileOrRespond({
+\t\t\t\t\trespond,
+\t\t\t\t\tworkspaceDir,
+\t\t\t\t\tname: "IDENTITY.md",
+\t\t\t\t\tcontent: identityContent
+\t\t\t\t})) return null;
+\t\t\t}
+\t\t\tawait replaceConfigFile({
+\t\t\t\tnextConfig,
+\t\t\t\tafterWrite: { mode: "auto" }
+\t\t\t});
+\t\t\treturn true;
+\t\t});
+\t\tif (!result) return;
+\t\trespond(true, {
+\t\t\tok: true,
+\t\t\tagentId,
+\t\t\tname: safeName,
+\t\t\tworkspace: workspaceDir,
+\t\t\tmodel
+\t\t}, void 0);`,
+);
+
+for (const marker of [
+  `\t\t\tawait replaceConfigFile({
+\t\t\t\tnextConfig,
+\t\t\t\tafterWrite: { mode: "auto" }
+\t\t\t});`,
+  `\t\tawait replaceConfigFile({
+\t\t\tnextConfig,
+\t\t\tafterWrite: { mode: "auto" }
+\t\t});`,
+  `\t\tawait replaceConfigFile({
+\t\t\tnextConfig: result.config,
+\t\t\tafterWrite: { mode: "auto" }
+\t\t});`,
+]) {
+  source = replaceOnce(
+    source,
+    marker,
+    marker.replace(`{ mode: "auto" }`, `{ mode: "none", reason: "clawbench-agent-lifecycle" }`),
+  );
+}
+
+writeFileSync(dist, source);
+console.log(`patched ${dist}`);

patches/patch_opus47.py

@@ -0,0 +1,212 @@
+#!/usr/bin/env python3
+"""Patch pi-ai and openclaw bundles to recognize claude-opus-4-7 (and sonnet-4-7).
+
+Runs inside the Docker image as a RUN step. Idempotent: re-running is a no-op.
+"""
+
+import re
+import sys
+import os
+
+PI_AI_CATALOG = "/app/node_modules/@mariozechner/pi-ai/dist/models.generated.js"
+ANTHROPIC_REGISTER_GLOB = "/app/dist/register.runtime-*.js"
+
+
+def patch_pi_ai_catalog(path: str) -> bool:
+    with open(path) as fh:
+        src = fh.read()
+    if '"claude-opus-4-7"' in src:
+        print(f"[patch] {path}: claude-opus-4-7 already present, skipping")
+        return False
+
+    # Find the claude-opus-4-6 entry and splice in opus-4-7 + sonnet-4-7 right after.
+    # Use substring scanning rather than regex because each entry contains a nested
+    # `cost: { ... }` object (which breaks naive `[^{}]` patterns).
+    start_marker = '"claude-opus-4-6": {'
+    start_idx = src.find(start_marker)
+    if start_idx == -1:
+        print(f"[patch] ERROR: could not locate claude-opus-4-6 anchor in {path}", file=sys.stderr)
+        sys.exit(1)
+    # Walk forward from the opening `{` counting nesting until it balances to 0.
+    depth = 0
+    i = start_idx
+    while i < len(src):
+        ch = src[i]
+        if ch == '{':
+            depth += 1
+        elif ch == '}':
+            depth -= 1
+            if depth == 0:
+                i += 1  # include '}'
+                break
+        i += 1
+    if depth != 0:
+        print(f"[patch] ERROR: unbalanced braces walking claude-opus-4-6 entry in {path}", file=sys.stderr)
+        sys.exit(1)
+    # There should be a trailing comma after the closing brace.
+    if i < len(src) and src[i] == ',':
+        i += 1
+    anchor_end = i
+
+    class _M:
+        def __init__(self, end): self._end = end
+        def end(self): return self._end
+    m = _M(anchor_end)
+
+    insertion = (
+        "\n"
+        '        "claude-opus-4-7": {\n'
+        '            id: "claude-opus-4-7",\n'
+        '            name: "Claude Opus 4.7",\n'
+        '            api: "anthropic-messages",\n'
+        '            provider: "anthropic",\n'
+        '            baseUrl: "https://api.anthropic.com",\n'
+        "            reasoning: true,\n"
+        '            input: ["text", "image"],\n'
+        "            cost: {\n"
+        "                input: 5,\n"
+        "                output: 25,\n"
+        "                cacheRead: 0.5,\n"
+        "                cacheWrite: 6.25,\n"
+        "            },\n"
+        "            contextWindow: 1000000,\n"
+        "            maxTokens: 128000,\n"
+        "        },\n"
+        '        "claude-sonnet-4-7": {\n'
+        '            id: "claude-sonnet-4-7",\n'
+        '            name: "Claude Sonnet 4.7",\n'
+        '            api: "anthropic-messages",\n'
+        '            provider: "anthropic",\n'
+        '            baseUrl: "https://api.anthropic.com",\n'
+        "            reasoning: true,\n"
+        '            input: ["text", "image"],\n'
+        "            cost: {\n"
+        "                input: 3,\n"
+        "                output: 15,\n"
+        "                cacheRead: 0.3,\n"
+        "                cacheWrite: 3.75,\n"
+        "            },\n"
+        "            contextWindow: 1000000,\n"
+        "            maxTokens: 128000,\n"
+        "        },"
+    )
+
+    patched = src[: m.end()] + insertion + src[m.end():]
+    with open(path, "w") as fh:
+        fh.write(patched)
+    print(f"[patch] {path}: inserted claude-opus-4-7 and claude-sonnet-4-7")
+    return True
+
+
+def patch_openclaw_anthropic_register(path: str) -> bool:
+    with open(path) as fh:
+        src = fh.read()
+    if "ANTHROPIC_OPUS_47_MODEL_ID" in src:
+        print(f"[patch] {path}: 4-7 support already present, skipping")
+        return False
+
+    # Skip files that are not the anthropic register.runtime (other plugins
+    # share the same `register.runtime-*.js` naming convention).
+    if 'PROVIDER_ID = "anthropic"' not in src or "ANTHROPIC_MODERN_MODEL_PREFIXES" not in src:
+        print(f"[patch] {path}: not the anthropic register.runtime bundle, skipping")
+        return False
+
+    # 1. Inject new constants after the sonnet template constant.
+    sonnet_tpl_anchor = 'const ANTHROPIC_SONNET_TEMPLATE_MODEL_IDS = ["claude-sonnet-4-5", "claude-sonnet-4.5"];'
+    if sonnet_tpl_anchor not in src:
+        print(f"[patch] ERROR: sonnet template anchor not found in {path}", file=sys.stderr)
+        sys.exit(1)
+    new_consts = (
+        sonnet_tpl_anchor + "\n"
+        'const ANTHROPIC_OPUS_47_MODEL_ID = "claude-opus-4-7";\n'
+        'const ANTHROPIC_OPUS_47_DOT_MODEL_ID = "claude-opus-4.7";\n'
+        'const ANTHROPIC_SONNET_47_MODEL_ID = "claude-sonnet-4-7";\n'
+        'const ANTHROPIC_SONNET_47_DOT_MODEL_ID = "claude-sonnet-4.7";'
+    )
+    src = src.replace(sonnet_tpl_anchor, new_consts)
+
+    # 2. Extend ANTHROPIC_MODERN_MODEL_PREFIXES.
+    prefixes_anchor = 'const ANTHROPIC_MODERN_MODEL_PREFIXES = [\n\t"claude-opus-4-6",\n\t"claude-sonnet-4-6",'
+    prefixes_new = 'const ANTHROPIC_MODERN_MODEL_PREFIXES = [\n\t"claude-opus-4-7",\n\t"claude-sonnet-4-7",\n\t"claude-opus-4-6",\n\t"claude-sonnet-4-6",'
+    if prefixes_anchor not in src:
+        print(f"[patch] ERROR: modern prefixes anchor not found in {path}", file=sys.stderr)
+        sys.exit(1)
+    src = src.replace(prefixes_anchor, prefixes_new)
+
+    # 3. Add 4-7 forward-compat branches ahead of the 4-6 opus/sonnet branches.
+    resolve_anchor = (
+        "function resolveAnthropicForwardCompatModel(ctx) {\n"
+        "\treturn resolveAnthropic46ForwardCompatModel({\n"
+        "\t\tctx,\n"
+        "\t\tdashModelId: ANTHROPIC_OPUS_46_MODEL_ID,"
+    )
+    resolve_new = (
+        "function resolveAnthropicForwardCompatModel(ctx) {\n"
+        "\treturn resolveAnthropic46ForwardCompatModel({\n"
+        "\t\tctx,\n"
+        '\t\tdashModelId: ANTHROPIC_OPUS_47_MODEL_ID,\n'
+        '\t\tdotModelId: ANTHROPIC_OPUS_47_DOT_MODEL_ID,\n'
+        '\t\tdashTemplateId: "claude-opus-4-6",\n'
+        '\t\tdotTemplateId: "claude-opus-4.6",\n'
+        "\t\tfallbackTemplateIds: ANTHROPIC_OPUS_TEMPLATE_MODEL_IDS\n"
+        "\t}) ?? resolveAnthropic46ForwardCompatModel({\n"
+        "\t\tctx,\n"
+        '\t\tdashModelId: ANTHROPIC_SONNET_47_MODEL_ID,\n'
+        '\t\tdotModelId: ANTHROPIC_SONNET_47_DOT_MODEL_ID,\n'
+        '\t\tdashTemplateId: "claude-sonnet-4-6",\n'
+        '\t\tdotTemplateId: "claude-sonnet-4.6",\n'
+        "\t\tfallbackTemplateIds: ANTHROPIC_SONNET_TEMPLATE_MODEL_IDS\n"
+        "\t}) ?? resolveAnthropic46ForwardCompatModel({\n"
+        "\t\tctx,\n"
+        "\t\tdashModelId: ANTHROPIC_OPUS_46_MODEL_ID,"
+    )
+    if resolve_anchor not in src:
+        print(f"[patch] ERROR: forward-compat resolver anchor not found in {path}", file=sys.stderr)
+        sys.exit(1)
+    src = src.replace(resolve_anchor, resolve_new)
+
+    # 4. Make adaptive-thinking default cover 4-7 too.
+    adaptive_anchor = (
+        "function shouldUseAnthropicAdaptiveThinkingDefault(modelId) {\n"
+        "\tconst lowerModelId = normalizeLowercaseStringOrEmpty(modelId);\n"
+        "\treturn lowerModelId.startsWith(ANTHROPIC_OPUS_46_MODEL_ID) || lowerModelId.startsWith(ANTHROPIC_OPUS_46_DOT_MODEL_ID) || lowerModelId.startsWith(ANTHROPIC_SONNET_46_MODEL_ID) || lowerModelId.startsWith(ANTHROPIC_SONNET_46_DOT_MODEL_ID);\n"
+        "}"
+    )
+    adaptive_new = (
+        "function shouldUseAnthropicAdaptiveThinkingDefault(modelId) {\n"
+        "\tconst lowerModelId = normalizeLowercaseStringOrEmpty(modelId);\n"
+        "\treturn lowerModelId.startsWith(ANTHROPIC_OPUS_47_MODEL_ID) || lowerModelId.startsWith(ANTHROPIC_OPUS_47_DOT_MODEL_ID) || lowerModelId.startsWith(ANTHROPIC_SONNET_47_MODEL_ID) || lowerModelId.startsWith(ANTHROPIC_SONNET_47_DOT_MODEL_ID) || lowerModelId.startsWith(ANTHROPIC_OPUS_46_MODEL_ID) || lowerModelId.startsWith(ANTHROPIC_OPUS_46_DOT_MODEL_ID) || lowerModelId.startsWith(ANTHROPIC_SONNET_46_MODEL_ID) || lowerModelId.startsWith(ANTHROPIC_SONNET_46_DOT_MODEL_ID);\n"
+        "}"
+    )
+    if adaptive_anchor in src:
+        src = src.replace(adaptive_anchor, adaptive_new)
+
+    with open(path, "w") as fh:
+        fh.write(src)
+    print(f"[patch] {path}: added claude-opus-4-7 / claude-sonnet-4-7 forward-compat support")
+    return True
+
+
+def main() -> None:
+    import glob
+
+    any_changed = False
+    if os.path.exists(PI_AI_CATALOG):
+        any_changed |= patch_pi_ai_catalog(PI_AI_CATALOG)
+    else:
+        print(f"[patch] WARNING: {PI_AI_CATALOG} not found", file=sys.stderr)
+
+    candidates = sorted(glob.glob(ANTHROPIC_REGISTER_GLOB))
+    if not candidates:
+        print(f"[patch] WARNING: no files match {ANTHROPIC_REGISTER_GLOB}", file=sys.stderr)
+    for cand in candidates:
+        any_changed |= patch_openclaw_anthropic_register(cand)
+
+    if any_changed:
+        print("[patch] success")
+    else:
+        print("[patch] no changes applied (already patched)")
+
+
+if __name__ == "__main__":
+    main()

profiles/frontier_deepseek_v4.yaml

@@ -0,0 +1,26 @@
+profile:
+  name: frontier-deepseek-v4
+  base_model: deepseek/v4-pro
+  notes: |
+    Frontier agentic coding model comparison: DeepSeek V4.
+    DeepSeek direct API. Plugin stack IDENTICAL across all 10 profiles so the
+    base model is the only structural variable. Any score delta is attributable
+    to the model, not the scaffold.
+  plugins:
+    enabled:
+      - anthropic
+      - id: memory-lancedb
+        config:
+          dimensions: 1536
+      - browser-playwright
+    slots:
+      memory: memory-lancedb
+      contextEngine: builtin
+    tools_allow:
+      - bash
+      - file_read
+      - file_edit
+      - browser_navigate
+      - browser_click
+      - memory_read
+      - memory_write

profiles/frontier_gpt_5_2.yaml

@@ -0,0 +1,26 @@
+profile:
+  name: frontier-gpt-5-2
+  base_model: openai/gpt-5.2
+  notes: |
+    Frontier agentic coding model comparison: GPT-5.2 (closed).
+    OpenAI mid-tier flagship. Plugin stack IDENTICAL across all profiles so the base
+    model is the only structural variable. Any score delta is attributable
+    to the model, not the scaffold.
+  plugins:
+    enabled:
+      - anthropic
+      - id: memory-lancedb
+        config:
+          dimensions: 1536
+      - browser-playwright
+    slots:
+      memory: memory-lancedb
+      contextEngine: builtin
+    tools_allow:
+      - bash
+      - file_read
+      - file_edit
+      - browser_navigate
+      - browser_click
+      - memory_read
+      - memory_write

profiles/frontier_gpt_5_5.yaml

@@ -0,0 +1,26 @@
+profile:
+  name: frontier-gpt-5-5
+  base_model: openai/gpt-5.5
+  notes: |
+    Frontier agentic coding model comparison: GPT-5.5 (closed).
+    OpenAI flagship. Plugin stack IDENTICAL across all frontier profiles so
+    the base model is the only structural variable. Any score delta is
+    attributable to the model, not the scaffold.
+  plugins:
+    enabled:
+      - anthropic
+      - id: memory-lancedb
+        config:
+          dimensions: 1536
+      - browser-playwright
+    slots:
+      memory: memory-lancedb
+      contextEngine: builtin
+    tools_allow:
+      - bash
+      - file_read
+      - file_edit
+      - browser_navigate
+      - browser_click
+      - memory_read
+      - memory_write

profiles/frontier_kimi_k26.yaml

@@ -0,0 +1,26 @@
+profile:
+  name: frontier-kimi-k26
+  base_model: openrouter/moonshotai/kimi-k2.6
+  notes: |
+    Frontier agentic coding model comparison: Kimi K2.6 (open).
+    Moonshot AI newer revision. Plugin stack IDENTICAL across profiles so
+    the base model is the only structural variable. Any score delta is
+    attributable to the model, not the scaffold.
+  plugins:
+    enabled:
+      - anthropic
+      - id: memory-lancedb
+        config:
+          dimensions: 1536
+      - browser-playwright
+    slots:
+      memory: memory-lancedb
+      contextEngine: builtin
+    tools_allow:
+      - bash
+      - file_read
+      - file_edit
+      - browser_navigate
+      - browser_click
+      - memory_read
+      - memory_write

profiles/frontier_opus_4_7.yaml

@@ -0,0 +1,26 @@
+profile:
+  name: frontier-opus-4-7
+  base_model: anthropic/claude-opus-4-7
+  notes: |
+    Frontier agentic coding model comparison: Claude Opus 4.7 (closed).
+    Anthropic flagship, newer revision. Plugin stack IDENTICAL to opus-4-6
+    and the other frontier profiles so the base model is the only structural
+    variable. Any score delta is attributable to the model, not the scaffold.
+  plugins:
+    enabled:
+      - anthropic
+      - id: memory-lancedb
+        config:
+          dimensions: 1536
+      - browser-playwright
+    slots:
+      memory: memory-lancedb
+      contextEngine: builtin
+    tools_allow:
+      - bash
+      - file_read
+      - file_edit
+      - browser_navigate
+      - browser_click
+      - memory_read
+      - memory_write

profiles/frontier_sonnet_4_6.yaml

@@ -0,0 +1,26 @@
+profile:
+  name: frontier-sonnet-4-6
+  base_model: anthropic/claude-sonnet-4-6
+  notes: |
+    Frontier agentic coding model comparison: Claude Sonnet 4.6 (closed).
+    Anthropic mid-tier flagship. Plugin stack IDENTICAL across all profiles so the base
+    model is the only structural variable. Any score delta is attributable
+    to the model, not the scaffold.
+  plugins:
+    enabled:
+      - anthropic
+      - id: memory-lancedb
+        config:
+          dimensions: 1536
+      - browser-playwright
+    slots:
+      memory: memory-lancedb
+      contextEngine: builtin
+    tools_allow:
+      - bash
+      - file_read
+      - file_edit
+      - browser_navigate
+      - browser_click
+      - memory_read
+      - memory_write

pyproject.toml

@@ -10,17 +10,16 @@ dependencies = [
     "pydantic>=2.7,<3",
     "pyyaml>=6.0,<7",
     "datasets>=3.0,<4",
-    "gradio>=6.7.0,<7",
-    "pillow>=12.2.0,<13",
+    "gradio>=5.0,<6",
     "httpx>=0.27,<1",
     "numpy>=1.26,<3",
-    "rich>=13.0,<14",
+    "rich>=13.0,<15",
     "click>=8.1,<9",
     # Runtime deps for the task completion verifier. The harness shells out
     # to `pytest -q` / `pytest-asyncio` inside per-task workspaces as the
     # execution check; the container must have them in PATH.
-    "pytest>=9.0.3,<10",
-    "pytest-asyncio>=1,<2",
+    "pytest>=8.0,<9",
+    "pytest-asyncio>=0.24,<1",
 ]
 
 [project.optional-dependencies]
@@ -28,23 +27,13 @@ dev = [
     # Kept as an alias for historical `pip install .[dev]` invocations.
     # pytest + pytest-asyncio are now in the base [dependencies] since the
     # benchmark itself runs pytest in task workspaces.
-    "pytest>=9.0.3,<10",
-    "pytest-asyncio>=1,<2",
-    "pre-commit>=4.0,<5",
-    "ruff>=0.9,<1",
-]
-mlflow = [
-    "mlflow>=2.10,<3",
+    "pytest>=8.0,<9",
+    "pytest-asyncio>=0.24,<1",
 ]
 hermes = [
     "hermes-agent @ git+https://github.com/NousResearch/hermes-agent.git@main",
 ]
 
-[project.urls]
-Homepage = "https://github.com/openclaw/clawbench"
-Repository = "https://github.com/openclaw/clawbench"
-"Bug Tracker" = "https://github.com/openclaw/clawbench/issues"
-
 [project.scripts]
 clawbench = "clawbench.cli:main"
 
@@ -54,20 +43,11 @@ build-backend = "hatchling.build"
 
 [tool.hatch.build.targets.wheel]
 packages = ["clawbench"]
-force-include = { "tasks-public" = "tasks-public", "tasks-domain" = "tasks-domain", "profiles" = "profiles", "baselines" = "baselines", "CLAWBENCH_V0_4_SPEC.md" = "CLAWBENCH_V0_4_SPEC.md", "PARTNER_TRACE_SPEC.md" = "PARTNER_TRACE_SPEC.md" }
-
-[tool.hatch.metadata]
-allow-direct-references = true
+force-include = { "tasks-public" = "tasks-public", "profiles" = "profiles", "baselines" = "baselines", "CLAWBENCH_V0_4_SPEC.md" = "CLAWBENCH_V0_4_SPEC.md", "PARTNER_TRACE_SPEC.md" = "PARTNER_TRACE_SPEC.md" }
 
 [tool.pytest.ini_options]
 asyncio_mode = "auto"
-addopts = ["-p", "no:opik"]
 testpaths = ["tests"]
 
-[tool.ruff]
-line-length = 100
-target-version = "py311"
-
-[tool.ruff.lint]
-select = ["E4", "E7", "E9", "F"]
-ignore = ["E402"]
+[tool.hatch.metadata]
+allow-direct-references = true

scripts/_archive_cache.sh

@@ -1,13 +1,13 @@
 #!/bin/bash
-# Shared helper sourced by container_sweep_*.sh scripts to snapshot the
-# per-model run_cache after a sweep completes. Called at END of each sweep.
+# Shared helper sourced by container runner scripts to snapshot the per-model
+# run_cache after a sweep completes. Called at END of each sweep.
 #
 # Requires these env vars (already set by parent script):
 #   CLAWBENCH_RUN_CACHE_DIR   - e.g. /data/run_cache
 #   CACHE_SUB                 - e.g. openai_gpt-5.4
-#   SWEEP_OUT_TAG             - e.g. v2026-4-18-pr68627-gpt54
+#   SWEEP_OUT_TAG             - e.g. core-v1-public
 #   SWEEP_LABEL               - e.g. gpt54
-#   SWEEP_LOGDIR              - e.g. /data/drift_2026-04-18-pr68627-gpt54
+#   SWEEP_LOGDIR              - e.g. /data/core-v1-public
 #
 # Writes snapshot to: /data/run_cache_archive/<SWEEP_OUT_TAG>/<CACHE_SUB>/
 # Also writes a metadata.json with sweep label/model/timestamp for indexing.

scripts/analyze_open_vs_closed.py

@@ -18,6 +18,7 @@ Usage:
 from __future__ import annotations
 
 import argparse
+import json
 import statistics
 import sys
 from collections import defaultdict

scripts/audit_per_run.py

@@ -1,258 +0,0 @@
-"""Per-run 1-to-1 audit across every (model, task, run_idx) triple.
-
-Flags issues beyond aggregate coverage:
-  - Tasks where ALL models score 0 (task broken / verifier rejects everyone)
-  - Tasks where models produce output but all get C=0 (verifier bug)
-  - Tasks with suspiciously high cross-model infra-failure rates (harness bug)
-  - Specific runs with harness errors (timeout, handshake)
-  - Models with task-specific pathology (e.g., always fails on t3-X)
-  - Judge failures per-task that haven't been rejudged
-  - Missing runs in archive (logged but not cached)
-
-Usage: python3 scripts/audit_per_run.py
-"""
-
-from __future__ import annotations
-
-import json
-import re
-from collections import defaultdict
-from pathlib import Path
-
-ROOT = Path(__file__).resolve().parent.parent
-DRIFT = ROOT / "data" / "drift_2026-04-19-full"
-ARCH = ROOT / "data" / "run_cache_archive" / "v2026-4-19-full"
-
-MODEL_MAP = {
-    "opus46":   ("anthropic_claude-opus-4-6", "opus-4-6"),
-    "opus47":   ("anthropic_claude-opus-4-7", "opus-4-7"),
-    "sonnet46": ("anthropic_claude-sonnet-4-6", "sonnet-4-6"),
-    "gpt54":    ("openai_gpt-5.4", "gpt-5.4"),
-    "gemini":   ("google_gemini-3.1-pro-preview", "gemini-3.1-pro"),
-    "glm":      ("openrouter_z-ai_glm-5.1", "glm-5.1"),
-    "minimax":  ("openrouter_minimax_minimax-m2.7", "minimax-m2.7"),
-    "kimi":     ("openrouter_moonshotai_kimi-k2.5", "kimi-k2.5"),
-    "qwen":     ("openrouter_qwen_qwen3.6-plus", "qwen-3.6-plus"),
-}
-
-LOG_LINE = re.compile(
-    r"^\[(\d+)/120\]\s+(\S+)\s+\([^)]+\)\s+run\s+(\d+):\s+([+\-~])\s+([\d.]+)"
-)
-HARNESS_ERR = re.compile(r"ERROR clawbench\.harness: Run (\S+)/(\d+) failed")
-JUDGE_INFRA_PHRASES = [
-    "gateway is restarting", "judge execution failed", "judge failed to run",
-    "judge call failed", "judge timed out",
-]
-
-
-def parse_log(log_path: Path):
-    runs = {}
-    errors = {}
-    if not log_path.exists():
-        return runs, errors
-    src = log_path.read_text(errors="ignore")
-    for line in src.splitlines():
-        m = LOG_LINE.match(line.strip())
-        if m:
-            seq, task, run_idx, outcome, score = m.groups()
-            runs[(task, int(run_idx) - 1)] = {"score": float(score), "outcome": outcome}
-        h = HARNESS_ERR.search(line)
-        if h:
-            errors[(h.group(1), int(h.group(2)))] = "harness_error"
-    return runs, errors
-
-
-def scan_archive(cache_dir: Path):
-    out = {}
-    if not cache_dir.exists():
-        return out
-    for tdir in cache_dir.iterdir():
-        if not tdir.is_dir():
-            continue
-        for rf in tdir.glob("run*.json"):
-            m = re.match(r"run(\d+)\.json", rf.name)
-            if not m:
-                continue
-            try:
-                d = json.load(open(rf))
-            except Exception:
-                continue
-            jr = d.get("judge_result", {}) or {}
-            reason = (jr.get("reason") or "").lower()
-            # Don't flag rejudged runs as infra-failed even if reason is empty —
-            # a rejudged run has a real judge call behind it (rejudged_at field).
-            judge_infra = (
-                jr.get("enabled")
-                and "rejudged_at" not in jr
-                and (
-                    any(p in reason for p in JUDGE_INFRA_PHRASES)
-                    or jr.get("error")
-                    or (not reason.strip() and jr.get("score", 0) == 0)
-                )
-            )
-            out[(tdir.name, int(m.group(1)))] = {
-                "run_score": d.get("run_score", 0),
-                "c": d.get("completion_result", {}).get("score", 0),
-                "t": d.get("trajectory_result", {}).get("score", 0),
-                "b": d.get("behavior_result", {}).get("score", 0),
-                "j": jr.get("score", 0) if jr.get("enabled") else None,
-                "judge_infra_failed": bool(judge_infra),
-                "rejudged": "rejudged_at" in jr,
-                "delivery": d.get("delivery_outcome"),
-                "failure_mode": d.get("failure_mode"),
-                "error": d.get("error"),
-                "n_messages": len(d.get("transcript", {}).get("messages", [])),
-                "has_assistant_text": any(
-                    m.get("role") == "assistant" and m.get("text")
-                    for m in d.get("transcript", {}).get("messages", [])
-                ),
-            }
-    return out
-
-
-def main():
-    # Gather everything
-    per_model = {}
-    for label, (sub, pretty) in MODEL_MAP.items():
-        log_p = DRIFT / f"docker_{label}_v2026-4-19-full.log"
-        arch_d = ARCH / sub
-        logged, errors = parse_log(log_p)
-        archived = scan_archive(arch_d)
-        per_model[pretty] = {
-            "logged": logged, "errors": errors, "archived": archived,
-        }
-
-    # Build per-task cross-model view
-    all_tasks = set()
-    for m in per_model.values():
-        for key in m["archived"]:
-            all_tasks.add(key[0])
-        for key in m["logged"]:
-            all_tasks.add(key[0])
-
-    # Issue classification
-    issues = defaultdict(list)
-
-    for task in sorted(all_tasks):
-        # Collect all runs for this task across models
-        task_runs_by_model = {}
-        for pretty, data in per_model.items():
-            task_runs = []
-            for run_idx in range(3):
-                key = (task, run_idx)
-                a = data["archived"].get(key)
-                logged = data["logged"].get(key)
-                err = (key in data["errors"])
-                task_runs.append({"archived": a, "logged": logged, "harness_err": err})
-            task_runs_by_model[pretty] = task_runs
-
-        # Compute cross-model stats
-        all_scores = []
-        all_cs = []
-        all_outputs = []  # model produced assistant text?
-        all_judge_infra = 0
-        all_harness_err = 0
-        for pretty, runs in task_runs_by_model.items():
-            for r in runs:
-                a = r["archived"]
-                if a:
-                    all_scores.append(a["run_score"])
-                    all_cs.append(a["c"])
-                    all_outputs.append(a["has_assistant_text"])
-                    if a["judge_infra_failed"]:
-                        all_judge_infra += 1
-                elif r["logged"]:
-                    all_scores.append(r["logged"]["score"])
-                if r["harness_err"]:
-                    all_harness_err += 1
-
-        if not all_scores:
-            continue
-        mean_score = sum(all_scores) / len(all_scores)
-        mean_c = sum(all_cs) / len(all_cs) if all_cs else 0
-        output_rate = sum(all_outputs) / len(all_outputs) if all_outputs else 0
-
-        # Flag issues
-        if mean_score < 0.1:
-            issues["task_fails_all_models"].append((task, mean_score, output_rate))
-        if mean_c < 0.05 and output_rate > 0.5:
-            issues["verifier_rejects_valid_outputs"].append((task, mean_c, output_rate))
-        if all_harness_err >= 5:
-            issues["harness_errors_cluster"].append((task, all_harness_err))
-        if all_judge_infra >= 5:
-            issues["judge_infra_cluster"].append((task, all_judge_infra))
-
-    # Print issues
-    print("=" * 70)
-    print("ISSUE: Tasks where ALL models score near-zero (broken verifier or task)")
-    print("=" * 70)
-    for task, mean, out_rate in sorted(issues["task_fails_all_models"]):
-        print(f"  {task:<40}  mean_score={mean:.3f}  assistant_output_rate={out_rate:.1%}")
-
-    print()
-    print("=" * 70)
-    print("ISSUE: Verifier rejects valid outputs (model produced text but C=0)")
-    print("=" * 70)
-    for task, mean_c, out_rate in sorted(issues["verifier_rejects_valid_outputs"]):
-        print(f"  {task:<40}  mean_completion={mean_c:.3f}  assistant_output_rate={out_rate:.1%}")
-
-    print()
-    print("=" * 70)
-    print("ISSUE: Harness-error clusters (gateway failures per task)")
-    print("=" * 70)
-    for task, n in sorted(issues["harness_errors_cluster"], key=lambda x: -x[1]):
-        print(f"  {task:<40}  harness_error_count={n}")
-
-    print()
-    print("=" * 70)
-    print("ISSUE: Judge-infra clusters (judge failing per task)")
-    print("=" * 70)
-    for task, n in sorted(issues["judge_infra_cluster"], key=lambda x: -x[1]):
-        print(f"  {task:<40}  judge_infra_failures={n}  (should be rejudged)")
-
-    # Per-model per-task pathologies
-    print()
-    print("=" * 70)
-    print("ISSUE: Model-specific task pathologies (all 3 runs of a task scored 0 on one model)")
-    print("=" * 70)
-    for pretty, data in per_model.items():
-        zero_tasks = []
-        for task in sorted(all_tasks):
-            all_three_zero = True
-            any_attempted = False
-            for run_idx in range(3):
-                key = (task, run_idx)
-                a = data["archived"].get(key)
-                logged = data["logged"].get(key)
-                if a:
-                    any_attempted = True
-                    if a["run_score"] > 0.01:
-                        all_three_zero = False
-                elif logged:
-                    any_attempted = True
-                    if logged["score"] > 0.01:
-                        all_three_zero = False
-                else:
-                    all_three_zero = False  # can't confirm
-                    any_attempted = False
-            if any_attempted and all_three_zero:
-                zero_tasks.append(task)
-        if zero_tasks:
-            print(f"  {pretty:<18}: all-zero on {len(zero_tasks)} tasks")
-            for t in zero_tasks[:6]:
-                print(f"    - {t}")
-
-    # Task coverage mismatches
-    print()
-    print("=" * 70)
-    print("COVERAGE: Models with non-complete coverage (logged != 120 or archived != 120)")
-    print("=" * 70)
-    for pretty, data in per_model.items():
-        n_log = len(data["logged"])
-        n_arch = len(data["archived"])
-        if n_log < 120 or n_arch < 120:
-            print(f"  {pretty:<18}  logged={n_log:<4}  archived={n_arch:<4}  missing={120 - max(n_log, n_arch)}")
-
-
-if __name__ == "__main__":
-    main()

scripts/audit_runs.py

@@ -1,204 +0,0 @@
-"""Comprehensive per-run audit across all models in drift_2026-04-19-full.
-
-For each model, cross-references:
-  1. Log file (docker_<label>_<tag>.log) — all [N/120] run attempts + their scores
-  2. Archived per-run JSONs (run_cache_archive/<tag>/<cache_sub>/<task>/runN.json)
-  3. Judge status per cached run (rejudged via direct API or not)
-
-Outputs a fair-comparison table: coverage %, infra-failure %, clean mean,
-coverage-normalized score, judge coverage.
-
-Usage:
-  python3 scripts/audit_runs.py
-"""
-
-from __future__ import annotations
-
-import json
-import re
-from pathlib import Path
-
-ROOT = Path(__file__).resolve().parent.parent
-DRIFT = ROOT / "data" / "drift_2026-04-19-full"
-ARCH = ROOT / "data" / "run_cache_archive" / "v2026-4-19-full"
-
-# Model label (in log filenames) → (cache_sub, pretty name)
-MODEL_MAP = {
-    "opus46":   ("anthropic_claude-opus-4-6", "opus-4-6"),
-    "opus47":   ("anthropic_claude-opus-4-7", "opus-4-7"),
-    "sonnet46": ("anthropic_claude-sonnet-4-6", "sonnet-4-6"),
-    "gpt54":    ("openai_gpt-5.4", "gpt-5.4"),
-    "gemini":   ("google_gemini-3.1-pro-preview", "gemini-3.1-pro"),
-    "glm":      ("openrouter_z-ai_glm-5.1", "glm-5.1"),
-    "minimax":  ("openrouter_minimax_minimax-m2.7", "minimax-m2.7"),
-    "kimi":     ("openrouter_moonshotai_kimi-k2.5", "kimi-k2.5"),
-    "qwen":     ("openrouter_qwen_qwen3.6-plus", "qwen-3.6-plus"),
-}
-
-# Regex to parse "[N/120] task (tier/family) run R: + 0.93 C=1.00 T=0.90 ..."
-LOG_LINE = re.compile(
-    r"^\[(\d+)/120\]\s+(\S+)\s+\([^)]+\)\s+run\s+(\d+):\s+([+\-~])\s+([\d.]+)"
-)
-JUDGE_INFRA_PHRASES = [
-    "gateway is restarting",
-    "judge execution failed",
-    "judge failed to run",
-    "judge call failed",
-    "judge timed out",
-]
-
-
-def parse_log(path: Path) -> dict:
-    """Return: {(task_id, run_idx): {"score": float, "outcome": "+/-/~"}} from log file."""
-    runs = {}
-    if not path.exists():
-        return runs
-    for line in path.read_text(errors="ignore").splitlines():
-        m = LOG_LINE.match(line.strip())
-        if not m:
-            continue
-        seq, task, run_idx, outcome, score = m.groups()
-        # Log uses 1-indexed run numbers; archive uses 0-indexed runN.json.
-        # Normalize to 0-indexed so keys cross-reference correctly.
-        key = (task, int(run_idx) - 1)
-        # Later entries overwrite earlier (retry semantics)
-        runs[key] = {"score": float(score), "outcome": outcome, "seq": int(seq)}
-    return runs
-
-
-def scan_archive(cache_dir: Path) -> dict:
-    """Return: {(task_id, run_idx): {"run_score": float, "c": float, "judge_err": bool, "rejudged": bool}}"""
-    out = {}
-    if not cache_dir.exists():
-        return out
-    for tdir in cache_dir.iterdir():
-        if not tdir.is_dir():
-            continue
-        for rf in tdir.glob("run*.json"):
-            try:
-                d = json.load(open(rf))
-            except Exception:
-                continue
-            m_run = re.match(r"run(\d+)\.json", rf.name)
-            if not m_run:
-                continue
-            run_idx = int(m_run.group(1))
-            jr = d.get("judge_result", {}) or {}
-            reason = (jr.get("reason") or "").lower()
-            judge_infra = (
-                any(p in reason for p in JUDGE_INFRA_PHRASES)
-                or jr.get("error")
-                or (not reason.strip() and jr.get("score", 0) == 0)
-            )
-            out[(tdir.name, run_idx)] = {
-                "run_score": d.get("run_score", 0),
-                "completion": d.get("completion_result", {}).get("score", 0),
-                "judge_score": jr.get("score", 0) if jr.get("enabled") else None,
-                "judge_infra_failed": bool(judge_infra and jr.get("enabled")),
-                "rejudged": "rejudged_at" in jr,
-                "delivery": d.get("delivery_outcome"),
-                "failure_mode": d.get("failure_mode"),
-            }
-    return out
-
-
-def audit_model(label: str, cache_sub: str, pretty: str) -> dict:
-    log_path = DRIFT / f"docker_{label}_v2026-4-19-full.log"
-    cache_dir = ARCH / cache_sub
-    logged = parse_log(log_path)
-    archived = scan_archive(cache_dir)
-
-    n_log = len(logged)
-    n_arch = len(archived)
-    not_archived = [k for k in logged.keys() if k not in archived]
-    # Classify runs
-    clean_runs = []                 # logged + archived + not-infra-zero + judge-OK
-    infra_zero_runs = []            # logged 0.00 (infra) — never landed in archive
-    archived_zero = []              # archived but run_score = 0 (infra/capability)
-    judge_infra = []                # archived with judge_infra_failed
-    rejudged = []                   # archived with rejudged_at
-
-    for k, a in archived.items():
-        if a["judge_infra_failed"] and not a["rejudged"]:
-            judge_infra.append(k)
-        if a["rejudged"]:
-            rejudged.append(k)
-        if a["run_score"] < 0.01:
-            archived_zero.append(k)
-        else:
-            clean_runs.append((k, a["run_score"]))
-
-    # Runs that got logged at 0.00 but weren't archived are pure infra-failures
-    for k in not_archived:
-        if logged[k]["score"] < 0.01:
-            infra_zero_runs.append(k)
-        else:
-            clean_runs.append((k, logged[k]["score"]))
-
-    # Score computations
-    all_scores = []
-    for k, a in archived.items():
-        all_scores.append(a["run_score"])
-    for k in not_archived:
-        all_scores.append(logged[k]["score"])
-
-    expected = 120
-
-    clean_scores = [s for _, s in clean_runs]
-    clean_mean = sum(clean_scores) / len(clean_scores) if clean_scores else 0
-
-    all_mean = sum(all_scores) / len(all_scores) if all_scores else 0
-    # Coverage-normalized: clean_mean with gap-penalty (missing runs count as 0)
-    coverage_normalized = (sum(clean_scores) + 0 * max(0, expected - len(clean_scores))) / expected
-
-    return {
-        "label": label,
-        "pretty": pretty,
-        "n_log_entries": n_log,
-        "n_archived": n_arch,
-        "n_missing_from_archive": len(not_archived),
-        "n_clean_runs": len(clean_runs),
-        "n_archived_zero": len(archived_zero),
-        "n_logged_infra_zero": len(infra_zero_runs),
-        "n_judge_infra_failed": len(judge_infra),
-        "n_rejudged": len(rejudged),
-        "coverage_pct": 100.0 * len(clean_runs) / expected,
-        "clean_mean": clean_mean,
-        "all_mean": all_mean,
-        "coverage_normalized": coverage_normalized,
-    }
-
-
-def main():
-    print(f"{'Model':<16} {'Logged':>7} {'Archv':>6} {'Clean':>6} {'Cov%':>5}  {'all_mean':>8} {'clean':>7} {'cov_norm':>8} {'infra_0':>8} {'j_rejdg':>8} {'j_failed':>8}")
-    print(f"{'-'*16} {'-'*7} {'-'*6} {'-'*6} {'-'*5}  {'-'*8} {'-'*7} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")
-    rows = []
-    for label, (cache_sub, pretty) in MODEL_MAP.items():
-        r = audit_model(label, cache_sub, pretty)
-        rows.append(r)
-
-    # Sort by coverage-normalized score
-    rows.sort(key=lambda r: -r["coverage_normalized"])
-    for r in rows:
-        print(
-            f"  {r['pretty']:<14} {r['n_log_entries']:>7} {r['n_archived']:>6} "
-            f"{r['n_clean_runs']:>6} {r['coverage_pct']:>4.0f}%  "
-            f"{r['all_mean']:>8.4f} {r['clean_mean']:>7.4f} "
-            f"{r['coverage_normalized']:>8.4f} "
-            f"{r['n_logged_infra_zero']+r['n_archived_zero']:>8} "
-            f"{r['n_rejudged']:>8} {r['n_judge_infra_failed']:>8}"
-        )
-
-    # Show gaps explicitly
-    print()
-    print("Legend:")
-    print("  all_mean      = mean of ALL attempts (log+archive merged; infra-zeros pull this DOWN)")
-    print("  clean         = mean excluding infra-failed runs (shows capability ceiling)")
-    print("  cov_norm      = clean*coverage + 0*missing; all models scored against 120-run denominator")
-    print("  infra_0       = runs that scored 0 due to infrastructure (gateway/state/handshake failures)")
-    print("  j_rejdg       = judge scores that have been rejudged via direct Anthropic API")
-    print("  j_failed      = judge infra-failures that have NOT been rejudged")
-
-
-if __name__ == "__main__":
-    main()

scripts/classify_regimes.py

@@ -19,7 +19,7 @@ import argparse
 import json
 import re
 import sys
-from collections import Counter
+from collections import Counter, defaultdict
 from pathlib import Path
 
 import numpy as np

scripts/container_adapter_eval.sh

@@ -0,0 +1,460 @@
+#!/bin/bash
+# Fair adapter lane runner.
+#
+# Runs one adapter/model pair inside a container-owned workspace/state dir.
+# Use docker run with full container privileges when measuring harnesses:
+#   docker run --rm --privileged --cap-add=ALL \
+#     --security-opt seccomp=unconfined --security-opt apparmor=unconfined \
+#     --user root --env-file .tmp/docker_eval.env \
+#     -e SWEEP_ADAPTER=hermes -e SWEEP_MODEL=openai/gpt-5.4 \
+#     -e SWEEP_LABEL=hermes-gpt54 -e SWEEP_OUT_TAG=fair-20260425 \
+#     -v "$PWD/data/fair-container:/data" \
+#     -v "$PWD/data/container-home-openclaw:/config/openclaw:ro" \
+#     clawbench-fair:latest
+
+set -u
+
+: "${SWEEP_ADAPTER:?SWEEP_ADAPTER required (openclaw|hermes)}"
+: "${SWEEP_MODEL:?SWEEP_MODEL required (e.g. openai/gpt-5.4)}"
+: "${SWEEP_LABEL:?SWEEP_LABEL required}"
+: "${SWEEP_OUT_TAG:=fair-container}"
+: "${SWEEP_LOGDIR:=/data/fair_results}"
+: "${SWEEP_RUNS:=1}"
+: "${SWEEP_CONCURRENCY:=1}"
+: "${SWEEP_BROWSER_CONCURRENCY:=1}"
+: "${CLAWBENCH_PER_RUN_BUDGET_SECONDS:=300}"
+: "${CLAWBENCH_PER_TURN_TIMEOUT_SECONDS:=180}"
+: "${HERMES_MAX_ITERATIONS:=90}"
+: "${HERMES_STEP_TIMEOUT_SECONDS:=60}"
+: "${OPENCLAW_EXEC_HOST:=gateway}"
+
+cd /home/node/app
+mkdir -p "$SWEEP_LOGDIR" /data/run_cache
+
+export OPENCLAW_GATEWAY_TOKEN="${OPENCLAW_GATEWAY_TOKEN:-local-dev-token-for-testing}"
+export OPENCLAW_GATEWAY_URL="${OPENCLAW_GATEWAY_URL:-ws://127.0.0.1:18789}"
+export OPENCLAW_SKIP_GMAIL_WATCHER=1
+export OPENCLAW_SKIP_CANVAS_HOST=1
+export OPENCLAW_NO_RESPAWN=1
+export CLAWBENCH_DISABLE_GATEWAY_DEVICE_IDENTITY=1
+export NODE_OPTIONS="${NODE_OPTIONS:-"--max-old-space-size=4096"}"
+if command -v npm >/dev/null 2>&1; then
+  export NODE_PATH="${NODE_PATH:-$(npm root -g 2>/dev/null || true)}"
+fi
+export CLAWBENCH_PER_RUN_BUDGET_SECONDS
+export CLAWBENCH_PER_TURN_TIMEOUT_SECONDS
+export HERMES_AGENT_REPO="${HERMES_AGENT_REPO:-/opt/hermes-agent}"
+export HERMES_DRIVER="${HERMES_DRIVER:-ai_agent}"
+export HERMES_TOOLSETS="${HERMES_TOOLSETS:-hermes-api-server}"
+export HERMES_MAX_ITERATIONS
+export HERMES_STEP_TIMEOUT_SECONDS
+export TERMINAL_ENV="${TERMINAL_ENV:-local}"
+
+safe_model="${SWEEP_MODEL//\//_}"
+safe_model="${safe_model//:/_}"
+safe_label="${SWEEP_LABEL//\//_}"
+safe_label="${safe_label//:/_}"
+export CLAWBENCH_RUN_CACHE_DIR="/data/run_cache/$safe_label"
+mkdir -p "$CLAWBENCH_RUN_CACHE_DIR"
+cache_sub="${SWEEP_ADAPTER}-${safe_model}"
+cache_paths=("$CLAWBENCH_RUN_CACHE_DIR/$cache_sub")
+if [ "$SWEEP_ADAPTER" = "openclaw" ]; then
+  cache_paths+=("$CLAWBENCH_RUN_CACHE_DIR/$safe_model")
+fi
+
+SRC_STATE="${OPENCLAW_CONFIG_SOURCE:-/config/openclaw}"
+if [ ! -d "$SRC_STATE" ]; then
+  SRC_STATE="/home/node/.openclaw"
+fi
+FRESH_HOME="/tmp/openclaw-home-${SWEEP_LABEL}-$$"
+FRESH_STATE="$FRESH_HOME/.openclaw"
+rm -rf "$FRESH_HOME"
+mkdir -p "$FRESH_STATE" "$FRESH_HOME/.config"
+if [ -f "$SRC_STATE/openclaw.json" ]; then
+  cp "$SRC_STATE/openclaw.json" "$FRESH_STATE/openclaw.json"
+fi
+mkdir -p \
+  "$FRESH_STATE/agents" \
+  "$FRESH_STATE/workspace" \
+  "$FRESH_STATE/logs" \
+  "$FRESH_STATE/memory" \
+  "$FRESH_STATE/cache" \
+  "$FRESH_STATE/identity" \
+  "$FRESH_STATE/devices" \
+  "$FRESH_STATE/tasks" \
+  "$FRESH_STATE/subagents" \
+  "$FRESH_STATE/flows" \
+  "$FRESH_STATE/cron"
+chmod -R 777 "$FRESH_STATE" 2>/dev/null || true
+export HOME="$FRESH_HOME"
+export OPENCLAW_HOME="$FRESH_HOME"
+export OPENCLAW_STATE_DIR="$FRESH_STATE"
+export OPENCLAW_CONFIG_PATH="$FRESH_STATE/openclaw.json"
+export OPENCLAW_REPO="${OPENCLAW_REPO:-/app}"
+export XDG_CONFIG_HOME="$FRESH_HOME/.config"
+export HERMES_HOME_BASE="${HERMES_HOME_BASE:-$FRESH_HOME/.hermes}"
+export HERMES_HOME="$HERMES_HOME_BASE"
+mkdir -p "$HERMES_HOME"
+
+if [ "$SWEEP_ADAPTER" = "hermes" ]; then
+  unset HERMES_PROVIDER
+  case "$SWEEP_MODEL" in
+    openai/*)
+      if [ -z "${OPENAI_API_KEY:-}" ] && [ -n "${HERMES_API_KEY:-}" ]; then
+        export OPENAI_API_KEY="$HERMES_API_KEY"
+      fi
+      export HERMES_BASE_URL="${HERMES_BASE_URL:-${OPENAI_BASE_URL:-https://api.openai.com/v1}}"
+      export OPENAI_BASE_URL="$HERMES_BASE_URL"
+      if [ -n "${OPENAI_API_KEY:-}" ]; then
+        export HERMES_API_KEY="$OPENAI_API_KEY"
+      fi
+      unset ANTHROPIC_API_KEY ANTHROPIC_TOKEN CLAUDE_CODE_OAUTH_TOKEN OPENROUTER_API_KEY
+      ;;
+    anthropic/*)
+      unset OPENAI_API_KEY OPENAI_BASE_URL HERMES_API_KEY HERMES_BASE_URL OPENROUTER_API_KEY
+      ;;
+    *)
+      if [ -n "${HERMES_BASE_URL:-}" ]; then
+        export OPENAI_BASE_URL="$HERMES_BASE_URL"
+      elif [ -z "${OPENAI_BASE_URL:-}" ] && [ -n "${OPENAI_API_KEY:-}" ]; then
+        export OPENAI_BASE_URL="https://api.openai.com/v1"
+      fi
+      if [ -n "${HERMES_API_KEY:-}" ] && [ -z "${OPENAI_API_KEY:-}" ]; then
+        export OPENAI_API_KEY="$HERMES_API_KEY"
+      fi
+      ;;
+  esac
+fi
+
+python - <<'PY'
+import json
+import os
+from pathlib import Path
+
+cfg_path = Path(os.environ["OPENCLAW_CONFIG_PATH"])
+if not cfg_path.exists():
+    raise SystemExit(0)
+
+data = json.loads(cfg_path.read_text(encoding="utf-8"))
+
+agents = data.get("agents")
+if isinstance(agents, dict):
+    # Keep static defaults, but never seed eval containers with old session-specific
+    # agent records from the developer machine.
+    agents["list"] = []
+
+channels = data.get("channels")
+if isinstance(channels, dict):
+    for channel in channels.values():
+        if isinstance(channel, dict):
+            channel["enabled"] = False
+            exec_approvals = channel.get("execApprovals")
+            if not isinstance(exec_approvals, dict):
+                exec_approvals = {}
+                channel["execApprovals"] = exec_approvals
+            exec_approvals["enabled"] = False
+
+plugins = data.get("plugins")
+if isinstance(plugins, dict):
+    stale = {"marxbiotech-git-tools", "lab"}
+    allow = plugins.get("allow")
+    if isinstance(allow, list):
+        plugins["allow"] = [item for item in allow if item not in stale]
+    entries = plugins.get("entries")
+    if isinstance(entries, dict):
+        for item in stale:
+            entries.pop(item, None)
+
+
+def set_nested(root, dotted, value):
+    cursor = root
+    parts = dotted.split(".")
+    for part in parts[:-1]:
+        child = cursor.get(part)
+        if not isinstance(child, dict):
+            child = {}
+            cursor[part] = child
+        cursor = child
+    cursor[parts[-1]] = value
+
+
+set_nested(data, "browser.headless", True)
+set_nested(data, "browser.noSandbox", True)
+set_nested(data, "gateway.reload.mode", "off")
+set_nested(data, "agents.defaults.skipBootstrap", True)
+set_nested(data, "agents.defaults.sandbox.mode", "off")
+exec_host = os.environ.get("OPENCLAW_EXEC_HOST", "gateway").strip().lower()
+if exec_host not in {"auto", "gateway", "sandbox", "node"}:
+    raise SystemExit(f"invalid OPENCLAW_EXEC_HOST={exec_host!r}")
+set_nested(data, "tools.exec.host", exec_host)
+set_nested(data, "tools.exec.security", "full")
+set_nested(data, "tools.exec.ask", "off")
+set_nested(data, "approvals.exec.enabled", False)
+model = os.environ.get("SWEEP_MODEL", "").strip()
+if model:
+    set_nested(data, "agents.defaults.model.primary", model)
+    set_nested(data, "agents.defaults.subagents.model.primary", model)
+
+tmp_path = cfg_path.with_suffix(".json.tmp")
+tmp_path.write_text(json.dumps(data, indent=2), encoding="utf-8")
+tmp_path.replace(cfg_path)
+
+approvals_path = cfg_path.with_name("exec-approvals.json")
+approvals = {
+    "version": 1,
+    "socket": {
+        "path": str(approvals_path.with_suffix(".sock")),
+        "token": "container-eval-token",
+    },
+    "defaults": {
+        "security": "full",
+        "ask": "off",
+        "askFallback": "full",
+    },
+    "agents": {
+        "*": {
+            "security": "full",
+            "ask": "off",
+            "askFallback": "full",
+        }
+    },
+}
+approvals_path.write_text(json.dumps(approvals, indent=2), encoding="utf-8")
+PY
+
+if [ "$SWEEP_ADAPTER" = "hermes" ]; then
+python - <<'PY'
+import os
+from pathlib import Path
+from urllib.parse import urlparse
+
+model = os.environ["SWEEP_MODEL"].strip()
+base_url = (os.environ.get("HERMES_BASE_URL") or os.environ.get("OPENAI_BASE_URL") or "").strip()
+
+provider = "custom"
+effective_model = model
+aux_base_url = ""
+aux_api_mode = ""
+if model.startswith("anthropic/"):
+    provider = "anthropic"
+elif urlparse(base_url).hostname == "api.openai.com" and model.startswith("openai/"):
+    effective_model = model.split("/", 1)[1]
+    aux_base_url = base_url
+    if effective_model.lower().startswith("gpt-5"):
+        aux_api_mode = "codex_responses"
+elif base_url:
+    aux_base_url = base_url
+
+tasks = [
+    "vision",
+    "web_extract",
+    "compression",
+    "session_search",
+    "skills_hub",
+    "approval",
+    "mcp",
+    "title_generation",
+]
+
+lines = [
+    "model:",
+    f"  provider: {provider}",
+    f"  default: {effective_model}",
+]
+if aux_base_url:
+    lines.append(f"  base_url: {aux_base_url}")
+if aux_api_mode:
+    lines.append(f"  api_mode: {aux_api_mode}")
+lines.append("auxiliary:")
+for task in tasks:
+    timeout = 360 if task == "web_extract" else 120 if task in {"vision", "compression"} else 30
+    lines.extend([
+        f"  {task}:",
+        "    provider: main",
+        f"    model: {effective_model}",
+        f"    timeout: {timeout}",
+    ])
+    if aux_base_url:
+        lines.append(f"    base_url: {aux_base_url}")
+    if aux_api_mode:
+        lines.append(f"    api_mode: {aux_api_mode}")
+    if task == "session_search":
+        lines.append("    max_concurrency: 1")
+
+path = Path(os.environ["HERMES_HOME"]) / "config.yaml"
+path.parent.mkdir(parents=True, exist_ok=True)
+path.write_text("\n".join(lines) + "\n", encoding="utf-8")
+PY
+fi
+
+OUT="$SWEEP_LOGDIR/${SWEEP_LABEL}_${SWEEP_ADAPTER}_${safe_model}_${SWEEP_OUT_TAG}.json"
+LOG="$SWEEP_LOGDIR/${SWEEP_LABEL}_${SWEEP_ADAPTER}_${safe_model}_${SWEEP_OUT_TAG}.log"
+GWLOG="$SWEEP_LOGDIR/gateway_${SWEEP_LABEL}_${SWEEP_OUT_TAG}.log"
+HERMES_AGENT_LOG="$SWEEP_LOGDIR/hermes_agent_${SWEEP_LABEL}_${SWEEP_OUT_TAG}.log"
+HERMES_ERROR_LOG="$SWEEP_LOGDIR/hermes_errors_${SWEEP_LABEL}_${SWEEP_OUT_TAG}.log"
+
+echo "===== CONTAINER ADAPTER EVAL START $(date '+%Y-%m-%d %H:%M:%S') ====="
+echo "uid:      $(id -u) ($(id -un 2>/dev/null || true))"
+echo "adapter:  $SWEEP_ADAPTER"
+echo "model:    $SWEEP_MODEL"
+echo "runs:     $SWEEP_RUNS"
+echo "execHost: $OPENCLAW_EXEC_HOST"
+echo "out:      $OUT"
+echo "cache:    ${cache_paths[*]}"
+echo "home:     $HOME"
+echo "state:    $OPENCLAW_STATE_DIR"
+echo "hermes:   ${HERMES_HOME:-}"
+openclaw --version 2>/dev/null || true
+python - <<'PY' 2>/dev/null || true
+import os, subprocess
+repo = os.environ.get("HERMES_AGENT_REPO", "")
+if repo:
+    try:
+        sha = subprocess.check_output(["git", "-C", repo, "rev-parse", "HEAD"], text=True).strip()
+        print(f"Hermes git: {sha}")
+    except Exception:
+        print(f"Hermes repo: {repo}")
+PY
+
+rm -rf "${cache_paths[@]}"
+rm -f "$OUT" "$LOG"
+
+GATEWAY_PID=""
+preserve_hermes_logs() {
+  if [ -f "${HERMES_HOME:-}/logs/agent.log" ]; then
+    cp "${HERMES_HOME:-}/logs/agent.log" "$HERMES_AGENT_LOG" 2>/dev/null || true
+  fi
+  if [ -f "${HERMES_HOME:-}/logs/errors.log" ]; then
+    cp "${HERMES_HOME:-}/logs/errors.log" "$HERMES_ERROR_LOG" 2>/dev/null || true
+  fi
+}
+
+cleanup() {
+  preserve_hermes_logs
+  if [ -n "${GATEWAY_PID:-}" ]; then
+    kill "$GATEWAY_PID" 2>/dev/null || true
+    wait "$GATEWAY_PID" 2>/dev/null || true
+  fi
+  rm -rf "${FRESH_HOME:-}" 2>/dev/null || true
+}
+trap cleanup EXIT
+
+if [ "$SWEEP_ADAPTER" = "openclaw" ]; then
+  echo "Starting OpenClaw gateway on :18789 ..."
+  HOME="$FRESH_HOME" \
+  OPENCLAW_HOME="$FRESH_HOME" \
+  OPENCLAW_STATE_DIR="$FRESH_STATE" \
+  OPENCLAW_CONFIG_PATH="$FRESH_STATE/openclaw.json" \
+  XDG_CONFIG_HOME="$FRESH_HOME/.config" \
+    openclaw gateway run \
+    --allow-unconfigured \
+    --dev \
+    --bind loopback \
+    --port 18789 \
+    --auth token \
+    --token "$OPENCLAW_GATEWAY_TOKEN" \
+    --compact \
+    > "$GWLOG" 2>&1 &
+  GATEWAY_PID=$!
+  ready=0
+  for i in $(seq 1 180); do
+    if curl -sf -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" http://127.0.0.1:18789/healthz > /dev/null 2>&1; then
+      echo "Gateway healthy after ${i}s"
+      ready=1
+      break
+    fi
+    sleep 1
+  done
+  if [ "$ready" -ne 1 ]; then
+    echo "ERROR: gateway failed to become healthy"
+    tail -80 "$GWLOG" 2>/dev/null || true
+    exit 1
+  fi
+  if [ -r "/proc/$GATEWAY_PID/environ" ]; then
+    actual_home="$(tr '\0' '\n' < "/proc/$GATEWAY_PID/environ" | awk -F= '$1 == "HOME" { print $2; exit }')"
+    if [ "$actual_home" != "$FRESH_HOME" ]; then
+      echo "ERROR: gateway HOME escaped container eval home: ${actual_home:-<unset>} != $FRESH_HOME"
+      tail -120 "$GWLOG" 2>/dev/null || true
+      exit 1
+    fi
+  fi
+  if [ ! -f "$FRESH_STATE/exec-approvals.json" ] || grep -q '/home/node/.openclaw' "$FRESH_STATE/exec-approvals.json"; then
+    echo "ERROR: exec approvals are not isolated in $FRESH_STATE"
+    exit 1
+  fi
+  echo "Waiting for OpenClaw session control plane ..."
+  python - <<'PY'
+import asyncio
+import os
+import sys
+import time
+
+from clawbench.client import GatewayClient, GatewayConfig
+
+
+async def probe_once(attempt: int) -> None:
+    config = GatewayConfig(
+        url=os.environ["OPENCLAW_GATEWAY_URL"],
+        token=os.environ["OPENCLAW_GATEWAY_TOKEN"],
+        connect_timeout=30.0,
+        request_timeout=30.0,
+    )
+    async with GatewayClient(config) as client:
+        key = await client.create_session(
+            model=os.environ["SWEEP_MODEL"],
+            label=f"clawbench-readiness-probe-{os.getpid()}-{attempt}",
+        )
+        await client.delete_session(key)
+
+
+async def main() -> int:
+    deadline = time.monotonic() + 240
+    attempt = 0
+    last_error = ""
+    while time.monotonic() < deadline:
+        attempt += 1
+        try:
+            await probe_once(attempt)
+            print(f"Gateway session control plane ready after {attempt} attempt(s)")
+            return 0
+        except Exception as exc:
+            last_error = f"{type(exc).__name__}: {exc}"
+            print(f"Gateway control probe {attempt} not ready: {last_error}")
+            await asyncio.sleep(5)
+    print(f"ERROR: gateway session control plane did not become ready: {last_error}", file=sys.stderr)
+    return 1
+
+
+raise SystemExit(asyncio.run(main()))
+PY
+  if [ "$?" -ne 0 ]; then
+    tail -120 "$GWLOG" 2>/dev/null || true
+    exit 1
+  fi
+fi
+
+TASK_ARGS=()
+if [ -n "${CHERRY_TASKS:-}" ]; then
+  IFS=',' read -ra TASK_ARR <<< "$CHERRY_TASKS"
+  for task_id in "${TASK_ARR[@]}"; do
+    TASK_ARGS+=("--task" "$task_id")
+  done
+fi
+
+clawbench run \
+  --adapter "$SWEEP_ADAPTER" \
+  --model "$SWEEP_MODEL" \
+  --runs "$SWEEP_RUNS" \
+  --concurrency "$SWEEP_CONCURRENCY" \
+  --browser-concurrency "$SWEEP_BROWSER_CONCURRENCY" \
+  --no-randomize \
+  "${TASK_ARGS[@]}" \
+  --output "$OUT" \
+  > "$LOG" 2>&1
+status=$?
+preserve_hermes_logs
+
+echo "===== clawbench exit=$status $(date '+%Y-%m-%d %H:%M:%S') ====="
+tail -80 "$LOG" 2>/dev/null || true
+
+exit "$status"

scripts/container_cherry_single.sh

@@ -1,198 +0,0 @@
-#!/bin/bash
-# Cherry-pick variant of container_sweep_single.sh: runs ONLY the tasks listed
-# in $CHERRY_TASKS (comma-separated task IDs), with state-dir isolation.
-#
-# Required env vars:
-#   SWEEP_LABEL   (e.g. opus47)
-#   SWEEP_MODEL   (e.g. anthropic/claude-opus-4-7)
-#   SWEEP_PROFILE (absolute path in container)
-#   SWEEP_LOGDIR  (default /data/drift_2026-04-20-cherry)
-#   SWEEP_OUT_TAG (default v2026-4-20-cherry)
-#   CHERRY_TASKS  (comma-separated task IDs, e.g. "t2-ctx-pronoun-resolve,t3-fin-budget-monthly")
-
-set -u
-
-: "${SWEEP_LABEL:?SWEEP_LABEL required}"
-: "${SWEEP_MODEL:?SWEEP_MODEL required}"
-: "${SWEEP_PROFILE:?SWEEP_PROFILE required}"
-: "${CHERRY_TASKS:?CHERRY_TASKS required (comma-separated task IDs)}"
-
-: "${SWEEP_LOGDIR:=/data/drift_2026-04-20-cherry}"
-: "${SWEEP_OUT_TAG:=v2026-4-20-cherry}"
-
-cd /data
-
-LOGDIR="$SWEEP_LOGDIR"
-mkdir -p "$LOGDIR"
-
-export OPENCLAW_GATEWAY_TOKEN="local-dev-token-for-testing"
-export CLAWBENCH_RUN_CACHE_DIR="/data/run_cache"
-mkdir -p "$CLAWBENCH_RUN_CACHE_DIR"
-export NODE_OPTIONS="--max-old-space-size=4096"
-# OpenClaw 4.22+ has slower agents.create / sessions.create on cold start
-# (we observed 72s for opus-4-7). Bump RPC timeouts so the harness doesn't
-# cancel mid-flight. Override defaults of 30s / 60s respectively.
-export CLAWBENCH_CONNECT_TIMEOUT="${CLAWBENCH_CONNECT_TIMEOUT:-120}"
-export CLAWBENCH_REQUEST_TIMEOUT="${CLAWBENCH_REQUEST_TIMEOUT:-300}"
-export CLAWBENCH_PER_RUN_BUDGET_SECONDS="${CLAWBENCH_PER_RUN_BUDGET_SECONDS:-900}"
-export HERMES_STEP_TIMEOUT_SECONDS="${HERMES_STEP_TIMEOUT_SECONDS:-180}"
-
-# State-dir isolation (same as container_sweep_single.sh)
-SRC_STATE="/home/node/.openclaw"
-FRESH_STATE="/tmp/openclaw-state-${SWEEP_LABEL}-$$"
-echo "[state-isolate] cloning config from $SRC_STATE to $FRESH_STATE"
-mkdir -p "$FRESH_STATE"
-[ -f "$SRC_STATE/openclaw.json" ] && cp "$SRC_STATE/openclaw.json" "$FRESH_STATE/openclaw.json"
-[ -f "$SRC_STATE/exec-approvals.json" ] && cp "$SRC_STATE/exec-approvals.json" "$FRESH_STATE/exec-approvals.json"
-for d in identity devices tasks subagents flows cron; do
-  [ -d "$SRC_STATE/$d" ] && cp -r "$SRC_STATE/$d" "$FRESH_STATE/$d"
-done
-mkdir -p "$FRESH_STATE/agents" "$FRESH_STATE/workspace" "$FRESH_STATE/logs" "$FRESH_STATE/memory" "$FRESH_STATE/cache"
-export OPENCLAW_STATE_DIR="$FRESH_STATE"
-export OPENCLAW_CONFIG_PATH="$FRESH_STATE/openclaw.json"
-echo "[state-isolate] OPENCLAW_STATE_DIR=$OPENCLAW_STATE_DIR"
-
-python - <<'PY'
-import json
-import os
-from pathlib import Path
-
-cfg_path = Path(os.environ["OPENCLAW_CONFIG_PATH"])
-data = json.loads(cfg_path.read_text(encoding="utf-8")) if cfg_path.exists() else {}
-
-def set_nested(root, dotted, value):
-    cursor = root
-    parts = dotted.split(".")
-    for part in parts[:-1]:
-        child = cursor.get(part)
-        if not isinstance(child, dict):
-            child = {}
-            cursor[part] = child
-        cursor = child
-    cursor[parts[-1]] = value
-
-exec_host = os.environ.get("OPENCLAW_EXEC_HOST", "gateway").strip().lower()
-if exec_host not in {"auto", "gateway", "sandbox", "node"}:
-    raise SystemExit(f"invalid OPENCLAW_EXEC_HOST={exec_host!r}")
-
-set_nested(data, "tools.exec.host", exec_host)
-set_nested(data, "tools.exec.security", "full")
-set_nested(data, "tools.exec.ask", "off")
-set_nested(data, "approvals.exec.enabled", False)
-cfg_path.write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8")
-
-approvals_path = cfg_path.with_name("exec-approvals.json")
-approvals = {
-    "version": 1,
-    "socket": {
-        "path": str(approvals_path.with_suffix(".sock")),
-        "token": "container-cherry-eval-token",
-    },
-    "defaults": {"security": "full", "ask": "off", "askFallback": "full"},
-    "agents": {"*": {"security": "full", "ask": "off", "askFallback": "full"}},
-}
-approvals_path.write_text(json.dumps(approvals, indent=2) + "\n", encoding="utf-8")
-PY
-
-# Map model to cache subdir (for archiving)
-case "$SWEEP_MODEL" in
-  anthropic/claude-opus-4-7)        CACHE_SUB="anthropic_claude-opus-4-7" ;;
-  anthropic/claude-opus-4-6)        CACHE_SUB="anthropic_claude-opus-4-6" ;;
-  anthropic/claude-sonnet-4-6)      CACHE_SUB="anthropic_claude-sonnet-4-6" ;;
-  openai/gpt-5.5)                   CACHE_SUB="openai_gpt-5.5" ;;
-  openai/gpt-5.4)                   CACHE_SUB="openai_gpt-5.4" ;;
-  google/gemini-3.1-pro-preview)    CACHE_SUB="google_gemini-3.1-pro-preview" ;;
-  openrouter/z-ai/glm-5.1)          CACHE_SUB="openrouter_z-ai_glm-5.1" ;;
-  openrouter/qwen/qwen3.6-plus)     CACHE_SUB="openrouter_qwen_qwen3.6-plus" ;;
-  openrouter/minimax/minimax-m2.7)  CACHE_SUB="openrouter_minimax_minimax-m2.7" ;;
-  openrouter/moonshotai/kimi-k2.6)  CACHE_SUB="openrouter_moonshotai_kimi-k2.6" ;;
-  openrouter/moonshotai/kimi-k2.5)  CACHE_SUB="openrouter_moonshotai_kimi-k2.5" ;;
-  openrouter/deepseek/deepseek-v4-pro) CACHE_SUB="openrouter_deepseek_deepseek-v4-pro" ;;
-  deepseek/deepseek-v4-pro)         CACHE_SUB="deepseek_deepseek-v4-pro" ;;
-  deepseek/v4-pro)                  CACHE_SUB="deepseek_v4-pro" ;;
-  *) CACHE_SUB="" ;;
-esac
-
-OUT="$LOGDIR/docker_${SWEEP_LABEL}_${SWEEP_OUT_TAG}.json"
-LOG="$LOGDIR/docker_${SWEEP_LABEL}_${SWEEP_OUT_TAG}.log"
-GWLOG="$LOGDIR/gateway_${SWEEP_LABEL}.log"
-
-echo "===== CHERRY-PICK SWEEP $(date '+%Y-%m-%d %H:%M:%S') ====="
-echo "label:   $SWEEP_LABEL"
-echo "model:   $SWEEP_MODEL"
-echo "tasks:   $CHERRY_TASKS"
-echo "out:     $OUT"
-
-# Force-clear this model's run_cache (including fixed-task slots — so they
-# actually re-run against the new image instead of hitting old cache).
-if [ -n "$CACHE_SUB" ] && [ -d "$CLAWBENCH_RUN_CACHE_DIR/$CACHE_SUB" ]; then
-  echo "clearing cache: $CLAWBENCH_RUN_CACHE_DIR/$CACHE_SUB"
-  rm -rf "$CLAWBENCH_RUN_CACHE_DIR/$CACHE_SUB"
-fi
-[ -f "$OUT" ] && rm -f "$OUT"
-
-# Start gateway with bumped heap
-echo "Starting gateway on :18789 (heap=4GB) ..."
-openclaw gateway --port 18789 > "$GWLOG" 2>&1 &
-GATEWAY_PID=$!
-ready=0
-for i in $(seq 1 120); do
-  if curl -sf -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" http://127.0.0.1:18789/ready > /dev/null 2>&1; then
-    echo "Gateway ready after ${i}s"
-    ready=1
-    break
-  fi
-  sleep 1
-done
-if [ $ready -ne 1 ]; then
-  echo "ERROR: gateway failed to become ready within 120s"
-  tail -30 "$GWLOG"
-  exit 1
-fi
-
-# Build -t args from comma-separated list
-TASK_ARGS=()
-IFS=',' read -ra TASK_ARR <<< "$CHERRY_TASKS"
-for t in "${TASK_ARR[@]}"; do
-  TASK_ARGS+=("-t" "$t")
-done
-
-echo "===== $(date '+%H:%M:%S') running clawbench with tasks: ${TASK_ARR[*]} ====="
-# NOTE: --profile intentionally OMITTED. The legacy frontier_*.yaml profile
-# format is incompatible with OpenClaw 4.22+ (loads n_tools_total=0,
-# starves the agent of tools, all runs fail with environment_unavailable
-# or timeout). Running with the default openclaw tool stack — same for
-# all models, so the comparison stays apples-to-apples.
-PROFILE_ARG=""
-if [ -n "${USE_PROFILE:-}" ] && [ -f "$SWEEP_PROFILE" ]; then
-  PROFILE_ARG="--profile $SWEEP_PROFILE"
-fi
-clawbench run \
-  --model "$SWEEP_MODEL" \
-  --runs 3 \
-  --concurrency "${CLAWBENCH_CONCURRENCY:-1}" \
-  $PROFILE_ARG \
-  --judge-model "anthropic/claude-sonnet-4-6" \
-  "${TASK_ARGS[@]}" \
-  -o "$OUT" \
-  > "$LOG" 2>&1
-status=$?
-
-if [ $status -eq 0 ]; then
-  echo "===== $(date '+%H:%M:%S') done $SWEEP_LABEL (exit 0) ====="
-else
-  echo "===== $(date '+%H:%M:%S') FAILED $SWEEP_LABEL (exit $status) ====="
-  tail -20 "$LOG"
-fi
-
-# Archive cache to v2026-4-20-cherry tag
-# shellcheck disable=SC1091
-source "$(dirname "$0")/_archive_cache.sh" 2>/dev/null && archive_run_cache || echo "[archive] helper missing"
-
-kill $GATEWAY_PID 2>/dev/null
-wait $GATEWAY_PID 2>/dev/null
-
-# Clean up isolated state dir
-[ -n "${FRESH_STATE:-}" ] && [ -d "$FRESH_STATE" ] && rm -rf "$FRESH_STATE"
-
-exit $status

scripts/container_lane_eval.sh

@@ -131,27 +131,11 @@ set_nested(data, "agents.defaults.skipBootstrap", True)
 set_nested(data, "agents.defaults.sandbox.mode", "off")
 set_nested(data, "agents.defaults.model.primary", os.environ["SWEEP_MODEL"])
 set_nested(data, "agents.defaults.subagents.model.primary", os.environ["SWEEP_MODEL"])
-set_nested(
-    data,
-    "agents.defaults.systemPromptOverride",
-    "You are running an OpenClaw benchmark task. Complete the user's request in the current "
-    "workspace using the available tools when needed. For file, code, browser, shell, or memory "
-    "tasks, make the requested changes directly and verify them when practical. Do not ask "
-    "follow-up questions during the benchmark. Keep any final reply brief.",
-)
 set_nested(data, "tools.exec.host", os.environ.get("OPENCLAW_EXEC_HOST", "gateway"))
 set_nested(data, "tools.exec.security", "full")
 set_nested(data, "tools.exec.ask", "off")
 set_nested(data, "approvals.exec.enabled", False)
 
-models = data.setdefault("agents", {}).setdefault("defaults", {}).setdefault("models", {})
-model_entry = models.setdefault(os.environ["SWEEP_MODEL"], {})
-params = model_entry.setdefault("params", {})
-params["fastMode"] = True
-if os.environ["SWEEP_MODEL"].startswith("openai/"):
-    params["transport"] = "sse"
-    params["openaiWsWarmup"] = False
-
 cfg_path.write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8")
 
 approvals_path = cfg_path.with_name("exec-approvals.json")
@@ -167,6 +151,11 @@ approvals = {
 approvals_path.write_text(json.dumps(approvals, indent=2) + "\n", encoding="utf-8")
 PY
 
+if [ "${CLAWBENCH_ENABLE_GBRAIN:-0}" = "1" ]; then
+  export CLAWBENCH_LANE_PREPARE_CMD="${CLAWBENCH_LANE_PREPARE_CMD:-/home/node/app/scripts/setup_gbrain_runtime.sh}"
+  "$CLAWBENCH_LANE_PREPARE_CMD"
+fi
+
 echo "===== CONTAINER LANE EVAL START $(date '+%Y-%m-%d %H:%M:%S') ====="
 echo "label:    $SWEEP_LABEL"
 echo "model:    $SWEEP_MODEL"

scripts/container_sweep_minimal.sh

@@ -1,98 +0,0 @@
-#!/bin/bash
-# Minimal single-model sweep — 1 run per task (not 3) for fast validation.
-# Used to quickly test if an openrouter-stream fix actually works without
-# committing to a full 60-minute 3-run sweep.
-#
-# Invocation (from host):
-#   docker run -d --name clawbench-<LABEL> \
-#     -e SWEEP_LABEL=<label> -e SWEEP_MODEL=<routed-model> \
-#     -e SWEEP_PROFILE=<abs-profile-path> \
-#     -e SWEEP_LOGDIR=<output-dir-in-container> \
-#     -e SWEEP_OUT_TAG=<tag> \
-#     -v .../scripts:/home/node/app/scripts:ro \
-#     -v .../data:/data \
-#     -v .../data/container-home-openclaw:/home/node/.openclaw \
-#     -v .../profiles:/home/node/app/profiles:ro \
-#     --memory 8g \
-#     <image> \
-#     bash /home/node/app/scripts/container_sweep_minimal.sh
-
-set -u
-
-: "${SWEEP_LABEL:?SWEEP_LABEL required}"
-: "${SWEEP_MODEL:?SWEEP_MODEL required}"
-: "${SWEEP_PROFILE:?SWEEP_PROFILE required}"
-: "${SWEEP_LOGDIR:?SWEEP_LOGDIR required}"
-: "${SWEEP_OUT_TAG:?SWEEP_OUT_TAG required}"
-
-cd /data
-mkdir -p "$SWEEP_LOGDIR"
-
-export OPENCLAW_GATEWAY_TOKEN="local-dev-token-for-testing"
-export CLAWBENCH_RUN_CACHE_DIR="/data/run_cache"
-mkdir -p "$CLAWBENCH_RUN_CACHE_DIR"
-export NODE_OPTIONS="--max-old-space-size=4096"
-
-# Clear cache for target model
-case "$SWEEP_MODEL" in
-  openrouter/z-ai/glm-5.1)          CACHE_SUB="openrouter_z-ai_glm-5.1" ;;
-  openrouter/minimax/minimax-m2.7)  CACHE_SUB="openrouter_minimax_minimax-m2.7" ;;
-  openrouter/moonshotai/kimi-k2.5)  CACHE_SUB="openrouter_moonshotai_kimi-k2.5" ;;
-  *) CACHE_SUB="" ;;
-esac
-if [ -n "$CACHE_SUB" ] && [ -d "$CLAWBENCH_RUN_CACHE_DIR/$CACHE_SUB" ]; then
-  echo "clearing cache: $CLAWBENCH_RUN_CACHE_DIR/$CACHE_SUB"
-  rm -rf "$CLAWBENCH_RUN_CACHE_DIR/$CACHE_SUB"
-fi
-
-OUT="$SWEEP_LOGDIR/docker_${SWEEP_LABEL}_${SWEEP_OUT_TAG}.json"
-LOG="$SWEEP_LOGDIR/docker_${SWEEP_LABEL}_${SWEEP_OUT_TAG}.log"
-GWLOG="$SWEEP_LOGDIR/gateway_${SWEEP_LABEL}.log"
-
-rm -f "$OUT"
-
-echo "===== MINIMAL SWEEP START $(date '+%Y-%m-%d %H:%M:%S') ====="
-echo "label:   $SWEEP_LABEL"
-echo "model:   $SWEEP_MODEL"
-echo "profile: $SWEEP_PROFILE"
-echo "out:     $OUT"
-echo "runs:    1 per task (MINIMAL)"
-
-echo "Starting gateway on :18789 (heap=4GB) ..."
-openclaw gateway --port 18789 > "$GWLOG" 2>&1 &
-GATEWAY_PID=$!
-
-ready=0
-for i in $(seq 1 120); do
-  if curl -sf -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" http://127.0.0.1:18789/health > /dev/null 2>&1; then
-    echo "Gateway healthy after ${i}s"
-    ready=1
-    break
-  fi
-  sleep 1
-done
-if [ $ready -ne 1 ]; then
-  echo "ERROR: gateway failed to come up"
-  exit 1
-fi
-
-echo "===== $(date '+%H:%M:%S') starting $SWEEP_LABEL ($SWEEP_MODEL) ====="
-clawbench run \
-  --model "$SWEEP_MODEL" \
-  --runs 1 \
-  --concurrency 4 \
-  --profile "$SWEEP_PROFILE" \
-  --judge-model "anthropic/claude-sonnet-4-6" \
-  -o "$OUT" \
-  > "$LOG" 2>&1
-status=$?
-
-echo "===== $(date '+%H:%M:%S') done $SWEEP_LABEL (exit $status) ====="
-
-# Archive the cache for future audits
-# shellcheck disable=SC1091
-source "$(dirname "$0")/_archive_cache.sh" 2>/dev/null && archive_run_cache || echo "[archive] helper missing, skipping"
-
-kill $GATEWAY_PID 2>/dev/null
-wait $GATEWAY_PID 2>/dev/null
-exit $status

scripts/container_sweep_single.sh

@@ -1,235 +0,0 @@
-#!/bin/bash
-# Single-model sweep with fresh gateway + bumped Node heap to prevent OOM.
-#
-# Invocation (from host):
-#   docker run -d --name clawbench-sweep-<LABEL> \
-#     -e SWEEP_LABEL=<label> -e SWEEP_MODEL=<routed-model> -e SWEEP_PROFILE=<abs-profile-path> \
-#     -v .../scripts:/home/node/app/scripts:ro \
-#     -v .../data:/data \
-#     -v .../data/container-home-openclaw:/home/node/.openclaw \
-#     -v .../profiles:/home/node/app/profiles:ro \
-#     --memory 8g \
-#     clawbench-clawbench:latest \
-#     bash /home/node/app/scripts/container_sweep_single.sh
-#
-# Differences vs container_sweep.sh:
-# - Bumps gateway Node.js heap via NODE_OPTIONS=--max-old-space-size=4096 (prevents 2GB OOM we saw at ~4h)
-# - One model per container (no shared-gateway drift between models)
-# - Force-clears run_cache for THIS model before running (prevents cache-replay masking)
-# - Writes to the same $LOGDIR/docker_${label}_${SWEEP_OUT_TAG}.json as the original sweep
-#   so generate_drift_report.py picks it up without changes
-
-set -u
-
-: "${SWEEP_LABEL:?SWEEP_LABEL required (e.g. glm, minimax, kimi)}"
-: "${SWEEP_MODEL:?SWEEP_MODEL required (e.g. openrouter/z-ai/glm-5.1)}"
-: "${SWEEP_PROFILE:?SWEEP_PROFILE required (absolute path in container)}"
-
-# Optional overrides (defaults target the v4.14 drift sweep):
-#   SWEEP_LOGDIR — where JSONs and logs go (default /data/drift_2026-04-14)
-#   SWEEP_OUT_TAG — tag embedded in output filename (default v2026-4-14)
-: "${SWEEP_LOGDIR:=/data/drift_2026-04-14}"
-: "${SWEEP_OUT_TAG:=v2026-4-14}"
-
-cd /data
-
-LOGDIR="$SWEEP_LOGDIR"
-mkdir -p "$LOGDIR"
-
-export OPENCLAW_GATEWAY_TOKEN="local-dev-token-for-testing"
-export CLAWBENCH_RUN_CACHE_DIR="/data/run_cache"
-mkdir -p "$CLAWBENCH_RUN_CACHE_DIR"
-
-# OOM fix: give the gateway Node process a 4GB old-space ceiling instead of the default ~2GB.
-# Scoped via env so we don't stomp on other Node processes (clawbench itself is python).
-export NODE_OPTIONS="--max-old-space-size=4096"
-# OpenClaw 4.22+ has slower agents.create / sessions.create on cold start
-# (we observed 72s for opus-4-7). Bump RPC timeouts so the harness doesn't
-# cancel mid-flight. Override defaults of 30s / 60s respectively.
-export CLAWBENCH_CONNECT_TIMEOUT="${CLAWBENCH_CONNECT_TIMEOUT:-120}"
-export CLAWBENCH_REQUEST_TIMEOUT="${CLAWBENCH_REQUEST_TIMEOUT:-300}"
-export CLAWBENCH_PER_RUN_BUDGET_SECONDS="${CLAWBENCH_PER_RUN_BUDGET_SECONDS:-900}"
-export HERMES_STEP_TIMEOUT_SECONDS="${HERMES_STEP_TIMEOUT_SECONDS:-180}"
-
-# State-dir isolation: the shared /home/node/.openclaw mount accumulates cruft
-# across sweeps (agents/, workspace/, logs/, memory/, stale openclaw.json.*.tmp)
-# which triggers gateway hot-reload churn and cascading `RPC agents.create timed
-# out after 60s` failures. Give each sweep a pristine state dir that carries
-# over only the config (openclaw.json, identity/, devices/, exec-approvals.json,
-# tasks/, subagents/, flows/, cron/) and leaves runtime state empty.
-SRC_STATE="/home/node/.openclaw"
-FRESH_STATE="/tmp/openclaw-state-${SWEEP_LABEL}-$$"
-echo "[state-isolate] cloning config from $SRC_STATE to $FRESH_STATE"
-mkdir -p "$FRESH_STATE"
-# Copy the main config (skip the .tmp/.bak/.clobbered/.pre-* cruft that can
-# confuse the loader — only the canonical openclaw.json is needed).
-if [ -f "$SRC_STATE/openclaw.json" ]; then
-  cp "$SRC_STATE/openclaw.json" "$FRESH_STATE/openclaw.json"
-fi
-if [ -f "$SRC_STATE/exec-approvals.json" ]; then
-  cp "$SRC_STATE/exec-approvals.json" "$FRESH_STATE/exec-approvals.json"
-fi
-# Carry over static config dirs — these are read-mostly and don't accumulate
-# per-run cruft. SKIP: agents/ workspace*/ logs/ memory/ cache/ browser/ canvas/
-# which all grow unboundedly across sweeps.
-for d in identity devices tasks subagents flows cron; do
-  if [ -d "$SRC_STATE/$d" ]; then
-    cp -r "$SRC_STATE/$d" "$FRESH_STATE/$d"
-  fi
-done
-# Ensure runtime dirs exist but are empty
-mkdir -p "$FRESH_STATE/agents" "$FRESH_STATE/workspace" "$FRESH_STATE/logs" "$FRESH_STATE/memory" "$FRESH_STATE/cache"
-export OPENCLAW_STATE_DIR="$FRESH_STATE"
-export OPENCLAW_CONFIG_PATH="$FRESH_STATE/openclaw.json"
-echo "[state-isolate] OPENCLAW_STATE_DIR=$OPENCLAW_STATE_DIR"
-du -sh "$FRESH_STATE" 2>/dev/null | sed 's/^/[state-isolate] size: /'
-
-python - <<'PY'
-import json
-import os
-from pathlib import Path
-
-cfg_path = Path(os.environ["OPENCLAW_CONFIG_PATH"])
-data = json.loads(cfg_path.read_text(encoding="utf-8")) if cfg_path.exists() else {}
-
-def set_nested(root, dotted, value):
-    cursor = root
-    parts = dotted.split(".")
-    for part in parts[:-1]:
-        child = cursor.get(part)
-        if not isinstance(child, dict):
-            child = {}
-            cursor[part] = child
-        cursor = child
-    cursor[parts[-1]] = value
-
-exec_host = os.environ.get("OPENCLAW_EXEC_HOST", "gateway").strip().lower()
-if exec_host not in {"auto", "gateway", "sandbox", "node"}:
-    raise SystemExit(f"invalid OPENCLAW_EXEC_HOST={exec_host!r}")
-
-set_nested(data, "tools.exec.host", exec_host)
-set_nested(data, "tools.exec.security", "full")
-set_nested(data, "tools.exec.ask", "off")
-set_nested(data, "approvals.exec.enabled", False)
-cfg_path.write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8")
-
-approvals_path = cfg_path.with_name("exec-approvals.json")
-approvals = {
-    "version": 1,
-    "socket": {
-        "path": str(approvals_path.with_suffix(".sock")),
-        "token": "container-single-eval-token",
-    },
-    "defaults": {"security": "full", "ask": "off", "askFallback": "full"},
-    "agents": {"*": {"security": "full", "ask": "off", "askFallback": "full"}},
-}
-approvals_path.write_text(json.dumps(approvals, indent=2) + "\n", encoding="utf-8")
-PY
-
-# Map label -> cache subdir (matches what clawbench writes)
-case "$SWEEP_MODEL" in
-  anthropic/claude-opus-4-7)        CACHE_SUB="anthropic_claude-opus-4-7" ;;
-  anthropic/claude-sonnet-4-7)      CACHE_SUB="anthropic_claude-sonnet-4-7" ;;
-  anthropic/claude-opus-4-6)        CACHE_SUB="anthropic_claude-opus-4-6" ;;
-  anthropic/claude-sonnet-4-6)      CACHE_SUB="anthropic_claude-sonnet-4-6" ;;
-  openai/gpt-5.5)                   CACHE_SUB="openai_gpt-5.5" ;;
-  openai/gpt-5.4)                   CACHE_SUB="openai_gpt-5.4" ;;
-  openai/gpt-5.2)                   CACHE_SUB="openai_gpt-5.2" ;;
-  google/gemini-3.1-pro-preview)    CACHE_SUB="google_gemini-3.1-pro-preview" ;;
-  openrouter/z-ai/glm-5.1)          CACHE_SUB="openrouter_z-ai_glm-5.1" ;;
-  openrouter/qwen/qwen3.6-plus)     CACHE_SUB="openrouter_qwen_qwen3.6-plus" ;;
-  openrouter/minimax/minimax-m2.7)  CACHE_SUB="openrouter_minimax_minimax-m2.7" ;;
-  openrouter/moonshotai/kimi-k2.6)  CACHE_SUB="openrouter_moonshotai_kimi-k2.6" ;;
-  openrouter/moonshotai/kimi-k2.5)  CACHE_SUB="openrouter_moonshotai_kimi-k2.5" ;;
-  deepseek/v4-pro)                  CACHE_SUB="deepseek_v4-pro" ;;
-  *) CACHE_SUB="" ;;
-esac
-
-OUT="$LOGDIR/docker_${SWEEP_LABEL}_${SWEEP_OUT_TAG}.json"
-LOG="$LOGDIR/docker_${SWEEP_LABEL}_${SWEEP_OUT_TAG}.log"
-GWLOG="$LOGDIR/gateway_${SWEEP_LABEL}.log"
-
-echo "===== SINGLE-MODEL SWEEP START $(date '+%Y-%m-%d %H:%M:%S') ====="
-echo "label:   $SWEEP_LABEL"
-echo "model:   $SWEEP_MODEL"
-echo "profile: $SWEEP_PROFILE"
-echo "out:     $OUT"
-echo "gwlog:   $GWLOG"
-echo "NODE_OPTIONS: $NODE_OPTIONS"
-
-# Force-clear this model's run_cache so we actually re-run (no replays)
-if [ -n "$CACHE_SUB" ] && [ -d "$CLAWBENCH_RUN_CACHE_DIR/$CACHE_SUB" ]; then
-  echo "clearing cache: $CLAWBENCH_RUN_CACHE_DIR/$CACHE_SUB"
-  rm -rf "$CLAWBENCH_RUN_CACHE_DIR/$CACHE_SUB"
-fi
-
-# Also remove any stale result JSON so we don't skip-on-idempotence
-if [ -f "$OUT" ]; then
-  echo "removing stale result: $OUT"
-  rm -f "$OUT"
-fi
-
-# Start gateway with bumped heap
-echo "Starting gateway on :18789 (heap=4GB) ..."
-openclaw gateway --port 18789 > "$GWLOG" 2>&1 &
-GATEWAY_PID=$!
-echo "gateway pid=$GATEWAY_PID"
-
-ready=0
-for i in $(seq 1 120); do
-  if curl -sf -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" http://127.0.0.1:18789/health > /dev/null 2>&1; then
-    echo "Gateway healthy after ${i}s"
-    ready=1
-    break
-  fi
-  sleep 1
-done
-if [ $ready -ne 1 ]; then
-  echo "ERROR: gateway failed to come up within 120s"
-  tail -30 "$GWLOG"
-  exit 1
-fi
-
-echo "===== $(date '+%H:%M:%S') starting $SWEEP_LABEL ($SWEEP_MODEL) ====="
-# NOTE: --profile intentionally OMITTED unless USE_PROFILE=1 is set. The
-# legacy frontier_*.yaml profile format is incompatible with OpenClaw
-# 4.22+ (loads n_tools_total=0). Running with the default openclaw tool
-# stack — identical across all models, so comparisons stay valid.
-PROFILE_ARG=""
-if [ -n "${USE_PROFILE:-}" ] && [ -f "$SWEEP_PROFILE" ]; then
-  PROFILE_ARG="--profile $SWEEP_PROFILE"
-fi
-clawbench run \
-  --model "$SWEEP_MODEL" \
-  --runs 3 \
-  --concurrency "${CLAWBENCH_CONCURRENCY:-1}" \
-  $PROFILE_ARG \
-  --judge-model "anthropic/claude-sonnet-4-6" \
-  -o "$OUT" \
-  > "$LOG" 2>&1
-status=$?
-
-if [ $status -eq 0 ]; then
-  echo "===== $(date '+%H:%M:%S') done $SWEEP_LABEL (exit 0) ====="
-else
-  echo "===== $(date '+%H:%M:%S') FAILED $SWEEP_LABEL (exit $status) ====="
-  tail -20 "$LOG"
-fi
-
-# Archive the cache for future audits (preserves transcripts per sweep tag)
-# shellcheck disable=SC1091
-source "$(dirname "$0")/_archive_cache.sh" 2>/dev/null && archive_run_cache || echo "[archive] helper missing, skipping"
-
-echo ""
-echo "===== SINGLE-MODEL SWEEP END $(date '+%Y-%m-%d %H:%M:%S') ====="
-kill $GATEWAY_PID 2>/dev/null
-wait $GATEWAY_PID 2>/dev/null
-echo "gateway stopped"
-
-# Clean up the isolated state dir (don't accumulate /tmp cruft across sweeps).
-if [ -n "${FRESH_STATE:-}" ] && [ -d "$FRESH_STATE" ]; then
-  echo "[state-isolate] removing $FRESH_STATE"
-  rm -rf "$FRESH_STATE"
-fi
-
-exit $status

scripts/generate_fair_report.py

@@ -1,254 +0,0 @@
-"""Fair 9-model comparison report generator for the v2026-4-19 full sweep.
-
-Reads the per-run archive at data/run_cache_archive/<tag>/<cache_sub>/<task>/runN.json
-and computes, per model:
-  - Coverage % (archived runs / 120)
-  - Overall mean, clean mean (excl. infra-zeros), coverage-normalized score
-  - Per-tier mean (tier1-5)
-  - Judge-infra failures remaining (should be 0 after rejudge pass)
-
-Writes markdown to reports/EVAL_REPORT_9MODEL_FAIR_<tag>.md.
-
-Usage:
-    python3 scripts/generate_fair_report.py \\
-        --tag v2026-4-19-full \\
-        [--out reports/EVAL_REPORT_9MODEL_FAIR_v2026-4-19-full.md]
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import re
-from collections import defaultdict
-from pathlib import Path
-from statistics import mean
-
-ROOT = Path(__file__).resolve().parent.parent
-
-MODEL_MAP = {
-    "opus47":   ("anthropic_claude-opus-4-7", "Claude Opus 4.7"),
-    "opus46":   ("anthropic_claude-opus-4-6", "Claude Opus 4.6"),
-    "sonnet46": ("anthropic_claude-sonnet-4-6", "Claude Sonnet 4.6"),
-    "gpt54":    ("openai_gpt-5.4", "GPT 5.4"),
-    "gemini":   ("google_gemini-3.1-pro-preview", "Gemini 3.1 Pro"),
-    "glm":      ("openrouter_z-ai_glm-5.1", "GLM 5.1"),
-    "minimax":  ("openrouter_minimax_minimax-m2.7", "MiniMax M2.7"),
-    "kimi25":   ("openrouter_moonshotai_kimi-k2.5", "Kimi K2.5"),
-    "qwen":     ("openrouter_qwen_qwen3.6-plus", "Qwen 3.6 Plus"),
-}
-
-JUDGE_INFRA_PHRASES = [
-    "gateway is restarting", "judge execution failed", "judge failed to run",
-    "judge call failed", "judge timed out",
-]
-
-
-def tier_of(task_id: str) -> str:
-    m = re.match(r"t(\d)-", task_id)
-    return f"tier{m.group(1)}" if m else "other"
-
-
-def scan_archive(cache_dir: Path) -> list[dict]:
-    rows = []
-    if not cache_dir.exists():
-        return rows
-    for tdir in sorted(cache_dir.iterdir()):
-        if not tdir.is_dir():
-            continue
-        for rf in sorted(tdir.glob("run*.json")):
-            try:
-                d = json.loads(rf.read_text())
-            except Exception:
-                continue
-            jr = d.get("judge_result", {}) or {}
-            reason = (jr.get("reason") or "").lower()
-            judge_infra = (
-                jr.get("enabled")
-                and "rejudged_at" not in jr
-                and (
-                    any(p in reason for p in JUDGE_INFRA_PHRASES)
-                    or jr.get("error")
-                    or (not reason.strip() and jr.get("score", 0) == 0)
-                )
-            )
-            rows.append({
-                "task": tdir.name,
-                "tier": tier_of(tdir.name),
-                "run_score": d.get("run_score", 0),
-                "c": d.get("completion_result", {}).get("score", 0),
-                "t": d.get("trajectory_result", {}).get("score", 0),
-                "b": d.get("behavior_result", {}).get("score", 0),
-                "j": jr.get("score", 0) if jr.get("enabled") else None,
-                "judge_infra": bool(judge_infra),
-                "rejudged": "rejudged_at" in jr,
-                "is_infra_zero": d.get("run_score", 0) < 0.01,
-            })
-    return rows
-
-
-def summarize(label: str, cache_sub: str, pretty: str, tag: str) -> dict:
-    cache_dir = ROOT / "data" / "run_cache_archive" / tag / cache_sub
-    rows = scan_archive(cache_dir)
-    n = len(rows)
-    if n == 0:
-        return {"label": label, "pretty": pretty, "n": 0, "missing": 120}
-
-    all_scores = [r["run_score"] for r in rows]
-    clean_rows = [r for r in rows if not r["is_infra_zero"]]
-    clean_scores = [r["run_score"] for r in clean_rows]
-    overall = mean(all_scores) if all_scores else 0
-    clean = mean(clean_scores) if clean_scores else 0
-    cov_norm = sum(clean_scores) / 120
-    coverage_pct = 100.0 * len(clean_rows) / 120
-
-    per_tier = defaultdict(list)
-    for r in rows:
-        per_tier[r["tier"]].append(r["run_score"])
-    tier_means = {t: mean(v) for t, v in per_tier.items() if v}
-
-    # Judge-only score (how well model does purely on LLM judgment)
-    judge_scores = [r["j"] for r in rows if r["j"] is not None]
-    judge_mean = mean(judge_scores) if judge_scores else None
-
-    # C=1.0 pass count
-    c_pass_count = sum(1 for r in rows if r["c"] >= 0.9999)
-
-    return {
-        "label": label,
-        "pretty": pretty,
-        "n": n,
-        "missing": max(0, 120 - n),
-        "n_clean": len(clean_rows),
-        "coverage_pct": coverage_pct,
-        "overall": overall,
-        "clean": clean,
-        "cov_norm": cov_norm,
-        "tier_means": tier_means,
-        "judge_mean": judge_mean,
-        "c_pass_count": c_pass_count,
-        "judge_infra_remaining": sum(1 for r in rows if r["judge_infra"]),
-        "rejudged": sum(1 for r in rows if r["rejudged"]),
-    }
-
-
-def build_markdown(summaries: list[dict], tag: str) -> str:
-    summaries = [s for s in summaries if s["n"] > 0]
-    summaries.sort(key=lambda s: -s.get("clean", 0))
-
-    L = []
-    L.append(f"# ClawBench Fair 9-Model Comparison — {tag}")
-    L.append("")
-    L.append("All 9 models at 120/120 coverage after gap-fill. Rankings use")
-    L.append("**clean mean run_score** — mean across all 120 archived runs per model.")
-    L.append("")
-    L.append("## Ranking (clean mean run_score, 0–1 scale)")
-    L.append("")
-    L.append("| Rank | Model | Clean | Judge-only | C=1.0 tasks | Coverage |")
-    L.append("|---:|---|---:|---:|---:|---:|")
-    for rank, s in enumerate(summaries, 1):
-        jm = f"{s['judge_mean']:.3f}" if s.get("judge_mean") is not None else "—"
-        cpct = s.get("c_pass_count", 0)
-        L.append(f"| {rank} | **{s['pretty']}** | **{s['clean']:.4f}** | "
-                 f"{jm} | {cpct}/{s['n']} | {s['n']}/120 |")
-    L.append("")
-
-    L.append("## Fairness audit — passed")
-    L.append("")
-    L.append("All 9 models subjected to **identical** evaluation conditions:")
-    L.append("")
-    L.append("- **Same 40 tasks × 3 runs = 120 expected runs per model** (all from v4-19-full sweep)")
-    L.append("- **Same completion/trajectory/behavior verifiers** for every model")
-    L.append("- **Same Docker image** (openclaw 2026-04-16 baseline)")
-    L.append("- **Same judge model** (Claude Sonnet 4.6)")
-    L.append("- **Judge infra failures all rejudged** via direct Anthropic API (0 left)")
-    L.append("- **Coverage parity**: 97-99% across all models (within ~3 runs)")
-    L.append("")
-    # Coverage table
-    L.append("### Coverage detail")
-    L.append("")
-    L.append("| Model | Archived | Missing | Rejudged via API |")
-    L.append("|---|---:|---:|---:|")
-    for s in summaries:
-        L.append(f"| {s['pretty']} | {s['n']}/120 | {s['missing']} | {s['rejudged']} |")
-    L.append("")
-
-    # Per-tier
-    L.append("## Per-tier mean run_score")
-    L.append("")
-    L.append("| Model | Tier 1 | Tier 2 | Tier 3 | Tier 4 | Tier 5 |")
-    L.append("|---|---:|---:|---:|---:|---:|")
-    for s in summaries:
-        tm = s.get("tier_means", {})
-        row = [s["pretty"]]
-        for t in ("tier1", "tier2", "tier3", "tier4", "tier5"):
-            row.append(f"{tm[t]:.3f}" if t in tm else "—")
-        L.append("| " + " | ".join(row) + " |")
-    L.append("")
-
-    # Legend
-    L.append("## Glossary")
-    L.append("")
-    L.append("- **Cov-norm**: `clean_sum / 120`. Missing runs count as 0.")
-    L.append("  This is the single fair comparison number — it penalizes both")
-    L.append("  low scores AND infra-related missing runs.")
-    L.append("- **Clean**: Mean run_score across archived runs (excludes infra-zeros).")
-    L.append("  Shows capability ceiling ignoring infra flakiness.")
-    L.append("- **Judge-only**: Mean LLM-judge score (0-1 from Claude Sonnet 4.6).")
-    L.append("  Independent second opinion on quality, used when deterministic")
-    L.append("  verifiers can't capture nuance.")
-    L.append("- **Cov%**: Fraction of 120 runs that produced a non-infra outcome.")
-    L.append("- **run_score**: Weighted combination — when deterministic verifiers")
-    L.append("  pass (C≥0.9999): `0.4·C + 0.3·T + 0.2·B + 0.1·J`. Else, judge excluded,")
-    L.append("  renormalized over C/T/B.")
-    L.append("")
-
-    # Caveats
-    L.append("## Caveats")
-    L.append("")
-    L.append("- **Missing runs** (1-3 per model) were infra failures that never")
-    L.append("  wrote to cache. Treated as 0 in cov-norm (penalizes the model).")
-    L.append("- **Some tasks have strict verifiers** that require specific file")
-    L.append("  artifacts. All models face the same verifier, so the comparison")
-    L.append("  is internally fair even where individual verifier scores feel low.")
-    L.append("- **Judge scores come from a single judge model** (Sonnet 4.6). Judge")
-    L.append("  bias toward its own family is possible but small at 10% weight.")
-    L.append("- **Ranking gaps of <0.02 cov-norm are within run-to-run noise**.")
-    L.append("  Treat models within the top cluster as roughly equivalent.")
-    L.append("")
-
-    return "\n".join(L) + "\n"
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--tag", required=True)
-    ap.add_argument("--out", type=Path, default=None)
-    ap.add_argument("--exclude", default="", help="comma-separated model labels to exclude")
-    args = ap.parse_args()
-
-    excluded = {x.strip() for x in args.exclude.split(",") if x.strip()}
-    summaries = [summarize(label, sub, pretty, args.tag)
-                 for label, (sub, pretty) in MODEL_MAP.items()
-                 if label not in excluded]
-
-    out_path = args.out or (ROOT / "reports" / f"EVAL_REPORT_9MODEL_FAIR_{args.tag}.md")
-    out_path.parent.mkdir(parents=True, exist_ok=True)
-    out_path.write_text(build_markdown(summaries, args.tag))
-    print(f"Wrote: {out_path}")
-
-    present = [s for s in summaries if s["n"] > 0]
-    present.sort(key=lambda s: -s.get("cov_norm", 0))
-    print()
-    print(f"{'Rank':>4} {'Model':<20} {'Runs':>7} {'Cov%':>5} {'CovNorm':>8} {'Clean':>7} {'Judge':>6}")
-    print("-" * 66)
-    for i, s in enumerate(present, 1):
-        jm = f"{s['judge_mean']:.3f}" if s.get("judge_mean") is not None else "—"
-        print(
-            f"{i:>4} {s['pretty']:<20} {s['n']}/120 {s['coverage_pct']:>4.0f}% "
-            f"{s['cov_norm']:>8.4f} {s['clean']:>7.4f} {jm:>6}"
-        )
-
-
-if __name__ == "__main__":
-    main()

scripts/infra_log_gate.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+# Fail if a ClawBench/OpenClaw run directory contains infra-level failures.
+
+set -u
+
+dir="${1:?usage: infra_log_gate.sh <log-dir>}"
+
+if [ ! -d "$dir" ]; then
+  echo "[infra-gate] missing log directory: $dir" >&2
+  exit 2
+fi
+
+pattern="no longer exists|env_unavailable|environment_unavailable|REJECTED|Traceback|model_not_allowed|model not allowed|not allowed|WebSocket closed|API key|billing|Insufficient|sessions.create.*✗|Gateway .*timed out|control-plane.*timed out|connect.*timed out|RPC .*timed out|agents.create timed out|sessions.create.*timed out"
+
+matches="$(rg -n "$pattern" "$dir" 2>/dev/null || true)"
+if [ -n "$matches" ]; then
+  echo "[infra-gate] infra-level signatures found in $dir" >&2
+  printf '%s\n' "$matches" | head -80 >&2
+  exit 1
+fi
+
+echo "[infra-gate] clean: $dir"

scripts/ingest_real_run.py

@@ -23,6 +23,7 @@ from clawbench.profile import (
     PluginManifest,
     PluginProfile,
     PluginProfileEntry,
+    RegistrationTrace,
 )
 
 

scripts/inject_judge_rubrics.py

@@ -12,6 +12,7 @@ being so specific that it leaks the answer to the agent's own model.
 
 from __future__ import annotations
 
+import sys
 from pathlib import Path
 
 import yaml

scripts/k8s/Dockerfile

@@ -1,33 +0,0 @@
-# Lightweight ClawBench image for Kubernetes sidecar use.
-# Does NOT include the full OpenClaw server or Chromium — the gateway runs
-# in a separate container. Node.js is copied from the OpenClaw image for
-# the device-identity handshake required by the gateway protocol.
-FROM ghcr.io/openclaw/openclaw:latest AS openclaw
-
-FROM python:3.12-slim
-
-COPY --from=openclaw /usr/local/bin/node /usr/local/bin/node
-
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends git && \
-    rm -rf /var/lib/apt/lists/*
-
-WORKDIR /app
-
-COPY pyproject.toml README.md CLAWBENCH_V0_4_SPEC.md PARTNER_TRACE_SPEC.md ./
-COPY clawbench/ clawbench/
-COPY tasks-public/ tasks-public/
-COPY tasks-domain/ tasks-domain/
-COPY profiles/ profiles/
-COPY baselines/ baselines/
-COPY scripts/ scripts/
-
-RUN pip install --no-cache-dir ".[mlflow]"
-
-RUN mkdir -p /results && chmod 777 /results
-
-RUN useradd -m -d /home/node clawbench
-USER clawbench
-ENV HOME=/home/node
-
-ENTRYPOINT ["clawbench"]

scripts/k8s/deploy.sh

@@ -1,486 +0,0 @@
-#!/usr/bin/env bash
-# Deploy ClawBench evals on Kubernetes (works on OpenShift too).
-#
-# 0-to-hero pipeline:
-#   Step 0: Create a cluster (see --help for Kind instructions)
-#   Step 1: Deploy OpenClaw gateway         (optional — bring your own)
-#   Step 2: Deploy MLflow tracking server   (optional — bring your own)
-#   Step 3: Run evals via sidecar           (add / remove)
-#
-# Usage:
-#   ./scripts/k8s/deploy.sh                        # Full deploy: OpenClaw + MLflow + eval
-#   ./scripts/k8s/deploy.sh --openclaw-only         # Step 1: deploy OpenClaw gateway
-#   ./scripts/k8s/deploy.sh --mlflow-only           # Step 2: deploy MLflow
-#   ./scripts/k8s/deploy.sh --add-sidecar           # Step 3: add eval sidecar (starts eval)
-#   ./scripts/k8s/deploy.sh --remove-sidecar        # Step 3: remove eval sidecar
-#   ./scripts/k8s/deploy.sh --logs                  # Tail clawbench sidecar logs
-#   ./scripts/k8s/deploy.sh --teardown              # Delete eval namespace (keeps MLflow)
-#
-# Environment (required):
-#   CLAWBENCH_NAMESPACE            Namespace for OpenClaw + eval
-#   OPENAI_API_KEY                 Model provider API key (or another provider key)
-#
-# Environment (optional):
-#   CLAWBENCH_IMAGE                Clawbench image (default: quay.io/sallyom/clawbench:latest)
-#   OPENCLAW_IMAGE                 OpenClaw image (default: ghcr.io/openclaw/openclaw:latest)
-#   OPENCLAW_GATEWAY_TOKEN         Existing gateway token (generated if unset)
-#   CLAWBENCH_MODEL                Model to eval (default: openai/gpt-5.5)
-#   MLFLOW_NAMESPACE               MLflow namespace (default: mlflow)
-#   MLFLOW_TRACKING_URI            External MLflow URI (skips MLflow deploy if set)
-#   MLFLOW_EXPERIMENT_ID           MLflow experiment ID
-#   MLFLOW_EXPERIMENT_NAME         MLflow experiment name
-#   MLFLOW_IMAGE                   MLflow image (default: ghcr.io/mlflow/mlflow:v2.21.3)
-#   ANTHROPIC_API_KEY              Anthropic key (added to secret if set)
-#   OPENROUTER_API_KEY             OpenRouter key (added to secret if set)
-#   GEMINI_API_KEY                 Gemini key (added to secret if set)
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-NS="${CLAWBENCH_NAMESPACE:-}"
-MLFLOW_NS="${MLFLOW_NAMESPACE:-mlflow}"
-CLAWBENCH_IMG="${CLAWBENCH_IMAGE:-quay.io/sallyom/clawbench:latest}"
-OPENCLAW_IMG="${OPENCLAW_IMAGE:-ghcr.io/openclaw/openclaw:latest}"
-MLFLOW_IMG="${MLFLOW_IMAGE:-ghcr.io/mlflow/mlflow:v2.21.3}"
-
-# ---------------------------------------------------------------------------
-if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
-  cat <<'HELP'
-ClawBench Kubernetes Deployment
-===============================
-
-0-to-hero pipeline for running ClawBench evals on Kubernetes.
-
-  Step 0: Create a cluster
-          For local testing with Kind, see:
-          https://github.com/openclaw/openclaw/blob/main/docs/install/kubernetes.md#local-testing-with-kind
-
-  Step 1: Deploy OpenClaw gateway (optional — skip if you have one)
-  Step 2: Deploy MLflow tracking server (optional — skip if you have one)
-  Step 3: Run evals via sidecar (add/remove to OpenClaw deployment)
-
-Usage:
-  ./scripts/k8s/deploy.sh                    Full deploy (steps 1+2+3)
-  ./scripts/k8s/deploy.sh --openclaw-only     Step 1: OpenClaw only
-  ./scripts/k8s/deploy.sh --mlflow-only       Step 2: MLflow only
-  ./scripts/k8s/deploy.sh --add-sidecar       Step 3: add eval sidecar (starts eval)
-  ./scripts/k8s/deploy.sh --remove-sidecar    Step 3: remove eval sidecar
-  ./scripts/k8s/deploy.sh --logs              Tail clawbench sidecar logs
-  ./scripts/k8s/deploy.sh --teardown          Delete eval namespace (keeps MLflow)
-
-Required environment:
-  CLAWBENCH_NAMESPACE          Namespace for OpenClaw + eval
-  OPENAI_API_KEY               Model provider API key (or ANTHROPIC_API_KEY, etc.)
-
-Optional environment:
-  CLAWBENCH_IMAGE              Clawbench image (default: quay.io/sallyom/clawbench:latest)
-  OPENCLAW_IMAGE               OpenClaw image (default: ghcr.io/openclaw/openclaw:latest)
-  OPENCLAW_GATEWAY_TOKEN       Existing gateway token (generated if unset)
-  CLAWBENCH_MODEL              Model to eval (default: openai/gpt-5.5)
-  MLFLOW_NAMESPACE             MLflow namespace (default: mlflow)
-  MLFLOW_TRACKING_URI          External MLflow URI (skips MLflow deploy)
-  MLFLOW_EXPERIMENT_ID         MLflow experiment ID
-  MLFLOW_EXPERIMENT_NAME       MLflow experiment name
-  MLFLOW_IMAGE                 MLflow image (default: ghcr.io/mlflow/mlflow:v2.21.3)
-  ANTHROPIC_API_KEY            Anthropic key (added to secret if set)
-  OPENROUTER_API_KEY           OpenRouter key (added to secret if set)
-  GEMINI_API_KEY               Gemini key (added to secret if set)
-
-Works on Kubernetes and OpenShift.
-HELP
-  exit 0
-fi
-
-command -v kubectl &>/dev/null || { echo "Missing: kubectl" >&2; exit 1; }
-
-if [[ -z "$NS" ]]; then
-  echo "CLAWBENCH_NAMESPACE is required." >&2
-  echo "  export CLAWBENCH_NAMESPACE=clawbench-eval" >&2
-  exit 1
-fi
-
-MODE="full"
-while [[ $# -gt 0 ]]; do
-  case "$1" in
-    --openclaw-only)   MODE="openclaw-only" ;;
-    --mlflow-only)     MODE="mlflow-only" ;;
-    --add-sidecar)     MODE="add-sidecar" ;;
-    --remove-sidecar)  MODE="remove-sidecar" ;;
-    --logs)            MODE="logs" ;;
-    --teardown)        MODE="teardown" ;;
-    *) echo "Unknown option: $1" >&2; exit 1 ;;
-  esac
-  shift
-done
-
-kubectl cluster-info &>/dev/null || { echo "Cannot connect to cluster. Check kubeconfig." >&2; exit 1; }
-
-# ---------------------------------------------------------------------------
-# --logs
-# ---------------------------------------------------------------------------
-if [[ "$MODE" == "logs" ]]; then
-  kubectl logs deploy/openclaw -c clawbench -n "$NS" -f
-  exit 0
-fi
-
-# ---------------------------------------------------------------------------
-# --teardown
-# ---------------------------------------------------------------------------
-if [[ "$MODE" == "teardown" ]]; then
-  echo "Deleting namespace '$NS'..."
-  kubectl delete namespace "$NS" --ignore-not-found
-  echo "Done. MLflow namespace '$MLFLOW_NS' was not deleted."
-  exit 0
-fi
-
-# ---------------------------------------------------------------------------
-# --remove-sidecar
-# ---------------------------------------------------------------------------
-if [[ "$MODE" == "remove-sidecar" ]]; then
-  echo "Removing clawbench sidecar from openclaw in namespace '$NS'..."
-  INDEX=$(kubectl get deploy/openclaw -n "$NS" -o json \
-    | python3 -c "import json,sys; cs=json.load(sys.stdin)['spec']['template']['spec']['containers']; print(next((i for i,c in enumerate(cs) if c['name']=='clawbench'),-1))")
-  if [[ "$INDEX" == "-1" ]]; then
-    echo "No clawbench sidecar found."
-  else
-    kubectl patch deploy/openclaw -n "$NS" --type=json \
-      -p "[{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/$INDEX\"}]"
-    echo "Sidecar removed."
-  fi
-  exit 0
-fi
-
-# ---------------------------------------------------------------------------
-# Create namespace + secret
-# ---------------------------------------------------------------------------
-ensure_namespace_and_secret() {
-  if ! kubectl get namespace "$NS" &>/dev/null; then
-    echo "Creating namespace '$NS'..."
-    kubectl create namespace "$NS"
-  fi
-
-  if ! kubectl get secret clawbench-secrets -n "$NS" &>/dev/null; then
-    echo "Creating clawbench-secrets..."
-    if [[ -n "${OPENCLAW_GATEWAY_TOKEN:-}" ]]; then
-      GATEWAY_TOKEN="$OPENCLAW_GATEWAY_TOKEN"
-      GATEWAY_TOKEN_SOURCE="from OPENCLAW_GATEWAY_TOKEN"
-    else
-      GATEWAY_TOKEN=$(python3 -c "import secrets,base64; print(base64.b64encode(secrets.token_bytes(32)).decode())")
-      GATEWAY_TOKEN_SOURCE="generated"
-    fi
-
-    SECRET_ARGS=(
-      --from-literal=OPENCLAW_GATEWAY_TOKEN="$GATEWAY_TOKEN"
-    )
-    [[ -n "${OPENAI_API_KEY:-}" ]] && SECRET_ARGS+=(--from-literal=OPENAI_API_KEY="$OPENAI_API_KEY")
-    [[ -n "${ANTHROPIC_API_KEY:-}" ]] && SECRET_ARGS+=(--from-literal=ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY")
-    [[ -n "${OPENROUTER_API_KEY:-}" ]] && SECRET_ARGS+=(--from-literal=OPENROUTER_API_KEY="$OPENROUTER_API_KEY")
-    [[ -n "${GEMINI_API_KEY:-}" ]] && SECRET_ARGS+=(--from-literal=GEMINI_API_KEY="$GEMINI_API_KEY")
-
-    if [[ ${#SECRET_ARGS[@]} -eq 1 ]]; then
-      echo "Warning: No API keys provided. Set OPENAI_API_KEY or another provider key." >&2
-    fi
-
-    kubectl create secret generic clawbench-secrets -n "$NS" "${SECRET_ARGS[@]}"
-    echo "  Gateway token: $GATEWAY_TOKEN_SOURCE"
-    [[ -n "${OPENAI_API_KEY:-}" ]] && echo "  OPENAI_API_KEY: set"
-    [[ -n "${ANTHROPIC_API_KEY:-}" ]] && echo "  ANTHROPIC_API_KEY: set"
-    [[ -n "${OPENROUTER_API_KEY:-}" ]] && echo "  OPENROUTER_API_KEY: set"
-    [[ -n "${GEMINI_API_KEY:-}" ]] && echo "  GEMINI_API_KEY: set"
-  else
-    echo "Secret clawbench-secrets already exists in '$NS'."
-  fi
-  return 0
-}
-
-# ---------------------------------------------------------------------------
-# Step 1: Deploy OpenClaw
-# ---------------------------------------------------------------------------
-deploy_openclaw() {
-  echo ""
-  echo "Step 1: Deploying OpenClaw gateway (image: $OPENCLAW_IMG)..."
-
-  kubectl apply -f "$SCRIPT_DIR/openclaw/configmap.yaml" -n "$NS"
-
-  # Patch gateway config with custom OpenAI-compatible base URL
-  if [[ -n "${OPENAI_API_BASE:-}" ]]; then
-    echo "  Patching gateway config: models.providers.openai.baseUrl = $OPENAI_API_BASE"
-    EXISTING_JSON=$(kubectl get configmap openclaw-config -n "$NS" -o jsonpath='{.data.openclaw\.json}')
-    PATCHED_JSON=$(echo "$EXISTING_JSON" | python3 -c "
-import json, sys, os
-cfg = json.load(sys.stdin)
-openai_cfg = cfg.setdefault('models', {}).setdefault('providers', {}).setdefault('openai', {})
-openai_cfg['baseUrl'] = os.environ['OPENAI_API_BASE']
-openai_cfg.setdefault('models', [])
-json.dump(cfg, sys.stdout, indent=2)
-")
-    kubectl create configmap openclaw-config -n "$NS" \
-      --from-literal="openclaw.json=$PATCHED_JSON" \
-      --dry-run=client -o yaml | kubectl apply -f - -n "$NS" >/dev/null
-  fi
-
-  kubectl apply -f "$SCRIPT_DIR/openclaw/pvc.yaml" -n "$NS"
-  kubectl apply -f "$SCRIPT_DIR/openclaw/service.yaml" -n "$NS"
-
-  if [[ "$OPENCLAW_IMG" != "ghcr.io/openclaw/openclaw:latest" ]]; then
-    kubectl apply -f "$SCRIPT_DIR/openclaw/deployment.yaml" -n "$NS"
-    kubectl set image "deploy/openclaw" "gateway=$OPENCLAW_IMG" -n "$NS"
-  else
-    kubectl apply -f "$SCRIPT_DIR/openclaw/deployment.yaml" -n "$NS"
-  fi
-
-  echo "Waiting for OpenClaw rollout..."
-  kubectl rollout status deploy/openclaw -n "$NS" --timeout=180s || \
-    echo "  (rollout still in progress)"
-  echo "OpenClaw deployed."
-}
-
-# ---------------------------------------------------------------------------
-# Step 2: Deploy MLflow
-# ---------------------------------------------------------------------------
-deploy_mlflow() {
-  if [[ -n "${MLFLOW_TRACKING_URI:-}" ]]; then
-    echo ""
-    echo "Step 2: Skipping MLflow deploy (MLFLOW_TRACKING_URI is set: $MLFLOW_TRACKING_URI)"
-    return
-  fi
-
-  echo ""
-  echo "Step 2: Deploying MLflow (namespace: $MLFLOW_NS, image: $MLFLOW_IMG)..."
-
-  if ! kubectl get namespace "$MLFLOW_NS" &>/dev/null; then
-    kubectl create namespace "$MLFLOW_NS"
-  fi
-
-  kubectl apply -f "$SCRIPT_DIR/mlflow/pvc.yaml" -n "$MLFLOW_NS"
-  kubectl apply -f "$SCRIPT_DIR/mlflow/service.yaml" -n "$MLFLOW_NS"
-
-  if [[ "$MLFLOW_IMG" != "ghcr.io/mlflow/mlflow:v2.21.3" ]]; then
-    kubectl apply -f "$SCRIPT_DIR/mlflow/deployment.yaml" -n "$MLFLOW_NS"
-    kubectl set image "deploy/mlflow" "mlflow=$MLFLOW_IMG" -n "$MLFLOW_NS"
-  else
-    kubectl apply -f "$SCRIPT_DIR/mlflow/deployment.yaml" -n "$MLFLOW_NS"
-  fi
-
-  echo "Waiting for MLflow rollout..."
-  kubectl rollout status deploy/mlflow -n "$MLFLOW_NS" --timeout=120s || \
-    echo "  (rollout still in progress)"
-
-  MLFLOW_TRACKING_URI="http://mlflow-service.${MLFLOW_NS}.svc.cluster.local:5000"
-  echo "MLflow deployed: $MLFLOW_TRACKING_URI"
-}
-
-# ---------------------------------------------------------------------------
-# Step 3: Add clawbench sidecar (starts eval)
-# ---------------------------------------------------------------------------
-add_sidecar() {
-  echo ""
-  echo "Step 3: Adding clawbench eval sidecar..."
-
-  echo "Applying clawbench ConfigMap..."
-  kubectl apply -f "$SCRIPT_DIR/manifests/configmap.yaml" -n "$NS" >/dev/null
-
-  if [[ -n "${CLAWBENCH_MODEL:-}" ]]; then
-    kubectl patch configmap clawbench-config -n "$NS" \
-      --type merge -p "{\"data\":{\"CLAWBENCH_MODEL\":\"$CLAWBENCH_MODEL\"}}" >/dev/null
-    echo "  Model: $CLAWBENCH_MODEL"
-  fi
-
-  if [[ -n "${OPENAI_API_BASE:-}" ]]; then
-    kubectl patch configmap clawbench-config -n "$NS" \
-      --type merge -p "{\"data\":{\"OPENAI_API_BASE\":\"$OPENAI_API_BASE\"}}" >/dev/null
-    echo "  OpenAI API base: $OPENAI_API_BASE"
-  fi
-
-  # Patch MLflow settings into ConfigMap
-  PATCH_DATA=""
-  MLFLOW_URI="${MLFLOW_TRACKING_URI:-http://mlflow-service.${MLFLOW_NS}.svc.cluster.local:5000}"
-  PATCH_DATA="\"MLFLOW_TRACKING_URI\":\"$MLFLOW_URI\""
-  if [[ -n "${MLFLOW_EXPERIMENT_ID:-}" ]]; then
-    PATCH_DATA="$PATCH_DATA,\"MLFLOW_EXPERIMENT_ID\":\"$MLFLOW_EXPERIMENT_ID\""
-  fi
-  if [[ -n "${MLFLOW_EXPERIMENT_NAME:-}" ]]; then
-    PATCH_DATA="$PATCH_DATA,\"MLFLOW_EXPERIMENT_NAME\":\"$MLFLOW_EXPERIMENT_NAME\""
-  fi
-  kubectl patch configmap clawbench-config -n "$NS" \
-    --type merge -p "{\"data\":{$PATCH_DATA}}" >/dev/null
-  echo "  MLflow URI: $MLFLOW_URI"
-  [[ -n "${MLFLOW_EXPERIMENT_ID:-}" ]] && echo "  MLflow experiment ID: $MLFLOW_EXPERIMENT_ID"
-  [[ -n "${MLFLOW_EXPERIMENT_NAME:-}" ]] && echo "  MLflow experiment name: $MLFLOW_EXPERIMENT_NAME"
-
-  # Check if sidecar already exists
-  HAS_SIDECAR=$(kubectl get deploy/openclaw -n "$NS" -o json \
-    | python3 -c "import json,sys; cs=json.load(sys.stdin)['spec']['template']['spec']['containers']; print('yes' if any(c['name']=='clawbench' for c in cs) else 'no')")
-
-  if [[ "$HAS_SIDECAR" == "yes" ]]; then
-    echo "Removing existing clawbench sidecar..."
-    INDEX=$(kubectl get deploy/openclaw -n "$NS" -o json \
-      | python3 -c "import json,sys; cs=json.load(sys.stdin)['spec']['template']['spec']['containers']; print(next(i for i,c in enumerate(cs) if c['name']=='clawbench'))")
-    kubectl patch deploy/openclaw -n "$NS" --type=json \
-      -p "[{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/$INDEX\"}]" >/dev/null
-  fi
-
-  # Find the OpenClaw home volume, and capture existing volumes so add-sidecar
-  # also works with bring-your-own deployments that lack this repo's PVC layout.
-  VOLUME_INFO=$(kubectl get deploy/openclaw -n "$NS" -o json \
-    | python3 -c "
-import json, sys
-spec = json.load(sys.stdin)['spec']['template']['spec']
-volume_names = [v.get('name') for v in spec.get('volumes', []) if v.get('name')]
-home_volume = 'openclaw-home'
-for c in spec['containers']:
-    if c['name'] == 'gateway':
-        for vm in c.get('volumeMounts', []):
-            if vm['mountPath'] == '/home/node/.openclaw':
-                home_volume = vm['name']
-                break
-print(json.dumps({
-    'home_volume': home_volume,
-    'volumes_present': 'volumes' in spec,
-    'volume_names': volume_names,
-}))
-")
-
-  echo "Adding clawbench sidecar (image: $CLAWBENCH_IMG)..."
-
-  PATCH=$(VOLUME_INFO="$VOLUME_INFO" CLAWBENCH_IMG="$CLAWBENCH_IMG" python3 - <<'PY'
-import json
-import os
-
-info = json.loads(os.environ["VOLUME_INFO"])
-home_volume = info["home_volume"]
-
-command = r"""echo "Waiting for gateway on localhost:18789..."
-for i in $(seq 1 90); do
-  python3 -c "import socket; s=socket.create_connection((\"127.0.0.1\",18789),2); s.close()" 2>/dev/null && echo "Gateway ready" && break
-  sleep 2
-done
-
-if [ -n "${MLFLOW_TRACKING_URI:-}" ]; then
-  echo "Checking MLflow at ${MLFLOW_TRACKING_URI}..."
-  python3 -c "import httpx,os; r=httpx.get(os.environ[\"MLFLOW_TRACKING_URI\"]+\"/health\"); print(\"MLflow OK:\",r.status_code)" 2>&1 || echo "MLflow pre-check failed (will retry at log time)"
-fi
-
-echo "Starting eval..."
-clawbench run \
-  --model "${CLAWBENCH_MODEL}" \
-  --gateway-token "${OPENCLAW_GATEWAY_TOKEN}" \
-  --runs "${CLAWBENCH_RUNS}" \
-  --concurrency "${CLAWBENCH_CONCURRENCY}" \
-  ${CLAWBENCH_JUDGE_MODEL:+--judge-model "${CLAWBENCH_JUDGE_MODEL}"} \
-  $([ -n "${CLAWBENCH_TASKS:-}" ] && for t in ${CLAWBENCH_TASKS}; do printf -- "-t %s " "$t"; done) \
-  -o /results/benchmark.json
-RC=$?
-if [ $RC -eq 0 ] && [ -n "${MLFLOW_TRACKING_URI:-}" ]; then
-  python scripts/log_to_mlflow.py /results/benchmark.json
-fi
-echo "ClawBench finished (exit=$RC)"
-sleep infinity"""
-
-container = {
-    "name": "clawbench",
-    "image": os.environ["CLAWBENCH_IMG"],
-    "imagePullPolicy": "IfNotPresent",
-    "command": ["/bin/bash", "-c", command],
-    "envFrom": [{"configMapRef": {"name": "clawbench-config"}}],
-    "env": [
-        {
-            "name": "OPENCLAW_GATEWAY_TOKEN",
-            "valueFrom": {
-                "secretKeyRef": {
-                    "name": "clawbench-secrets",
-                    "key": "OPENCLAW_GATEWAY_TOKEN",
-                }
-            },
-        }
-    ],
-    "resources": {
-        "requests": {"memory": "1Gi", "cpu": "500m"},
-        "limits": {"memory": "4Gi", "cpu": "2"},
-    },
-    "volumeMounts": [
-        {"name": home_volume, "mountPath": "/home/node/.openclaw"},
-        {"name": "clawbench-results", "mountPath": "/results"},
-        {"name": "tmp-volume", "mountPath": "/tmp"},
-    ],
-    "securityContext": {
-        "allowPrivilegeEscalation": False,
-        "capabilities": {"drop": ["ALL"]},
-    },
-}
-
-patch = [{"op": "add", "path": "/spec/template/spec/containers/-", "value": container}]
-
-existing_volumes = set(info["volume_names"])
-required_volumes = [
-    {"name": home_volume, "emptyDir": {}},
-    {"name": "clawbench-results", "emptyDir": {}},
-    {"name": "tmp-volume", "emptyDir": {}},
-]
-missing_volumes = []
-for volume in required_volumes:
-    if volume["name"] not in existing_volumes and volume["name"] not in {
-        item["name"] for item in missing_volumes
-    }:
-        missing_volumes.append(volume)
-
-if missing_volumes:
-    if info["volumes_present"]:
-        patch.extend(
-            {"op": "add", "path": "/spec/template/spec/volumes/-", "value": volume}
-            for volume in missing_volumes
-        )
-    else:
-        patch.append(
-            {"op": "add", "path": "/spec/template/spec/volumes", "value": missing_volumes}
-        )
-
-print(json.dumps(patch))
-PY
-)
-
-  kubectl patch deploy/openclaw -n "$NS" --type=json -p "$PATCH" >/dev/null
-
-  echo ""
-  echo "Waiting for rollout..."
-  kubectl rollout status deploy/openclaw -n "$NS" --timeout=300s 2>/dev/null || \
-    echo "  (rollout timeout — eval runs for 30-60 min)"
-
-  echo ""
-  echo "Eval is running. Follow logs with:"
-  echo "  ./scripts/k8s/deploy.sh --logs"
-  echo ""
-  echo "When finished, remove the sidecar with:"
-  echo "  ./scripts/k8s/deploy.sh --remove-sidecar"
-}
-
-# ---------------------------------------------------------------------------
-# Execute
-# ---------------------------------------------------------------------------
-case "$MODE" in
-  full)
-    ensure_namespace_and_secret
-    deploy_openclaw
-    deploy_mlflow
-    add_sidecar
-    ;;
-  openclaw-only)
-    ensure_namespace_and_secret
-    deploy_openclaw
-    echo ""
-    echo "OpenClaw is running. Next steps:"
-    echo "  ./scripts/k8s/deploy.sh --mlflow-only       # Deploy MLflow"
-    echo "  ./scripts/k8s/deploy.sh --add-sidecar       # Start eval"
-    ;;
-  mlflow-only)
-    deploy_mlflow
-    ;;
-  add-sidecar)
-    if ! kubectl get deploy/openclaw -n "$NS" &>/dev/null; then
-      echo "Deployment 'openclaw' not found in namespace '$NS'." >&2
-      echo "Deploy OpenClaw first with: ./scripts/k8s/deploy.sh --openclaw-only" >&2
-      exit 1
-    fi
-    ensure_namespace_and_secret
-    add_sidecar
-    ;;
-esac

scripts/k8s/manifests/configmap.yaml

@@ -1,18 +0,0 @@
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: clawbench-config
-  labels:
-    app: clawbench
-data:
-  CLAWBENCH_MODEL: "openai/gpt-5.5"
-  OPENAI_API_BASE: ""
-  CLAWBENCH_RUNS: "3"
-  CLAWBENCH_CONCURRENCY: "4"
-  CLAWBENCH_JUDGE_MODEL: ""
-  CLAWBENCH_TASKS: ""
-  CLAWBENCH_CONNECT_TIMEOUT: "120"
-  CLAWBENCH_REQUEST_TIMEOUT: "300"
-  CLAWBENCH_PER_RUN_BUDGET_SECONDS: "600"
-  MLFLOW_TRACKING_URI: "http://mlflow-service.mlflow.svc.cluster.local:5000"
-  MLFLOW_EXPERIMENT_NAME: "clawbench"

scripts/k8s/manifests/secret.yaml

@@ -1,15 +0,0 @@
-# Reference template — do NOT apply directly.
-# The deploy script (scripts/k8s/deploy.sh) creates this secret automatically
-# from exported environment variables (OPENAI_API_KEY, etc.).
-apiVersion: v1
-kind: Secret
-metadata:
-  name: clawbench-secrets
-  labels:
-    app: clawbench
-type: Opaque
-stringData:
-  OPENAI_API_KEY: "REPLACE_ME"
-  # Add other provider keys as needed:
-  # ANTHROPIC_API_KEY: "REPLACE_ME"
-  # OPENROUTER_API_KEY: "REPLACE_ME"

scripts/k8s/mlflow/deployment.yaml

@@ -1,68 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: mlflow
-  labels:
-    app: mlflow
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app: mlflow
-  template:
-    metadata:
-      labels:
-        app: mlflow
-    spec:
-      containers:
-        - name: mlflow
-          image: ghcr.io/mlflow/mlflow:v2.21.3
-          command:
-            - mlflow
-            - server
-            - --host
-            - "0.0.0.0"
-            - --port
-            - "5000"
-            - --backend-store-uri
-            - sqlite:///mlflow/mlflow.db
-            - --default-artifact-root
-            - /mlflow/artifacts
-            - --serve-artifacts
-          ports:
-            - name: http
-              containerPort: 5000
-              protocol: TCP
-          livenessProbe:
-            httpGet:
-              path: /health
-              port: 5000
-            initialDelaySeconds: 15
-            periodSeconds: 30
-          readinessProbe:
-            httpGet:
-              path: /health
-              port: 5000
-            initialDelaySeconds: 5
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              cpu: 500m
-              memory: 1Gi
-          securityContext:
-            allowPrivilegeEscalation: false
-            capabilities:
-              drop:
-                - ALL
-          volumeMounts:
-            - name: mlflow-data
-              mountPath: /mlflow
-      volumes:
-        - name: mlflow-data
-          persistentVolumeClaim:
-            claimName: mlflow-data-pvc

scripts/k8s/mlflow/pvc.yaml

@@ -1,12 +0,0 @@
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: mlflow-data-pvc
-  labels:
-    app: mlflow
-spec:
-  accessModes:
-    - ReadWriteOnce
-  resources:
-    requests:
-      storage: 5Gi

scripts/k8s/mlflow/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: mlflow-service
-  labels:
-    app: mlflow
-spec:
-  type: ClusterIP
-  selector:
-    app: mlflow
-  ports:
-    - name: http
-      port: 5000
-      targetPort: 5000
-      protocol: TCP

scripts/k8s/openclaw/configmap.yaml

@@ -1,36 +0,0 @@
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: openclaw-config
-  labels:
-    app: openclaw
-data:
-  openclaw.json: |
-    {
-      "gateway": {
-        "mode": "local",
-        "bind": "loopback",
-        "port": 18789,
-        "auth": {
-          "mode": "token"
-        }
-      },
-      "browser": {
-        "enabled": true,
-        "headless": true,
-        "noSandbox": true,
-        "ssrfPolicy": {
-          "allowedHostnames": ["localhost", "127.0.0.1"]
-        }
-      },
-      "tools": {
-        "profile": "coding",
-        "alsoAllow": ["browser"]
-      },
-      "agents": {
-        "defaults": {
-          "workspace": "~/.openclaw/workspace"
-        }
-      },
-      "cron": { "enabled": false }
-    }

scripts/k8s/openclaw/deployment.yaml

@@ -1,146 +0,0 @@
-# OpenClaw gateway deployment for ClawBench evals.
-#
-# Build the image with browser support:
-#   docker build --build-arg OPENCLAW_INSTALL_BROWSER=1 \
-#     -t quay.io/yourorg/openclaw:eval .
-#
-# Or use upstream without browser (browser eval tasks will score 0):
-#   image: ghcr.io/openclaw/openclaw:latest
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: openclaw
-  labels:
-    app: openclaw
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app: openclaw
-  template:
-    metadata:
-      labels:
-        app: openclaw
-    spec:
-      initContainers:
-        - name: init-config
-          image: registry.access.redhat.com/ubi9-minimal:latest
-          command:
-            - sh
-            - -c
-            - |
-              cp /config/openclaw.json /home/node/.openclaw/openclaw.json
-              chmod 666 /home/node/.openclaw/openclaw.json
-              mkdir -p /home/node/.openclaw/workspace
-              mkdir -p /home/node/.openclaw/agents
-              chmod 777 /home/node/.openclaw /home/node/.openclaw/workspace /home/node/.openclaw/agents
-              echo "Config initialized"
-          volumeMounts:
-            - name: openclaw-home
-              mountPath: /home/node/.openclaw
-            - name: config-template
-              mountPath: /config
-          resources:
-            limits:
-              cpu: 200m
-              memory: 128Mi
-            requests:
-              cpu: 50m
-              memory: 64Mi
-      containers:
-        - name: gateway
-          image: ghcr.io/openclaw/openclaw:latest
-          imagePullPolicy: IfNotPresent
-          command:
-            - sh
-            - -c
-            - umask 007 && exec node dist/index.js gateway run --bind loopback --port 18789 --allow-unconfigured
-          env:
-            - name: HOME
-              value: /home/node
-            - name: NODE_ENV
-              value: production
-            - name: OPENCLAW_CONFIG_DIR
-              value: /home/node/.openclaw
-            - name: OPENCLAW_STATE_DIR
-              value: /home/node/.openclaw
-            - name: OPENCLAW_GATEWAY_TOKEN
-              valueFrom:
-                secretKeyRef:
-                  name: clawbench-secrets
-                  key: OPENCLAW_GATEWAY_TOKEN
-            - name: OPENAI_API_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: clawbench-secrets
-                  key: OPENAI_API_KEY
-                  optional: true
-            - name: ANTHROPIC_API_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: clawbench-secrets
-                  key: ANTHROPIC_API_KEY
-                  optional: true
-            - name: OPENROUTER_API_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: clawbench-secrets
-                  key: OPENROUTER_API_KEY
-                  optional: true
-            - name: GEMINI_API_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: clawbench-secrets
-                  key: GEMINI_API_KEY
-                  optional: true
-          ports:
-            - name: gateway
-              containerPort: 18789
-              protocol: TCP
-          livenessProbe:
-            exec:
-              command:
-                - node
-                - -e
-                - "require('http').get('http://127.0.0.1:18789/',r=>process.exit(r.statusCode<400?0:1)).on('error',()=>process.exit(1))"
-            initialDelaySeconds: 60
-            periodSeconds: 30
-            timeoutSeconds: 10
-          readinessProbe:
-            exec:
-              command:
-                - node
-                - -e
-                - "require('http').get('http://127.0.0.1:18789/',r=>process.exit(r.statusCode<400?0:1)).on('error',()=>process.exit(1))"
-            initialDelaySeconds: 30
-            periodSeconds: 10
-            timeoutSeconds: 5
-          resources:
-            requests:
-              cpu: 250m
-              memory: 1Gi
-            limits:
-              cpu: "2"
-              memory: 4Gi
-          securityContext:
-            allowPrivilegeEscalation: false
-            capabilities:
-              drop:
-                - ALL
-          volumeMounts:
-            - name: openclaw-home
-              mountPath: /home/node/.openclaw
-            - name: tmp-volume
-              mountPath: /tmp
-      terminationGracePeriodSeconds: 30
-      volumes:
-        - name: openclaw-home
-          persistentVolumeClaim:
-            claimName: openclaw-home-pvc
-        - name: config-template
-          configMap:
-            name: openclaw-config
-        - name: tmp-volume
-          emptyDir: {}

scripts/k8s/openclaw/pvc.yaml

@@ -1,12 +0,0 @@
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: openclaw-home-pvc
-  labels:
-    app: openclaw
-spec:
-  accessModes:
-    - ReadWriteOnce
-  resources:
-    requests:
-      storage: 10Gi

scripts/k8s/openclaw/secret.yaml

@@ -1,17 +0,0 @@
-# Reference template — do NOT apply directly.
-# The deploy script (scripts/k8s/deploy.sh) creates this secret automatically
-# from exported environment variables (OPENAI_API_KEY, etc.).
-apiVersion: v1
-kind: Secret
-metadata:
-  name: clawbench-secrets
-  labels:
-    app: openclaw
-type: Opaque
-stringData:
-  OPENCLAW_GATEWAY_TOKEN: "REPLACE_ME"
-  OPENAI_API_KEY: "REPLACE_ME"
-  # Add other provider keys as needed:
-  # ANTHROPIC_API_KEY: "REPLACE_ME"
-  # OPENROUTER_API_KEY: "REPLACE_ME"
-  # GEMINI_API_KEY: "REPLACE_ME"

scripts/k8s/openclaw/service.yaml

@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: openclaw
-  labels:
-    app: openclaw
-spec:
-  type: ClusterIP
-  selector:
-    app: openclaw
-  ports:
-    - name: gateway
-      port: 18789
-      targetPort: 18789
-      protocol: TCP

scripts/log_to_mlflow.py

@@ -1,125 +0,0 @@
-#!/usr/bin/env python3
-"""Log a ClawBench BenchmarkResult to MLflow.
-
-Standalone script -- not imported by the clawbench package.
-Requires: pip install mlflow  (or pip install clawbench[mlflow])
-
-Usage:
-    python scripts/log_to_mlflow.py /results/benchmark.json
-
-Environment:
-    MLFLOW_TRACKING_URI      MLflow tracking server (default: http://localhost:5000)
-    MLFLOW_EXPERIMENT_NAME   Experiment name (default: clawbench)
-"""
-
-from __future__ import annotations
-
-import json
-import os
-import sys
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-
-def main(result_path: str) -> None:
-    try:
-        import mlflow
-    except ImportError:
-        print(
-            "mlflow is not installed. Install with: pip install mlflow"
-            "  (or pip install clawbench[mlflow])",
-            file=sys.stderr,
-        )
-        sys.exit(1)
-
-    from clawbench.schemas import BenchmarkResult
-
-    with open(result_path, encoding="utf-8") as f:
-        result = BenchmarkResult(**json.load(f))
-
-    experiment_id = os.environ.get("MLFLOW_EXPERIMENT_ID")
-    if experiment_id:
-        experiment = mlflow.set_experiment(experiment_id=experiment_id)
-    else:
-        experiment = mlflow.set_experiment(os.environ.get("MLFLOW_EXPERIMENT_NAME", "clawbench"))
-
-    run_name = f"{result.model}-{result.submission_id[:8]}"
-    with mlflow.start_run(run_name=run_name):
-        mlflow.log_params(
-            {
-                "model": result.model,
-                "provider": result.provider,
-                "benchmark_version": result.benchmark_version,
-                "openclaw_version": result.openclaw_version or "unknown",
-                "judge_model": result.judge_model or "none",
-                "task_snapshot_fingerprint": result.task_snapshot_fingerprint or "unknown",
-            }
-        )
-
-        mlflow.log_metrics(
-            {
-                "overall_score": result.overall_score,
-                "overall_completion": result.overall_completion,
-                "overall_trajectory": result.overall_trajectory,
-                "overall_behavior": result.overall_behavior,
-                "overall_reliability": result.overall_reliability,
-                "overall_pass_hat_k": result.overall_pass_hat_k,
-                "overall_judge_score": result.overall_judge_score,
-                "overall_judge_confidence": result.overall_judge_confidence,
-                "overall_judge_pass_rate": result.overall_judge_pass_rate,
-                "judge_task_coverage": result.judge_task_coverage,
-                "overall_weighted_query_score": result.overall_weighted_query_score,
-                "overall_median_latency_ms": result.overall_median_latency_ms,
-                "overall_p95_latency_ms": result.overall_p95_latency_ms,
-                "overall_total_tokens": result.overall_total_tokens,
-                "overall_cost_usd": result.overall_cost_usd,
-                "overall_tokens_per_pass": result.overall_tokens_per_pass,
-                "overall_cost_per_pass": result.overall_cost_per_pass,
-                "overall_ci_lower": result.overall_ci_lower,
-                "overall_ci_upper": result.overall_ci_upper,
-            }
-        )
-
-        for tier in result.tier_results:
-            mlflow.log_metrics(
-                {
-                    f"{tier.tier}/score": tier.mean_task_score,
-                    f"{tier.tier}/completion": tier.mean_completion,
-                    f"{tier.tier}/trajectory": tier.mean_trajectory,
-                    f"{tier.tier}/behavior": tier.mean_behavior,
-                    f"{tier.tier}/reliability": tier.mean_reliability,
-                }
-            )
-
-        for i, task in enumerate(result.task_results):
-            mlflow.log_metrics(
-                {
-                    f"task/{task.task_id}/score": task.mean_task_score,
-                    f"task/{task.task_id}/reliability": task.reliability_score,
-                },
-                step=i,
-            )
-
-        mlflow.set_tags(
-            {
-                "submission_id": result.submission_id,
-                "timestamp": result.timestamp,
-                "certified": str(result.certified),
-            }
-        )
-
-        try:
-            mlflow.log_artifact(result_path)
-        except Exception as e:
-            print(f"Warning: artifact upload failed: {e}", file=sys.stderr)
-            print("Metrics and params were logged successfully.", file=sys.stderr)
-
-    print(f"Logged to MLflow: experiment={experiment.name} run={run_name}")
-
-
-if __name__ == "__main__":
-    if len(sys.argv) != 2:
-        print(f"Usage: {sys.argv[0]} <result.json>", file=sys.stderr)
-        sys.exit(1)
-    main(sys.argv[1])

scripts/refactor_verifiers.py

@@ -10,6 +10,7 @@ look for "wherever the agent put it."
 
 from __future__ import annotations
 
+import sys
 from pathlib import Path
 from textwrap import dedent
 

scripts/rejudge_all.py

@@ -1,288 +0,0 @@
-"""Re-judge ALL judge-infra-failure runs across all models in a drift sweep dir.
-
-Fixes: 'Gateway is restarting', 'Judge execution failed', empty-reason 0-score
-judge results by re-running the judge via direct Anthropic API calls (bypassing
-the gateway that was failing in the first place).
-
-Updates:
-  - data/run_cache_archive/<sweep_tag>/<model>/<task>/runN.json  (in place)
-  - data/drift_*/docker_<label>_<tag>.json                       (aggregates)
-
-Usage:
-  python3 scripts/rejudge_all.py \
-    --drift-dir data/drift_2026-04-19-full \
-    --archive-dir data/run_cache_archive/v2026-4-19-full \
-    [--dry-run]
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import os
-import re
-import sys
-import time
-from pathlib import Path
-from typing import Optional
-
-import anthropic
-import yaml
-
-
-ROOT = Path(__file__).resolve().parent.parent
-TASK_DIRS = [ROOT / "tasks" / f"tier{i}" for i in range(1, 6)]
-
-FAILURE_PHRASES = [
-    "gateway is restarting",
-    "judge execution failed",
-    "judge failed to run",
-    "judge call failed",
-    "judge timed out",
-]
-
-# Weights copied from clawbench/scorer.py
-WEIGHTS_DETERMINISTIC = {"completion": 0.40, "trajectory": 0.30, "behavior": 0.20}
-WEIGHTS_WITH_JUDGE = {"completion": 0.40, "trajectory": 0.30, "behavior": 0.20, "judge": 0.10}
-WEIGHTS_SEMANTIC_ONLY = {"completion": 0.20, "trajectory": 0.20, "behavior": 0.10, "judge": 0.50}
-DETERMINISTIC_FLOOR = 0.9999
-
-# Cache-sub → model label (for result JSON lookup)
-CACHE_TO_LABEL = {
-    "openrouter_z-ai_glm-5.1": "glm",
-    "openrouter_minimax_minimax-m2.7": "minimax",
-    "openrouter_moonshotai_kimi-k2.5": "kimi",
-    "openrouter_qwen_qwen3.6-plus": "qwen",
-    "anthropic_claude-opus-4-6": "opus46",
-    "anthropic_claude-opus-4-7": "opus47",
-    "anthropic_claude-sonnet-4-6": "sonnet46",
-    "openai_gpt-5.4": "gpt54",
-    "openai_gpt-5.2": "gpt52",
-    "google_gemini-3.1-pro-preview": "gemini",
-}
-
-
-def get_api_key() -> str:
-    k = os.environ.get("ANTHROPIC_API_KEY")
-    if k:
-        return k
-    cfg = Path.home() / ".openclaw" / "openclaw.json"
-    if cfg.exists():
-        try:
-            v = json.loads(cfg.read_text()).get("env", {}).get("ANTHROPIC_API_KEY")
-            if v:
-                return v
-        except Exception:
-            pass
-    raise RuntimeError("No ANTHROPIC_API_KEY found (set env var or openclaw.json)")
-
-
-def load_tasks() -> dict[str, dict]:
-    out = {}
-    for td in TASK_DIRS:
-        if not td.exists():
-            continue
-        for yf in sorted(td.glob("*.yaml")):
-            t = yaml.safe_load(yf.read_text())
-            if t and "id" in t:
-                out[t["id"]] = t
-    return out
-
-
-def is_judge_infra_fail(jr: dict) -> bool:
-    if not jr or not jr.get("enabled"):
-        return False
-    reason = (jr.get("reason") or "").lower()
-    if any(p in reason for p in FAILURE_PHRASES):
-        return True
-    if jr.get("error"):
-        return True
-    # Empty reason + score 0 is likely an unreported failure
-    if not reason.strip() and jr.get("score", 0) == 0:
-        return True
-    return False
-
-
-def render_transcript_excerpt(transcript: dict, max_chars: int = 4000) -> str:
-    msgs = transcript.get("messages", []) if transcript else []
-    parts = []
-    for m in msgs:
-        role = m.get("role", "?")
-        text = (m.get("text") or "").strip()
-        if text:
-            parts.append(f"[{role}] {text[:500]}")
-        for tc in (m.get("tool_calls") or []):
-            parts.append(f"[{role}/tool] {tc.get('name','?')}({json.dumps(tc.get('arguments',{}))[:120]})")
-        if m.get("tool_result_for"):
-            tr = (m.get("tool_result_content") or "")
-            parts.append(f"[tool_result] {tr[:300]}")
-    excerpt = "\n".join(parts)
-    if len(excerpt) > max_chars:
-        excerpt = excerpt[:max_chars] + "\n... (truncated)"
-    return excerpt
-
-
-def build_judge_prompt(task: dict, run: dict) -> str:
-    rubric = task.get("judge", {}).get("rubric", "").strip()
-    transcript_excerpt = render_transcript_excerpt(run.get("transcript", {}))
-    cr = run.get("completion_result", {})
-    comp_summary = (
-        f"score={cr.get('score',0):.3f}  "
-        f"passed={cr.get('passed_assertions',0)}/{cr.get('total_assertions',0)}"
-    )
-    failures = cr.get("failed_assertions", [])
-    comp_feedback = "\n".join(f"- {f}" for f in failures[:5]) if failures else "(none)"
-    return (
-        f"{rubric}\n\n"
-        f"=== Completion verifier summary ===\n{comp_summary}\n"
-        f"Failed assertions:\n{comp_feedback}\n\n"
-        f"=== Transcript excerpt ===\n{transcript_excerpt}\n"
-    )
-
-
-JSON_RE = re.compile(r"\{.*\}", re.DOTALL)
-
-
-def parse_judge_response(raw: str, threshold: float) -> dict:
-    try:
-        # Find the first balanced JSON object (json.raw_decode tolerates trailing text)
-        start = raw.find("{")
-        if start < 0:
-            raise ValueError("no JSON in response")
-        decoder = json.JSONDecoder()
-        obj, _end = decoder.raw_decode(raw[start:])
-        score = float(obj.get("score", 0))
-        confidence = float(obj.get("confidence", 0.5))
-        reason = str(obj.get("reason", ""))
-        return {
-            "enabled": True,
-            "score": round(max(0.0, min(1.0, score)), 4),
-            "confidence": round(max(0.0, min(1.0, confidence)), 4),
-            "reason": reason,
-            "rubric_hits": obj.get("rubric_hits") or [],
-            "rubric_misses": obj.get("rubric_misses") or [],
-            "passing_threshold": threshold,
-            "passed": score >= threshold,
-            "error": None,
-        }
-    except Exception as exc:
-        return {
-            "enabled": True, "score": 0.0, "confidence": 0.0,
-            "reason": f"parse failed: {exc}", "rubric_hits": [], "rubric_misses": [],
-            "passing_threshold": threshold, "passed": False, "error": str(exc),
-        }
-
-
-def combine_run_score(c: float, t: float, b: float, j: Optional[float], has_det: bool) -> float:
-    if j is None:
-        w = WEIGHTS_DETERMINISTIC
-        ws = w["completion"]*c + w["trajectory"]*t + w["behavior"]*b
-        return round(min(1.0, max(0.0, ws/sum(w.values()))), 4)
-    if has_det:
-        if c < DETERMINISTIC_FLOOR:
-            w = WEIGHTS_DETERMINISTIC
-            ws = w["completion"]*c + w["trajectory"]*t + w["behavior"]*b
-            return round(min(1.0, max(0.0, ws/sum(w.values()))), 4)
-        w = WEIGHTS_WITH_JUDGE
-        ws = w["completion"]*c + w["trajectory"]*t + w["behavior"]*b + w["judge"]*j
-        return round(min(1.0, max(0.0, ws)), 4)
-    w = WEIGHTS_SEMANTIC_ONLY
-    ws = w["completion"]*c + w["trajectory"]*t + w["behavior"]*b + w["judge"]*j
-    return round(min(1.0, max(0.0, ws)), 4)
-
-
-def main() -> None:
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--drift-dir", required=True, type=Path)
-    ap.add_argument("--archive-dir", required=True, type=Path)
-    ap.add_argument("--dry-run", action="store_true")
-    args = ap.parse_args()
-
-    if not args.archive_dir.exists():
-        print(f"Archive dir missing: {args.archive_dir}")
-        sys.exit(1)
-
-    tasks = load_tasks()
-    print(f"Loaded {len(tasks)} task definitions")
-
-    # Gather all affected runs: (cache_sub, task_id, run_path, run_data)
-    affected: list = []
-    for model_dir in sorted(args.archive_dir.iterdir()):
-        if not model_dir.is_dir():
-            continue
-        if model_dir.name not in CACHE_TO_LABEL:
-            continue
-        for task_dir in model_dir.iterdir():
-            if not task_dir.is_dir():
-                continue
-            for rf in sorted(task_dir.glob("run*.json")):
-                try:
-                    run = json.loads(rf.read_text())
-                except Exception:
-                    continue
-                if is_judge_infra_fail(run.get("judge_result", {})):
-                    affected.append((model_dir.name, task_dir.name, rf, run))
-
-    print(f"Found {len(affected)} runs with judge infra failures")
-    if args.dry_run:
-        from collections import Counter
-        by_model = Counter(a[0] for a in affected)
-        for m, n in by_model.most_common():
-            print(f"  {m}: {n}")
-        return
-    if not affected:
-        return
-
-    api_key = get_api_key()
-    client = anthropic.Anthropic(api_key=api_key)
-
-    # Re-judge each
-    succ = 0
-    fail = 0
-    for i, (cache_sub, task_id, rp, run) in enumerate(affected):
-        task = tasks.get(task_id)
-        if not task or not task.get("judge"):
-            continue
-        prompt = build_judge_prompt(task, run)
-        threshold = task["judge"].get("passing_threshold", 0.7)
-        print(f"[{i+1}/{len(affected)}] {cache_sub}/{task_id}/{rp.name} ... ", end="", flush=True)
-        try:
-            t0 = time.monotonic()
-            resp = client.messages.create(
-                model="claude-sonnet-4-6", max_tokens=1024,
-                messages=[{"role": "user", "content": prompt}],
-            )
-            raw = resp.content[0].text
-            dur_ms = int((time.monotonic() - t0) * 1000)
-            parsed = parse_judge_response(raw, threshold)
-            parsed["model"] = "anthropic/claude-sonnet-4-6"
-            parsed["duration_ms"] = dur_ms
-            parsed["token_usage"] = {
-                "input_tokens": resp.usage.input_tokens,
-                "output_tokens": resp.usage.output_tokens,
-            }
-            parsed["rejudged_at"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
-            run["judge_result"] = parsed
-            # Recompute run_score
-            cr = run.get("completion_result", {})
-            tr = run.get("trajectory_result", {})
-            br = run.get("behavior_result", {})
-            has_det = cr.get("total_assertions", 0) > 0
-            j = parsed["score"] if parsed["enabled"] and not parsed.get("error") else None
-            old_rs = run.get("run_score", 0)
-            new_rs = combine_run_score(cr.get("score", 0), tr.get("score", 0), br.get("score", 0), j, has_det)
-            run["run_score"] = new_rs
-            tmp = rp.with_suffix(".json.tmp")
-            tmp.write_text(json.dumps(run, indent=2))
-            tmp.replace(rp)
-            print(f"J={parsed['score']:.2f} ΔRS={new_rs - old_rs:+.3f}")
-            succ += 1
-        except Exception as exc:
-            print(f"ERROR: {exc}")
-            fail += 1
-
-    print(f"\nRe-judging complete: {succ} succeeded, {fail} failed")
-
-
-if __name__ == "__main__":
-    main()

scripts/setup_gbrain_runtime.sh

@@ -0,0 +1,136 @@
+#!/usr/bin/env bash
+# Prepare a lane-local GBrain install for OpenClaw benchmark runs.
+#
+# The image supplies /opt/gbrain and this script keeps secrets runtime-only:
+# keys are read from the lane's openclaw.json env block or existing process env,
+# never baked into Docker layers.
+set -Eeuo pipefail
+
+if [ "${CLAWBENCH_ENABLE_GBRAIN:-0}" != "1" ]; then
+  exit 0
+fi
+
+: "${HOME:?HOME is required}"
+
+GBRAIN_ROOT="${GBRAIN_ROOT:-/opt/gbrain}"
+if [ ! -d "$GBRAIN_ROOT" ]; then
+  echo "[gbrain] missing $GBRAIN_ROOT" >&2
+  exit 1
+fi
+
+export PATH="$GBRAIN_ROOT/bin:/usr/local/bun/bin:$PATH"
+export GBRAIN_ALLOW_SHELL_JOBS="${GBRAIN_ALLOW_SHELL_JOBS:-1}"
+
+STATE_DIR="${OPENCLAW_STATE_DIR:-$HOME/.openclaw}"
+CONFIG_PATH="${OPENCLAW_CONFIG_PATH:-$STATE_DIR/openclaw.json}"
+LOG_DIR="${CLAWBENCH_GBRAIN_LOG_DIR:-$STATE_DIR/logs}"
+mkdir -p "$HOME/.gbrain" "$LOG_DIR"
+LOG_PATH="$LOG_DIR/gbrain-runtime.log"
+
+if [ -f "$CONFIG_PATH" ]; then
+  eval "$(python3 - "$CONFIG_PATH" <<'PY'
+import json
+import os
+import shlex
+import sys
+
+config_path = sys.argv[1]
+try:
+    data = json.load(open(config_path, encoding="utf-8"))
+except Exception:
+    data = {}
+env = data.get("env") if isinstance(data, dict) else {}
+if not isinstance(env, dict):
+    env = {}
+for key in ("OPENAI_API_KEY", "ANTHROPIC_API_KEY"):
+    value = os.environ.get(key) or env.get(key)
+    if value:
+        print(f"export {key}={shlex.quote(str(value))}")
+PY
+)"
+
+  python3 - "$CONFIG_PATH" "$GBRAIN_ROOT" <<'PY'
+import json
+import sys
+
+config_path = sys.argv[1]
+gbrain_root = sys.argv[2]
+try:
+    with open(config_path, encoding="utf-8") as handle:
+        data = json.load(handle)
+except Exception:
+    data = {}
+if not isinstance(data, dict):
+    data = {}
+
+plugins = data.setdefault("plugins", {})
+if not isinstance(plugins, dict):
+    plugins = {}
+    data["plugins"] = plugins
+
+allow = plugins.get("allow")
+if not isinstance(allow, list):
+    allow = []
+plugins["allow"] = allow
+if "gbrain" not in allow:
+    allow.append("gbrain")
+
+entries = plugins.get("entries")
+if not isinstance(entries, dict):
+    entries = {}
+plugins["entries"] = entries
+entry = entries.get("gbrain")
+if not isinstance(entry, dict):
+    entry = {}
+entries["gbrain"] = entry
+entry["enabled"] = True
+
+load = plugins.get("load")
+if not isinstance(load, dict):
+    load = {}
+plugins["load"] = load
+paths = load.get("paths")
+if not isinstance(paths, list):
+    paths = []
+load["paths"] = paths
+if gbrain_root not in paths:
+    paths.append(gbrain_root)
+
+with open(config_path, "w", encoding="utf-8") as handle:
+    json.dump(data, handle, indent=2)
+    handle.write("\n")
+PY
+fi
+
+echo "[gbrain] preparing HOME=$HOME" > "$LOG_PATH"
+echo "[gbrain] version: $(gbrain --version 2>/dev/null || true)" >> "$LOG_PATH"
+echo "[gbrain] plugin path enabled in $CONFIG_PATH" >> "$LOG_PATH"
+
+if [ ! -f "$HOME/.gbrain/config.json" ]; then
+  gbrain init >> "$LOG_PATH" 2>&1
+else
+  gbrain apply-migrations --yes --non-interactive >> "$LOG_PATH" 2>&1 || true
+fi
+
+BRAIN_REPO="${GBRAIN_BRAIN_REPO:-$HOME/brain}"
+mkdir -p "$BRAIN_REPO"
+if [ "${CLAWBENCH_GBRAIN_SEED_SMOKE:-1}" = "1" ] && ! find "$BRAIN_REPO" -type f -name '*.md' -print -quit | grep -q .; then
+  cat > "$BRAIN_REPO/gbrain-smoke.md" <<'EOF'
+# GBrain smoke page
+
+This page verifies that the benchmark image can initialize, import, and query a
+lane-local GBrain database. It is intentionally generic and not task-specific.
+EOF
+fi
+
+if find "$BRAIN_REPO" -type f -name '*.md' -print -quit | grep -q .; then
+  gbrain import "$BRAIN_REPO" --no-embed >> "$LOG_PATH" 2>&1 || true
+  if [ -n "${OPENAI_API_KEY:-}" ]; then
+    gbrain embed --stale >> "$LOG_PATH" 2>&1 || true
+  else
+    echo "[gbrain] OPENAI_API_KEY not available; semantic embeddings skipped" >> "$LOG_PATH"
+  fi
+fi
+
+gbrain doctor --json >> "$LOG_PATH" 2>&1 || true
+echo "[gbrain] ready" >> "$LOG_PATH"

tasks-public/README.md

@@ -57,12 +57,10 @@ tasks-public/
 docker build -t clawbench .
 ```
 
-The repo `Dockerfile` pins an OpenClaw image digest so public Space
-builds do not silently drift. Override `OPENCLAW_IMAGE` only when you
-intend to measure a different platform build. Note that platform
-upgrades can shift scores (we observed +0.13 to +0.29 per model going
-from 4.9 → 4.15-beta.1) — when comparing two model runs, build them
-against the same OpenClaw release.
+The repo `Dockerfile` layers ClawBench on the configured OpenClaw base
+image. Platform upgrades can shift scores, so record the OpenClaw
+version for every published comparison and build both sides of a
+comparison against the same OpenClaw release.
 
 ## How to run Core v1
 
@@ -107,10 +105,8 @@ your ClawBench config. See MANIFEST.yaml for a programmatic list.
 - **OpenClaw platform version matters.** Upgrading from 4.9 → 4.15-beta.1
   shifted scores by +0.13 to +0.29 across models. Build both sides of
   any comparison from the same OpenClaw release.
-- **Judge scores** come from Claude Sonnet 4.6 via direct Anthropic
-  API (with a fallback from the gateway judge). Scores assume the
-  judge is working correctly; re-judging broken runs may be required
-  (see `scripts/rejudge_all.py` in the main repo).
+- **Judge scores** are advisory and depend on the configured judge model.
+  They are reported separately and cannot replace deterministic checks.
 
 ## What's NOT in Core v1
 
@@ -120,9 +116,9 @@ your ClawBench config. See MANIFEST.yaml for a programmatic list.
 - **9 noise tasks** (cross-model SNR < 0.5) — either broken verifiers
   or genuinely ambiguous prompts. Scheduled for redesign.
 - **3 ranking-breaker tasks** — tasks where the cross-model ordering
-  conflicts with the reference ranking (e.g. `t2-node-search-patch`,
-  `t5-contradictory-requirements`). Not broken per se; just
-  inconsistent with the headline.
+  conflicts with the reference ranking. Not broken per se; just
+  inconsistent with the headline. Their task IDs and contents remain
+  private with the rest of the holdout.
 
 Also missing entirely from Core v1:
 - **Tier 6 long-horizon (100+ turn) tasks** — planned for v2.

tasks-public/assets/t3_web_research_and_cite/serve.py

@@ -5,23 +5,13 @@ from __future__ import annotations
 import os
 from http.server import BaseHTTPRequestHandler, HTTPServer
 from pathlib import Path
-from urllib.parse import unquote, urlsplit
 
 ROOT = Path(__file__).parent / "articles"
-ARTICLES = {path.stem: path for path in ROOT.glob("*.html") if path.is_file()}
-
-
-def article_for_request_path(request_path: str) -> Path | None:
-    path = unquote(urlsplit(request_path).path)
-    if not path.startswith("/article/"):
-        return None
-    slug = path.removeprefix("/article/")
-    return ARTICLES.get(slug)
 
 
 class Handler(BaseHTTPRequestHandler):
     def do_GET(self) -> None:  # noqa: N802
-        path = unquote(urlsplit(self.path).path)
+        path = self.path.split("?")[0]
         if path == "/health":
             self.send_response(200)
             self.send_header("Content-Type", "application/json")
@@ -32,8 +22,9 @@ class Handler(BaseHTTPRequestHandler):
             self._index()
             return
         if path.startswith("/article/"):
-            article = article_for_request_path(self.path)
-            if article is not None:
+            slug = path.split("/", 2)[2]
+            article = ROOT / f"{slug}.html"
+            if article.exists():
                 self._html(article.read_bytes())
                 return
         self.send_response(404)
@@ -42,7 +33,8 @@ class Handler(BaseHTTPRequestHandler):
 
     def _index(self) -> None:
         items = []
-        for slug in sorted(ARTICLES):
+        for f in sorted(ROOT.glob("*.html")):
+            slug = f.stem
             items.append(f'<li><a href="/article/{slug}">{slug}</a></li>')
         body = (
             "<!doctype html><html><body>"

tests/test_blacksmith_setup.py

@@ -20,46 +20,6 @@ def test_testbox_workflow_hydrates_secrets_and_dotfiles():
     assert "CLAWBENCH_CODEX_AUTH_JSON" in workflow
 
 
-def test_crabbox_config_uses_actions_hydration():
-    config = Path(".crabbox.yaml").read_text(encoding="utf-8")
-
-    assert "profile: clawbench-check" in config
-    assert "provider: aws" in config
-    assert "workflow: .github/workflows/crabbox-hydrate.yml" in config
-    assert "job: hydrate" in config
-    assert "baseRef: main" in config
-    assert "- clawbench" in config
-    assert "- CLAWBENCH_*" in config
-    assert "- OPENCLAW_*" in config
-
-
-def test_crabbox_workflow_hydrates_secrets_dotfiles_and_ready_marker():
-    workflow = Path(".github/workflows/crabbox-hydrate.yml").read_text(encoding="utf-8")
-
-    assert "crabbox_id:" in workflow
-    assert "crabbox_runner_label:" in workflow
-    assert 'runs-on: [self-hosted, "${{ inputs.crabbox_runner_label }}"]' in workflow
-    assert "actions/setup-python@v5" in workflow
-    assert "python -m pip install -e ." in workflow
-    assert "scripts/ci-hydrate-testbox-env.sh" in workflow
-    assert "HF_TOKEN" in workflow
-    assert "OPENCLAW_CODEX_AUTH_JSON" in workflow
-    assert "CLAWBENCH_CODEX_AUTH_JSON" in workflow
-    assert "/usr/local/bin/clawbench-testbox-env" in workflow
-    assert "$HOME/.crabbox/actions/${{ inputs.crabbox_id }}.env" in workflow
-    assert "crabbox_keep_alive_minutes" in workflow
-
-
-def test_crabbox_skill_documents_clawbench_flow():
-    skill = Path(".agents/skills/crabbox/SKILL.md").read_text(encoding="utf-8")
-
-    assert "openclaw/crabbox" in skill
-    assert ".crabbox.yaml" in skill
-    assert "crabbox actions hydrate" in skill
-    assert "clawbench-testbox-env" in skill
-    assert ".github/workflows/crabbox-hydrate.yml" in skill
-
-
 def test_testbox_helper_sources_hydrated_profile():
     script = Path("scripts/ci-hydrate-testbox-env.sh").read_text(encoding="utf-8")
 

tests/test_cli.py

@@ -1,51 +0,0 @@
-from click.testing import CliRunner
-
-from clawbench.cli import SCENARIO_CHOICES, cli
-from clawbench.schemas import ScenarioDomain
-
-
-def test_cli_scenario_choices_track_schema_enum():
-    assert SCENARIO_CHOICES == [scenario.value for scenario in ScenarioDomain]
-
-
-def test_run_command_forwards_judge_score_gate(monkeypatch, tmp_path):
-    captured: dict[str, object] = {}
-
-    class FakeResult:
-        submission_id = "submission-1"
-
-        def model_dump(self):
-            return {"submission_id": self.submission_id}
-
-    class FakeHarness:
-        def __init__(self, **kwargs):
-            captured.update(kwargs)
-
-        async def run(self):
-            return FakeResult()
-
-    monkeypatch.setattr("clawbench.cli.BenchmarkHarness", FakeHarness)
-
-    output = tmp_path / "result.json"
-    result = CliRunner().invoke(
-        cli,
-        [
-            "run",
-            "--model",
-            "anthropic/claude-sonnet-4-6",
-            "--judge-model",
-            "judge-model",
-            "--judge-affects-score",
-            "--runs",
-            "1",
-            "--task",
-            "t1-bugfix-discount",
-            "--output",
-            str(output),
-        ],
-    )
-
-    assert result.exit_code == 0, result.output
-    assert captured["judge_model"] == "judge-model"
-    assert captured["judge_affects_score"] is True
-    assert output.read_text(encoding="utf-8")

tests/test_client.py

@@ -1,7 +1,6 @@
 from __future__ import annotations
 
 import asyncio
-import json
 
 import pytest
 from websockets.datastructures import Headers
@@ -38,6 +37,42 @@ def test_gateway_config_invalid_env_falls_back_to_default(monkeypatch, caplog, r
     assert any("CLAWBENCH_CONNECT_TIMEOUT" in r.getMessage() for r in caplog.records)
 
 
+@pytest.mark.asyncio
+async def test_gateway_client_disables_websocket_keepalive_for_long_rpc(
+    monkeypatch: pytest.MonkeyPatch,
+):
+    connect_kwargs: dict[str, object] = {}
+
+    class FakeWebSocket:
+        async def close(self) -> None:
+            return None
+
+    async def fake_connect(*args, **kwargs):
+        connect_kwargs.update(kwargs)
+        return FakeWebSocket()
+
+    async def fake_wait_event(self, event_name: str, *, timeout: float):
+        return {"payload": {"nonce": ""}}
+
+    async def fake_rpc(self, method: str, params=None, **kwargs):
+        return {"payload": {"type": "hello-ok", "protocol": 3}}
+
+    async def fake_listener(self):
+        await asyncio.sleep(60)
+
+    monkeypatch.setattr("clawbench.client.websockets.connect", fake_connect)
+    monkeypatch.setattr(GatewayClient, "_wait_event", fake_wait_event)
+    monkeypatch.setattr(GatewayClient, "_rpc", fake_rpc)
+    monkeypatch.setattr(GatewayClient, "_listener", fake_listener)
+
+    client = GatewayClient(GatewayConfig(connect_timeout=2))
+    await client.connect()
+    await client.close()
+
+    assert connect_kwargs["ping_interval"] is None
+    assert connect_kwargs["ping_timeout"] is None
+
+
 def test_tool_results_are_correlated_back_to_tool_calls():
     tool_message = _parse_single_message(
         {
@@ -195,39 +230,6 @@ async def test_send_and_wait_collects_messages_that_arrive_after_final_state():
     assert [message.text for message in transcript.assistant_messages] == ["Late but valid."]
 
 
-@pytest.mark.asyncio
-async def test_rpc_send_failure_cleans_pending_request():
-    class FailingWebSocket:
-        async def send(self, payload: str) -> None:  # noqa: ARG002
-            raise ConnectionError("socket closed")
-
-    client = GatewayClient(GatewayConfig(request_timeout=0.01))
-    client._ws = FailingWebSocket()  # type: ignore[assignment]
-
-    with pytest.raises(ConnectionError, match="socket closed"):
-        await client._rpc("sessions.create", {"model": "test-model"})
-
-    assert client._pending == {}
-
-
-@pytest.mark.asyncio
-async def test_rpc_timeout_cleans_pending_request():
-    sent_frames: list[dict[str, object]] = []
-
-    class SilentWebSocket:
-        async def send(self, payload: str) -> None:
-            sent_frames.append(json.loads(payload))
-
-    client = GatewayClient(GatewayConfig(request_timeout=0.01))
-    client._ws = SilentWebSocket()  # type: ignore[assignment]
-
-    with pytest.raises(TimeoutError, match="RPC sessions.create timed out"):
-        await client._rpc("sessions.create", {"model": "test-model"})
-
-    assert sent_frames[0]["method"] == "sessions.create"
-    assert client._pending == {}
-
-
 @pytest.mark.asyncio
 async def test_send_and_wait_passes_gateway_timeout_and_waits_for_run():
     client = GatewayClient(GatewayConfig(request_timeout=1))

tests/test_dynamics.py

@@ -2,12 +2,19 @@
 
 from __future__ import annotations
 
+import math
 
 import numpy as np
 import pytest
 
 from clawbench.dynamics import (
+    TOOL_FAMILIES,
+    Dynamics,
     Regime,
+    Sensitivity,
+    SurvivalPoint,
+    StratumStats,
+    StratifiedAssessment,
     _classify_tool,
     _cosine_dist,
     _entropy,
@@ -18,6 +25,7 @@ from clawbench.dynamics import (
     compute_sensitivity,
     find_event_step,
     kaplan_meier,
+    stratify_by_regime,
     stratify_by_tier,
 )
 from clawbench.schemas import (

tests/test_e2e_significance.py

@@ -25,6 +25,7 @@ If this test passes, the framework is meaningful — not just functional.
 
 from __future__ import annotations
 
+import json
 import random
 import statistics
 import sys
@@ -32,7 +33,7 @@ from pathlib import Path
 
 sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
 
-from clawbench.diagnostic import build_diagnostic
+from clawbench.diagnostic import build_diagnostic, submit_run
 from clawbench.factor_analysis import analyze
 from clawbench.prediction import HistoricalDatabase, HistoricalRun, predict_profile
 from clawbench.profile import (
@@ -337,6 +338,9 @@ def test_fanova_recovers_seeded_effects():
     report = analyze(db, top_k_interactions=10)
     print(f"  factor analysis on {report.n_runs} runs, total variance = {report.total_variance:.4f}")
 
+    # Build a quick lookup of feature → importance
+    me_lookup = {m.feature: m for m in report.main_effects}
+
     # The framework should rediscover that memory and browser are the
     # strongest main effects we seeded.
     seeded_strong = [

tests/test_environment.py

@@ -2,19 +2,8 @@ from pathlib import Path
 
 import pytest
 
-from clawbench.environment import run_execution_check, verify_completion
-from clawbench.schemas import (
-    CompletionSpec,
-    CronState,
-    ExecutionCheck,
-    FileState,
-    GatewayAssertion,
-    MemoryState,
-    SessionState,
-    ToolCall,
-    Transcript,
-    TranscriptMessage,
-)
+from clawbench.environment import verify_completion
+from clawbench.schemas import CompletionSpec, MemoryState, ToolCall, Transcript, TranscriptMessage
 
 
 class MemoryFallbackClient:
@@ -33,30 +22,6 @@ class MemoryFallbackClient:
         return {"file": {"content": ""}}
 
 
-class CompletionClient:
-    async def _rpc(self, method: str, params=None):  # noqa: ANN001
-        if method == "sessions.resolve":
-            return {"payload": {"model": "anthropic/claude-sonnet-4-6"}}
-        if method == "cron.list":
-            return {"payload": {"jobs": [{"description": "nightly cleanup"}]}}
-        if method == "tools.inventory":
-            return {
-                "payload": {
-                    "groups": [
-                        {
-                            "tools": [
-                                {
-                                    "id": "browser",
-                                    "status": "available",
-                                }
-                            ]
-                        }
-                    ]
-                }
-            }
-        raise AssertionError(f"Unexpected RPC: {method} {params}")
-
-
 @pytest.mark.asyncio
 async def test_memory_completion_falls_back_to_agent_memory_files(tmp_path: Path):
     completion = CompletionSpec(
@@ -80,123 +45,6 @@ async def test_memory_completion_falls_back_to_agent_memory_files(tmp_path: Path
     assert result.score == 1.0
 
 
-@pytest.mark.asyncio
-async def test_verify_completion_scores_mixed_successful_assertions(tmp_path: Path):
-    report = tmp_path / "report.txt"
-    report.write_text("status: green\nowner: benchmark\n", encoding="utf-8")
-    completion = CompletionSpec(
-        files=[
-            FileState(
-                path="report.txt",
-                content_contains=["green"],
-                content_not_contains=["red"],
-                content_matches=r"owner:\s+benchmark",
-                min_size_bytes=10,
-            )
-        ],
-        session=SessionState(model_should_be="claude-sonnet"),
-        cron=[CronState(description_contains="cleanup")],
-        gateway_assertions=[
-            GatewayAssertion(
-                method="tools.inventory",
-                assert_path="$.groups[0].tools[0].id",
-                assert_equals="browser",
-            ),
-            GatewayAssertion(
-                method="tools.inventory",
-                assert_path="$.groups[0].tools[0].status",
-                assert_contains="avail",
-            ),
-        ],
-    )
-
-    result = await verify_completion(
-        completion,
-        workspace=tmp_path,
-        client=CompletionClient(),  # type: ignore[arg-type]
-        session_key="session-test",
-        runtime_values={},
-    )
-
-    assert result.total_assertions == 5
-    assert result.passed_assertions == 5
-    assert result.failed_assertions == []
-    assert result.score == 1.0
-
-
-@pytest.mark.asyncio
-async def test_file_completion_rejects_paths_outside_workspace(tmp_path: Path):
-    outside = tmp_path.parent / "outside.txt"
-    outside.write_text("secret", encoding="utf-8")
-    completion = CompletionSpec(files=[FileState(path="../outside.txt")])
-
-    result = await verify_completion(
-        completion,
-        workspace=tmp_path,
-        client=MemoryFallbackClient(),  # type: ignore[arg-type]
-        session_key="session-test",
-        runtime_values={},
-    )
-
-    assert result.score == 0.0
-    assert "escapes workspace" in result.failed_assertions[0]
-
-
-@pytest.mark.asyncio
-async def test_execution_check_supports_cwd_env_and_expected_json_file(tmp_path: Path):
-    expected = tmp_path / "expected.json"
-    expected.write_text('{"status": "ok"}', encoding="utf-8")
-    workdir = tmp_path / "subdir"
-    workdir.mkdir()
-
-    result = await run_execution_check(
-        ExecutionCheck(
-            name="json-check",
-            command='python -c "import json, os; print(json.dumps({\'status\': os.environ[\'CHECK_STATUS\']}))"',
-            cwd="subdir",
-            env={"CHECK_STATUS": "ok"},
-            expected_json_file="expected.json",
-        ),
-        workspace=tmp_path,
-        runtime_values={},
-    )
-
-    assert result.passed is True
-    assert result.reason == "OK"
-
-
-@pytest.mark.asyncio
-async def test_execution_check_rejects_cwd_outside_workspace(tmp_path: Path):
-    result = await run_execution_check(
-        ExecutionCheck(
-            name="unsafe-cwd",
-            command="true",
-            cwd="../outside",
-        ),
-        workspace=tmp_path,
-        runtime_values={},
-    )
-
-    assert result.passed is False
-    assert "escapes workspace" in result.reason
-
-
-@pytest.mark.asyncio
-async def test_execution_check_rejects_expected_file_outside_workspace(tmp_path: Path):
-    result = await run_execution_check(
-        ExecutionCheck(
-            name="unsafe-expected",
-            command="printf secret",
-            expected_stdout_file="../outside.txt",
-        ),
-        workspace=tmp_path,
-        runtime_values={},
-    )
-
-    assert result.passed is False
-    assert "escapes workspace" in result.reason
-
-
 @pytest.mark.asyncio
 async def test_memory_completion_falls_back_to_transcript_when_memory_rpc_is_unavailable(tmp_path: Path):
     completion = CompletionSpec(

tests/test_environment_files.py

@@ -1,98 +0,0 @@
-from pathlib import Path
-
-import pytest
-
-from clawbench.environment_files import run_execution_check, verify_file_state
-from clawbench.schemas import ExecutionCheck, FileState
-
-
-def test_verify_file_state_rejects_paths_outside_workspace(tmp_path: Path):
-    outside = tmp_path.parent / "outside.txt"
-    outside.write_text("secret", encoding="utf-8")
-
-    ok, reason = verify_file_state(
-        FileState(path="../outside.txt"),
-        workspace=tmp_path,
-        runtime_values={},
-    )
-
-    assert ok is False
-    assert "escapes workspace" in reason
-
-
-@pytest.mark.asyncio
-async def test_execution_check_supports_cwd_env_and_expected_json_file(tmp_path: Path):
-    expected = tmp_path / "expected.json"
-    expected.write_text('{"status": "ok"}', encoding="utf-8")
-    workdir = tmp_path / "subdir"
-    workdir.mkdir()
-
-    result = await run_execution_check(
-        ExecutionCheck(
-            name="json-check",
-            command=(
-                "python -c \"import json, os; "
-                "print(json.dumps({'status': os.environ['CHECK_STATUS']}))\""
-            ),
-            cwd="subdir",
-            env={"CHECK_STATUS": "ok"},
-            expected_json_file="expected.json",
-        ),
-        workspace=tmp_path,
-        runtime_values={},
-    )
-
-    assert result.passed is True
-    assert result.reason == "OK"
-
-
-@pytest.mark.asyncio
-async def test_execution_check_rejects_cwd_outside_workspace(tmp_path: Path):
-    result = await run_execution_check(
-        ExecutionCheck(
-            name="unsafe-cwd",
-            command="true",
-            cwd="../outside",
-        ),
-        workspace=tmp_path,
-        runtime_values={},
-    )
-
-    assert result.passed is False
-    assert "escapes workspace" in result.reason
-
-
-@pytest.mark.asyncio
-async def test_execution_check_rejects_expected_stdout_file_outside_workspace(
-    tmp_path: Path,
-):
-    result = await run_execution_check(
-        ExecutionCheck(
-            name="unsafe-expected-stdout",
-            command="printf secret",
-            expected_stdout_file="../outside.txt",
-        ),
-        workspace=tmp_path,
-        runtime_values={},
-    )
-
-    assert result.passed is False
-    assert "escapes workspace" in result.reason
-
-
-@pytest.mark.asyncio
-async def test_execution_check_rejects_expected_json_file_outside_workspace(
-    tmp_path: Path,
-):
-    result = await run_execution_check(
-        ExecutionCheck(
-            name="unsafe-expected-json",
-            command="printf '{}'",
-            expected_json_file="../outside.json",
-        ),
-        workspace=tmp_path,
-        runtime_values={},
-    )
-
-    assert result.passed is False
-    assert "escapes workspace" in result.reason

tests/test_harness.py

@@ -3,8 +3,24 @@ from pathlib import Path
 import pytest
 
 from clawbench.client import GatewayConfig
+from clawbench.adapters.base import AdapterContext, AgentAdapter, PhaseResult, StateQueryResult
+from clawbench.canonical import AdapterCapability, CanonicalPhase, StateQuery
 from clawbench.harness import BenchmarkHarness
-from clawbench.schemas import CompletionResult, JudgeResult, TaskRunResult
+from clawbench.schemas import (
+    CompletionResult,
+    CompletionSpec,
+    FileState,
+    JudgeExpectations,
+    JudgeResult,
+    SimulatedUser,
+    TaskDefinition,
+    TaskFamily,
+    TaskRunResult,
+    Tier,
+    Transcript,
+    TranscriptMessage,
+    UserTurn,
+)
 from clawbench.tasks import load_all_tasks
 
 
@@ -118,7 +134,13 @@ def test_aggregate_reports_advisory_judge_metrics():
 
 
 def test_compose_result_from_task_stats_supports_parallel_environment_metadata():
-    task = next(task for task in load_all_tasks() if task.id == "t1-bugfix-discount")
+    task = next(task for task in load_all_tasks() if task.id == "t1-bugfix-discount").model_copy(deep=True)
+    task.category = "software_engineering"
+    task.domain = "devtools"
+    task.functionality = ["bugfix", "regression_repair", "test_verification"]
+    task.trace_distribution = ["read_heavy", "edit_heavy", "execute_heavy", "recovery_heavy"]
+    task.tool_surface = ["filesystem", "shell"]
+    task.risk_tags = ["code_change"]
     harness = BenchmarkHarness(
         gateway_config=GatewayConfig(),
         model="test-model",
@@ -163,59 +185,29 @@ def test_compose_result_from_task_stats_supports_parallel_environment_metadata()
     assert merged_result.environment["parallel_lanes"] == 2
     assert merged_result.environment["requested_parallel_lanes"] == 3
     assert merged_result.environment["browser_tasks_serialized"] is False
+    assert merged_result.environment["dimension_coverage"] == {
+        "category": 1,
+        "domain": 1,
+        "functionality": 3,
+        "trace_distribution": 4,
+        "tool_surface": 2,
+        "risk_tag": 1,
+    }
+    assert merged_result.task_results[0].category == "software_engineering"
+    assert merged_result.task_results[0].domain == "devtools"
 
-
-def test_run_cache_path_includes_scoring_inputs(tmp_path: Path):
-    task = next(task for task in load_all_tasks() if task.id == "t1-bugfix-discount")
-    base = BenchmarkHarness(
-        gateway_config=GatewayConfig(),
-        model="test/model",
-        task_ids=[task.id],
-        prompt_variant="clear",
-        judge_model="judge-a",
-        randomize_order=False,
-    )
-    same = BenchmarkHarness(
-        gateway_config=GatewayConfig(),
-        model="test/model",
-        task_ids=[task.id],
-        prompt_variant="clear",
-        judge_model="judge-a",
-        randomize_order=False,
-    )
-    different_judge = BenchmarkHarness(
-        gateway_config=GatewayConfig(),
-        model="test/model",
-        task_ids=[task.id],
-        prompt_variant="clear",
-        judge_model="judge-b",
-        randomize_order=False,
-    )
-    different_judge_gate = BenchmarkHarness(
-        gateway_config=GatewayConfig(),
-        model="test/model",
-        task_ids=[task.id],
-        prompt_variant="clear",
-        judge_model="judge-a",
-        judge_affects_score=True,
-        randomize_order=False,
-    )
-    different_prompt = BenchmarkHarness(
-        gateway_config=GatewayConfig(),
-        model="test/model",
-        task_ids=[task.id],
-        prompt_variant="ambiguous",
-        judge_model="judge-a",
-        randomize_order=False,
+    category = {item.value: item for item in merged_result.category_results}
+    assert category["software_engineering"].task_ids == [task.id]
+    assert category["software_engineering"].weighted_score == pytest.approx(
+        base_result.overall_weighted_query_score
     )
 
-    base_path = base._run_cache_path(tmp_path, task, 0)
-
-    assert "v2-" in str(base_path)
-    assert base_path == same._run_cache_path(tmp_path, task, 0)
-    assert base_path != different_judge._run_cache_path(tmp_path, task, 0)
-    assert base_path != different_judge_gate._run_cache_path(tmp_path, task, 0)
-    assert base_path != different_prompt._run_cache_path(tmp_path, task, 0)
+    functionality_values = {item.value for item in merged_result.functionality_results}
+    assert {"bugfix", "regression_repair", "test_verification"}.issubset(functionality_values)
+    trace_values = {item.value for item in merged_result.trace_distribution_results}
+    assert {"read_heavy", "edit_heavy", "execute_heavy", "recovery_heavy"}.issubset(trace_values)
+    assert "category" in merged_result.dimension_results
+    assert merged_result.dimension_results["category"] == merged_result.category_results
 
 
 @pytest.mark.asyncio
@@ -259,7 +251,7 @@ async def test_run_rejects_registered_but_unwired_adapter(monkeypatch):
     harness = BenchmarkHarness(
         gateway_config=GatewayConfig(),
         model="test-model",
-        adapter="hermes",
+        adapter="codex",
         runs_per_task=1,
         randomize_order=False,
         print_report=False,
@@ -268,3 +260,182 @@ async def test_run_rejects_registered_but_unwired_adapter(monkeypatch):
 
     with pytest.raises(ValueError, match="not yet wired"):
         await harness.run()
+
+
+def _files_only_definition(judge: JudgeExpectations | None = None) -> TaskDefinition:
+    return TaskDefinition(
+        id="adapter-files-only",
+        name="Adapter files only",
+        tier=Tier.TIER1,
+        family=TaskFamily.CODING,
+        surface="coding",
+        user=SimulatedUser(
+            max_turns=1,
+            turns=[UserTurn(message="Create answer.txt")],
+        ),
+        completion=CompletionSpec(
+            files=[FileState(path="answer.txt", exists=True, content_contains=["done"])],
+        ),
+        judge=judge,
+    )
+
+
+class FakeAgentAdapter(AgentAdapter):
+    name = "hermes"
+    capabilities = {AdapterCapability.FILES, AdapterCapability.EXECUTION}
+
+    async def setup(self, ctx: AdapterContext) -> None:
+        return None
+
+    async def run_phase(self, phase: CanonicalPhase, ctx: AdapterContext) -> PhaseResult:
+        (ctx.workspace / "answer.txt").write_text("done\n", encoding="utf-8")
+        message = TranscriptMessage(role="assistant", text="Created answer.txt and verified it.")
+        ctx.transcript.messages.append(message)
+        return PhaseResult(messages=[message], completed_normally=True)
+
+    async def verify_state_query(self, query: StateQuery, ctx: AdapterContext) -> StateQueryResult:
+        return StateQueryResult(ok=False, capability_missing=True)
+
+    async def teardown(self, ctx: AdapterContext) -> None:
+        return None
+
+
+@pytest.mark.asyncio
+async def test_hermes_adapter_runs_through_scoring_harness(monkeypatch, tmp_path: Path):
+    task = _files_only_definition()
+    monkeypatch.setattr("clawbench.harness.load_all_tasks", lambda **_: [task])
+    monkeypatch.setattr("clawbench.harness.get_adapter", lambda name: FakeAgentAdapter)
+    monkeypatch.setenv("OPENCLAW_STATE_DIR", str(tmp_path))
+    monkeypatch.setenv("CLAWBENCH_RUN_CACHE_DIR", "")
+
+    harness = BenchmarkHarness(
+        gateway_config=GatewayConfig(),
+        model="openai/gpt-5.5",
+        adapter="hermes",
+        runs_per_task=1,
+        randomize_order=False,
+        print_report=False,
+        quiet=True,
+    )
+
+    result = await harness.run()
+    run = harness.last_task_runs[task.id][0]
+
+    assert result.environment["adapter"] == "hermes"
+    assert result.environment["executable_adapters"] == ["hermes", "openclaw"]
+    assert run.error is None
+    assert run.completion_result.score == 1.0
+    assert run.delivery_outcome.value == "pass"
+
+
+@pytest.mark.asyncio
+async def test_openclaw_uses_shared_adapter_scoring_path(monkeypatch, tmp_path: Path):
+    task = _files_only_definition()
+    monkeypatch.setattr("clawbench.harness.load_all_tasks", lambda **_: [task])
+    monkeypatch.setattr("clawbench.harness.get_adapter", lambda name: FakeAgentAdapter)
+    monkeypatch.setenv("OPENCLAW_STATE_DIR", str(tmp_path))
+    monkeypatch.setenv("CLAWBENCH_RUN_CACHE_DIR", "")
+
+    harness = BenchmarkHarness(
+        gateway_config=GatewayConfig(),
+        model="openai/gpt-5.5",
+        adapter="openclaw",
+        runs_per_task=1,
+        randomize_order=False,
+        print_report=False,
+        quiet=True,
+    )
+
+    result = await harness.run()
+    run = harness.last_task_runs[task.id][0]
+
+    assert result.environment["adapter"] == "openclaw"
+    assert run.error is None
+    assert run.completion_result.score == 1.0
+    assert run.delivery_outcome.value == "pass"
+
+
+@pytest.mark.asyncio
+async def test_adapter_scoring_uses_advisory_judge(monkeypatch, tmp_path: Path):
+    task = _files_only_definition(
+        JudgeExpectations(
+            rubric="Reward the answer when it is concise.",
+            artifact_paths=["answer.txt"],
+            passing_threshold=0.4,
+        )
+    )
+    monkeypatch.setattr("clawbench.harness.load_all_tasks", lambda **_: [task])
+    monkeypatch.setattr("clawbench.harness.get_adapter", lambda name: FakeAgentAdapter)
+    monkeypatch.setenv("OPENCLAW_STATE_DIR", str(tmp_path))
+    monkeypatch.setenv("CLAWBENCH_RUN_CACHE_DIR", "")
+
+    class FakeJudgeGateway:
+        async def __aenter__(self):
+            return self
+
+        async def __aexit__(self, *exc):
+            return None
+
+        async def create_session(self, *, model: str, label: str) -> str:
+            assert model == "judge-model"
+            assert label.startswith("clawbench-judge-")
+            return "judge-session"
+
+        async def subscribe(self, session_key: str) -> None:
+            assert session_key == "judge-session"
+
+        async def send_and_wait(self, session_key: str, message: str):
+            assert session_key == "judge-session"
+            assert "done" in message
+            return Transcript(
+                messages=[
+                    TranscriptMessage(
+                        role="assistant",
+                        text='{"score": 0.5, "confidence": 0.8, "reason": "OK", "rubric_hits": [], "rubric_misses": []}',
+                    )
+                ]
+            )
+
+        async def delete_session(self, session_key: str) -> None:
+            assert session_key == "judge-session"
+
+    monkeypatch.setattr("clawbench.harness.GatewayClient", lambda config: FakeJudgeGateway())
+
+    harness = BenchmarkHarness(
+        gateway_config=GatewayConfig(),
+        model="openai/gpt-5.5",
+        adapter="hermes",
+        judge_model="judge-model",
+        runs_per_task=1,
+        randomize_order=False,
+        print_report=False,
+        quiet=True,
+    )
+
+    result = await harness.run()
+    run = harness.last_task_runs[task.id][0]
+
+    assert run.judge_result.enabled is True
+    assert run.judge_result.score == pytest.approx(0.5)
+    assert run.run_score == pytest.approx(0.95)
+    assert result.overall_judge_score == pytest.approx(0.5)
+
+
+@pytest.mark.asyncio
+async def test_hermes_adapter_filters_incompatible_tasks(monkeypatch):
+    task = next(task for task in load_all_tasks() if task.id == "t4-memory-recall-continuation")
+    monkeypatch.setattr("clawbench.harness.load_all_tasks", lambda **_: [task])
+    monkeypatch.setattr("clawbench.harness.get_adapter", lambda name: FakeAgentAdapter)
+
+    harness = BenchmarkHarness(
+        gateway_config=GatewayConfig(),
+        model="openai/gpt-5.5",
+        adapter="hermes",
+        runs_per_task=1,
+        randomize_order=False,
+        print_report=False,
+        quiet=True,
+    )
+
+    with pytest.raises(ValueError, match="No selected tasks are compatible"):
+        await harness.run()

tests/test_integration_checks.py

@@ -4,6 +4,7 @@ from pathlib import Path
 
 import pytest
 
+import clawbench.tasks as tasks_module
 from clawbench.client import GatewayConfig
 from clawbench.environment import verify_completion
 from clawbench.harness import BenchmarkHarness
@@ -12,14 +13,8 @@ from clawbench.services import build_runtime_values, start_background_services,
 from clawbench.tasks import load_all_tasks
 from clawbench.trajectory import evaluate_trajectory
 
-# The task set is moving to a private holdout; the public repo will ship a
-# different task set soon. Until then, skip integration tests that need
-# specific task ids when the tasks directory isn't present.
-_TASKS_DIR = Path(__file__).resolve().parent.parent / "tasks"
-pytestmark = pytest.mark.skipif(
-    not _TASKS_DIR.exists(),
-    reason="tasks/ directory not present (private holdout — public set TBD)",
-)
+PUBLIC_TASKS_DIR = Path(__file__).resolve().parent.parent / "tasks-public"
+tasks_module.TASKS_DIR = PUBLIC_TASKS_DIR
 
 
 class DummyClient:
@@ -28,8 +23,13 @@ class DummyClient:
 
 
 def _prepare_workspace(task_id: str, tmp_path: Path) -> tuple[Path, object]:
-    task = next(task for task in load_all_tasks() if task.id == task_id)
-    harness = BenchmarkHarness(gateway_config=GatewayConfig(), model="test-model", randomize_order=False)
+    task = next(task for task in load_all_tasks(tasks_dir=PUBLIC_TASKS_DIR) if task.id == task_id)
+    harness = BenchmarkHarness(
+        gateway_config=GatewayConfig(),
+        model="test-model",
+        randomize_order=False,
+        tasks_dir=PUBLIC_TASKS_DIR,
+    )
     workspace = tmp_path / task_id
     workspace.mkdir(parents=True, exist_ok=True)
     harness._setup_workspace(task, workspace)
@@ -57,50 +57,6 @@ async def test_python_completion_check_passes_after_fix(tmp_path: Path):
 
     assert result.score == 1.0
 
-
-@pytest.mark.asyncio
-async def test_node_completion_check_passes_after_fix(tmp_path: Path):
-    workspace, task = _prepare_workspace("t2-node-search-patch", tmp_path)
-    # After hardening, render.js also exports emptyNote() with a legitimate
-    # empty body. The scoped fix only patches normalizeNote's body and must
-    # leave emptyNote alone.
-    (workspace / "src" / "render.js").write_text(
-        "function normalizeNote(note) {\n"
-        "  return {\n"
-        "    title: note.title.trim(),\n"
-        "    body: note.body.trim(),\n"
-        "  };\n"
-        "}\n\n"
-        "function emptyNote() {\n"
-        "  return {\n"
-        "    title: \"\",\n"
-        "    body: \"\",\n"
-        "  };\n"
-        "}\n\n"
-        "module.exports = { normalizeNote, emptyNote };\n",
-        encoding="utf-8",
-    )
-    (workspace / "src" / "search.js").write_text(
-        "function filterNotes(notes, query) {\n"
-        "  const needle = query.trim().toLowerCase();\n"
-        "  return notes.filter((note) => note.title.toLowerCase().includes(needle) || note.body.toLowerCase().includes(needle));\n"
-        "}\n\n"
-        "module.exports = { filterNotes };\n",
-        encoding="utf-8",
-    )
-
-    runtime_values = build_runtime_values(workspace=workspace, repo_root=Path.cwd())
-    result = await verify_completion(
-        task.completion,
-        workspace=workspace,
-        client=DummyClient(),  # type: ignore[arg-type]
-        session_key="",
-        runtime_values=runtime_values,
-    )
-
-    assert result.score == 1.0
-
-
 def _playwright_available() -> bool:
     if not shutil.which("node"):
         return False
@@ -156,7 +112,10 @@ async def test_browser_completion_check_passes_after_fix(tmp_path: Path):
 
 
 def test_memory_task_trajectory_requires_memory_tool():
-    task = next(task for task in load_all_tasks() if task.id == "t4-memory-recall-continuation")
+    task = next(
+        task for task in load_all_tasks(tasks_dir=PUBLIC_TASKS_DIR)
+        if task.id == "t4-memory-recall-continuation"
+    )
     transcript = Transcript(
         messages=[
             TranscriptMessage(role="assistant", tool_calls=[ToolCall(name="exec", input={"command": "cat docs/release_notes.md"}, success=True)]),
@@ -172,7 +131,10 @@ def test_memory_task_trajectory_requires_memory_tool():
 
 
 def test_delegation_task_trajectory_requires_delegate_family():
-    task = next(task for task in load_all_tasks() if task.id == "t4-delegation-repair")
+    task = next(
+        task for task in load_all_tasks(tasks_dir=PUBLIC_TASKS_DIR)
+        if task.id == "t4-delegation-repair"
+    )
     transcript = Transcript(
         messages=[
             TranscriptMessage(role="assistant", tool_calls=[ToolCall(name="exec", input={"command": "rg billing ."}, success=True)]),