openclaw/docs
openclaw/docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore(sync): mirror docs from openclaw/openclaw@dd1a94f089

Browse Source
This commit is contained in:
openclaw-docs-sync[bot] 2026-04-27 17:43:05 +00:00
parent 068672d2fb
commit 3982ff9da4
7 changed files with 333 additions and 202 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.openclaw-sync/source.json
View File

@ -1,5 +1,5 @@
{
"repository": "openclaw/openclaw",
"sha": "e3ad82d86df0ecebbddf17d98f22921a60d149f6",
"syncedAt": "2026-04-27T17:25:18.298Z"
"sha": "dd1a94f0893bafcffac875cf3e66cd9865a25c78",
"syncedAt": "2026-04-27T17:41:21.179Z"
}

12
docs/.i18n/glossary.zh-CN.json
View File

@ -503,6 +503,18 @@
"source": "QA Refactor",
"target": "QA 重构"
},
{
"source": "Matrix QA",
"target": "Matrix QA"
},
{
"source": "QA overview",
"target": "QA overview"
},
{
"source": "QA channel",
"target": "QA channel"
},
{
"source": "Rich Output Protocol",
"target": "富输出协议"

88
docs/channels/qa-channel.md
View File

@ -7,27 +7,16 @@ read_when:
- You are iterating on end-to-end QA automation
---
`qa-channel` is a bundled synthetic message transport for automated OpenClaw QA.
`qa-channel` is a bundled synthetic message transport for automated OpenClaw QA. It is not a production channel — it exists to exercise the same channel plugin boundary used by real transports while keeping state deterministic and fully inspectable.
It is not a production channel. It exists to exercise the same channel plugin
boundary used by real transports while keeping state deterministic and fully
inspectable.
## What it does today
## What it does
- Slack-class target grammar:
- `dm:<user>`
- `channel:<room>`
- `thread:<room>/<thread>`
- HTTP-backed synthetic bus for:
- inbound message injection
- outbound transcript capture
- thread creation
- reactions
- edits
- deletes
- search and read actions
- Bundled host-side self-check runner that writes a Markdown report
- HTTP-backed synthetic bus for inbound message injection, outbound transcript capture, thread creation, reactions, edits, deletes, and search/read actions.
- Host-side self-check runner that writes a Markdown report to `.artifacts/qa-e2e/`.
## Config
@ -45,68 +34,53 @@ inspectable.
}
```
Supported account keys:
Account keys:
- `baseUrl`
- `botUserId`
- `botDisplayName`
- `pollTimeoutMs`
- `allowFrom`
- `defaultTo`
- `actions.messages`
- `actions.reactions`
- `actions.search`
- `actions.threads`
- `enabled` — master toggle for this account.
- `name` — optional display label.
- `baseUrl` — synthetic bus URL.
- `botUserId` — Matrix-style bot user id used in target grammar.
- `botDisplayName` — display name for outbound messages.
- `pollTimeoutMs` — long-poll wait window. Integer between 100 and 30000.
- `allowFrom` — sender allowlist (user ids or `"*"`).
- `defaultTo` — fallback target when none is supplied.
- `actions.messages` / `actions.reactions` / `actions.search` / `actions.threads` — per-action tool gating.
## Runner
Multi-account keys at the top level:
Current vertical slice:
- `accounts` — record of named per-account overrides keyed by account id.
- `defaultAccount` — preferred account id when multiple are configured.
## Runners
Host-side self-check (writes a Markdown report under `.artifacts/qa-e2e/`):
```bash
pnpm qa:e2e
```
This now routes through the bundled `qa-lab` extension. It starts the in-repo
QA bus, boots the bundled `qa-channel` runtime slice, runs a deterministic
self-check, and writes a Markdown report under `.artifacts/qa-e2e/`.
This routes through `qa-lab`, starts the in-repo QA bus, boots the bundled `qa-channel` runtime slice, and runs a deterministic self-check.
Private debugger UI:
```bash
pnpm qa:lab:up
```
That one command builds the QA site, starts the Docker-backed gateway + QA Lab
stack, and prints the QA Lab URL. From that site you can pick scenarios, choose
the model lane, launch individual runs, and watch results live.
Full repo-backed QA suite:
Full repo-backed scenario suite:
```bash
pnpm openclaw qa suite
```
That launches the private QA debugger at a local URL, separate from the
shipped Control UI bundle.
Runs scenarios in parallel against the QA gateway lane. See [QA overview](/concepts/qa-e2e-automation) for scenarios, profiles, and provider modes.
## Scope
Docker-backed QA site (gateway + QA Lab debugger UI in one stack):
Current scope is intentionally narrow:
```bash
pnpm qa:lab:up
```
- bus + plugin transport
- threaded routing grammar
- channel-owned message actions
- Markdown reporting
- Docker-backed QA site with run controls
Follow-up work will add:
- provider/model matrix execution
- richer scenario discovery
- OpenClaw-native orchestration later
Builds the QA site, starts the Docker-backed gateway + QA Lab stack, and prints the QA Lab URL. From there you can pick scenarios, choose the model lane, launch individual runs, and watch results live. The QA Lab debugger is separate from the shipped Control UI bundle.
## Related
- [QA overview](/concepts/qa-e2e-automation) — overall stack, transport adapters, scenario authoring
- [Matrix QA](/concepts/qa-matrix) — example live-transport runner that drives a real channel
- [Pairing](/channels/pairing)
- [Groups](/channels/groups)
- [Channels overview](/channels)

147
docs/concepts/qa-e2e-automation.md
View File

@ -1,10 +1,11 @@
---
summary: "Private QA automation shape for qa-lab, qa-channel, seeded scenarios, and protocol reports"
summary: "QA stack overview: qa-lab, qa-channel, repo-backed scenarios, live transport lanes, transport adapters, and reporting."
read_when:
- Extending qa-lab or qa-channel
- Understanding how the QA stack fits together
- Extending qa-lab, qa-channel, or a transport adapter
- Adding repo-backed QA scenarios
- Building higher-realism QA automation around the Gateway dashboard
title: "QA E2E automation"
title: "QA overview"
---
The private QA stack is meant to exercise OpenClaw in a more realistic,
@ -16,9 +17,37 @@ Current pieces:
reaction, edit, and delete surfaces.
- `extensions/qa-lab`: debugger UI and QA bus for observing the transcript,
injecting inbound messages, and exporting a Markdown report.
- `extensions/qa-matrix`, future runner plugins: live-transport adapters that
drive a real channel inside a child QA gateway.
- `qa/`: repo-backed seed assets for the kickoff task and baseline QA
scenarios.
## Command surface
Every QA flow runs under `pnpm openclaw qa <subcommand>`. Many have `pnpm qa:*`
script aliases; both forms are supported.
| Command | Purpose |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `qa run` | Bundled QA self-check; writes a Markdown report. |
| `qa suite` | Run repo-backed scenarios against the QA gateway lane. Aliases: `pnpm openclaw qa suite --runner multipass` for a disposable Linux VM. |
| `qa coverage` | Print the markdown scenario-coverage inventory (`--json` for machine output). |
| `qa parity-report` | Compare two `qa-suite-summary.json` files and write the agentic parity-gate report. |
| `qa character-eval` | Run the character QA scenario across multiple live models with a judged report. See [Reporting](#reporting). |
| `qa manual` | Run a one-off prompt against the selected provider/model lane. |
| `qa ui` | Start the QA debugger UI and local QA bus (alias: `pnpm qa:lab:ui`). |
| `qa docker-build-image` | Build the prebaked QA Docker image. |
| `qa docker-scaffold` | Write a docker-compose scaffold for the QA dashboard + gateway lane. |
| `qa up` | Build the QA site, start the Docker-backed stack, print the URL (alias: `pnpm qa:lab:up`; `:fast` variant adds `--use-prebuilt-image --bind-ui-dist --skip-ui-build`). |
| `qa aimock` | Start only the AIMock provider server. |
| `qa mock-openai` | Start only the scenario-aware `mock-openai` provider server. |
| `qa credentials doctor` / `add` / `list` / `remove` | Manage the shared Convex credential pool. |
| `qa matrix` | Live transport lane against a disposable Tuwunel homeserver. See [Matrix QA](/concepts/qa-matrix). |
| `qa telegram` | Live transport lane against a real private Telegram group. |
| `qa discord` | Live transport lane against a real private Discord guild channel. |
## Operator flow
The current QA operator flow is a two-pane QA site:
- Left: Gateway dashboard (Control UI) with the agent.
@ -76,23 +105,7 @@ For a transport-real Matrix smoke lane, run:
pnpm openclaw qa matrix --profile fast --fail-fast
```
That lane provisions a disposable Tuwunel homeserver in Docker, registers
temporary driver, SUT, and observer users, creates one private room, then runs
the real Matrix plugin inside a QA gateway child. The live transport lane keeps
the child config scoped to the transport under test, so Matrix runs without
`qa-channel` in the child config. It writes the structured report artifacts and
a combined stdout/stderr log into the selected Matrix QA output directory. To
capture the outer `scripts/run-node.mjs` build/launcher output too, set
`OPENCLAW_RUN_NODE_OUTPUT_LOG=<path>` to a repo-local log file.
Matrix progress is printed by default. The CLI default profile is `all`, so
plain `pnpm openclaw qa matrix` still runs the full catalog. Use `--profile
fast` for the release-critical transport contract, or shard full coverage with
`transport`, `media`, `e2ee-smoke`, `e2ee-deep`, and `e2ee-cli`. `--fail-fast`
stops after the first failed scenario when you want a release gate instead of a
full inventory. `OPENCLAW_QA_MATRIX_TIMEOUT_MS` bounds the full run,
`OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS` can shorten no-reply quiet windows for
CI, and `OPENCLAW_QA_MATRIX_CLEANUP_TIMEOUT_MS` bounds cleanup so a stuck
Docker teardown reports the exact recovery command instead of hanging.
The full CLI reference, profile/scenario catalog, env vars, and artifact layout for this lane live in [Matrix QA](/concepts/qa-matrix). At a glance: it provisions a disposable Tuwunel homeserver in Docker, registers temporary driver/SUT/observer users, runs the real Matrix plugin inside a child QA gateway scoped to that transport (no `qa-channel`), then writes a Markdown report, JSON summary, observed-events artifact, and combined output log under `.artifacts/qa-e2e/matrix-<timestamp>/`.
For a transport-real Telegram smoke lane, run:
@ -106,7 +119,8 @@ disposable server. It requires `OPENCLAW_QA_TELEGRAM_GROUP_ID`,
`OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN`, plus two distinct bots in the same
private group. The SUT bot must have a Telegram username, and bot-to-bot
observation works best when both bots have Bot-to-Bot Communication Mode
enabled in `@BotFather`.
enabled in `@BotFather`. Set `OPENCLAW_QA_TELEGRAM_CAPTURE_CONTENT=1` to keep
message bodies in observed-message artifacts (default redacts).
The command exits non-zero when any scenario fails. Use `--allow-failures` when
you want artifacts without a failing exit code.
The Telegram report and summary include per-reply RTT from the driver message
@ -133,17 +147,17 @@ driver bot controlled by the harness and a SUT bot started by the child
OpenClaw gateway through the bundled Discord plugin. It requires
`OPENCLAW_QA_DISCORD_GUILD_ID`, `OPENCLAW_QA_DISCORD_CHANNEL_ID`,
`OPENCLAW_QA_DISCORD_DRIVER_BOT_TOKEN`, `OPENCLAW_QA_DISCORD_SUT_BOT_TOKEN`,
and `OPENCLAW_QA_DISCORD_SUT_APPLICATION_ID` when using env credentials.
and `OPENCLAW_QA_DISCORD_SUT_APPLICATION_ID` when using env credentials. Set
`OPENCLAW_QA_DISCORD_CAPTURE_CONTENT=1` to keep message bodies in
observed-message artifacts (default redacts).
The lane verifies channel mention handling and checks that the SUT bot has
registered the native `/help` command with Discord.
The command exits non-zero when any scenario fails. Use `--allow-failures` when
you want artifacts without a failing exit code.
Live transport lanes now share one smaller contract instead of each inventing
their own scenario list shape:
## Live transport coverage
`qa-channel` remains the broad synthetic product-behavior suite and is not part
of the live transport coverage matrix.
Live transport lanes share one contract instead of each inventing their own scenario list shape. `qa-channel` is the broad synthetic product-behavior suite and is not part of the live transport coverage matrix.
| Lane | Canary | Mention gating | Allowlist block | Top-level reply | Restart resume | Thread follow-up | Thread isolation | Reaction observation | Help command | Native command registration |
| -------- | ------ | -------------- | --------------- | --------------- | -------------- | ---------------- | ---------------- | -------------------- | ------------ | --------------------------- |
@ -235,19 +249,79 @@ provider names.
## Transport adapters
`qa-lab` owns a generic transport seam for markdown QA scenarios.
`qa-channel` is the first adapter on that seam, but the design target is wider:
future real or synthetic channels should plug into the same suite runner
instead of adding a transport-specific QA runner.
`qa-lab` owns a generic transport seam for markdown QA scenarios. `qa-channel` is the first adapter on that seam, but the design target is wider: future real or synthetic channels should plug into the same suite runner instead of adding a transport-specific QA runner.
At the architecture level, the split is:
- `qa-lab` owns generic scenario execution, worker concurrency, artifact writing, and reporting.
- the transport adapter owns gateway config, readiness, inbound and outbound observation, transport actions, and normalized transport state.
- markdown scenario files under `qa/scenarios/` define the test run; `qa-lab` provides the reusable runtime surface that executes them.
- The transport adapter owns gateway config, readiness, inbound and outbound observation, transport actions, and normalized transport state.
- Markdown scenario files under `qa/scenarios/` define the test run; `qa-lab` provides the reusable runtime surface that executes them.
Maintainer-facing adoption guidance for new channel adapters lives in
[Testing](/help/testing#adding-a-channel-to-qa).
### Adding a channel
Adding a channel to the markdown QA system requires exactly two things:
1. A transport adapter for the channel.
2. A scenario pack that exercises the channel contract.
Do not add a new top-level QA command root when the shared `qa-lab` host can own the flow.
`qa-lab` owns the shared host mechanics:
- the `openclaw qa` command root
- suite startup and teardown
- worker concurrency
- artifact writing
- report generation
- scenario execution
- compatibility aliases for older `qa-channel` scenarios
Runner plugins own the transport contract:
- how `openclaw qa <runner>` is mounted beneath the shared `qa` root
- how the gateway is configured for that transport
- how readiness is checked
- how inbound events are injected
- how outbound messages are observed
- how transcripts and normalized transport state are exposed
- how transport-backed actions are executed
- how transport-specific reset or cleanup is handled
The minimum adoption bar for a new channel:
1. Keep `qa-lab` as the owner of the shared `qa` root.
2. Implement the transport runner on the shared `qa-lab` host seam.
3. Keep transport-specific mechanics inside the runner plugin or channel harness.
4. Mount the runner as `openclaw qa <runner>` instead of registering a competing root command. Runner plugins should declare `qaRunners` in `openclaw.plugin.json` and export a matching `qaRunnerCliRegistrations` array from `runtime-api.ts`. Keep `runtime-api.ts` light; lazy CLI and runner execution should stay behind separate entrypoints.
5. Author or adapt markdown scenarios under the themed `qa/scenarios/` directories.
6. Use the generic scenario helpers for new scenarios.
7. Keep existing compatibility aliases working unless the repo is doing an intentional migration.
The decision rule is strict:
- If behavior can be expressed once in `qa-lab`, put it in `qa-lab`.
- If behavior depends on one channel transport, keep it in that runner plugin or plugin harness.
- If a scenario needs a new capability that more than one channel can use, add a generic helper instead of a channel-specific branch in `suite.ts`.
- If a behavior is only meaningful for one transport, keep the scenario transport-specific and make that explicit in the scenario contract.
### Scenario helper names
Preferred generic helpers for new scenarios:
- `waitForTransportReady`
- `waitForChannelReady`
- `injectInboundMessage`
- `injectOutboundMessage`
- `waitForTransportOutboundMessage`
- `waitForChannelOutboundMessage`
- `waitForNoTransportOutbound`
- `getTransportSnapshot`
- `readTransportMessage`
- `readTransportTranscript`
- `formatTransportTranscript`
- `resetTransport`
Compatibility aliases remain available for existing scenarios — `waitForQaChannelReady`, `waitForOutboundMessage`, `waitForNoOutbound`, `formatConversationTranscript`, `resetBus` — but new scenario authoring should use the generic names. The aliases exist to avoid a flag-day migration, not as the model going forward.
## Reporting
@ -259,6 +333,8 @@ The report should answer:
- What stayed blocked
- What follow-up scenarios are worth adding
For the inventory of available scenarios — useful when sizing follow-up work or wiring a new transport — run `pnpm openclaw qa coverage` (add `--json` for machine-readable output).
For character and style checks, run the same scenario across multiple live model
refs and write a judged Markdown report:
@ -314,6 +390,7 @@ When no `--judge-model` is passed, the judges default to
## Related docs
- [Testing](/help/testing)
- [Matrix QA](/concepts/qa-matrix)
- [QA Channel](/channels/qa-channel)
- [Testing](/help/testing)
- [Dashboard](/web/dashboard)

136
docs/concepts/qa-matrix.md Normal file
View File

@ -0,0 +1,136 @@
---
summary: "Maintainer reference for the Docker-backed Matrix live QA lane: CLI, profiles, env vars, scenarios, and output artifacts."
read_when:
- Running pnpm openclaw qa matrix locally
- Adding or selecting Matrix QA scenarios
- Triaging Matrix QA failures, timeouts, or stuck cleanup
title: "Matrix QA"
---
The Matrix QA lane runs the bundled `@openclaw/matrix` plugin against a disposable Tuwunel homeserver in Docker, with temporary driver, SUT, and observer accounts plus seeded rooms. It is the live transport-real coverage for Matrix.
This is maintainer-only tooling. Packaged OpenClaw releases intentionally omit `qa-lab`, so `openclaw qa` is only available from a source checkout. Source checkouts load the bundled runner directly — no plugin install step is needed.
For broader QA framework context, see [QA overview](/concepts/qa-e2e-automation).
## Quick start
```bash
pnpm openclaw qa matrix --profile fast --fail-fast
```
Plain `pnpm openclaw qa matrix` runs `--profile all` and does not stop on first failure. Use `--profile fast --fail-fast` for a release gate; shard the catalog with `--profile transport|media|e2ee-smoke|e2ee-deep|e2ee-cli` when running the full inventory in parallel.
## What the lane does
1. Provisions a disposable Tuwunel homeserver in Docker (default image `ghcr.io/matrix-construct/tuwunel:v1.5.1`, server name `matrix-qa.test`, port `28008`).
2. Registers three temporary users — `driver` (sends inbound traffic), `sut` (the OpenClaw Matrix account under test), `observer` (third-party traffic capture).
3. Seeds rooms required by the selected scenarios (main, threading, media, restart, secondary, allowlist, E2EE, verification DM, etc.).
4. Starts a child OpenClaw gateway with the real Matrix plugin scoped to the SUT account; `qa-channel` is not loaded in the child.
5. Runs scenarios in sequence, observing events through the driver/observer Matrix clients.
6. Tears down the homeserver, writes report and summary artifacts, then exits.
## CLI
```text
pnpm openclaw qa matrix [options]
```
### Common flags
| Flag | Default | Description |
| --------------------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `--profile <profile>` | `all` | Scenario profile. See [Profiles](#profiles). |
| `--fail-fast` | off | Stop after the first failed check or scenario. |
| `--scenario <id>` | — | Run only this scenario. Repeatable. See [Scenarios](#scenarios). |
| `--output-dir <path>` | `<repo>/.artifacts/qa-e2e/matrix-<timestamp>` | Where reports, summary, observed events, and the output log are written. Relative paths resolve against `--repo-root`. |
| `--repo-root <path>` | `process.cwd()` | Repository root when invoking from a neutral working directory. |
| `--sut-account <id>` | `sut` | Matrix account id inside the QA gateway config. |
### Provider flags
The lane uses a real Matrix transport but the model provider is configurable:
| Flag | Default | Description |
| ------------------------ | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `--provider-mode <mode>` | `live-frontier` | `mock-openai` for deterministic mock dispatch or `live-frontier` for live frontier providers. The legacy alias `live-openai` still works. |
| `--model <ref>` | provider default | Primary `provider/model` ref. |
| `--alt-model <ref>` | provider default | Alternate `provider/model` ref where scenarios switch mid-run. |
| `--fast` | off | Enable provider fast mode where supported. |
Matrix QA does not accept `--credential-source` or `--credential-role`. The lane provisions disposable users locally; there is no shared credential pool to lease against.
## Profiles
The selected profile decides which scenarios run.
| Profile | Use it for |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `all` (default) | Full catalog. Slow but exhaustive. |
| `fast` | Release-gate subset that exercises the live transport contract: canary, mention gating, allowlist block, reply shape, restart resume, thread follow-up, thread isolation, reaction observation. |
| `transport` | Transport-level threading, DM, room, autojoin, mention/allowlist scenarios. |
| `media` | Image, audio, video, PDF, EPUB attachment coverage. |
| `e2ee-smoke` | Minimum E2EE coverage — basic encrypted reply, thread follow-up, bootstrap success. |
| `e2ee-deep` | Exhaustive E2EE state-loss, backup, key, and recovery scenarios. |
| `e2ee-cli` | `openclaw matrix encryption setup` and `verify *` CLI scenarios driven through the QA harness. |
The exact mapping lives in `extensions/qa-matrix/src/runners/contract/scenario-catalog.ts`.
## Scenarios
The full scenario id list is the `MatrixQaScenarioId` union in `extensions/qa-matrix/src/runners/contract/scenario-catalog.ts:15`. Categories include:
- threading — `matrix-thread-*`, `matrix-subagent-thread-spawn`
- top-level / DM / room — `matrix-top-level-reply-shape`, `matrix-room-*`, `matrix-dm-*`
- media — `matrix-media-type-coverage`, `matrix-room-image-understanding-attachment`, `matrix-attachment-only-ignored`, `matrix-unsupported-media-safe`
- routing — `matrix-room-autojoin-invite`, `matrix-secondary-room-*`
- reactions — `matrix-reaction-*`
- restart and replay — `matrix-restart-*`, `matrix-stale-sync-replay-dedupe`, `matrix-room-membership-loss`, `matrix-homeserver-restart-resume`, `matrix-initial-catchup-then-incremental`
- mention gating and allowlists — `matrix-mention-*`, `matrix-allowlist-*`, `matrix-multi-actor-ordering`, `matrix-inbound-edit-*`, `matrix-mxid-prefixed-command-block`, `matrix-observer-allowlist-override`
- E2EE — `matrix-e2ee-*` (basic reply, thread follow-up, bootstrap, recovery key lifecycle, state-loss variants, server backup behavior, device hygiene, SAS / QR / DM verification, restart, artifact redaction)
- E2EE CLI — `matrix-e2ee-cli-*` (encryption setup, idempotent setup, bootstrap failure, recovery-key lifecycle, multi-account, gateway-reply round-trip, self-verification)
Pass `--scenario <id>` (repeatable) to run a hand-picked set; combine with `--profile all` to ignore profile gating.
## Environment variables
| Variable | Default | Effect |
| --------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OPENCLAW_QA_MATRIX_TIMEOUT_MS` | `1800000` (30 min) | Hard upper bound on the entire run. |
| `OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS` | `8000` | Quiet window for negative no-reply assertions. Clamped to `≤` the run timeout. |
| `OPENCLAW_QA_MATRIX_CLEANUP_TIMEOUT_MS` | `90000` | Bound for Docker teardown. Failure surfaces include the recovery `docker compose ... down --remove-orphans` command. |
| `OPENCLAW_QA_MATRIX_TUWUNEL_IMAGE` | `ghcr.io/matrix-construct/tuwunel:v1.5.1` | Override the homeserver image when validating against a different Tuwunel version. |
| `OPENCLAW_QA_MATRIX_PROGRESS` | on | `0` silences `[matrix-qa] ...` progress lines on stderr. `1` forces them on. |
| `OPENCLAW_QA_MATRIX_CAPTURE_CONTENT` | redacted | `1` keeps message body and `formatted_body` in `matrix-qa-observed-events.json`. Default redacts to keep CI artifacts safe. |
| `OPENCLAW_QA_MATRIX_DISABLE_FORCE_EXIT` | off | `1` skips the deterministic `process.exit` after artifact write. The default forces exit because matrix-js-sdk's native crypto handles can keep the event loop alive past artifact completion. |
| `OPENCLAW_RUN_NODE_OUTPUT_LOG` | unset | When set by an outer launcher (e.g. `scripts/run-node.mjs`), Matrix QA reuses that log path instead of starting its own tee. |
## Output artifacts
Written to `--output-dir`:
- `matrix-qa-report.md` — Markdown protocol report (what passed, failed, was skipped, and why).
- `matrix-qa-summary.json` — Structured summary suitable for CI parsing and dashboards.
- `matrix-qa-observed-events.json` — Observed Matrix events from the driver and observer clients. Bodies are redacted unless `OPENCLAW_QA_MATRIX_CAPTURE_CONTENT=1`.
- `matrix-qa-output.log` — Combined stdout/stderr from the run. If `OPENCLAW_RUN_NODE_OUTPUT_LOG` is set, the outer launcher's log is reused instead.
The default output dir is `<repo>/.artifacts/qa-e2e/matrix-<timestamp>` so successive runs do not overwrite each other.
## Triage tips
- **Run hangs near the end:** `matrix-js-sdk` native crypto handles can outlive the harness. The default forces a clean `process.exit` after artifact write; if you have unset `OPENCLAW_QA_MATRIX_DISABLE_FORCE_EXIT=1`, expect the process to linger.
- **Cleanup error:** look for the printed recovery command (a `docker compose ... down --remove-orphans` invocation) and run it manually to release the homeserver port.
- **Flaky negative-assertion windows in CI:** lower `OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS` (default 8 s) when CI is fast; raise it on slow shared runners.
- **Need redacted bodies for a bug report:** rerun with `OPENCLAW_QA_MATRIX_CAPTURE_CONTENT=1` and attach `matrix-qa-observed-events.json`. Treat the resulting artifact as sensitive.
- **Different Tuwunel version:** point `OPENCLAW_QA_MATRIX_TUWUNEL_IMAGE` at the version under test. The lane checks in only the pinned default image.
## Live transport contract
Matrix is one of three live transport lanes (Matrix, Telegram, Discord) that share a single contract checklist defined in [QA overview → Live transport coverage](/concepts/qa-e2e-automation#live-transport-coverage). `qa-channel` remains the broad synthetic suite and is intentionally not part of that matrix.
## Related
- [QA overview](/concepts/qa-e2e-automation) — overall QA stack and live transport contract
- [QA Channel](/channels/qa-channel) — synthetic channel adapter for repo-backed scenarios
- [Testing](/help/testing) — running tests and adding QA coverage
- [Matrix](/channels/matrix) — the channel plugin under test

39
docs/docs.json
View File

@ -1134,7 +1134,8 @@
"concepts/oauth",
"start/bootstrapping",
"concepts/experimental-features",
"concepts/qa-e2e-automation"
"concepts/qa-e2e-automation",
"concepts/qa-matrix"
]
},
{
@ -2596,7 +2597,8 @@
"ja-JP/concepts/oauth",
"ja-JP/start/bootstrapping",
"ja-JP/concepts/experimental-features",
"ja-JP/concepts/qa-e2e-automation"
"ja-JP/concepts/qa-e2e-automation",
"ja-JP/concepts/qa-matrix"
]
},
{
@ -3443,7 +3445,8 @@
"es/concepts/oauth",
"es/start/bootstrapping",
"es/concepts/experimental-features",
"es/concepts/qa-e2e-automation"
"es/concepts/qa-e2e-automation",
"es/concepts/qa-matrix"
]
},
{
@ -4290,7 +4293,8 @@
"pt-BR/concepts/oauth",
"pt-BR/start/bootstrapping",
"pt-BR/concepts/experimental-features",
"pt-BR/concepts/qa-e2e-automation"
"pt-BR/concepts/qa-e2e-automation",
"pt-BR/concepts/qa-matrix"
]
},
{
@ -5137,7 +5141,8 @@
"ko/concepts/oauth",
"ko/start/bootstrapping",
"ko/concepts/experimental-features",
"ko/concepts/qa-e2e-automation"
"ko/concepts/qa-e2e-automation",
"ko/concepts/qa-matrix"
]
},
{
@ -5984,7 +5989,8 @@
"de/concepts/oauth",
"de/start/bootstrapping",
"de/concepts/experimental-features",
"de/concepts/qa-e2e-automation"
"de/concepts/qa-e2e-automation",
"de/concepts/qa-matrix"
]
},
{
@ -6831,7 +6837,8 @@
"fr/concepts/oauth",
"fr/start/bootstrapping",
"fr/concepts/experimental-features",
"fr/concepts/qa-e2e-automation"
"fr/concepts/qa-e2e-automation",
"fr/concepts/qa-matrix"
]
},
{
@ -7678,7 +7685,8 @@
"ar/concepts/oauth",
"ar/start/bootstrapping",
"ar/concepts/experimental-features",
"ar/concepts/qa-e2e-automation"
"ar/concepts/qa-e2e-automation",
"ar/concepts/qa-matrix"
]
},
{
@ -8525,7 +8533,8 @@
"it/concepts/oauth",
"it/start/bootstrapping",
"it/concepts/experimental-features",
"it/concepts/qa-e2e-automation"
"it/concepts/qa-e2e-automation",
"it/concepts/qa-matrix"
]
},
{
@ -9372,7 +9381,8 @@
"tr/concepts/oauth",
"tr/start/bootstrapping",
"tr/concepts/experimental-features",
"tr/concepts/qa-e2e-automation"
"tr/concepts/qa-e2e-automation",
"tr/concepts/qa-matrix"
]
},
{
@ -10219,7 +10229,8 @@
"uk/concepts/oauth",
"uk/start/bootstrapping",
"uk/concepts/experimental-features",
"uk/concepts/qa-e2e-automation"
"uk/concepts/qa-e2e-automation",
"uk/concepts/qa-matrix"
]
},
{
@ -11066,7 +11077,8 @@
"id/concepts/oauth",
"id/start/bootstrapping",
"id/concepts/experimental-features",
"id/concepts/qa-e2e-automation"
"id/concepts/qa-e2e-automation",
"id/concepts/qa-matrix"
]
},
{
@ -11913,7 +11925,8 @@
"pl/concepts/oauth",
"pl/start/bootstrapping",
"pl/concepts/experimental-features",
"pl/concepts/qa-e2e-automation"
"pl/concepts/qa-e2e-automation",
"pl/concepts/qa-matrix"
]
},
{

109
docs/help/testing.md
View File

@ -15,6 +15,16 @@ of Docker runners. This doc is a "how we test" guide:
- How live tests discover credentials and select models/providers.
- How to add regressions for real-world model/provider issues.
<Note>
**QA stack (qa-lab, qa-channel, live transport lanes)** is documented separately:
- [QA overview](/concepts/qa-e2e-automation) — architecture, command surface, scenario authoring.
- [Matrix QA](/concepts/qa-matrix) — reference for `pnpm openclaw qa matrix`.
- [QA channel](/channels/qa-channel) — the synthetic transport plugin used by repo-backed scenarios.
This page covers running the regular test suites and Docker/Parallels runners. The QA-specific runners section below ([QA-specific runners](#qa-specific-runners)) lists the concrete `qa` invocations and points back at the references above.
</Note>
## Quick start
Most days:
@ -248,17 +258,8 @@ gh workflow run package-acceptance.yml --ref main \
- Starts only the local AIMock provider server for direct protocol smoke
testing.
- `pnpm openclaw qa matrix`
- Runs the Matrix live QA lane against a disposable Docker-backed Tuwunel homeserver.
- This QA host is repo/dev-only today. Packaged OpenClaw installs do not ship
`qa-lab`, so they do not expose `openclaw qa`.
- Repo checkouts load the bundled runner directly; no separate plugin install
step is needed.
- Provisions three temporary Matrix users (`driver`, `sut`, `observer`) plus one private room, then starts a QA gateway child with the real Matrix plugin as the SUT transport.
- Defaults to `--profile all`. Use `--profile fast --fail-fast` for release-critical transport proof, or `--profile transport|media|e2ee-smoke|e2ee-deep|e2ee-cli` when sharding the full catalog.
- Uses the pinned stable Tuwunel image `ghcr.io/matrix-construct/tuwunel:v1.5.1` by default. Override with `OPENCLAW_QA_MATRIX_TUWUNEL_IMAGE` when you need to test a different image.
- Matrix does not expose shared credential-source flags because the lane provisions disposable users locally.
- Writes a Matrix QA report, summary, observed-events artifact, and combined stdout/stderr output log under `.artifacts/qa-e2e/...`.
- Emits progress by default and enforces a hard run timeout with `OPENCLAW_QA_MATRIX_TIMEOUT_MS` (default 30 minutes). `OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS` tunes negative no-reply quiet windows, and cleanup is bounded by `OPENCLAW_QA_MATRIX_CLEANUP_TIMEOUT_MS` with failures including the recovery `docker compose ... down --remove-orphans` command.
- Runs the Matrix live QA lane against a disposable Docker-backed Tuwunel homeserver. Source-checkout only — packaged installs do not ship `qa-lab`.
- Full CLI, profile/scenario catalog, env vars, and artifact layout: [Matrix QA](/concepts/qa-matrix).
- `pnpm openclaw qa telegram`
- Runs the Telegram live QA lane against a real private group using the driver and SUT bot tokens from env.
- Requires `OPENCLAW_QA_TELEGRAM_GROUP_ID`, `OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN`, and `OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN`. The group id must be the numeric Telegram chat id.
@ -269,16 +270,7 @@ gh workflow run package-acceptance.yml --ref main \
- For stable bot-to-bot observation, enable Bot-to-Bot Communication Mode in `@BotFather` for both bots and ensure the driver bot can observe group bot traffic.
- Writes a Telegram QA report, summary, and observed-messages artifact under `.artifacts/qa-e2e/...`. Replying scenarios include RTT from driver send request to observed SUT reply.
Live transport lanes share one standard contract so new transports do not drift:
`qa-channel` remains the broad synthetic QA suite and is not part of the live
transport coverage matrix.
| Lane | Canary | Mention gating | Allowlist block | Top-level reply | Restart resume | Thread follow-up | Thread isolation | Reaction observation | Help command | Native command registration |
| -------- | ------ | -------------- | --------------- | --------------- | -------------- | ---------------- | ---------------- | -------------------- | ------------ | --------------------------- |
| Matrix | x | x | x | x | x | x | x | x | | |
| Telegram | x | x | | | | | | | x | |
| Discord | x | x | | | | | | | | x |
Live transport lanes share one standard contract so new transports do not drift; the per-lane coverage matrix lives in [QA overview → Live transport coverage](/concepts/qa-e2e-automation#live-transport-coverage). `qa-channel` is the broad synthetic suite and is not part of that matrix.
### Shared Telegram credentials via Convex (v1)
@ -360,80 +352,7 @@ Payload shape for Telegram kind:
### Adding a channel to QA
Adding a channel to the markdown QA system requires exactly two things:
1. A transport adapter for the channel.
2. A scenario pack that exercises the channel contract.
Do not add a new top-level QA command root when the shared `qa-lab` host can
own the flow.
`qa-lab` owns the shared host mechanics:
- the `openclaw qa` command root
- suite startup and teardown
- worker concurrency
- artifact writing
- report generation
- scenario execution
- compatibility aliases for older `qa-channel` scenarios
Runner plugins own the transport contract:
- how `openclaw qa <runner>` is mounted beneath the shared `qa` root
- how the gateway is configured for that transport
- how readiness is checked
- how inbound events are injected
- how outbound messages are observed
- how transcripts and normalized transport state are exposed
- how transport-backed actions are executed
- how transport-specific reset or cleanup is handled
The minimum adoption bar for a new channel is:
1. Keep `qa-lab` as the owner of the shared `qa` root.
2. Implement the transport runner on the shared `qa-lab` host seam.
3. Keep transport-specific mechanics inside the runner plugin or channel harness.
4. Mount the runner as `openclaw qa <runner>` instead of registering a competing root command.
Runner plugins should declare `qaRunners` in `openclaw.plugin.json` and export a matching `qaRunnerCliRegistrations` array from `runtime-api.ts`.
Keep `runtime-api.ts` light; lazy CLI and runner execution should stay behind separate entrypoints.
5. Author or adapt markdown scenarios under the themed `qa/scenarios/` directories.
6. Use the generic scenario helpers for new scenarios.
7. Keep existing compatibility aliases working unless the repo is doing an intentional migration.
The decision rule is strict:
- If behavior can be expressed once in `qa-lab`, put it in `qa-lab`.
- If behavior depends on one channel transport, keep it in that runner plugin or plugin harness.
- If a scenario needs a new capability that more than one channel can use, add a generic helper instead of a channel-specific branch in `suite.ts`.
- If a behavior is only meaningful for one transport, keep the scenario transport-specific and make that explicit in the scenario contract.
Preferred generic helper names for new scenarios are:
- `waitForTransportReady`
- `waitForChannelReady`
- `injectInboundMessage`
- `injectOutboundMessage`
- `waitForTransportOutboundMessage`
- `waitForChannelOutboundMessage`
- `waitForNoTransportOutbound`
- `getTransportSnapshot`
- `readTransportMessage`
- `readTransportTranscript`
- `formatTransportTranscript`
- `resetTransport`
Compatibility aliases remain available for existing scenarios, including:
- `waitForQaChannelReady`
- `waitForOutboundMessage`
- `waitForNoOutbound`
- `formatConversationTranscript`
- `resetBus`
New channel work should use the generic helper names.
Compatibility aliases exist to avoid a flag day migration, not as the model for
new scenario authoring.
The architecture and scenario-helper names for new channel adapters live in [QA overview → Adding a channel](/concepts/qa-e2e-automation#adding-a-channel). The minimum bar: implement the transport runner on the shared `qa-lab` host seam, declare `qaRunners` in the plugin manifest, mount as `openclaw qa <runner>`, and author scenarios under `qa/scenarios/`.
## Test suites (what runs where)