Compare commits
1 Commits
main
...
fix/health
| Author | SHA1 | Date | |
|---|---|---|---|
|
522b257955 |
22
README.md
22
README.md
@ -30,9 +30,8 @@ Unit tests can say code passed. Kova answers the release question:
|
||||
and plugin load failures.
|
||||
- **Failure containment**: provider timeouts, malformed responses, streaming
|
||||
stalls, recovery, gateway health after failure, and leaked child processes.
|
||||
- **Human and agent reports**: verdict-first Markdown for people, compact
|
||||
`*.summary.json` for agents/CI, full JSON evidence for audits, plus artifact
|
||||
bundles for handoff.
|
||||
- **Human and agent reports**: concise Markdown for people, structured JSON for
|
||||
agents/CI, plus artifact bundles for handoff.
|
||||
|
||||
Kova uses OCM to create isolated OpenClaw labs. Kova is not testing OCM. OCM is
|
||||
the harness; OpenClaw is the product under test.
|
||||
@ -52,7 +51,7 @@ release-runtime-startup/fresh
|
||||
build-tooling peak RSS: 2409 MB
|
||||
missing dependency: @homebridge/ciao from bundled bonjour
|
||||
|
||||
gateway-session-send-turn/mock-openai-provider
|
||||
dashboard-session-send-turn/mock-openai-provider
|
||||
agent turn: 9.2s
|
||||
pre-provider OpenClaw time: 8.9s
|
||||
provider time: 1ms
|
||||
@ -61,7 +60,7 @@ gateway-session-send-turn/mock-openai-provider
|
||||
health: gateway had post-command health failures
|
||||
|
||||
Fixer brief:
|
||||
Area: plugins/runtime deps, Gateway session agent path
|
||||
Area: plugins/runtime deps, dashboard session agent path
|
||||
Why it matters: users can start successfully but hit plugin dependency errors
|
||||
and slow first replies unrelated to provider latency.
|
||||
```
|
||||
@ -89,10 +88,6 @@ node bin/kova.mjs setup --ci --json
|
||||
Kova data lives in `~/.kova` by default: credentials, reports, artifacts, and
|
||||
baselines.
|
||||
|
||||
The scenario model separates the shared Gateway session message path from
|
||||
dashboard-specific checks, channel adapter checks, and Agent CLI checks. See
|
||||
[Kova Scenario Hierarchy](docs/SCENARIO_HIERARCHY.md).
|
||||
|
||||
## Run The Important Checks
|
||||
|
||||
### Test A Local OpenClaw Checkout Like A Release
|
||||
@ -127,7 +122,7 @@ durable artifact bundle for failed gates.
|
||||
```sh
|
||||
node bin/kova.mjs run \
|
||||
--target local-build:/path/to/openclaw \
|
||||
--scenario gateway-session-send-turn \
|
||||
--scenario dashboard-session-send-turn \
|
||||
--execute \
|
||||
--json
|
||||
```
|
||||
@ -235,17 +230,10 @@ Agents should use JSON plans and reports:
|
||||
|
||||
```sh
|
||||
node bin/kova.mjs plan --json
|
||||
node bin/kova.mjs inventory plan --openclaw-bin openclaw --openclaw-repo /path/to/openclaw --json
|
||||
node bin/kova.mjs matrix plan --profile smoke --target runtime:stable --json
|
||||
node bin/kova.mjs matrix run --profile smoke --target runtime:stable --execute --json
|
||||
```
|
||||
|
||||
`inventory plan` is planner-only. It compares discovered OpenClaw CLI commands,
|
||||
product-relevant package scripts, and manifests with Kova surfaces and reports
|
||||
unmodeled capabilities as warnings. Package-script discovery defaults to
|
||||
`--script-scope product`; use `--script-scope all` only when auditing repo
|
||||
maintenance scripts too.
|
||||
|
||||
Kova ships repo-local agent skills:
|
||||
|
||||
- `.agents/skills/kova-operator`
|
||||
|
||||
@ -35,18 +35,11 @@ node bin/kova.mjs self-check --json
|
||||
|
||||
```sh
|
||||
node bin/kova.mjs plan --json
|
||||
node bin/kova.mjs inventory plan --openclaw-bin openclaw --openclaw-repo /path/to/openclaw --json
|
||||
node bin/kova.mjs plan --scenario fresh-install --state missing-plugin-index --json
|
||||
node bin/kova.mjs matrix plan --profile smoke --target runtime:stable --json
|
||||
node bin/kova.mjs matrix plan --profile release --target runtime:stable --include tag:plugins --exclude state:broken-plugin-deps --json
|
||||
```
|
||||
|
||||
Use `inventory plan` when checking whether Kova is missing OpenClaw command,
|
||||
product script, plugin, or extension surfaces. Package-script discovery defaults
|
||||
to `--script-scope product` so internal repo maintenance scripts do not drown
|
||||
out OpenClaw capability coverage. Treat unmodeled inventory entries as planning
|
||||
warnings until they are intentionally promoted to gate policy.
|
||||
|
||||
3. Dry-run the intended scenario:
|
||||
|
||||
```sh
|
||||
@ -118,11 +111,9 @@ or baseline comparison is not blocking. Do not save baselines from
|
||||
`--profile-on-failure` runs; those are instrumented diagnostic runs and their
|
||||
resource numbers can include profiler overhead.
|
||||
|
||||
5. Read the generated `*.summary.json` first. Use the Markdown report for the
|
||||
human decision summary and the full JSON report only when raw phase, command,
|
||||
or collector evidence is needed. For failures, start with `decision`,
|
||||
`findings`, and `failureBrief` in `report summarize --json` or the `Failure
|
||||
Brief` section from `report paste`.
|
||||
5. Read the generated JSON report first. Use the Markdown report for the human
|
||||
summary. For failures, start with `failureBrief` in `report summarize --json`
|
||||
or the `Failure Brief` section from `report paste`.
|
||||
|
||||
6. Produce a compact handoff when needed:
|
||||
|
||||
|
||||
@ -14,25 +14,20 @@ Required fields:
|
||||
- `title`: short human name.
|
||||
- `ownerArea`: OpenClaw subsystem likely to own failures.
|
||||
- `description`: what OpenClaw behavior this surface proves.
|
||||
- `purposes`: Kova purposes this surface is relevant to, such as `release`,
|
||||
`regression`, `diagnostic`, `performance`, `upgrade`, `plugin`, `provider`,
|
||||
or `soak`.
|
||||
- `requiredStates`: state ids or traits expected for meaningful coverage.
|
||||
- `targetKinds`: target kinds that can run the surface.
|
||||
- `requiredMetrics`: metric ids from `metrics/known.json`.
|
||||
- `processRoles`: role ids from `process-roles/*.json`.
|
||||
- `thresholds`: default pass/fail thresholds for the surface.
|
||||
- `diagnostics`: source-build timeline expectations when available.
|
||||
- `requirements`: stable requirement ids for this surface. Each requirement owns
|
||||
the states or state traits, target kinds, and metrics that prove that part of
|
||||
the surface contract.
|
||||
|
||||
Then:
|
||||
|
||||
1. Add or update one scenario in `scenarios/*.json` with `"surface": "<id>"`
|
||||
and `proves` entries for the requirement ids it exercises.
|
||||
1. Add or update one scenario in `scenarios/*.json` with `"surface": "<id>"`.
|
||||
2. Add missing metric ids to `metrics/known.json`.
|
||||
3. Add state fixture hooks or traits in `states/*.json` only when the
|
||||
requirement needs a new user condition.
|
||||
4. Add profile requirement coverage in `profiles/*.json` only when the surface
|
||||
requirement should be part of that profile.
|
||||
3. Add the surface to compatible states in `states/*.json`.
|
||||
4. Add profile coverage in `profiles/*.json` only when the surface should be
|
||||
part of that profile.
|
||||
5. Run `node bin/kova.mjs plan --json`.
|
||||
6. Run a dry-run for the scenario.
|
||||
7. Add self-check coverage if the surface introduces new evidence parsing.
|
||||
@ -50,6 +45,8 @@ Required fields:
|
||||
- `title`: short human name.
|
||||
- `objective`: what user history or degraded condition this state models.
|
||||
- `traits`: known traits validated by Kova.
|
||||
- `compatibleSurfaces`: surface ids this state can be paired with.
|
||||
- `incompatibleSurfaces`: surface ids this state must not be paired with.
|
||||
- `riskArea`: what can break when this state is used.
|
||||
- `ownerArea`: OpenClaw subsystem most likely to own state-specific failures.
|
||||
- `setupEvidence`: what proves setup happened.
|
||||
@ -60,16 +57,11 @@ inside disposable Kova envs and make the evidence explicit. Existing user state
|
||||
must be represented through clone/import metadata, not direct mutation of a
|
||||
durable env.
|
||||
|
||||
Put positive state compatibility on surface requirements through `states` or
|
||||
`stateTraits`. Add `incompatibleSurfaces` only for hard safety blocks where a
|
||||
fixture must never run against a surface; do not store empty compatibility
|
||||
lists.
|
||||
|
||||
Then:
|
||||
|
||||
1. Pair the state with compatible scenarios or profile entries.
|
||||
2. Add or update surface requirements when this state becomes required proof
|
||||
for a surface.
|
||||
2. Add profile trait/state coverage only when it is required for release
|
||||
confidence.
|
||||
3. Run `node bin/kova.mjs plan --json`.
|
||||
4. Dry-run at least one scenario/state pair.
|
||||
5. Execute a disposable scenario when the state lifecycle mutates files,
|
||||
@ -80,17 +72,10 @@ Then:
|
||||
Self-check and plan validation must fail for:
|
||||
|
||||
- unknown surface, state, process role, metric, or profile references
|
||||
- unknown purpose references
|
||||
- scenarios that prove unknown surface requirements
|
||||
- surface requirements that reference unknown states, traits, target kinds, or
|
||||
metrics
|
||||
- invalid state traits
|
||||
- malformed lifecycle phases
|
||||
- scenario/state pairs that violate requirement state contracts or hard
|
||||
incompatibility blocks
|
||||
- scenario/state pairs that violate compatibility
|
||||
- profile entries that require unknown surfaces or states
|
||||
- profile gate coverage that uses derived policy fields instead of
|
||||
`requirements` or `platforms`
|
||||
|
||||
If a new surface or state needs exceptions to these rules, the contract is too
|
||||
loose. Tighten the JSON or add a focused validator.
|
||||
|
||||
@ -18,8 +18,7 @@ kova.report.v1
|
||||
"runId": "kova-2026-04-29T000000Z",
|
||||
"outputPaths": {
|
||||
"markdown": "/path/to/report.md",
|
||||
"json": "/path/to/report.json",
|
||||
"summary": "/path/to/report.summary.json"
|
||||
"json": "/path/to/report.json"
|
||||
},
|
||||
"mode": "dry-run",
|
||||
"profile": null,
|
||||
@ -83,9 +82,8 @@ lists every file staged into the bundle with relative path, byte size, and
|
||||
SHA-256 digest so agents can inspect evidence coverage without scraping raw log
|
||||
output or unpacking blindly.
|
||||
|
||||
`outputPaths` records the Markdown, full JSON, and compact summary JSON paths
|
||||
for the report itself. The matrix receipt also includes bundle and checksum
|
||||
paths after bundling.
|
||||
`outputPaths` records the Markdown and JSON paths for the report itself. The
|
||||
matrix receipt also includes bundle and checksum paths after bundling.
|
||||
|
||||
`gate` is normally `null`. When `kova matrix run --gate` is used, it contains
|
||||
the release gate verdict, blocking/warning counts, required scenario policy, and
|
||||
@ -226,181 +224,6 @@ Current metrics include:
|
||||
- runtime dependency staging grouped by bundled plugin when OpenClaw emits
|
||||
`runtimeDeps.stage` spans with `pluginId` attributes
|
||||
|
||||
## Agent Turn Evidence
|
||||
|
||||
Agent turns are reported under `records[*].measurements.agentTurns`. Gateway
|
||||
session turns use the active `sessions.send` window for `totalTurnMs`,
|
||||
`preProviderMs`, `providerFinalMs`, `postProviderMs`, cold/warm metrics, and
|
||||
threshold checks. The raw support-command duration is still preserved as
|
||||
`rawCommandDurationMs` so readers can separate runner/session setup overhead
|
||||
from the OpenClaw turn path.
|
||||
|
||||
Gateway/session turn entries include:
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": "kova.agentTurnEvidence.v1",
|
||||
"label": "cold",
|
||||
"totalTurnMs": 1260,
|
||||
"rawCommandDurationMs": 2100,
|
||||
"gatewaySession": {
|
||||
"schemaVersion": "kova.gatewaySessionTurn.v1",
|
||||
"method": "sessions.send",
|
||||
"createSession": true,
|
||||
"sessionKey": "kova-gateway-session-send",
|
||||
"activeStartedAtEpochMs": 1777536000000,
|
||||
"activeFinishedAtEpochMs": 1777536001260,
|
||||
"activeTurnMs": 1260,
|
||||
"sessionCreateDurationMs": 120,
|
||||
"sendDurationMs": 80,
|
||||
"timeToFirstAssistantMs": 900,
|
||||
"timeToMatchedAssistantMs": 1260,
|
||||
"historyPollCount": 3,
|
||||
"historyErrorCount": 0
|
||||
},
|
||||
"turnDiagnostics": {
|
||||
"schemaVersion": "kova.activeTurnDiagnostics.v1",
|
||||
"metadataScan": {
|
||||
"count": 1,
|
||||
"totalDurationMs": 45,
|
||||
"maxDurationMs": 45
|
||||
},
|
||||
"eventLoop": {
|
||||
"sampleCount": 1,
|
||||
"maxMs": 12
|
||||
},
|
||||
"sessionPolling": {
|
||||
"pollCount": 3,
|
||||
"errorCount": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Gateway session turns also include pre-provider attribution when an OpenClaw
|
||||
diagnostics timeline is available. Kova clips `gateway.chat_send*`,
|
||||
`auto_reply*`, and `reply.*` spans to the active `sessions.send` pre-provider
|
||||
window and reports the unioned known time so overlapping spans are not counted
|
||||
twice. Provider work remains separate.
|
||||
|
||||
```json
|
||||
{
|
||||
"gatewaySessionPreProviderAttribution": {
|
||||
"schemaVersion": "kova.gatewaySessionPreProviderAttribution.v1",
|
||||
"available": true,
|
||||
"label": "cold",
|
||||
"timelineArtifacts": ["/tmp/kova/openclaw/timeline.jsonl"],
|
||||
"window": {
|
||||
"startEpochMs": 1777536000000,
|
||||
"endEpochMs": 1777536000200,
|
||||
"durationMs": 200
|
||||
},
|
||||
"provider": {
|
||||
"totalDurationMs": 600,
|
||||
"firstByteLatencyMs": 25,
|
||||
"firstChunkLatencyMs": 30
|
||||
},
|
||||
"knownAttributedMs": 170,
|
||||
"unattributedMs": 30,
|
||||
"coverageRatio": 0.85,
|
||||
"spanSummaries": [
|
||||
{
|
||||
"name": "auto_reply.finalize_context",
|
||||
"count": 1,
|
||||
"errorCount": 0,
|
||||
"totalClippedDurationMs": 100,
|
||||
"maxClippedDurationMs": 100
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Repeat summaries expose machine-readable medians at
|
||||
`records[*].measurements.gatewaySessionPreProviderAttribution` plus flat comparison
|
||||
metrics such as `coldPreProviderAttributedMs`,
|
||||
`coldPreProviderUnattributedMs`, `warmPreProviderAttributedMs`, and
|
||||
`warmPreProviderUnattributedMs`.
|
||||
|
||||
Aggregate fields are also exposed on `measurements` for comparison and
|
||||
performance summaries:
|
||||
|
||||
- `agentMetadataScanCount`
|
||||
- `agentMetadataScanTotalMs`
|
||||
- `agentMetadataScanMaxMs`
|
||||
- `agentEventLoopMaxMs`
|
||||
- `agentEventLoopSampleCount`
|
||||
- `agentSessionPollCount`
|
||||
- `agentSessionPollErrorCount`
|
||||
|
||||
## Health And Readiness
|
||||
|
||||
Health/readiness data lives under `records[*].measurements.health`:
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": "kova.health.v1",
|
||||
"readiness": {
|
||||
"phaseId": "cold-start",
|
||||
"listeningReadyAtMs": 2536,
|
||||
"healthReadyAtMs": 3005,
|
||||
"classification": "ready",
|
||||
"severity": "pass",
|
||||
"reason": "gateway became healthy within the readiness threshold",
|
||||
"thresholdMs": 30000,
|
||||
"deadlineMs": 120000,
|
||||
"attempts": 4
|
||||
},
|
||||
"startupSamples": {
|
||||
"scope": "startup-sample",
|
||||
"count": 4,
|
||||
"okCount": 1,
|
||||
"failureCount": 3,
|
||||
"p95Ms": 120,
|
||||
"maxMs": 120,
|
||||
"slowestPhaseId": "cold-start"
|
||||
},
|
||||
"postReadySamples": {
|
||||
"scope": "post-ready",
|
||||
"count": 9,
|
||||
"okCount": 9,
|
||||
"failureCount": 0,
|
||||
"p95Ms": 469,
|
||||
"maxMs": 652,
|
||||
"slowestPhaseId": "api-latency"
|
||||
},
|
||||
"unknownSamples": {
|
||||
"scope": "unknown",
|
||||
"count": 0,
|
||||
"okCount": 0,
|
||||
"failureCount": 0,
|
||||
"p95Ms": null,
|
||||
"maxMs": null,
|
||||
"slowestPhaseId": null
|
||||
},
|
||||
"final": {
|
||||
"scope": "final",
|
||||
"gatewayState": "running",
|
||||
"ok": true,
|
||||
"healthOk": true,
|
||||
"failureCount": 0,
|
||||
"p95Ms": 90,
|
||||
"maxMs": 90,
|
||||
"slowestPhaseId": "final"
|
||||
},
|
||||
"slowestSample": {
|
||||
"scope": "post-ready",
|
||||
"phaseId": "api-latency",
|
||||
"durationMs": 652
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Scenario phases declare `healthScope` so the evaluator does not infer meaning
|
||||
from phase ids. Allowed values are `readiness`, `startup-sample`, `post-ready`,
|
||||
`final`, and `none`. Reports do not emit old top-level readiness or health p95
|
||||
fields; readers should use the scoped health object directly.
|
||||
|
||||
Role-specific thresholds can fail a scenario separately from total process-tree
|
||||
thresholds. For example, a report can show that `gateway` exceeded memory while
|
||||
`package-manager` stayed normal, or that `package-manager` spiked during local
|
||||
@ -453,9 +276,8 @@ Aggregate metric fields include:
|
||||
- `samples`
|
||||
|
||||
Current aggregate metrics include startup readiness, TCP listening, RSS, CPU,
|
||||
event-loop delay, agent turn latency, agent metadata scan count/time, active
|
||||
turn event-loop max, session poll count, startup health p95, post-ready health
|
||||
p95, and runtime dependency staging.
|
||||
event-loop delay, agent turn latency, health p95, and runtime dependency
|
||||
staging.
|
||||
|
||||
Baseline stores use schema `kova.baselines.v1`. Baseline read/write requires
|
||||
`--execute` so stored evidence comes from real OpenClaw runs, not dry-run plans.
|
||||
@ -517,90 +339,33 @@ definitions, state fixture definitions, surface definitions, process-role
|
||||
definitions, profile summaries, platform metadata, and supports filtering with
|
||||
`--scenario`, `--state`, and `--profile`.
|
||||
|
||||
Every scenario must declare a `surface` and the requirement ids it proves.
|
||||
Registry validation fails before plan, run, or matrix output if a scenario
|
||||
references an unknown surface or requirement, a surface references an unknown
|
||||
process role, or a profile references an unknown scenario/state/requirement.
|
||||
Every scenario must declare a `surface`. Registry validation fails before plan,
|
||||
run, or matrix output if a scenario references an unknown surface, a surface
|
||||
references an unknown process role, or a profile references an unknown
|
||||
scenario/state/surface.
|
||||
|
||||
Every state must declare traits, risk area, owner area, setup evidence, and
|
||||
cleanup guarantees. Positive surface compatibility is owned by surface
|
||||
requirements. Registry validation rejects unknown state traits, unknown hard
|
||||
incompatibility references, and profile entries that pair a scenario with a
|
||||
state that does not satisfy the scenario's proved requirements.
|
||||
Every state must declare traits, compatible surfaces, incompatible surfaces,
|
||||
risk area, owner area, setup evidence, and cleanup guarantees. Registry
|
||||
validation rejects unknown state traits, unknown surface references, and profile
|
||||
entries that pair a scenario with a state that is not allowed for the scenario's
|
||||
surface.
|
||||
|
||||
Plan JSON includes `coverage`:
|
||||
|
||||
- `surfaces`: each surface with scenario count and mapped scenarios
|
||||
- `scenarioSurfaceMap`: direct scenario-to-surface mappings
|
||||
- `surfacesWithoutScenarios`: declared surfaces with no scenario yet
|
||||
- `profiles`: per-profile selected surfaces, scenarios, states, requirement
|
||||
coverage, derived required coverage, coverage gaps, state trait coverage,
|
||||
state/surface pairs, and trait/surface coverage
|
||||
|
||||
`kova matrix plan --json` also includes `resolvedCoverage`. This is the pre-run
|
||||
contract resolver for the selected profile, target, filters, scenarios, and
|
||||
states. It does not change execution reports. It lists planned obligations as
|
||||
surface requirement, scenario, state, target kind, status, required states,
|
||||
required state traits, required target kinds, and required metrics. Invalid
|
||||
obligations, such as a scenario proving an unknown requirement or a selected
|
||||
state that cannot satisfy the requirement, fail planning before execution.
|
||||
|
||||
`kova inventory plan --json` is planner-only and does not write a run report. It
|
||||
uses schema `kova.inventory.plan.v1` and includes:
|
||||
|
||||
- `sources`: whether OpenClaw help, package scripts, and manifests were scanned
|
||||
- `modeledSurfaces`: current Kova surfaces
|
||||
- `capabilities`: discovered CLI commands, product-relevant package scripts,
|
||||
plugin manifests, and extension manifests with matched Kova surface ids when
|
||||
known
|
||||
- `coverage.warnings`: unmodeled discovered capabilities
|
||||
- `coverage.ambiguous`: discovered capabilities that match multiple Kova
|
||||
surfaces
|
||||
- `coverage.blockers`: selected missing or unmodeled capabilities when
|
||||
`--require-modeled <capability>` is used
|
||||
|
||||
Inventory warnings are discovery signal first. They do not block release gates
|
||||
until a later policy deliberately promotes them.
|
||||
|
||||
Package-script discovery defaults to `--script-scope product`. Use
|
||||
`--script-scope all` to include every package script or `--script-scope none` to
|
||||
scan only CLI help and manifests.
|
||||
- `profiles`: per-profile selected surfaces, scenarios, states, required
|
||||
coverage, coverage gaps, state trait coverage, state/surface pairs, and
|
||||
trait/surface coverage
|
||||
|
||||
## Summary Output
|
||||
|
||||
Each run also writes `<run>.summary.json`. `kova report summarize
|
||||
<report.json> --json` prints the same compact agent-facing contract:
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": "kova.report.summary.v1",
|
||||
"decision": {
|
||||
"verdict": "FAIL",
|
||||
"reason": "gateway peak RSS 701.8 MB exceeded threshold 700 MB",
|
||||
"blockingFindingCount": 1,
|
||||
"warningFindingCount": 0
|
||||
},
|
||||
"run": {
|
||||
"repeat": 3,
|
||||
"parallel": 1,
|
||||
"auth": {}
|
||||
},
|
||||
"coverage": {
|
||||
"recordCount": 3,
|
||||
"scenarioCount": 1,
|
||||
"stateCount": 1
|
||||
},
|
||||
"findings": [],
|
||||
"groups": [],
|
||||
"samples": [],
|
||||
"artifacts": []
|
||||
}
|
||||
```
|
||||
|
||||
Agents should use the summary before reading the full report when they only
|
||||
need pass/fail, findings, aggregate performance, sample-level evidence, and
|
||||
artifact paths. The full `kova.report.v1` JSON remains the audit trail with raw
|
||||
records, phases, commands, and collector evidence.
|
||||
`kova report summarize <report.json> --json` returns a compact agent-facing
|
||||
view of each scenario with status, cleanup, failed command, concise failure
|
||||
reason, violations, and a small measurement summary. Agents should use this
|
||||
before reading the full report when they only need pass/fail and high-signal
|
||||
performance evidence.
|
||||
|
||||
When a report contains failures, the structured summary also includes
|
||||
`failureBrief` with:
|
||||
@ -663,11 +428,9 @@ the existing matrix runner and adds:
|
||||
{
|
||||
"schemaVersion": "kova.gate.v1",
|
||||
"enabled": true,
|
||||
"purpose": "release",
|
||||
"profileId": "release",
|
||||
"policyId": "openclaw-release",
|
||||
"verdict": "DO_NOT_SHIP",
|
||||
"outcome": "DO_NOT_SHIP",
|
||||
"ok": false,
|
||||
"complete": true,
|
||||
"partial": false,
|
||||
@ -682,9 +445,6 @@ the existing matrix runner and adds:
|
||||
"blocking": ["darwin-arm64"],
|
||||
"warning": ["linux-x64", "linux-arm64", "wsl2"]
|
||||
},
|
||||
"requirements": {
|
||||
"blocking": ["release-runtime-startup:baseline"]
|
||||
},
|
||||
"states": {
|
||||
"blocking": ["fresh"]
|
||||
},
|
||||
@ -715,20 +475,13 @@ Verdicts:
|
||||
- `BLOCKED`: Kova cannot make a ship/no-ship decision, usually because the run
|
||||
was not executed, skipped, or blocked by harness/provisioning behavior.
|
||||
|
||||
`outcome` is purpose-aware. For `release` gates it matches `verdict`. For
|
||||
non-release purposes, a passing complete gate reports `PASS`, a blocking
|
||||
OpenClaw failure reports `FAIL`, and incomplete or harness-blocked gates report
|
||||
`PARTIAL` or `BLOCKED`.
|
||||
|
||||
Filtered gate slices are partial. They can produce `DO_NOT_SHIP` when a selected
|
||||
blocking scenario fails, but they cannot produce `SHIP` because required gate
|
||||
coverage is missing. A passing filtered slice remains `PARTIAL`.
|
||||
|
||||
Release profiles define explicit platform coverage and requirement coverage
|
||||
using `surface:requirement` ids. Surface, scenario, state, trait, and
|
||||
state-surface coverage views are derived from resolved obligations for report
|
||||
compatibility. Missing blocking requirement/platform coverage prevents `SHIP`;
|
||||
missing warning coverage creates warning cards. Platform coverage keys include
|
||||
Release profiles may define explicit platform/surface/scenario/state/trait and
|
||||
state-surface coverage. Missing blocking coverage prevents `SHIP`; missing
|
||||
warning coverage creates warning cards. Platform coverage keys include
|
||||
`darwin-arm64`, `linux-x64`, `linux-arm64`, and `wsl2` where detectable.
|
||||
|
||||
Gate cards are concise fixer records. They include severity, scenario/state,
|
||||
@ -781,9 +534,7 @@ retained-artifacts.json
|
||||
Comparison currently detects status regressions, missing scenario/state entries,
|
||||
and increases in peak RSS, health failures, health p95, missing dependency
|
||||
errors, plugin load failures, metadata scan mentions, and config normalization
|
||||
mentions. It also reports group-level status changes and finding deltas before
|
||||
metric deltas, so a comparison can say which failures were resolved, which
|
||||
new findings appeared, and whether repeat-run pass/fail counts improved.
|
||||
mentions.
|
||||
|
||||
## Artifact Bundle
|
||||
|
||||
|
||||
@ -1,43 +0,0 @@
|
||||
# Kova Scenario Hierarchy
|
||||
|
||||
Kova measures OpenClaw behavior by runtime path, not by whichever client first
|
||||
exposed that path.
|
||||
|
||||
## Gateway Session
|
||||
|
||||
`gateway-session-send-turn` is the shared Gateway session messaging benchmark.
|
||||
It calls `sessions.create`, `sessions.send`, and `chat.history` directly over
|
||||
Gateway RPC with mock auth/provider. Use it for the core Control UI, dashboard,
|
||||
and channel message path.
|
||||
|
||||
This scenario owns:
|
||||
|
||||
- cold and warm Gateway session turn timing
|
||||
- pre-provider, provider, and post-provider timing
|
||||
- Gateway session diagnostics timeline attribution
|
||||
- Gateway process RSS/CPU as the primary product resource signal
|
||||
- helper process cost under `gateway-session-client`
|
||||
|
||||
## Dashboard
|
||||
|
||||
`dashboard-readiness` is dashboard-specific coverage. It verifies the dashboard
|
||||
command, URL/websocket entry, and post-dashboard Gateway health. It does not own
|
||||
the shared message-turn benchmark.
|
||||
|
||||
## Channels
|
||||
|
||||
Channel scenarios should prove adapter behavior: the channel can create or
|
||||
route a user turn into the shared Gateway session path and recover/report
|
||||
adapter-specific failures. They should compare against `gateway-session-send-turn`
|
||||
instead of duplicating the core Gateway session benchmark.
|
||||
|
||||
## Agent CLI
|
||||
|
||||
Agent CLI scenarios measure CLI client behavior separately:
|
||||
|
||||
- `agent-cli-local-turn` covers short-lived local agent CLI turns.
|
||||
- `agent-gateway-rpc-turn` covers `openclaw agent` as a CLI client crossing the
|
||||
Gateway agent RPC path.
|
||||
|
||||
CLI process cost should stay on CLI roles. It should not be used as the primary
|
||||
acceptance signal for Control UI, dashboard, or channel users.
|
||||
@ -5,17 +5,10 @@
|
||||
"agentCleanupMs",
|
||||
"agentColdWarmDeltaMs",
|
||||
"agentContainmentHealthFailures",
|
||||
"agentEventLoopMaxMs",
|
||||
"agentEventLoopSampleCount",
|
||||
"agentMetadataScanCount",
|
||||
"agentMetadataScanMaxMs",
|
||||
"agentMetadataScanTotalMs",
|
||||
"agentPreProviderMaxMs",
|
||||
"agentPreProviderP95Ms",
|
||||
"agentProcessLeaks",
|
||||
"agentProviderFinalP95Ms",
|
||||
"agentSessionPollCount",
|
||||
"agentSessionPollErrorCount",
|
||||
"agentTurnMaxMs",
|
||||
"agentTurnMs",
|
||||
"agentTurnP95Ms",
|
||||
@ -29,10 +22,7 @@
|
||||
"browserTabCountMin",
|
||||
"browserTabsMs",
|
||||
"coldAgentTurnMs",
|
||||
"coldPreProviderAttributedMs",
|
||||
"coldPreProviderAttributionCoverage",
|
||||
"coldPreProviderMs",
|
||||
"coldPreProviderUnattributedMs",
|
||||
"coldReadyMs",
|
||||
"coldRuntimeDepsStagingMs",
|
||||
"coldWarmDeltaMs",
|
||||
@ -40,13 +30,14 @@
|
||||
"diagnosticPresent",
|
||||
"doctorFixMs",
|
||||
"eventLoopMaxMs",
|
||||
"finalHealthFailures",
|
||||
"gatewayReadyHardTimeoutMs",
|
||||
"gatewayReadyMs",
|
||||
"gatewayResponsive",
|
||||
"gatewayRssGrowthMb",
|
||||
"gatewaySurvives",
|
||||
"healthFailures",
|
||||
"healthMs",
|
||||
"healthP95Ms",
|
||||
"inputLagMs",
|
||||
"maxCpuPercent",
|
||||
"mediaDescribeMs",
|
||||
@ -67,14 +58,9 @@
|
||||
"networkTurnMs",
|
||||
"openclawSlowestSpanMs",
|
||||
"openclawTimelineParseErrors",
|
||||
"officialPluginInstallOk",
|
||||
"officialPluginSecurityBlocks",
|
||||
"peakRssMb",
|
||||
"pluginIndexPresent",
|
||||
"pluginInstallMs",
|
||||
"pluginLoadFailures",
|
||||
"postReadyHealthFailures",
|
||||
"postReadyHealthP95Ms",
|
||||
"pluginUpdateDryRunMs",
|
||||
"pluginsListMs",
|
||||
"preProviderDominanceRatio",
|
||||
@ -85,8 +71,6 @@
|
||||
"providerRequestCountMin",
|
||||
"providerSlowMinMs",
|
||||
"providerTimeoutMentions",
|
||||
"readinessHealthReadyMs",
|
||||
"readinessListeningMs",
|
||||
"restartCount",
|
||||
"restartReadyMs",
|
||||
"rssGrowthMb",
|
||||
@ -102,16 +86,11 @@
|
||||
"statusAfterFailureMs",
|
||||
"statusAfterModelsMs",
|
||||
"statusMs",
|
||||
"startupHealthFailures",
|
||||
"startupHealthP95Ms",
|
||||
"syncFsStallDetected",
|
||||
"tuiSmokeMs",
|
||||
"upgradeMs",
|
||||
"warmAgentTurnMs",
|
||||
"warmPreProviderAttributedMs",
|
||||
"warmPreProviderAttributionCoverage",
|
||||
"warmPreProviderMs",
|
||||
"warmPreProviderUnattributedMs",
|
||||
"warmReadyMs",
|
||||
"warmRuntimeDepsRestageCount",
|
||||
"warmRuntimeDepsStagingMs",
|
||||
|
||||
@ -2,6 +2,6 @@
|
||||
"id": "dashboard-cli",
|
||||
"title": "Dashboard CLI",
|
||||
"description": "OpenClaw dashboard command paths that produce or validate dashboard URLs and websocket configuration.",
|
||||
"commandPatterns": ["dashboard", "dashboard --no-open"],
|
||||
"processPatterns": ["openclaw.*dashboard"]
|
||||
"commandPatterns": ["dashboard", "dashboard --no-open", "run-dashboard-session-send-turn.mjs", "sessions.send"],
|
||||
"processPatterns": ["openclaw.*dashboard", "node\\s+.*run-dashboard-session-send-turn\\.mjs"]
|
||||
}
|
||||
|
||||
@ -1,7 +0,0 @@
|
||||
{
|
||||
"id": "gateway-session-client",
|
||||
"title": "Gateway Session Client",
|
||||
"description": "Kova helper process that drives the shared Gateway sessions.create, sessions.send, and chat.history path directly over Gateway RPC.",
|
||||
"commandPatterns": ["run-gateway-session-send-turn.mjs", "sessions.create", "sessions.send", "chat.history"],
|
||||
"processPatterns": ["node\\s+.*run-gateway-session-send-turn\\.mjs"]
|
||||
}
|
||||
@ -2,15 +2,8 @@
|
||||
"id": "channel-upgrade",
|
||||
"title": "Channel Upgrade Matrix",
|
||||
"objective": "Focused OpenClaw channel upgrade coverage for stable-to-beta transitions through real OCM-managed envs.",
|
||||
"targetKinds": [
|
||||
"channel"
|
||||
],
|
||||
"targetKinds": ["channel"],
|
||||
"entries": [
|
||||
{
|
||||
"scenario": "upgrade-stable-channel-to-beta",
|
||||
"state": "stable-channel-user",
|
||||
"timeoutMs": 240000
|
||||
}
|
||||
],
|
||||
"purpose": "upgrade"
|
||||
{ "scenario": "upgrade-stable-channel-to-beta", "state": "stable-channel-user", "timeoutMs": 240000 }
|
||||
]
|
||||
}
|
||||
|
||||
@ -2,14 +2,10 @@
|
||||
"id": "diagnostic",
|
||||
"title": "Source-Built Diagnostics",
|
||||
"objective": "Run release-shaped local OpenClaw builds with timeline diagnostics enabled so Kova can attribute startup, plugin, provider, agent, event-loop, CPU, and heap behavior to OpenClaw phases.",
|
||||
"targetKinds": [
|
||||
"local-build"
|
||||
],
|
||||
"targetKinds": ["local-build"],
|
||||
"diagnostics": {
|
||||
"timelineRequired": true,
|
||||
"timelineRequiredForTargetKinds": [
|
||||
"local-build"
|
||||
],
|
||||
"timelineRequiredForTargetKinds": ["local-build"],
|
||||
"requiredKeySpans": [
|
||||
"gateway.startup",
|
||||
"gateway.ready",
|
||||
@ -24,30 +20,12 @@
|
||||
},
|
||||
"calibration": {
|
||||
"roles": {
|
||||
"gateway": {
|
||||
"peakRssMb": 1000,
|
||||
"maxCpuPercent": 350
|
||||
},
|
||||
"command-tree": {
|
||||
"peakRssMb": 1400,
|
||||
"maxCpuPercent": 450
|
||||
},
|
||||
"runtime-staging": {
|
||||
"peakRssMb": 900,
|
||||
"maxCpuPercent": 350
|
||||
},
|
||||
"package-manager": {
|
||||
"peakRssMb": 900,
|
||||
"maxCpuPercent": 350
|
||||
},
|
||||
"agent-cli": {
|
||||
"peakRssMb": 1200,
|
||||
"maxCpuPercent": 350
|
||||
},
|
||||
"mock-provider": {
|
||||
"peakRssMb": 300,
|
||||
"maxCpuPercent": 150
|
||||
}
|
||||
"gateway": { "peakRssMb": 1000, "maxCpuPercent": 350 },
|
||||
"command-tree": { "peakRssMb": 1400, "maxCpuPercent": 450 },
|
||||
"runtime-staging": { "peakRssMb": 900, "maxCpuPercent": 350 },
|
||||
"package-manager": { "peakRssMb": 900, "maxCpuPercent": 350 },
|
||||
"agent-cli": { "peakRssMb": 1200, "maxCpuPercent": 350 },
|
||||
"mock-provider": { "peakRssMb": 300, "maxCpuPercent": 150 }
|
||||
},
|
||||
"surfaces": {
|
||||
"release-runtime-startup": {
|
||||
@ -78,7 +56,7 @@
|
||||
"openclawSlowestSpanMs": 45000
|
||||
}
|
||||
},
|
||||
"gateway-session-send-turn": {
|
||||
"dashboard-session-send-turn": {
|
||||
"thresholds": {
|
||||
"agentTurnMs": 60000,
|
||||
"preProviderMs": 15000,
|
||||
@ -115,7 +93,7 @@
|
||||
"gateway-performance": {
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 60000,
|
||||
"postReadyHealthP95Ms": 1000,
|
||||
"healthP95Ms": 1000,
|
||||
"peakRssMb": 1000,
|
||||
"openclawSlowestSpanMs": 45000
|
||||
}
|
||||
@ -125,52 +103,37 @@
|
||||
"gate": {
|
||||
"id": "openclaw-diagnostic",
|
||||
"coverage": {
|
||||
"requirements": {
|
||||
"surfaces": {
|
||||
"blocking": ["release-runtime-startup", "gateway-performance", "bundled-runtime-deps", "agent-cli-local-turn", "agent-gateway-rpc-turn", "dashboard-session-send-turn", "tui-message-turn", "openai-compatible-turn"]
|
||||
},
|
||||
"states": {
|
||||
"blocking": ["fresh", "missing-plugin-index", "many-bundled-plugins", "mock-openai-provider"]
|
||||
},
|
||||
"stateSurfaces": {
|
||||
"blocking": [
|
||||
"release-runtime-startup:baseline",
|
||||
"gateway-performance:baseline",
|
||||
"bundled-runtime-deps:baseline",
|
||||
"agent-cli-local-turn:baseline",
|
||||
"agent-gateway-rpc-turn:baseline",
|
||||
"gateway-session-send-turn:baseline",
|
||||
"tui-message-turn:baseline",
|
||||
"openai-compatible-turn:baseline"
|
||||
"release-runtime-startup:fresh",
|
||||
"gateway-performance:many-bundled-plugins",
|
||||
"bundled-runtime-deps:missing-plugin-index",
|
||||
"agent-cli-local-turn:mock-openai-provider",
|
||||
"agent-gateway-rpc-turn:mock-openai-provider",
|
||||
"dashboard-session-send-turn:mock-openai-provider",
|
||||
"tui-message-turn:mock-openai-provider",
|
||||
"openai-compatible-turn:mock-openai-provider"
|
||||
]
|
||||
},
|
||||
"scenarios": {
|
||||
"blocking": ["release-runtime-startup", "gateway-performance", "bundled-runtime-deps", "agent-cold-warm-message", "agent-gateway-rpc-turn", "dashboard-session-send-turn", "tui-message-turn", "openai-compatible-turn"]
|
||||
}
|
||||
},
|
||||
"blocking": [
|
||||
{
|
||||
"scenario": "release-runtime-startup",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-performance",
|
||||
"state": "many-bundled-plugins"
|
||||
},
|
||||
{
|
||||
"scenario": "bundled-runtime-deps",
|
||||
"state": "missing-plugin-index"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-cold-warm-message",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-gateway-rpc-turn",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-session-send-turn",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "tui-message-turn",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "openai-compatible-turn",
|
||||
"state": "mock-openai-provider"
|
||||
}
|
||||
{ "scenario": "release-runtime-startup", "state": "fresh" },
|
||||
{ "scenario": "gateway-performance", "state": "many-bundled-plugins" },
|
||||
{ "scenario": "bundled-runtime-deps", "state": "missing-plugin-index" },
|
||||
{ "scenario": "agent-cold-warm-message", "state": "mock-openai-provider" },
|
||||
{ "scenario": "agent-gateway-rpc-turn", "state": "mock-openai-provider" },
|
||||
{ "scenario": "dashboard-session-send-turn", "state": "mock-openai-provider" },
|
||||
{ "scenario": "tui-message-turn", "state": "mock-openai-provider" },
|
||||
{ "scenario": "openai-compatible-turn", "state": "mock-openai-provider" }
|
||||
]
|
||||
},
|
||||
"entries": [
|
||||
@ -200,7 +163,7 @@
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-session-send-turn",
|
||||
"scenario": "dashboard-session-send-turn",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
@ -214,6 +177,5 @@
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
}
|
||||
],
|
||||
"purpose": "diagnostic"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,236 +3,56 @@
|
||||
"title": "Exhaustive Matrix",
|
||||
"objective": "Maximum local OpenClaw coverage across install, upgrade, plugins, providers, config corruption, filesystem pressure, service recovery, and performance scenarios.",
|
||||
"entries": [
|
||||
{
|
||||
"scenario": "fresh-install",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "fresh-install",
|
||||
"state": "onboarded-user"
|
||||
},
|
||||
{
|
||||
"scenario": "fresh-install",
|
||||
"state": "plugin-index"
|
||||
},
|
||||
{
|
||||
"scenario": "fresh-install",
|
||||
"state": "old-config-keys"
|
||||
},
|
||||
{
|
||||
"scenario": "fresh-install",
|
||||
"state": "large-memory-session"
|
||||
},
|
||||
{
|
||||
"scenario": "fresh-install",
|
||||
"state": "corrupted-config"
|
||||
},
|
||||
{
|
||||
"scenario": "release-runtime-startup",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "upgrade-existing-user",
|
||||
"state": "old-release-user",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "upgrade-existing-user",
|
||||
"state": "failed-upgrade",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "bundled-runtime-deps",
|
||||
"state": "missing-plugin-index"
|
||||
},
|
||||
{
|
||||
"scenario": "bundled-runtime-deps",
|
||||
"state": "stale-runtime-deps"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-lifecycle",
|
||||
"state": "plugin-index"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-lifecycle",
|
||||
"state": "external-plugin"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-lifecycle",
|
||||
"state": "broken-plugin-deps"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-external-install",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "official-plugin-install",
|
||||
"state": "official-plugins",
|
||||
"timeoutMs": 240000
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-remove",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-update",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-bad-manifest",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-missing-runtime-deps",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "bundled-plugin-startup",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "provider-models",
|
||||
"state": "model-auth-configured"
|
||||
},
|
||||
{
|
||||
"scenario": "provider-models",
|
||||
"state": "model-auth-missing"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-cold-warm-message",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "agent-gateway-rpc-turn",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-session-send-turn",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "tui-message-turn",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "openai-compatible-turn",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "agent-auth-missing",
|
||||
"state": "agent-auth-missing",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "agent-long-session",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 360000
|
||||
},
|
||||
{
|
||||
"scenario": "agent-provider-slow",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "agent-provider-timeout",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "agent-provider-malformed",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "agent-provider-streaming-stall",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "agent-provider-concurrent",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 240000
|
||||
},
|
||||
{
|
||||
"scenario": "agent-provider-recovery",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 240000
|
||||
},
|
||||
{
|
||||
"scenario": "dashboard-readiness",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "tui-responsiveness",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "mcp-runtime-start-stop",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "browser-automation-smoke",
|
||||
"state": "fresh",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "media-understanding-timeout",
|
||||
"state": "fresh",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "agent-network-offline",
|
||||
"state": "fresh",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-performance",
|
||||
"state": "many-bundled-plugins"
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-performance",
|
||||
"state": "gateway-already-running"
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-performance",
|
||||
"state": "stale-service-state"
|
||||
},
|
||||
{
|
||||
"scenario": "failure-injection",
|
||||
"state": "broken-plugin-deps"
|
||||
},
|
||||
{
|
||||
"scenario": "failure-injection",
|
||||
"state": "corrupted-config"
|
||||
},
|
||||
{
|
||||
"scenario": "soak",
|
||||
"state": "large-workspace",
|
||||
"timeoutMs": 240000
|
||||
},
|
||||
{
|
||||
"scenario": "workspace-scan-pressure",
|
||||
"state": "large-workspace",
|
||||
"timeoutMs": 240000
|
||||
},
|
||||
{
|
||||
"scenario": "soak",
|
||||
"state": "large-memory-session",
|
||||
"timeoutMs": 240000
|
||||
},
|
||||
{
|
||||
"scenario": "cross-platform-smoke",
|
||||
"state": "slow-filesystem"
|
||||
},
|
||||
{
|
||||
"scenario": "cross-platform-smoke",
|
||||
"state": "channel-configured"
|
||||
}
|
||||
],
|
||||
"purpose": "regression"
|
||||
{ "scenario": "fresh-install", "state": "fresh" },
|
||||
{ "scenario": "fresh-install", "state": "onboarded-user" },
|
||||
{ "scenario": "fresh-install", "state": "plugin-index" },
|
||||
{ "scenario": "fresh-install", "state": "old-config-keys" },
|
||||
{ "scenario": "fresh-install", "state": "large-memory-session" },
|
||||
{ "scenario": "fresh-install", "state": "corrupted-config" },
|
||||
{ "scenario": "release-runtime-startup", "state": "fresh" },
|
||||
{ "scenario": "upgrade-existing-user", "state": "old-release-user", "timeoutMs": 180000 },
|
||||
{ "scenario": "upgrade-existing-user", "state": "failed-upgrade", "timeoutMs": 180000 },
|
||||
{ "scenario": "bundled-runtime-deps", "state": "missing-plugin-index" },
|
||||
{ "scenario": "bundled-runtime-deps", "state": "stale-runtime-deps" },
|
||||
{ "scenario": "plugin-lifecycle", "state": "plugin-index" },
|
||||
{ "scenario": "plugin-lifecycle", "state": "external-plugin" },
|
||||
{ "scenario": "plugin-lifecycle", "state": "broken-plugin-deps" },
|
||||
{ "scenario": "plugin-external-install", "state": "fresh" },
|
||||
{ "scenario": "plugin-remove", "state": "fresh" },
|
||||
{ "scenario": "plugin-update", "state": "fresh" },
|
||||
{ "scenario": "plugin-bad-manifest", "state": "fresh" },
|
||||
{ "scenario": "plugin-missing-runtime-deps", "state": "fresh" },
|
||||
{ "scenario": "bundled-plugin-startup", "state": "fresh" },
|
||||
{ "scenario": "provider-models", "state": "model-auth-configured" },
|
||||
{ "scenario": "provider-models", "state": "model-auth-missing" },
|
||||
{ "scenario": "agent-cold-warm-message", "state": "mock-openai-provider", "timeoutMs": 180000 },
|
||||
{ "scenario": "agent-gateway-rpc-turn", "state": "mock-openai-provider", "timeoutMs": 180000 },
|
||||
{ "scenario": "dashboard-session-send-turn", "state": "mock-openai-provider", "timeoutMs": 180000 },
|
||||
{ "scenario": "tui-message-turn", "state": "mock-openai-provider", "timeoutMs": 180000 },
|
||||
{ "scenario": "openai-compatible-turn", "state": "mock-openai-provider", "timeoutMs": 180000 },
|
||||
{ "scenario": "agent-auth-missing", "state": "agent-auth-missing", "timeoutMs": 180000 },
|
||||
{ "scenario": "agent-long-session", "state": "mock-openai-provider", "timeoutMs": 360000 },
|
||||
{ "scenario": "agent-provider-slow", "state": "mock-openai-provider", "timeoutMs": 180000 },
|
||||
{ "scenario": "agent-provider-timeout", "state": "mock-openai-provider", "timeoutMs": 180000 },
|
||||
{ "scenario": "agent-provider-malformed", "state": "mock-openai-provider", "timeoutMs": 180000 },
|
||||
{ "scenario": "agent-provider-streaming-stall", "state": "mock-openai-provider", "timeoutMs": 180000 },
|
||||
{ "scenario": "agent-provider-concurrent", "state": "mock-openai-provider", "timeoutMs": 240000 },
|
||||
{ "scenario": "agent-provider-recovery", "state": "mock-openai-provider", "timeoutMs": 240000 },
|
||||
{ "scenario": "dashboard-readiness", "state": "fresh" },
|
||||
{ "scenario": "tui-responsiveness", "state": "fresh" },
|
||||
{ "scenario": "mcp-runtime-start-stop", "state": "fresh" },
|
||||
{ "scenario": "browser-automation-smoke", "state": "fresh", "timeoutMs": 180000 },
|
||||
{ "scenario": "media-understanding-timeout", "state": "fresh", "timeoutMs": 180000 },
|
||||
{ "scenario": "agent-network-offline", "state": "fresh", "timeoutMs": 180000 },
|
||||
{ "scenario": "gateway-performance", "state": "many-bundled-plugins" },
|
||||
{ "scenario": "gateway-performance", "state": "gateway-already-running" },
|
||||
{ "scenario": "gateway-performance", "state": "stale-service-state" },
|
||||
{ "scenario": "failure-injection", "state": "broken-plugin-deps" },
|
||||
{ "scenario": "failure-injection", "state": "corrupted-config" },
|
||||
{ "scenario": "soak", "state": "large-workspace", "timeoutMs": 240000 },
|
||||
{ "scenario": "workspace-scan-pressure", "state": "large-workspace", "timeoutMs": 240000 },
|
||||
{ "scenario": "soak", "state": "large-memory-session", "timeoutMs": 240000 },
|
||||
{ "scenario": "cross-platform-smoke", "state": "slow-filesystem" },
|
||||
{ "scenario": "cross-platform-smoke", "state": "channel-configured" }
|
||||
]
|
||||
}
|
||||
|
||||
@ -2,25 +2,10 @@
|
||||
"id": "local-build-upgrade",
|
||||
"title": "Local Build Upgrade Matrix",
|
||||
"objective": "Focused OpenClaw upgrade coverage for release-shaped local builds against stable-channel and cloned existing-user state.",
|
||||
"targetKinds": [
|
||||
"local-build"
|
||||
],
|
||||
"targetKinds": ["local-build"],
|
||||
"entries": [
|
||||
{
|
||||
"scenario": "upgrade-stable-channel-to-local-build",
|
||||
"state": "stable-channel-user",
|
||||
"timeoutMs": 300000
|
||||
},
|
||||
{
|
||||
"scenario": "upgrade-durable-clone-to-local-build",
|
||||
"state": "onboarded-user",
|
||||
"timeoutMs": 300000
|
||||
},
|
||||
{
|
||||
"scenario": "upgrade-durable-clone-to-local-build",
|
||||
"state": "plugin-index",
|
||||
"timeoutMs": 300000
|
||||
}
|
||||
],
|
||||
"purpose": "upgrade"
|
||||
{ "scenario": "upgrade-stable-channel-to-local-build", "state": "stable-channel-user", "timeoutMs": 300000 },
|
||||
{ "scenario": "upgrade-durable-clone-to-local-build", "state": "onboarded-user", "timeoutMs": 300000 },
|
||||
{ "scenario": "upgrade-durable-clone-to-local-build", "state": "plugin-index", "timeoutMs": 300000 }
|
||||
]
|
||||
}
|
||||
|
||||
@ -1,30 +0,0 @@
|
||||
{
|
||||
"id": "official-plugins",
|
||||
"title": "Official Plugin Matrix",
|
||||
"objective": "Focused OpenClaw coverage for published official plugin install paths that users run from terminals.",
|
||||
"gate": {
|
||||
"id": "openclaw-official-plugins",
|
||||
"coverage": {
|
||||
"requirements": {
|
||||
"blocking": [
|
||||
"official-plugin-install:baseline"
|
||||
]
|
||||
}
|
||||
},
|
||||
"blocking": [
|
||||
{
|
||||
"scenario": "official-plugin-install",
|
||||
"state": "official-plugins"
|
||||
}
|
||||
],
|
||||
"warning": []
|
||||
},
|
||||
"entries": [
|
||||
{
|
||||
"scenario": "official-plugin-install",
|
||||
"state": "official-plugins",
|
||||
"timeoutMs": 240000
|
||||
}
|
||||
],
|
||||
"purpose": "plugin"
|
||||
}
|
||||
@ -4,70 +4,21 @@
|
||||
"objective": "Broad OpenClaw release confidence across install, upgrade, bundled plugins, model/provider, UI, failure, soak, and platform smoke scenarios.",
|
||||
"calibration": {
|
||||
"roles": {
|
||||
"gateway": {
|
||||
"peakRssMb": 900,
|
||||
"maxCpuPercent": 300
|
||||
},
|
||||
"command-tree": {
|
||||
"peakRssMb": 1200,
|
||||
"maxCpuPercent": 400
|
||||
},
|
||||
"runtime-management": {
|
||||
"peakRssMb": 900,
|
||||
"maxCpuPercent": 350
|
||||
},
|
||||
"package-manager": {
|
||||
"peakRssMb": 900,
|
||||
"maxCpuPercent": 350
|
||||
},
|
||||
"agent-cli": {
|
||||
"peakRssMb": 1100,
|
||||
"maxCpuPercent": 350
|
||||
},
|
||||
"agent-process": {
|
||||
"peakRssMb": 900,
|
||||
"maxCpuPercent": 300
|
||||
},
|
||||
"plugin-cli": {
|
||||
"peakRssMb": 650,
|
||||
"maxCpuPercent": 250
|
||||
},
|
||||
"model-cli": {
|
||||
"peakRssMb": 650,
|
||||
"maxCpuPercent": 250
|
||||
},
|
||||
"doctor-cli": {
|
||||
"peakRssMb": 700,
|
||||
"maxCpuPercent": 300
|
||||
},
|
||||
"tui-cli": {
|
||||
"peakRssMb": 650,
|
||||
"maxCpuPercent": 250
|
||||
},
|
||||
"dashboard-cli": {
|
||||
"peakRssMb": 650,
|
||||
"maxCpuPercent": 250
|
||||
},
|
||||
"gateway-session-client": {
|
||||
"peakRssMb": 650,
|
||||
"maxCpuPercent": 250
|
||||
},
|
||||
"openai-compatible-client": {
|
||||
"peakRssMb": 650,
|
||||
"maxCpuPercent": 250
|
||||
},
|
||||
"mcp-runtime": {
|
||||
"peakRssMb": 500,
|
||||
"maxCpuPercent": 200
|
||||
},
|
||||
"browser-sidecar": {
|
||||
"peakRssMb": 700,
|
||||
"maxCpuPercent": 250
|
||||
},
|
||||
"mock-provider": {
|
||||
"peakRssMb": 300,
|
||||
"maxCpuPercent": 150
|
||||
}
|
||||
"gateway": { "peakRssMb": 900, "maxCpuPercent": 300 },
|
||||
"command-tree": { "peakRssMb": 1200, "maxCpuPercent": 400 },
|
||||
"runtime-management": { "peakRssMb": 900, "maxCpuPercent": 350 },
|
||||
"package-manager": { "peakRssMb": 900, "maxCpuPercent": 350 },
|
||||
"agent-cli": { "peakRssMb": 1100, "maxCpuPercent": 350 },
|
||||
"agent-process": { "peakRssMb": 900, "maxCpuPercent": 300 },
|
||||
"plugin-cli": { "peakRssMb": 650, "maxCpuPercent": 250 },
|
||||
"model-cli": { "peakRssMb": 650, "maxCpuPercent": 250 },
|
||||
"doctor-cli": { "peakRssMb": 700, "maxCpuPercent": 300 },
|
||||
"tui-cli": { "peakRssMb": 650, "maxCpuPercent": 250 },
|
||||
"dashboard-cli": { "peakRssMb": 650, "maxCpuPercent": 250 },
|
||||
"openai-compatible-client": { "peakRssMb": 650, "maxCpuPercent": 250 },
|
||||
"mcp-runtime": { "peakRssMb": 500, "maxCpuPercent": 200 },
|
||||
"browser-sidecar": { "peakRssMb": 700, "maxCpuPercent": 250 },
|
||||
"mock-provider": { "peakRssMb": 300, "maxCpuPercent": 150 }
|
||||
},
|
||||
"surfaces": {
|
||||
"release-runtime-startup": {
|
||||
@ -108,7 +59,7 @@
|
||||
"agentProcessLeaks": 0
|
||||
}
|
||||
},
|
||||
"gateway-session-send-turn": {
|
||||
"dashboard-session-send-turn": {
|
||||
"thresholds": {
|
||||
"agentTurnMs": 45000,
|
||||
"preProviderMs": 10000,
|
||||
@ -141,16 +92,6 @@
|
||||
"pluginLoadFailures": 0
|
||||
}
|
||||
},
|
||||
"official-plugin-install": {
|
||||
"thresholds": {
|
||||
"pluginInstallMs": 120000,
|
||||
"officialPluginInstallOk": 1,
|
||||
"officialPluginSecurityBlocks": 0,
|
||||
"pluginsListMs": 10000,
|
||||
"missingDependencyErrors": 0,
|
||||
"pluginLoadFailures": 0
|
||||
}
|
||||
},
|
||||
"soak": {
|
||||
"thresholds": {
|
||||
"soakMinDurationMs": 60000,
|
||||
@ -221,221 +162,152 @@
|
||||
"id": "openclaw-release",
|
||||
"coverage": {
|
||||
"platforms": {
|
||||
"blocking": ["darwin-arm64"],
|
||||
"warning": ["linux-x64", "linux-arm64", "wsl2"]
|
||||
},
|
||||
"states": {
|
||||
"blocking": [
|
||||
"darwin-arm64"
|
||||
],
|
||||
"warning": [
|
||||
"linux-x64",
|
||||
"linux-arm64",
|
||||
"wsl2"
|
||||
"fresh",
|
||||
"onboarded-user",
|
||||
"old-release-user",
|
||||
"old-release-2026-4-20-user",
|
||||
"old-release-2026-4-24-user",
|
||||
"plugin-index",
|
||||
"many-bundled-plugins"
|
||||
]
|
||||
},
|
||||
"requirements": {
|
||||
"traits": {
|
||||
"blocking": ["fresh-user", "existing-user", "old-release", "plugin-pressure", "provider-pressure"],
|
||||
"warning": ["filesystem-pressure", "memory-pressure", "failure-state"]
|
||||
},
|
||||
"stateSurfaces": {
|
||||
"blocking": [
|
||||
"release-runtime-startup:baseline",
|
||||
"fresh-install:baseline",
|
||||
"upgrade-existing-user:baseline",
|
||||
"bundled-runtime-deps:baseline",
|
||||
"plugin-lifecycle:baseline",
|
||||
"plugin-external-install:baseline",
|
||||
"official-plugin-install:baseline",
|
||||
"plugin-remove:baseline",
|
||||
"plugin-update:baseline",
|
||||
"plugin-bad-manifest:baseline",
|
||||
"plugin-missing-runtime-deps:baseline",
|
||||
"bundled-plugin-startup:baseline",
|
||||
"provider-models:baseline",
|
||||
"agent-cli-local-turn:baseline",
|
||||
"agent-gateway-rpc-turn:baseline",
|
||||
"gateway-session-send-turn:baseline",
|
||||
"tui-message-turn:baseline",
|
||||
"openai-compatible-turn:baseline",
|
||||
"dashboard:baseline",
|
||||
"tui:baseline",
|
||||
"gateway-performance:baseline"
|
||||
"release-runtime-startup:fresh",
|
||||
"fresh-install:fresh",
|
||||
"fresh-install:onboarded-user",
|
||||
"upgrade-existing-user:old-release-user",
|
||||
"upgrade-existing-user:old-release-2026-4-20-user",
|
||||
"upgrade-existing-user:old-release-2026-4-24-user",
|
||||
"bundled-runtime-deps:missing-plugin-index",
|
||||
"plugin-lifecycle:plugin-index",
|
||||
"plugin-lifecycle:external-plugin",
|
||||
"gateway-performance:many-bundled-plugins",
|
||||
"agent-cli-local-turn:mock-openai-provider",
|
||||
"agent-gateway-rpc-turn:mock-openai-provider",
|
||||
"dashboard-session-send-turn:mock-openai-provider",
|
||||
"tui-message-turn:mock-openai-provider",
|
||||
"openai-compatible-turn:mock-openai-provider",
|
||||
"provider-models:model-auth-missing",
|
||||
"dashboard:fresh",
|
||||
"tui:fresh"
|
||||
],
|
||||
"warning": [
|
||||
"failure-containment:baseline",
|
||||
"soak:baseline",
|
||||
"workspace-scan:baseline",
|
||||
"mcp-runtime:baseline",
|
||||
"browser-automation:baseline",
|
||||
"media-understanding:baseline",
|
||||
"network-offline:baseline",
|
||||
"cross-platform-smoke:baseline"
|
||||
"failure-containment:broken-plugin-deps",
|
||||
"soak:large-workspace",
|
||||
"workspace-scan:large-workspace",
|
||||
"mcp-runtime:fresh",
|
||||
"browser-automation:fresh",
|
||||
"media-understanding:fresh",
|
||||
"network-offline:fresh",
|
||||
"cross-platform-smoke:slow-filesystem"
|
||||
]
|
||||
},
|
||||
"surfaces": {
|
||||
"blocking": [
|
||||
"release-runtime-startup",
|
||||
"fresh-install",
|
||||
"upgrade-existing-user",
|
||||
"bundled-runtime-deps",
|
||||
"plugin-lifecycle",
|
||||
"plugin-external-install",
|
||||
"plugin-remove",
|
||||
"plugin-update",
|
||||
"plugin-bad-manifest",
|
||||
"plugin-missing-runtime-deps",
|
||||
"bundled-plugin-startup",
|
||||
"provider-models",
|
||||
"agent-cli-local-turn",
|
||||
"agent-gateway-rpc-turn",
|
||||
"dashboard-session-send-turn",
|
||||
"tui-message-turn",
|
||||
"openai-compatible-turn",
|
||||
"dashboard",
|
||||
"tui",
|
||||
"gateway-performance"
|
||||
],
|
||||
"warning": ["failure-containment", "soak", "workspace-scan", "mcp-runtime", "browser-automation", "media-understanding", "network-offline", "cross-platform-smoke"]
|
||||
},
|
||||
"scenarios": {
|
||||
"blocking": [
|
||||
"release-runtime-startup",
|
||||
"fresh-install",
|
||||
"upgrade-existing-user",
|
||||
"upgrade-from-2026-4-20",
|
||||
"upgrade-from-2026-4-24",
|
||||
"bundled-runtime-deps",
|
||||
"plugin-lifecycle",
|
||||
"provider-models",
|
||||
"agent-cold-warm-message",
|
||||
"agent-gateway-rpc-turn",
|
||||
"dashboard-session-send-turn",
|
||||
"tui-message-turn",
|
||||
"openai-compatible-turn",
|
||||
"dashboard-readiness",
|
||||
"tui-responsiveness",
|
||||
"gateway-performance"
|
||||
],
|
||||
"warning": [
|
||||
"workspace-scan-pressure",
|
||||
"mcp-runtime-start-stop",
|
||||
"browser-automation-smoke",
|
||||
"media-understanding-timeout",
|
||||
"agent-network-offline"
|
||||
]
|
||||
}
|
||||
},
|
||||
"blocking": [
|
||||
{
|
||||
"scenario": "release-runtime-startup",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "fresh-install",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "fresh-install",
|
||||
"state": "onboarded-user"
|
||||
},
|
||||
{
|
||||
"scenario": "upgrade-existing-user",
|
||||
"state": "old-release-user"
|
||||
},
|
||||
{
|
||||
"scenario": "upgrade-from-2026-4-20",
|
||||
"state": "old-release-2026-4-20-user"
|
||||
},
|
||||
{
|
||||
"scenario": "upgrade-from-2026-4-24",
|
||||
"state": "old-release-2026-4-24-user"
|
||||
},
|
||||
{
|
||||
"scenario": "bundled-runtime-deps",
|
||||
"state": "missing-plugin-index"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-lifecycle",
|
||||
"state": "plugin-index"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-external-install",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "official-plugin-install",
|
||||
"state": "official-plugins"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-remove",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-update",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-bad-manifest",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-missing-runtime-deps",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "bundled-plugin-startup",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-lifecycle",
|
||||
"state": "external-plugin"
|
||||
},
|
||||
{
|
||||
"scenario": "provider-models",
|
||||
"state": "model-auth-missing"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-cold-warm-message",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-gateway-rpc-turn",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-session-send-turn",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "tui-message-turn",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "openai-compatible-turn",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "dashboard-readiness",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "tui-responsiveness",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-performance",
|
||||
"state": "many-bundled-plugins"
|
||||
}
|
||||
{ "scenario": "release-runtime-startup", "state": "fresh" },
|
||||
{ "scenario": "fresh-install", "state": "fresh" },
|
||||
{ "scenario": "fresh-install", "state": "onboarded-user" },
|
||||
{ "scenario": "upgrade-existing-user", "state": "old-release-user" },
|
||||
{ "scenario": "upgrade-from-2026-4-20", "state": "old-release-2026-4-20-user" },
|
||||
{ "scenario": "upgrade-from-2026-4-24", "state": "old-release-2026-4-24-user" },
|
||||
{ "scenario": "bundled-runtime-deps", "state": "missing-plugin-index" },
|
||||
{ "scenario": "plugin-lifecycle", "state": "plugin-index" },
|
||||
{ "scenario": "plugin-external-install", "state": "fresh" },
|
||||
{ "scenario": "plugin-remove", "state": "fresh" },
|
||||
{ "scenario": "plugin-update", "state": "fresh" },
|
||||
{ "scenario": "plugin-bad-manifest", "state": "fresh" },
|
||||
{ "scenario": "plugin-missing-runtime-deps", "state": "fresh" },
|
||||
{ "scenario": "bundled-plugin-startup", "state": "fresh" },
|
||||
{ "scenario": "plugin-lifecycle", "state": "external-plugin" },
|
||||
{ "scenario": "provider-models", "state": "model-auth-missing" },
|
||||
{ "scenario": "agent-cold-warm-message", "state": "mock-openai-provider" },
|
||||
{ "scenario": "agent-gateway-rpc-turn", "state": "mock-openai-provider" },
|
||||
{ "scenario": "dashboard-session-send-turn", "state": "mock-openai-provider" },
|
||||
{ "scenario": "tui-message-turn", "state": "mock-openai-provider" },
|
||||
{ "scenario": "openai-compatible-turn", "state": "mock-openai-provider" },
|
||||
{ "scenario": "dashboard-readiness", "state": "fresh" },
|
||||
{ "scenario": "tui-responsiveness", "state": "fresh" },
|
||||
{ "scenario": "gateway-performance", "state": "many-bundled-plugins" }
|
||||
],
|
||||
"warning": [
|
||||
{
|
||||
"scenario": "agent-provider-slow",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-provider-timeout",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-provider-malformed",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-provider-streaming-stall",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-provider-concurrent",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-provider-recovery",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-auth-missing",
|
||||
"state": "agent-auth-missing"
|
||||
},
|
||||
{
|
||||
"scenario": "agent-long-session",
|
||||
"state": "mock-openai-provider"
|
||||
},
|
||||
{
|
||||
"scenario": "failure-injection",
|
||||
"state": "broken-plugin-deps"
|
||||
},
|
||||
{
|
||||
"scenario": "soak",
|
||||
"state": "large-workspace"
|
||||
},
|
||||
{
|
||||
"scenario": "workspace-scan-pressure",
|
||||
"state": "large-workspace"
|
||||
},
|
||||
{
|
||||
"scenario": "mcp-runtime-start-stop",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "browser-automation-smoke",
|
||||
"state": "fresh",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "media-understanding-timeout",
|
||||
"state": "fresh",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "agent-network-offline",
|
||||
"state": "fresh",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "cross-platform-smoke",
|
||||
"state": "slow-filesystem"
|
||||
}
|
||||
{ "scenario": "agent-provider-slow", "state": "mock-openai-provider" },
|
||||
{ "scenario": "agent-provider-timeout", "state": "mock-openai-provider" },
|
||||
{ "scenario": "agent-provider-malformed", "state": "mock-openai-provider" },
|
||||
{ "scenario": "agent-provider-streaming-stall", "state": "mock-openai-provider" },
|
||||
{ "scenario": "agent-provider-concurrent", "state": "mock-openai-provider" },
|
||||
{ "scenario": "agent-provider-recovery", "state": "mock-openai-provider" },
|
||||
{ "scenario": "agent-auth-missing", "state": "agent-auth-missing" },
|
||||
{ "scenario": "agent-long-session", "state": "mock-openai-provider" },
|
||||
{ "scenario": "failure-injection", "state": "broken-plugin-deps" },
|
||||
{ "scenario": "soak", "state": "large-workspace" },
|
||||
{ "scenario": "workspace-scan-pressure", "state": "large-workspace" },
|
||||
{ "scenario": "mcp-runtime-start-stop", "state": "fresh" },
|
||||
{ "scenario": "browser-automation-smoke", "state": "fresh", "timeoutMs": 180000 },
|
||||
{ "scenario": "media-understanding-timeout", "state": "fresh", "timeoutMs": 180000 },
|
||||
{ "scenario": "agent-network-offline", "state": "fresh", "timeoutMs": 180000 },
|
||||
{ "scenario": "cross-platform-smoke", "state": "slow-filesystem" }
|
||||
]
|
||||
},
|
||||
"entries": [
|
||||
@ -478,11 +350,6 @@
|
||||
"scenario": "plugin-external-install",
|
||||
"state": "fresh"
|
||||
},
|
||||
{
|
||||
"scenario": "official-plugin-install",
|
||||
"state": "official-plugins",
|
||||
"timeoutMs": 240000
|
||||
},
|
||||
{
|
||||
"scenario": "plugin-remove",
|
||||
"state": "fresh"
|
||||
@ -522,7 +389,7 @@
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-session-send-turn",
|
||||
"scenario": "dashboard-session-send-turn",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
@ -625,6 +492,5 @@
|
||||
"scenario": "cross-platform-smoke",
|
||||
"state": "slow-filesystem"
|
||||
}
|
||||
],
|
||||
"purpose": "release"
|
||||
]
|
||||
}
|
||||
|
||||
@ -38,7 +38,7 @@
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-session-send-turn",
|
||||
"scenario": "dashboard-session-send-turn",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
@ -47,6 +47,5 @@
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
}
|
||||
],
|
||||
"purpose": "regression"
|
||||
]
|
||||
}
|
||||
|
||||
@ -44,7 +44,7 @@
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
{
|
||||
"scenario": "gateway-session-send-turn",
|
||||
"scenario": "dashboard-session-send-turn",
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 180000
|
||||
},
|
||||
@ -53,6 +53,5 @@
|
||||
"state": "mock-openai-provider",
|
||||
"timeoutMs": 360000
|
||||
}
|
||||
],
|
||||
"purpose": "soak"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,20 +3,10 @@
|
||||
"surface": "agent-cli-local-turn",
|
||||
"title": "Agent Auth Missing",
|
||||
"objective": "Prove OpenClaw fails a local agent turn clearly when model auth is missing, keeps the gateway responsive, and leaves no leaked child processes.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"auth",
|
||||
"failure",
|
||||
"containment"
|
||||
],
|
||||
"states": [
|
||||
"agent-auth-missing"
|
||||
],
|
||||
"tags": ["agent", "message", "auth", "failure", "containment"],
|
||||
"states": ["agent-auth-missing"],
|
||||
"timeoutMs": 180000,
|
||||
"auth": {
|
||||
"mode": "missing"
|
||||
},
|
||||
"auth": { "mode": "missing" },
|
||||
"agent": {
|
||||
"expectedFailure": true
|
||||
},
|
||||
@ -34,16 +24,8 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Missing-Auth Env",
|
||||
"intent": "Start a disposable OpenClaw gateway without Kova mock or live model credentials.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"startup readiness",
|
||||
"no Kova auth setup phase"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "startup readiness", "no Kova auth setup phase"]
|
||||
},
|
||||
{
|
||||
"id": "missing-auth-agent-turn",
|
||||
@ -53,32 +35,14 @@
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/expect-command-fails.mjs -- ocm @{env} -- agent --local --agent main --session-id kova-agent-auth-missing --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 30 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"clear auth failure",
|
||||
"no provider request",
|
||||
"process leak snapshot",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["clear auth failure", "no provider request", "process leak snapshot", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-auth-failure-health",
|
||||
"title": "Post-Auth-Failure Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after the missing-auth agent failure.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"auth failure logs",
|
||||
"plugin errors",
|
||||
"memory after auth failure"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "auth failure logs", "plugin errors", "memory after auth failure"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -2,14 +2,8 @@
|
||||
"id": "agent-cold-warm-message",
|
||||
"surface": "agent-cli-local-turn",
|
||||
"title": "Agent CLI Local Cold/Warm Message",
|
||||
"objective": "Send cold and warm simple messages through short-lived `openclaw agent --local` CLI processes, verify mock-provider responses, and attribute latency before, during, and after provider work.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"latency",
|
||||
"providers",
|
||||
"cold-warm"
|
||||
],
|
||||
"objective": "Send cold and warm simple messages through `openclaw agent --local`, verify mock-provider responses, and attribute latency before, during, and after provider work.",
|
||||
"tags": ["agent", "message", "latency", "providers", "cold-warm"],
|
||||
"timeoutMs": 240000,
|
||||
"agent": {
|
||||
"expectedText": "KOVA_AGENT_OK"
|
||||
@ -36,15 +30,8 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Agent Env",
|
||||
"intent": "Create a disposable OpenClaw env before wiring the model provider and sending local agent messages.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --no-service --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"env created without service"
|
||||
],
|
||||
"healthScope": "none"
|
||||
"commands": ["ocm start {env} {startSelector} --no-service --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "env created without service"]
|
||||
},
|
||||
{
|
||||
"id": "cold-agent-turn",
|
||||
@ -53,14 +40,7 @@
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --local --agent main --session-id kova-agent-cold-warm --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"cold command duration",
|
||||
"final assistant text",
|
||||
"mock provider request timing",
|
||||
"gateway health after cold turn",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["cold command duration", "final assistant text", "mock provider request timing", "gateway health after cold turn", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "warm-agent-turn",
|
||||
@ -69,31 +49,14 @@
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --local --agent main --session-id kova-agent-cold-warm --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"warm command duration",
|
||||
"final assistant text",
|
||||
"mock provider request timing",
|
||||
"cold/warm delta",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["warm command duration", "final assistant text", "mock provider request timing", "cold/warm delta", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-agent-health",
|
||||
"title": "Post-Agent Env Status",
|
||||
"intent": "Verify the env remains usable after both local agent turns and capture plugin diagnostics.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"env status",
|
||||
"plugin errors",
|
||||
"memory after agent turns"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status"],
|
||||
"evidence": ["env status", "plugin errors", "memory after agent turns"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -1,16 +1,9 @@
|
||||
{
|
||||
"id": "agent-gateway-rpc-turn",
|
||||
"surface": "agent-gateway-rpc-turn",
|
||||
"title": "Agent CLI Gateway RPC Turn",
|
||||
"objective": "Send a simple user message through `openclaw agent` without `--local`, measuring the CLI client path that crosses the Gateway agent RPC boundary. This is CLI surface coverage, not the primary direct Gateway/session benchmark.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"cli",
|
||||
"message",
|
||||
"gateway",
|
||||
"rpc",
|
||||
"providers"
|
||||
],
|
||||
"title": "Agent Gateway RPC Turn",
|
||||
"objective": "Send a simple user message through `openclaw agent` without `--local`, forcing the CLI to cross the Gateway agent RPC boundary and prove final response, provider timing, gateway health, and process containment.",
|
||||
"tags": ["agent", "message", "gateway", "rpc", "providers"],
|
||||
"timeoutMs": 180000,
|
||||
"agent": {
|
||||
"expectedText": "KOVA_AGENT_OK"
|
||||
@ -31,65 +24,31 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Gateway Env",
|
||||
"intent": "Create a disposable OpenClaw env without starting the gateway yet so Kova auth is applied before the gateway process boots.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --no-service --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"env created without service"
|
||||
],
|
||||
"healthScope": "none"
|
||||
"commands": ["ocm start {env} {startSelector} --no-service --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "env created without service"]
|
||||
},
|
||||
{
|
||||
"id": "gateway-start",
|
||||
"title": "Start Gateway",
|
||||
"intent": "Start the gateway after auth/provider config is already present in the OpenClaw home.",
|
||||
"commands": [
|
||||
"ocm service install {env} --json",
|
||||
"ocm service start {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway service installed",
|
||||
"gateway service started",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm service install {env} --json", "ocm service start {env} --json"],
|
||||
"evidence": ["gateway service installed", "gateway service started", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "gateway-agent-turn",
|
||||
"title": "Gateway-Backed Agent CLI Turn",
|
||||
"intent": "Send a user message through the Gateway-backed OpenClaw agent CLI path and keep that CLI cost separate from direct Gateway/session benchmarks.",
|
||||
"title": "Gateway Agent Turn",
|
||||
"intent": "Send a user message through the Gateway-backed OpenClaw agent CLI path.",
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --agent main --session-id kova-agent-gateway-rpc --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"command duration",
|
||||
"final assistant text",
|
||||
"mock provider request timing",
|
||||
"gateway health after turn",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["command duration", "final assistant text", "mock provider request timing", "gateway health after turn", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-agent-health",
|
||||
"title": "Post-Agent Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after the Gateway RPC agent turn.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"provider logs",
|
||||
"plugin errors",
|
||||
"memory after agent turn"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "provider logs", "plugin errors", "memory after agent turn"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,14 +3,7 @@
|
||||
"surface": "agent-cli-local-turn",
|
||||
"title": "Agent Long Session",
|
||||
"objective": "Send repeated simple messages through one OpenClaw session to catch latency drift, provider routing drift, resource growth, health degradation, and child-process leaks during normal assistant use.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"latency",
|
||||
"providers",
|
||||
"soak",
|
||||
"long-session"
|
||||
],
|
||||
"tags": ["agent", "message", "latency", "providers", "soak", "long-session"],
|
||||
"timeoutMs": 360000,
|
||||
"agent": {
|
||||
"expectedText": "KOVA_AGENT_OK"
|
||||
@ -39,15 +32,8 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Long Session Env",
|
||||
"intent": "Start a disposable OpenClaw gateway before wiring the model provider and sending repeated messages.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "cold-session-turn",
|
||||
@ -56,13 +42,7 @@
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --local --agent main --session-id kova-agent-long-session --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"cold command duration",
|
||||
"assistant text",
|
||||
"provider request timing",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["cold command duration", "assistant text", "provider request timing", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "warm-session-turn",
|
||||
@ -71,13 +51,7 @@
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --local --agent main --session-id kova-agent-long-session --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"warm command duration",
|
||||
"assistant text",
|
||||
"provider request timing",
|
||||
"cold/warm delta"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["warm command duration", "assistant text", "provider request timing", "cold/warm delta"]
|
||||
},
|
||||
{
|
||||
"id": "session-turn-3",
|
||||
@ -86,13 +60,7 @@
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --local --agent main --session-id kova-agent-long-session --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"turn duration",
|
||||
"assistant text",
|
||||
"provider request timing",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["turn duration", "assistant text", "provider request timing", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "session-turn-4",
|
||||
@ -101,13 +69,7 @@
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --local --agent main --session-id kova-agent-long-session --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"turn duration",
|
||||
"assistant text",
|
||||
"provider request timing",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["turn duration", "assistant text", "provider request timing", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "session-turn-5",
|
||||
@ -116,13 +78,7 @@
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --local --agent main --session-id kova-agent-long-session --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"turn duration",
|
||||
"assistant text",
|
||||
"provider request timing",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["turn duration", "assistant text", "provider request timing", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "session-turn-6",
|
||||
@ -131,33 +87,14 @@
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --local --agent main --session-id kova-agent-long-session --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"turn duration",
|
||||
"assistant text",
|
||||
"provider request timing",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["turn duration", "assistant text", "provider request timing", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-session-health",
|
||||
"title": "Post-Session Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after repeated agent turns and capture provider/plugin diagnostics.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 500 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"provider logs",
|
||||
"plugin errors",
|
||||
"memory after repeated turns",
|
||||
"process leak summary"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 500 --raw"],
|
||||
"evidence": ["gateway status", "provider logs", "plugin errors", "memory after repeated turns", "process leak summary"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,18 +3,9 @@
|
||||
"surface": "network-offline",
|
||||
"title": "Agent Network Offline",
|
||||
"objective": "Run a local OpenClaw agent turn against an unreachable provider endpoint and prove OpenClaw fails clearly without taking down the gateway.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"provider",
|
||||
"network",
|
||||
"offline",
|
||||
"containment"
|
||||
],
|
||||
"tags": ["agent", "message", "provider", "network", "offline", "containment"],
|
||||
"timeoutMs": 180000,
|
||||
"auth": {
|
||||
"mode": "none"
|
||||
},
|
||||
"auth": { "mode": "none" },
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
"networkFailureObserved": 1,
|
||||
@ -29,15 +20,8 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Network Offline Env",
|
||||
"intent": "Start a disposable OpenClaw gateway before configuring an unreachable provider endpoint.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "network-offline-turn",
|
||||
@ -46,31 +30,14 @@
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/agent-network-offline.mjs --env {env} --artifact-dir {artifactDir} --timeout-seconds 20 --max-command-ms 45000"
|
||||
],
|
||||
"evidence": [
|
||||
"bounded network failure",
|
||||
"gateway status after failure",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["bounded network failure", "gateway status after failure", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-network-health",
|
||||
"title": "Post-Network Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive and collect logs after the network failure path.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"network/provider failure logs",
|
||||
"plugin errors",
|
||||
"memory after network failure"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "network/provider failure logs", "plugin errors", "memory after network failure"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,17 +3,9 @@
|
||||
"surface": "agent-cli-local-turn",
|
||||
"title": "Agent Provider Concurrent Pressure",
|
||||
"objective": "Prove OpenClaw can process multiple overlapping local agent turns through the provider path, keep the gateway healthy, return correct responses, and leave no leaked child processes.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"provider-failure",
|
||||
"concurrency",
|
||||
"containment"
|
||||
],
|
||||
"tags": ["agent", "message", "provider-failure", "concurrency", "containment"],
|
||||
"timeoutMs": 240000,
|
||||
"auth": {
|
||||
"mode": "mock"
|
||||
},
|
||||
"auth": { "mode": "mock" },
|
||||
"mockProvider": {
|
||||
"mode": "concurrent-pressure",
|
||||
"delayMs": 1500,
|
||||
@ -40,15 +32,8 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Agent Env",
|
||||
"intent": "Start a disposable OpenClaw gateway before applying concurrent provider pressure.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "concurrent-provider-turns",
|
||||
@ -57,34 +42,14 @@
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/run-concurrent-agent-turns.mjs --env {env} --count 3 --session-prefix kova-agent-provider-concurrent --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --expected-text KOVA_AGENT_OK --timeout 120"
|
||||
],
|
||||
"evidence": [
|
||||
"assistant responses",
|
||||
"provider request count",
|
||||
"provider overlap timing",
|
||||
"pre-provider timing",
|
||||
"role resource samples",
|
||||
"process leak snapshot"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["assistant responses", "provider request count", "provider overlap timing", "pre-provider timing", "role resource samples", "process leak snapshot"]
|
||||
},
|
||||
{
|
||||
"id": "post-concurrency-health",
|
||||
"title": "Post-Concurrency Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after concurrent agent/provider work.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"provider logs",
|
||||
"plugin errors",
|
||||
"memory after concurrent turns"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "provider logs", "plugin errors", "memory after concurrent turns"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,17 +3,9 @@
|
||||
"surface": "agent-cli-local-turn",
|
||||
"title": "Agent Provider Malformed",
|
||||
"objective": "Prove OpenClaw handles malformed provider responses as contained agent failures without taking down the gateway.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"provider-failure",
|
||||
"malformed",
|
||||
"containment"
|
||||
],
|
||||
"tags": ["agent", "message", "provider-failure", "malformed", "containment"],
|
||||
"timeoutMs": 180000,
|
||||
"auth": {
|
||||
"mode": "mock"
|
||||
},
|
||||
"auth": { "mode": "mock" },
|
||||
"mockProvider": {
|
||||
"mode": "malformed"
|
||||
},
|
||||
@ -31,15 +23,8 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Agent Env",
|
||||
"intent": "Start a disposable OpenClaw gateway before wiring the malformed mock provider.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "malformed-provider-turn",
|
||||
@ -49,32 +34,14 @@
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/expect-command-fails.mjs -- ocm @{env} -- agent --local --agent main --session-id kova-agent-provider-malformed --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 60 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"clear command failure",
|
||||
"malformed provider evidence",
|
||||
"gateway remains supervised",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["clear command failure", "malformed provider evidence", "gateway remains supervised", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-failure-health",
|
||||
"title": "Post-Failure Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after the malformed provider response.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"provider logs",
|
||||
"plugin errors",
|
||||
"memory after malformed response"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "provider logs", "plugin errors", "memory after malformed response"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,17 +3,9 @@
|
||||
"surface": "agent-cli-local-turn",
|
||||
"title": "Agent Provider Recovery",
|
||||
"objective": "Prove a transient provider failure is contained and provider recovery produces a normal assistant response in the same OpenClaw env/session lifecycle.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"provider-failure",
|
||||
"recovery",
|
||||
"containment"
|
||||
],
|
||||
"tags": ["agent", "message", "provider-failure", "recovery", "containment"],
|
||||
"timeoutMs": 240000,
|
||||
"auth": {
|
||||
"mode": "mock"
|
||||
},
|
||||
"auth": { "mode": "mock" },
|
||||
"mockProvider": {
|
||||
"mode": "error-then-recover",
|
||||
"errorStatus": 503
|
||||
@ -35,15 +27,8 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Agent Env",
|
||||
"intent": "Start a disposable OpenClaw gateway before wiring the transient-failure mock provider.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "transient-provider-failure-turn",
|
||||
@ -52,14 +37,7 @@
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --local --agent main --session-id kova-agent-provider-recovery --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"assistant response",
|
||||
"provider 503 evidence",
|
||||
"provider 200 recovery evidence",
|
||||
"gateway remains supervised",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["assistant response", "provider 503 evidence", "provider 200 recovery evidence", "gateway remains supervised", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "recovery-provider-turn",
|
||||
@ -68,32 +46,14 @@
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --local --agent main --session-id kova-agent-provider-recovery --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"assistant response",
|
||||
"provider recovery timing",
|
||||
"gateway remains healthy",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["assistant response", "provider recovery timing", "gateway remains healthy", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-failure-health",
|
||||
"title": "Post-Failure Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after provider failure and recovery.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"provider logs",
|
||||
"plugin errors",
|
||||
"memory after recovery"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "provider logs", "plugin errors", "memory after recovery"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,17 +3,9 @@
|
||||
"surface": "agent-cli-local-turn",
|
||||
"title": "Agent Provider Slow",
|
||||
"objective": "Prove Kova can attribute a slow assistant turn to provider latency while confirming OpenClaw still returns a valid response and keeps the gateway healthy.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"provider-failure",
|
||||
"latency",
|
||||
"containment"
|
||||
],
|
||||
"tags": ["agent", "message", "provider-failure", "latency", "containment"],
|
||||
"timeoutMs": 180000,
|
||||
"auth": {
|
||||
"mode": "mock"
|
||||
},
|
||||
"auth": { "mode": "mock" },
|
||||
"mockProvider": {
|
||||
"mode": "slow",
|
||||
"delayMs": 2500
|
||||
@ -37,15 +29,8 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Agent Env",
|
||||
"intent": "Start a disposable OpenClaw gateway before wiring the slow mock provider.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "slow-provider-turn",
|
||||
@ -54,32 +39,14 @@
|
||||
"commands": [
|
||||
"ocm @{env} -- agent --local --agent main --session-id kova-agent-provider-slow --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 120 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"assistant response",
|
||||
"provider delay timing",
|
||||
"pre-provider timing",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["assistant response", "provider delay timing", "pre-provider timing", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-failure-health",
|
||||
"title": "Post-Failure Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after the slow provider turn.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"provider logs",
|
||||
"plugin errors",
|
||||
"memory after provider delay"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "provider logs", "plugin errors", "memory after provider delay"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,17 +3,9 @@
|
||||
"surface": "agent-cli-local-turn",
|
||||
"title": "Agent Provider Streaming Stall",
|
||||
"objective": "Prove OpenClaw contains a provider stream that starts but stalls, keeps the gateway responsive, and leaves no leaked agent/plugin child processes.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"provider-failure",
|
||||
"streaming-stall",
|
||||
"containment"
|
||||
],
|
||||
"tags": ["agent", "message", "provider-failure", "streaming-stall", "containment"],
|
||||
"timeoutMs": 180000,
|
||||
"auth": {
|
||||
"mode": "mock"
|
||||
},
|
||||
"auth": { "mode": "mock" },
|
||||
"mockProvider": {
|
||||
"mode": "streaming-stall",
|
||||
"stallMs": 45000
|
||||
@ -34,15 +26,8 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Agent Env",
|
||||
"intent": "Start a disposable OpenClaw gateway before wiring the streaming-stall mock provider.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "streaming-stall-provider-turn",
|
||||
@ -52,33 +37,14 @@
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/expect-command-fails.mjs -- ocm @{env} -- agent --local --agent main --session-id kova-agent-provider-streaming-stall --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 20 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"clear command failure or timeout",
|
||||
"streaming stall provider evidence",
|
||||
"process leak snapshot",
|
||||
"gateway remains supervised",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["clear command failure or timeout", "streaming stall provider evidence", "process leak snapshot", "gateway remains supervised", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-failure-health",
|
||||
"title": "Post-Failure Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after the stalled provider stream.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"provider logs",
|
||||
"plugin errors",
|
||||
"memory after streaming stall"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "provider logs", "plugin errors", "memory after streaming stall"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,17 +3,9 @@
|
||||
"surface": "agent-cli-local-turn",
|
||||
"title": "Agent Provider Timeout",
|
||||
"objective": "Prove OpenClaw surfaces provider timeout failure clearly, keeps the gateway healthy, and leaves no retained child process state after a failed agent turn.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"provider-failure",
|
||||
"timeout",
|
||||
"containment"
|
||||
],
|
||||
"tags": ["agent", "message", "provider-failure", "timeout", "containment"],
|
||||
"timeoutMs": 180000,
|
||||
"auth": {
|
||||
"mode": "mock"
|
||||
},
|
||||
"auth": { "mode": "mock" },
|
||||
"mockProvider": {
|
||||
"mode": "timeout",
|
||||
"stallMs": 45000
|
||||
@ -32,15 +24,8 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Agent Env",
|
||||
"intent": "Start a disposable OpenClaw gateway before wiring the timeout mock provider.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "timeout-provider-turn",
|
||||
@ -50,32 +35,14 @@
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/expect-command-fails.mjs -- ocm @{env} -- agent --local --agent main --session-id kova-agent-provider-timeout --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --thinking off --timeout 20 --json"
|
||||
],
|
||||
"evidence": [
|
||||
"clear command failure",
|
||||
"provider timeout/abort timing",
|
||||
"gateway remains supervised",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["clear command failure", "provider timeout/abort timing", "gateway remains supervised", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-failure-health",
|
||||
"title": "Post-Failure Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after the timeout failure.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"provider logs",
|
||||
"plugin errors",
|
||||
"memory after timeout"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "provider logs", "plugin errors", "memory after timeout"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "browser-automation",
|
||||
"title": "Browser Automation Smoke",
|
||||
"objective": "Start OpenClaw's real browser automation surface, open a managed browser tab, capture browser state, stop the profile, and verify the gateway remains healthy.",
|
||||
"tags": [
|
||||
"browser",
|
||||
"automation",
|
||||
"gateway",
|
||||
"plugins"
|
||||
],
|
||||
"tags": ["browser", "automation", "gateway", "plugins"],
|
||||
"timeoutMs": 180000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
@ -31,50 +26,22 @@
|
||||
"id": "gateway",
|
||||
"title": "Gateway Start",
|
||||
"intent": "Start the gateway and prove it is healthy before browser automation.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"gateway port",
|
||||
"readiness classification"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- status"],
|
||||
"evidence": ["gateway status", "gateway port", "readiness classification"]
|
||||
},
|
||||
{
|
||||
"id": "browser-smoke",
|
||||
"title": "Browser Automation Smoke",
|
||||
"intent": "Use OpenClaw's browser CLI to start a managed headless profile, open a tab, list tabs, snapshot, and stop.",
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/browser-automation-smoke.mjs --env {env} --artifact-dir {artifactDir} --timeout-ms 45000"
|
||||
],
|
||||
"evidence": [
|
||||
"browser start timing",
|
||||
"tabs timing",
|
||||
"opened tab count",
|
||||
"snapshot timing",
|
||||
"browser stop timing"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["node {kovaRoot}/support/browser-automation-smoke.mjs --env {env} --artifact-dir {artifactDir} --timeout-ms 45000"],
|
||||
"evidence": ["browser start timing", "tabs timing", "opened tab count", "snapshot timing", "browser stop timing"]
|
||||
},
|
||||
{
|
||||
"id": "post-browser-health",
|
||||
"title": "Post-Browser Gateway Health",
|
||||
"intent": "Verify browser automation did not leave the gateway degraded.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"status after browser automation",
|
||||
"browser plugin errors",
|
||||
"gateway errors"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["status after browser automation", "browser plugin errors", "gateway errors"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "bundled-plugin-startup",
|
||||
"title": "Bundled Plugin Startup",
|
||||
"objective": "Validate that OpenClaw's bundled plugins load during gateway startup without missing package/module errors or degraded plugin services.",
|
||||
"tags": [
|
||||
"plugins",
|
||||
"bundled",
|
||||
"runtime-deps",
|
||||
"startup"
|
||||
],
|
||||
"tags": ["plugins", "bundled", "runtime-deps", "startup"],
|
||||
"timeoutMs": 180000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
@ -23,51 +18,22 @@
|
||||
"id": "startup",
|
||||
"title": "Start Bundled Plugin Gateway",
|
||||
"intent": "Start OpenClaw and let bundled plugin bootstrap run in the same shape users get from the target runtime.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"bundled plugin count",
|
||||
"readiness classification",
|
||||
"dependency staging"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["bundled plugin count", "readiness classification", "dependency staging"]
|
||||
},
|
||||
{
|
||||
"id": "inspect",
|
||||
"title": "Inspect Bundled Plugins",
|
||||
"intent": "List and inspect plugin registry state after startup.",
|
||||
"commands": [
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- plugins registry --refresh --json",
|
||||
"ocm logs {env} --tail 400 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"plugin list",
|
||||
"registry refresh",
|
||||
"missing package/module errors",
|
||||
"plugin service failures"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- plugins list", "ocm @{env} -- plugins registry --refresh --json", "ocm logs {env} --tail 400 --raw"],
|
||||
"evidence": ["plugin list", "registry refresh", "missing package/module errors", "plugin service failures"]
|
||||
},
|
||||
{
|
||||
"id": "restart",
|
||||
"title": "Warm Restart Bundled Plugins",
|
||||
"intent": "Restart after dependency staging should be warm and verify bundled plugin services remain healthy.",
|
||||
"commands": [
|
||||
"ocm service restart {env}",
|
||||
"ocm service status {env} --json",
|
||||
"ocm logs {env} --tail 400 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"warm readiness",
|
||||
"bundled plugin reload",
|
||||
"runtime dependency reuse"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm service restart {env}", "ocm service status {env} --json", "ocm logs {env} --tail 400 --raw"],
|
||||
"evidence": ["warm readiness", "bundled plugin reload", "runtime dependency reuse"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "bundled-runtime-deps",
|
||||
"title": "Bundled Runtime Dependency Integrity",
|
||||
"objective": "Verify bundled plugin runtime dependencies stage correctly, remain reusable on warm starts, and do not produce missing dependency errors.",
|
||||
"tags": [
|
||||
"plugins",
|
||||
"runtime-deps",
|
||||
"cold-start",
|
||||
"warm-start"
|
||||
],
|
||||
"tags": ["plugins", "runtime-deps", "cold-start", "warm-start"],
|
||||
"thresholds": {
|
||||
"coldReadyMs": 45000,
|
||||
"warmReadyMs": 20000,
|
||||
@ -21,35 +16,15 @@
|
||||
"id": "cold-start",
|
||||
"title": "Cold Runtime Dependency Start",
|
||||
"intent": "Start a fresh env and capture bundled runtime dependency staging logs.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"dependency staging duration",
|
||||
"installed dependency list",
|
||||
"missing dependency errors"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["dependency staging duration", "installed dependency list", "missing dependency errors"]
|
||||
},
|
||||
{
|
||||
"id": "warm-restart",
|
||||
"title": "Warm Runtime Dependency Restart",
|
||||
"intent": "Restart with staged dependencies already present and verify no repeated expensive staging or missing dependency errors.",
|
||||
"commands": [
|
||||
"ocm service restart {env}",
|
||||
"ocm service status {env} --json",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"warm ready time",
|
||||
"dependency staging reuse",
|
||||
"missing dependency errors"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm service restart {env}", "ocm service status {env} --json", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["warm ready time", "dependency staging reuse", "missing dependency errors"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,18 +3,9 @@
|
||||
"surface": "cross-platform-smoke",
|
||||
"title": "Cross-Platform Runtime Smoke",
|
||||
"objective": "Run the core OpenClaw runtime scenarios across supported platforms and flag platform-specific startup, filesystem, and service behavior.",
|
||||
"tags": [
|
||||
"platform",
|
||||
"macos",
|
||||
"linux",
|
||||
"wsl2",
|
||||
"gateway"
|
||||
],
|
||||
"tags": ["platform", "macos", "linux", "wsl2", "gateway"],
|
||||
"platforms": {
|
||||
"include": [
|
||||
"darwin",
|
||||
"linux"
|
||||
]
|
||||
"include": ["darwin", "linux"]
|
||||
},
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 45000,
|
||||
@ -26,39 +17,15 @@
|
||||
"id": "platform",
|
||||
"title": "Platform Baseline",
|
||||
"intent": "Capture OS, architecture, Node, OCM, and OpenClaw runtime details.",
|
||||
"commands": [
|
||||
"node --version",
|
||||
"ocm --version",
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"OS",
|
||||
"architecture",
|
||||
"Node version",
|
||||
"runtime version",
|
||||
"gateway port"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["node --version", "ocm --version", "ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["OS", "architecture", "Node version", "runtime version", "gateway port"]
|
||||
},
|
||||
{
|
||||
"id": "core-smoke",
|
||||
"title": "Core Runtime Smoke",
|
||||
"intent": "Run the minimum user-facing checks used for every platform.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm logs {env} --tail 250 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"status",
|
||||
"plugin list",
|
||||
"filesystem stall logs",
|
||||
"health latency"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm @{env} -- plugins list", "ocm logs {env} --tail 250 --raw"],
|
||||
"evidence": ["status", "plugin list", "filesystem stall logs", "health latency"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "dashboard",
|
||||
"title": "Dashboard Readiness",
|
||||
"objective": "Verify OpenClaw can produce a usable dashboard URL and keep the gateway responsive after dashboard entry.",
|
||||
"tags": [
|
||||
"dashboard",
|
||||
"control-ui",
|
||||
"gateway",
|
||||
"websocket"
|
||||
],
|
||||
"tags": ["dashboard", "control-ui", "gateway", "websocket"],
|
||||
"timeoutMs": 120000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
@ -23,48 +18,22 @@
|
||||
"id": "gateway",
|
||||
"title": "Gateway Start",
|
||||
"intent": "Start the gateway and classify readiness before dashboard checks.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"gateway port",
|
||||
"readiness classification"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- status"],
|
||||
"evidence": ["gateway status", "gateway port", "readiness classification"]
|
||||
},
|
||||
{
|
||||
"id": "dashboard",
|
||||
"title": "Dashboard Link",
|
||||
"intent": "Use OpenClaw's dashboard command without opening a browser and require it to return promptly.",
|
||||
"commands": [
|
||||
"ocm @{env} -- dashboard --no-open"
|
||||
],
|
||||
"evidence": [
|
||||
"dashboard URL",
|
||||
"token handling",
|
||||
"command latency"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- dashboard --no-open"],
|
||||
"evidence": ["dashboard URL", "token handling", "command latency"]
|
||||
},
|
||||
{
|
||||
"id": "post-dashboard-health",
|
||||
"title": "Post-Dashboard Health",
|
||||
"intent": "Verify dashboard entry did not make the gateway or websocket layer unstable.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"status after dashboard command",
|
||||
"websocket disconnect logs",
|
||||
"gateway errors"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["status after dashboard command", "websocket disconnect logs", "gateway errors"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
62
scenarios/dashboard-session-send-turn-existing-user.json
Normal file
62
scenarios/dashboard-session-send-turn-existing-user.json
Normal file
@ -0,0 +1,62 @@
|
||||
{
|
||||
"id": "dashboard-session-send-turn-existing-user",
|
||||
"surface": "dashboard-session-send-turn",
|
||||
"title": "Dashboard Session Send Existing User Turn",
|
||||
"objective": "Clone an existing OpenClaw env, move the clone to the target runtime, send a dashboard-style user message through Gateway sessions.send, and measure response latency without mutating the durable source env.",
|
||||
"tags": ["agent", "message", "dashboard", "sessions", "gateway", "providers", "existing-user", "live"],
|
||||
"timeoutMs": 240000,
|
||||
"agent": {
|
||||
"expectedText": "KOVA_AGENT_OK"
|
||||
},
|
||||
"thresholds": {
|
||||
"upgradeMs": 120000,
|
||||
"gatewayReadyMs": 45000,
|
||||
"agentTurnMs": 45000,
|
||||
"preProviderMs": 10000,
|
||||
"providerFinalMs": 3000,
|
||||
"preProviderDominanceRatio": 0.8,
|
||||
"statusMs": 10000,
|
||||
"peakRssMb": 900,
|
||||
"missingDependencyErrors": 0,
|
||||
"pluginLoadFailures": 0
|
||||
},
|
||||
"phases": [
|
||||
{
|
||||
"id": "clone",
|
||||
"title": "Clone Existing Env",
|
||||
"intent": "Clone a durable existing user env into a disposable Kova env before runtime or message testing.",
|
||||
"commands": ["ocm env clone {sourceEnv} {env} --json"],
|
||||
"evidence": ["source env", "clone root", "cloned OpenClaw config"]
|
||||
},
|
||||
{
|
||||
"id": "upgrade",
|
||||
"title": "Move Clone To Target Runtime",
|
||||
"intent": "Run the real OCM upgrade path so the cloned user state runs the requested OpenClaw target runtime.",
|
||||
"commands": ["ocm upgrade {env} {upgradeSelector} --json", "ocm service status {env} --json"],
|
||||
"evidence": ["upgrade JSON", "runtime binding", "post-upgrade service state"]
|
||||
},
|
||||
{
|
||||
"id": "gateway-start",
|
||||
"title": "Start Gateway",
|
||||
"intent": "Start the cloned gateway after upgrade and wait for readiness before sending a dashboard-style message.",
|
||||
"commands": ["ocm service install {env} --json", "ocm service start {env} --json"],
|
||||
"evidence": ["gateway service installed", "gateway service started", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "dashboard-session-turn",
|
||||
"title": "Dashboard Session Message",
|
||||
"intent": "Exercise Gateway sessions.send on cloned user state and verify final assistant text appears in chat history.",
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/run-dashboard-session-send-turn.mjs --env {env} --session-key kova-dashboard-existing-user --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --expected-text KOVA_AGENT_OK --timeout 120000"
|
||||
],
|
||||
"evidence": ["sessions.create timing", "sessions.send timing", "time to assistant text", "provider timing", "gateway health after turn", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-dashboard-health",
|
||||
"title": "Post-Dashboard Gateway Health",
|
||||
"intent": "Verify the cloned gateway remains responsive after the dashboard-style turn and collect logs for embedded-run/liveness evidence.",
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "embedded-run traces", "liveness warnings", "plugin errors", "memory after dashboard turn"]
|
||||
}
|
||||
]
|
||||
}
|
||||
54
scenarios/dashboard-session-send-turn.json
Normal file
54
scenarios/dashboard-session-send-turn.json
Normal file
@ -0,0 +1,54 @@
|
||||
{
|
||||
"id": "dashboard-session-send-turn",
|
||||
"surface": "dashboard-session-send-turn",
|
||||
"title": "Dashboard Session Send Turn",
|
||||
"objective": "Create a dashboard session, send a user message through Gateway `sessions.send`, and wait for final assistant text in chat history.",
|
||||
"tags": ["agent", "message", "dashboard", "sessions", "gateway", "providers"],
|
||||
"timeoutMs": 180000,
|
||||
"agent": {
|
||||
"expectedText": "KOVA_AGENT_OK"
|
||||
},
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
"agentTurnMs": 45000,
|
||||
"preProviderMs": 10000,
|
||||
"providerFinalMs": 3000,
|
||||
"preProviderDominanceRatio": 0.8,
|
||||
"statusMs": 10000,
|
||||
"peakRssMb": 900,
|
||||
"missingDependencyErrors": 0,
|
||||
"pluginLoadFailures": 0
|
||||
},
|
||||
"phases": [
|
||||
{
|
||||
"id": "provision",
|
||||
"title": "Provision Dashboard Env",
|
||||
"intent": "Create a disposable OpenClaw env without starting the gateway yet so Kova auth is applied before the gateway process boots.",
|
||||
"commands": ["ocm start {env} {startSelector} --no-service --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "env created without service"]
|
||||
},
|
||||
{
|
||||
"id": "gateway-start",
|
||||
"title": "Start Gateway",
|
||||
"intent": "Start the gateway after auth/provider config is already present in the OpenClaw home.",
|
||||
"commands": ["ocm service install {env} --json", "ocm service start {env} --json"],
|
||||
"evidence": ["gateway service installed", "gateway service started", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "dashboard-session-turn",
|
||||
"title": "Dashboard Session Message",
|
||||
"intent": "Exercise Gateway `sessions.send` and verify the final assistant response is present in chat history.",
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/run-dashboard-session-send-turn.mjs --env {env} --session-key kova-dashboard-session-send --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --expected-text KOVA_AGENT_OK --timeout 120000"
|
||||
],
|
||||
"evidence": ["sessions.send command duration", "chat history final assistant text", "mock provider request timing", "gateway health after turn", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-dashboard-health",
|
||||
"title": "Post-Dashboard Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after the dashboard-style message turn.",
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "provider logs", "plugin errors", "memory after dashboard turn"]
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -3,13 +3,7 @@
|
||||
"surface": "failure-containment",
|
||||
"title": "OpenClaw Failure Containment",
|
||||
"objective": "Inject bad plugin/provider/runtime conditions and verify OpenClaw reports useful diagnostics without taking down the gateway.",
|
||||
"tags": [
|
||||
"failure",
|
||||
"plugins",
|
||||
"providers",
|
||||
"diagnostics",
|
||||
"gateway"
|
||||
],
|
||||
"tags": ["failure", "plugins", "providers", "diagnostics", "gateway"],
|
||||
"thresholds": {
|
||||
"gatewaySurvives": true,
|
||||
"diagnosticPresent": true,
|
||||
@ -20,33 +14,15 @@
|
||||
"id": "baseline",
|
||||
"title": "Baseline Gateway",
|
||||
"intent": "Start from a clean gateway before injecting failure conditions.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"baseline status",
|
||||
"gateway PID"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- status"],
|
||||
"evidence": ["baseline status", "gateway PID"]
|
||||
},
|
||||
{
|
||||
"id": "diagnostics",
|
||||
"title": "Diagnostic Capture",
|
||||
"intent": "Capture logs and status after injected failures. Scenario-specific injectors will be added as Kova grows.",
|
||||
"commands": [
|
||||
"ocm logs {env} --tail 300 --raw",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"error classification",
|
||||
"gateway survival",
|
||||
"recovery guidance"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm logs {env} --tail 300 --raw", "ocm @{env} -- status"],
|
||||
"evidence": ["error classification", "gateway survival", "recovery guidance"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,13 +3,7 @@
|
||||
"surface": "fresh-install",
|
||||
"title": "Fresh OpenClaw Install Baseline",
|
||||
"objective": "Create a disposable fresh OpenClaw home from the target runtime, start the gateway, and verify basic user-facing commands without onboarding or manual setup.",
|
||||
"tags": [
|
||||
"fresh-user",
|
||||
"gateway",
|
||||
"plugins",
|
||||
"models",
|
||||
"performance"
|
||||
],
|
||||
"tags": ["fresh-user", "gateway", "plugins", "models", "performance"],
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
"statusMs": 10000,
|
||||
@ -22,90 +16,43 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Fresh Env",
|
||||
"intent": "Create a disposable OpenClaw environment with the selected runtime and minimum local config.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"OCM start JSON",
|
||||
"env name",
|
||||
"runtime binding",
|
||||
"gateway port"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["OCM start JSON", "env name", "runtime binding", "gateway port"]
|
||||
},
|
||||
{
|
||||
"id": "readiness",
|
||||
"title": "Gateway Readiness",
|
||||
"intent": "Confirm the gateway reaches a usable running state within the threshold.",
|
||||
"commands": [
|
||||
"ocm service status {env} --json",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"ready time",
|
||||
"gateway state",
|
||||
"gateway PID",
|
||||
"health/status result"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm service status {env} --json", "ocm @{env} -- status"],
|
||||
"evidence": ["ready time", "gateway state", "gateway PID", "health/status result"]
|
||||
},
|
||||
{
|
||||
"id": "plugins",
|
||||
"title": "Plugin Baseline",
|
||||
"intent": "Verify OpenClaw can inspect installed/bundled plugin state without missing runtime dependency errors.",
|
||||
"commands": [
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- plugins update --all --dry-run"
|
||||
],
|
||||
"evidence": [
|
||||
"plugins list output",
|
||||
"plugin update dry-run output",
|
||||
"missing dependency log scan"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- plugins list", "ocm @{env} -- plugins update --all --dry-run"],
|
||||
"evidence": ["plugins list output", "plugin update dry-run output", "missing dependency log scan"]
|
||||
},
|
||||
{
|
||||
"id": "models",
|
||||
"title": "Model Baseline",
|
||||
"intent": "Verify model discovery does not stall the gateway or hang indefinitely.",
|
||||
"commands": [
|
||||
"ocm @{env} -- models list"
|
||||
],
|
||||
"evidence": [
|
||||
"models list duration",
|
||||
"timeout behavior",
|
||||
"gateway health after model list"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- models list"],
|
||||
"evidence": ["models list duration", "timeout behavior", "gateway health after model list"]
|
||||
},
|
||||
{
|
||||
"id": "logs",
|
||||
"title": "Startup Logs",
|
||||
"intent": "Capture startup logs for dependency staging, plugin loading, stalls, and warnings.",
|
||||
"commands": [
|
||||
"ocm logs {env} --tail 200 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"startup logs",
|
||||
"missing dependency errors",
|
||||
"plugin metadata scan warnings"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm logs {env} --tail 200 --raw"],
|
||||
"evidence": ["startup logs", "missing dependency errors", "plugin metadata scan warnings"]
|
||||
},
|
||||
{
|
||||
"id": "cleanup",
|
||||
"title": "Cleanup",
|
||||
"intent": "Destroy the disposable OpenClaw environment unless retained for debugging.",
|
||||
"commands": [
|
||||
"ocm env destroy {env} --yes"
|
||||
],
|
||||
"evidence": [
|
||||
"destroy result"
|
||||
],
|
||||
"healthScope": "none"
|
||||
"commands": ["ocm env destroy {env} --yes"],
|
||||
"evidence": ["destroy result"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,71 +3,35 @@
|
||||
"surface": "gateway-performance",
|
||||
"title": "Gateway Startup And Runtime Performance",
|
||||
"objective": "Measure cold start, warm start, health latency, memory, CPU, and user-facing command latency for a target OpenClaw runtime.",
|
||||
"tags": [
|
||||
"performance",
|
||||
"gateway",
|
||||
"memory",
|
||||
"cpu",
|
||||
"event-loop"
|
||||
],
|
||||
"tags": ["performance", "gateway", "memory", "cpu", "event-loop"],
|
||||
"thresholds": {
|
||||
"coldReadyMs": 30000,
|
||||
"warmReadyMs": 15000,
|
||||
"healthP95Ms": 1000,
|
||||
"peakRssMb": 900,
|
||||
"eventLoopMaxMs": 500,
|
||||
"postReadyHealthP95Ms": 1000
|
||||
"eventLoopMaxMs": 500
|
||||
},
|
||||
"phases": [
|
||||
{
|
||||
"id": "cold-start",
|
||||
"title": "Cold Start",
|
||||
"intent": "Start a fresh gateway and capture readiness timing, process state, and logs.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm service status {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"ready time",
|
||||
"PID",
|
||||
"RSS",
|
||||
"CPU",
|
||||
"startup logs"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm service status {env} --json"],
|
||||
"evidence": ["ready time", "PID", "RSS", "CPU", "startup logs"]
|
||||
},
|
||||
{
|
||||
"id": "api-latency",
|
||||
"title": "API Latency",
|
||||
"intent": "Run user-facing status, plugin, and model commands and capture duration and gateway health after each.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- models list"
|
||||
],
|
||||
"evidence": [
|
||||
"command durations",
|
||||
"health after each command",
|
||||
"logs"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm @{env} -- plugins list", "ocm @{env} -- models list"],
|
||||
"evidence": ["command durations", "health after each command", "logs"]
|
||||
},
|
||||
{
|
||||
"id": "warm-restart",
|
||||
"title": "Warm Restart",
|
||||
"intent": "Restart the gateway after runtime deps and registries are warm, then compare readiness and memory.",
|
||||
"commands": [
|
||||
"ocm service restart {env}",
|
||||
"ocm service status {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"warm ready time",
|
||||
"RSS delta",
|
||||
"startup log delta"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm service restart {env}", "ocm service status {env} --json"],
|
||||
"evidence": ["warm ready time", "RSS delta", "startup log delta"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -1,116 +0,0 @@
|
||||
{
|
||||
"id": "gateway-session-send-turn-existing-user",
|
||||
"surface": "gateway-session-send-turn",
|
||||
"title": "Gateway Session Send Existing User Turn",
|
||||
"objective": "Clone an existing OpenClaw env, move the clone to the target runtime, send a user message through Gateway sessions.send, and measure response latency without mutating the durable source env.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"sessions",
|
||||
"gateway",
|
||||
"control-ui",
|
||||
"channels",
|
||||
"providers",
|
||||
"existing-user",
|
||||
"live"
|
||||
],
|
||||
"timeoutMs": 240000,
|
||||
"agent": {
|
||||
"expectedText": "KOVA_AGENT_OK"
|
||||
},
|
||||
"thresholds": {
|
||||
"upgradeMs": 120000,
|
||||
"gatewayReadyMs": 45000,
|
||||
"agentTurnMs": 45000,
|
||||
"preProviderMs": 10000,
|
||||
"providerFinalMs": 3000,
|
||||
"preProviderDominanceRatio": 0.8,
|
||||
"statusMs": 10000,
|
||||
"peakRssMb": 900,
|
||||
"missingDependencyErrors": 0,
|
||||
"pluginLoadFailures": 0
|
||||
},
|
||||
"phases": [
|
||||
{
|
||||
"id": "clone",
|
||||
"title": "Clone Existing Env",
|
||||
"intent": "Clone a durable existing user env into a disposable Kova env before runtime or message testing.",
|
||||
"commands": [
|
||||
"ocm env clone {sourceEnv} {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"source env",
|
||||
"clone root",
|
||||
"cloned OpenClaw config"
|
||||
],
|
||||
"healthScope": "none"
|
||||
},
|
||||
{
|
||||
"id": "upgrade",
|
||||
"title": "Move Clone To Target Runtime",
|
||||
"intent": "Run the real OCM upgrade path so the cloned user state runs the requested OpenClaw target runtime.",
|
||||
"commands": [
|
||||
"ocm upgrade {env} {upgradeSelector} --json",
|
||||
"ocm service status {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"upgrade JSON",
|
||||
"runtime binding",
|
||||
"post-upgrade service state"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
},
|
||||
{
|
||||
"id": "gateway-start",
|
||||
"title": "Start Gateway",
|
||||
"intent": "Start the cloned gateway after upgrade and wait for readiness before sending a Gateway session message.",
|
||||
"commands": [
|
||||
"ocm service install {env} --json",
|
||||
"ocm service start {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway service installed",
|
||||
"gateway service started",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
},
|
||||
{
|
||||
"id": "gateway-session-turn",
|
||||
"title": "Gateway Session Message",
|
||||
"intent": "Exercise Gateway sessions.send on cloned user state and verify final assistant text appears in chat history.",
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/run-gateway-session-send-turn.mjs --env {env} --session-key kova-gateway-existing-user --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --expected-text KOVA_AGENT_OK --timeout 120000"
|
||||
],
|
||||
"evidence": [
|
||||
"sessions.create timing",
|
||||
"sessions.send timing",
|
||||
"time to assistant text",
|
||||
"provider timing",
|
||||
"gateway health after turn",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
},
|
||||
{
|
||||
"id": "post-gateway-session-health",
|
||||
"title": "Post-Gateway-Session Health",
|
||||
"intent": "Verify the cloned gateway remains responsive after the Gateway session turn and collect logs for embedded-run/liveness evidence.",
|
||||
"commands": [
|
||||
"ocm @{env} -- gateway status --json --require-rpc",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status probe",
|
||||
"embedded-run traces",
|
||||
"liveness warnings",
|
||||
"plugin errors",
|
||||
"memory after Gateway session turn"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
@ -1,123 +0,0 @@
|
||||
{
|
||||
"id": "gateway-session-send-turn",
|
||||
"surface": "gateway-session-send-turn",
|
||||
"title": "Gateway Session Cold/Warm Turns",
|
||||
"objective": "Start a normal gateway, send cold and warm user messages through the Gateway session API, and wait for final assistant text in chat history.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"sessions",
|
||||
"gateway",
|
||||
"control-ui",
|
||||
"channels",
|
||||
"providers",
|
||||
"cold-warm"
|
||||
],
|
||||
"timeoutMs": 180000,
|
||||
"agent": {
|
||||
"expectedText": "KOVA_AGENT_OK"
|
||||
},
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
"agentTurnMs": 45000,
|
||||
"coldAgentTurnMs": 45000,
|
||||
"warmAgentTurnMs": 15000,
|
||||
"coldWarmDeltaMs": 30000,
|
||||
"preProviderMs": 10000,
|
||||
"coldPreProviderMs": 10000,
|
||||
"warmPreProviderMs": 10000,
|
||||
"providerFinalMs": 3000,
|
||||
"statusMs": 10000,
|
||||
"peakRssMb": 900,
|
||||
"missingDependencyErrors": 0,
|
||||
"pluginLoadFailures": 0
|
||||
},
|
||||
"phases": [
|
||||
{
|
||||
"id": "provision",
|
||||
"title": "Provision Gateway Session Env",
|
||||
"intent": "Create a disposable OpenClaw env without starting the gateway yet so Kova auth is applied before the gateway process boots.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --no-service --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"env created without service"
|
||||
],
|
||||
"healthScope": "none"
|
||||
},
|
||||
{
|
||||
"id": "gateway-start",
|
||||
"title": "Start Gateway",
|
||||
"intent": "Start the gateway after auth/provider config is already present in the OpenClaw home.",
|
||||
"commands": [
|
||||
"ocm service install {env} --json",
|
||||
"ocm service start {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway service installed",
|
||||
"gateway service started",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
},
|
||||
{
|
||||
"id": "cold-gateway-session-turn",
|
||||
"title": "Cold Gateway Session Turn",
|
||||
"intent": "Create the Gateway session, send the first message through `sessions.send`, and verify the final assistant response is present in chat history.",
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/run-gateway-session-send-turn.mjs --env {env} --session-key kova-gateway-session-send --create-session true --min-assistant-count 1 --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --expected-text KOVA_AGENT_OK --timeout 120000"
|
||||
],
|
||||
"evidence": [
|
||||
"cold sessions.send active turn duration",
|
||||
"chat history final assistant text",
|
||||
"mock provider request timing",
|
||||
"metadata scans during active turn",
|
||||
"event-loop delay during active turn",
|
||||
"session history polling while active",
|
||||
"gateway health after turn",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
},
|
||||
{
|
||||
"id": "warm-gateway-session-turn",
|
||||
"title": "Warm Gateway Session Turn",
|
||||
"intent": "Reuse the same Gateway session, send a second message through `sessions.send`, and prove whether cold discovery/cache work disappears.",
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/run-gateway-session-send-turn.mjs --env {env} --session-key kova-gateway-session-send --create-session false --min-assistant-count 2 --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --expected-text KOVA_AGENT_OK --timeout 120000"
|
||||
],
|
||||
"evidence": [
|
||||
"warm sessions.send active turn duration",
|
||||
"chat history final assistant text",
|
||||
"mock provider request timing",
|
||||
"cold/warm delta",
|
||||
"metadata scans during active turn",
|
||||
"event-loop delay during active turn",
|
||||
"session history polling while active",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
},
|
||||
{
|
||||
"id": "post-gateway-session-health",
|
||||
"title": "Post-Gateway-Session Health",
|
||||
"intent": "Verify the gateway remains responsive after cold and warm Gateway session message turns.",
|
||||
"commands": [
|
||||
"ocm @{env} -- gateway status --json --require-rpc",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status probe",
|
||||
"provider logs",
|
||||
"plugin errors",
|
||||
"memory after Gateway session turn"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
@ -3,13 +3,7 @@
|
||||
"surface": "mcp-runtime",
|
||||
"title": "MCP Runtime Start/Stop",
|
||||
"objective": "Start OpenClaw's real MCP stdio bridge against the disposable gateway, perform a JSON-RPC initialize and tools/list smoke, then verify the bridge exits without leaking a runtime process.",
|
||||
"tags": [
|
||||
"mcp",
|
||||
"stdio",
|
||||
"gateway",
|
||||
"runtime",
|
||||
"start-stop"
|
||||
],
|
||||
"tags": ["mcp", "stdio", "gateway", "runtime", "start-stop"],
|
||||
"timeoutMs": 120000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
@ -29,49 +23,22 @@
|
||||
"id": "gateway",
|
||||
"title": "Gateway Start",
|
||||
"intent": "Start the gateway and confirm it is healthy before opening the MCP stdio bridge.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"gateway port",
|
||||
"readiness classification"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- status"],
|
||||
"evidence": ["gateway status", "gateway port", "readiness classification"]
|
||||
},
|
||||
{
|
||||
"id": "mcp-bridge",
|
||||
"title": "MCP Bridge Smoke",
|
||||
"intent": "Spawn the real OpenClaw MCP stdio bridge, initialize it, list tools, and close it cleanly.",
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/mcp-bridge-smoke.mjs --env {env} --artifact-dir {artifactDir} --timeout-ms 30000"
|
||||
],
|
||||
"evidence": [
|
||||
"MCP initialize timing",
|
||||
"tools/list timing",
|
||||
"tool count",
|
||||
"bridge process exit"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["node {kovaRoot}/support/mcp-bridge-smoke.mjs --env {env} --artifact-dir {artifactDir} --timeout-ms 30000"],
|
||||
"evidence": ["MCP initialize timing", "tools/list timing", "tool count", "bridge process exit"]
|
||||
},
|
||||
{
|
||||
"id": "post-mcp-health",
|
||||
"title": "Post-MCP Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after the MCP bridge starts and exits.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"status after MCP bridge",
|
||||
"MCP bridge errors",
|
||||
"gateway errors"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["status after MCP bridge", "MCP bridge errors", "gateway errors"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,14 +3,7 @@
|
||||
"surface": "media-understanding",
|
||||
"title": "Media Understanding Timeout",
|
||||
"objective": "Run OpenClaw's packaged image media-understanding capability against a deterministic mock provider timeout and prove the command fails quickly while the gateway remains healthy.",
|
||||
"tags": [
|
||||
"media",
|
||||
"image",
|
||||
"timeout",
|
||||
"provider",
|
||||
"gateway",
|
||||
"capability-cli"
|
||||
],
|
||||
"tags": ["media", "image", "timeout", "provider", "gateway", "capability-cli"],
|
||||
"timeoutMs": 180000,
|
||||
"mockProvider": {
|
||||
"mode": "timeout",
|
||||
@ -32,15 +25,8 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Media Env",
|
||||
"intent": "Start a disposable OpenClaw gateway before wiring mock auth and running media understanding.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "media-timeout",
|
||||
@ -49,32 +35,14 @@
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/media-understanding-timeout.mjs --env {env} --artifact-dir {artifactDir} --timeout-ms 1200 --max-command-ms 45000"
|
||||
],
|
||||
"evidence": [
|
||||
"image describe command duration",
|
||||
"provider timeout observed",
|
||||
"gateway status after timeout",
|
||||
"mock provider request log"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["image describe command duration", "provider timeout observed", "gateway status after timeout", "mock provider request log"]
|
||||
},
|
||||
{
|
||||
"id": "post-media-health",
|
||||
"title": "Post-Media Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive and collect logs after the media timeout path.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"provider timeout logs",
|
||||
"plugin errors",
|
||||
"memory after media timeout"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "provider timeout logs", "plugin errors", "memory after media timeout"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -1,79 +0,0 @@
|
||||
{
|
||||
"id": "official-plugin-install",
|
||||
"surface": "official-plugin-install",
|
||||
"title": "Official Plugin Install",
|
||||
"objective": "Install a real published official OpenClaw plugin exactly as users do, then prove registry/list persistence and gateway restart health.",
|
||||
"tags": [
|
||||
"plugins",
|
||||
"official-plugin",
|
||||
"install",
|
||||
"security-scan",
|
||||
"release"
|
||||
],
|
||||
"states": [
|
||||
"official-plugins"
|
||||
],
|
||||
"timeoutMs": 240000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 45000,
|
||||
"gatewayReadyHardTimeoutMs": 120000,
|
||||
"pluginInstallMs": 120000,
|
||||
"officialPluginInstallOk": 1,
|
||||
"officialPluginSecurityBlocks": 0,
|
||||
"pluginsListMs": 10000,
|
||||
"missingDependencyErrors": 0,
|
||||
"pluginLoadFailures": 0
|
||||
},
|
||||
"phases": [
|
||||
{
|
||||
"id": "provision",
|
||||
"title": "Provision Official Plugin Env",
|
||||
"intent": "Start a fresh disposable OpenClaw env before running the published plugin install path.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- plugins list"
|
||||
],
|
||||
"evidence": [
|
||||
"fresh env started",
|
||||
"baseline plugin list captured"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
},
|
||||
{
|
||||
"id": "install",
|
||||
"title": "Install Official Plugin",
|
||||
"intent": "Run the same plugin install paths users run from terminals for every required plugin declared by the active official plugin state.",
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/run-official-plugin-install.mjs --env {env} --state {kovaRoot}/states/official-plugins.json --artifact-dir {artifactDir} --timeout-ms 120000"
|
||||
],
|
||||
"evidence": [
|
||||
"published plugin install results",
|
||||
"security scanner results",
|
||||
"plugins appear in list",
|
||||
"registry refresh succeeds"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
},
|
||||
{
|
||||
"id": "restart",
|
||||
"title": "Restart After Official Plugin Install",
|
||||
"intent": "Restart the gateway so OpenClaw loads persisted official plugin state without plugin load or dependency errors.",
|
||||
"commands": [
|
||||
"ocm service restart {env}",
|
||||
"ocm service status {env} --json",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm logs {env} --tail 400 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"restart readiness",
|
||||
"official plugin remains installed",
|
||||
"plugin load logs",
|
||||
"missing dependency scan"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
@ -3,13 +3,7 @@
|
||||
"surface": "openai-compatible-turn",
|
||||
"title": "OpenAI-Compatible Turn",
|
||||
"objective": "Send a chat-completions request to OpenClaw's OpenAI-compatible HTTP endpoint and verify the final assistant response, provider timing, auth handling, and gateway health.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"openai-compatible",
|
||||
"http",
|
||||
"providers"
|
||||
],
|
||||
"tags": ["agent", "message", "openai-compatible", "http", "providers"],
|
||||
"timeoutMs": 180000,
|
||||
"agent": {
|
||||
"expectedText": "KOVA_AGENT_OK"
|
||||
@ -30,30 +24,15 @@
|
||||
"id": "provision",
|
||||
"title": "Provision HTTP Env",
|
||||
"intent": "Create a disposable OpenClaw env without starting the gateway yet so Kova auth is applied before the gateway process boots.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --no-service --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"env created without service"
|
||||
],
|
||||
"healthScope": "none"
|
||||
"commands": ["ocm start {env} {startSelector} --no-service --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "env created without service"]
|
||||
},
|
||||
{
|
||||
"id": "gateway-start",
|
||||
"title": "Start Gateway",
|
||||
"intent": "Start the gateway after auth/provider config is already present in the OpenClaw home.",
|
||||
"commands": [
|
||||
"ocm service install {env} --json",
|
||||
"ocm service start {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway service installed",
|
||||
"gateway service started",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm service install {env} --json", "ocm service start {env} --json"],
|
||||
"evidence": ["gateway service installed", "gateway service started", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "openai-compatible-turn",
|
||||
@ -62,33 +41,14 @@
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/run-openai-compatible-turn.mjs --env {env} --model openai/gpt-5.5 --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --expected-text KOVA_AGENT_OK --timeout 120000"
|
||||
],
|
||||
"evidence": [
|
||||
"HTTP status",
|
||||
"final assistant text",
|
||||
"mock provider request timing",
|
||||
"gateway health after turn",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["HTTP status", "final assistant text", "mock provider request timing", "gateway health after turn", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-http-health",
|
||||
"title": "Post-HTTP Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after the OpenAI-compatible request.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"provider logs",
|
||||
"plugin errors",
|
||||
"memory after HTTP turn"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "provider logs", "plugin errors", "memory after HTTP turn"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "plugin-bad-manifest",
|
||||
"title": "Bad Plugin Manifest",
|
||||
"objective": "Attempt to install an invalid plugin fixture, require OpenClaw to reject it cleanly, and verify the gateway remains healthy.",
|
||||
"tags": [
|
||||
"plugins",
|
||||
"manifest",
|
||||
"failure",
|
||||
"lifecycle"
|
||||
],
|
||||
"tags": ["plugins", "manifest", "failure", "lifecycle"],
|
||||
"timeoutMs": 180000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
@ -21,48 +16,22 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Plugin Env",
|
||||
"intent": "Start a clean gateway before attempting the invalid plugin install.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"baseline gateway status",
|
||||
"readiness classification"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- status"],
|
||||
"evidence": ["baseline gateway status", "readiness classification"]
|
||||
},
|
||||
{
|
||||
"id": "reject-invalid-plugin",
|
||||
"title": "Reject Invalid Plugin",
|
||||
"intent": "Run the real install command and require it to fail without corrupting plugin state.",
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/expect-command-fails.mjs -- ocm @{env} -- plugins install {kovaRoot}/support/plugins/kova-bad-manifest"
|
||||
],
|
||||
"evidence": [
|
||||
"install command rejected",
|
||||
"validation error",
|
||||
"no install record committed"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["node {kovaRoot}/support/expect-command-fails.mjs -- ocm @{env} -- plugins install {kovaRoot}/support/plugins/kova-bad-manifest"],
|
||||
"evidence": ["install command rejected", "validation error", "no install record committed"]
|
||||
},
|
||||
{
|
||||
"id": "post-failure-health",
|
||||
"title": "Post-Failure Health",
|
||||
"intent": "Verify OpenClaw still answers status and plugin list after rejecting the bad manifest.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"plugin list",
|
||||
"logs after invalid install"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm @{env} -- plugins list", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "plugin list", "logs after invalid install"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "plugin-external-install",
|
||||
"title": "External Plugin Install",
|
||||
"objective": "Install a real local external plugin fixture, refresh OpenClaw plugin state, and verify the gateway remains healthy without dependency or registry errors.",
|
||||
"tags": [
|
||||
"plugins",
|
||||
"external-plugin",
|
||||
"install",
|
||||
"lifecycle"
|
||||
],
|
||||
"tags": ["plugins", "external-plugin", "install", "lifecycle"],
|
||||
"timeoutMs": 180000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
@ -22,51 +17,22 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Plugin Env",
|
||||
"intent": "Start a clean OpenClaw env before installing an external plugin.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- plugins list"
|
||||
],
|
||||
"evidence": [
|
||||
"baseline plugin list",
|
||||
"gateway readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- plugins list"],
|
||||
"evidence": ["baseline plugin list", "gateway readiness"]
|
||||
},
|
||||
{
|
||||
"id": "install",
|
||||
"title": "Install Local Plugin",
|
||||
"intent": "Use OpenClaw's real plugin install command against a local external plugin fixture.",
|
||||
"commands": [
|
||||
"ocm @{env} -- plugins install {kovaRoot}/support/plugins/kova-basic --force",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- plugins registry --refresh --json"
|
||||
],
|
||||
"evidence": [
|
||||
"install output",
|
||||
"plugin index update",
|
||||
"registry refresh",
|
||||
"plugin appears in list"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- plugins install {kovaRoot}/support/plugins/kova-basic --force", "ocm @{env} -- plugins list", "ocm @{env} -- plugins registry --refresh --json"],
|
||||
"evidence": ["install output", "plugin index update", "registry refresh", "plugin appears in list"]
|
||||
},
|
||||
{
|
||||
"id": "restart",
|
||||
"title": "Restart After Install",
|
||||
"intent": "Restart the gateway so OpenClaw loads the installed external plugin from persisted state.",
|
||||
"commands": [
|
||||
"ocm service restart {env}",
|
||||
"ocm service status {env} --json",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"restart readiness",
|
||||
"plugin load logs",
|
||||
"missing dependency scan"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm service restart {env}", "ocm service status {env} --json", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["restart readiness", "plugin load logs", "missing dependency scan"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "plugin-lifecycle",
|
||||
"title": "OpenClaw Plugin Lifecycle",
|
||||
"objective": "Exercise plugin list, update, install/remove-ready paths, runtime dependency diagnostics, restart behavior, and registry consistency.",
|
||||
"tags": [
|
||||
"plugins",
|
||||
"runtime-deps",
|
||||
"gateway",
|
||||
"restart"
|
||||
],
|
||||
"tags": ["plugins", "runtime-deps", "gateway", "restart"],
|
||||
"thresholds": {
|
||||
"pluginsListMs": 10000,
|
||||
"pluginUpdateDryRunMs": 20000,
|
||||
@ -19,36 +14,15 @@
|
||||
"id": "baseline",
|
||||
"title": "Plugin Baseline",
|
||||
"intent": "Inspect plugin state immediately after gateway startup.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- plugins update --all --dry-run"
|
||||
],
|
||||
"evidence": [
|
||||
"plugin list",
|
||||
"update dry-run",
|
||||
"runtime dependency errors"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- plugins list", "ocm @{env} -- plugins update --all --dry-run"],
|
||||
"evidence": ["plugin list", "update dry-run", "runtime dependency errors"]
|
||||
},
|
||||
{
|
||||
"id": "restart",
|
||||
"title": "Restart After Plugin Inspection",
|
||||
"intent": "Verify plugin state survives gateway restart and does not create missing dependency errors.",
|
||||
"commands": [
|
||||
"ocm service restart {env}",
|
||||
"ocm service status {env} --json",
|
||||
"ocm logs {env} --tail 250 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"restart status",
|
||||
"logs",
|
||||
"missing dependency scan"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm service restart {env}", "ocm service status {env} --json", "ocm logs {env} --tail 250 --raw"],
|
||||
"evidence": ["restart status", "logs", "missing dependency scan"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "plugin-missing-runtime-deps",
|
||||
"title": "Plugin Missing Runtime Deps",
|
||||
"objective": "Install a plugin that imports an undeclared runtime dependency and verify OpenClaw reports the dependency failure without taking down the gateway.",
|
||||
"tags": [
|
||||
"plugins",
|
||||
"runtime-deps",
|
||||
"failure",
|
||||
"lifecycle"
|
||||
],
|
||||
"tags": ["plugins", "runtime-deps", "failure", "lifecycle"],
|
||||
"timeoutMs": 180000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
@ -22,50 +17,22 @@
|
||||
"id": "install",
|
||||
"title": "Install Missing-Dependency Plugin",
|
||||
"intent": "Install a fixture whose runtime entry imports a package it does not declare.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- plugins install {kovaRoot}/support/plugins/kova-missing-runtime-dep --force",
|
||||
"ocm @{env} -- plugins list"
|
||||
],
|
||||
"evidence": [
|
||||
"install result",
|
||||
"plugin entry registered",
|
||||
"gateway readiness before load"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- plugins install {kovaRoot}/support/plugins/kova-missing-runtime-dep --force", "ocm @{env} -- plugins list"],
|
||||
"evidence": ["install result", "plugin entry registered", "gateway readiness before load"]
|
||||
},
|
||||
{
|
||||
"id": "restart",
|
||||
"title": "Restart And Capture Failure",
|
||||
"intent": "Restart so OpenClaw attempts to load the plugin and emits missing dependency diagnostics.",
|
||||
"commands": [
|
||||
"ocm service restart {env}",
|
||||
"ocm service status {env} --json",
|
||||
"ocm logs {env} --tail 400 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"missing dependency diagnostic",
|
||||
"plugin load failure",
|
||||
"gateway remains supervised"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm service restart {env}", "ocm service status {env} --json", "ocm logs {env} --tail 400 --raw"],
|
||||
"evidence": ["missing dependency diagnostic", "plugin load failure", "gateway remains supervised"]
|
||||
},
|
||||
{
|
||||
"id": "survival",
|
||||
"title": "Gateway Survival Check",
|
||||
"intent": "Confirm one bad plugin does not make the user-facing gateway unusable.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm @{env} -- plugins list"
|
||||
],
|
||||
"evidence": [
|
||||
"status after plugin failure",
|
||||
"plugin list after failure"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm @{env} -- plugins list"],
|
||||
"evidence": ["status after plugin failure", "plugin list after failure"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,13 +3,7 @@
|
||||
"surface": "plugin-remove",
|
||||
"title": "Plugin Remove",
|
||||
"objective": "Install and uninstall a local external plugin, then verify OpenClaw removes plugin install records cleanly and survives restart.",
|
||||
"tags": [
|
||||
"plugins",
|
||||
"external-plugin",
|
||||
"remove",
|
||||
"uninstall",
|
||||
"lifecycle"
|
||||
],
|
||||
"tags": ["plugins", "external-plugin", "remove", "uninstall", "lifecycle"],
|
||||
"timeoutMs": 180000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
@ -23,51 +17,22 @@
|
||||
"id": "install",
|
||||
"title": "Install Plugin To Remove",
|
||||
"intent": "Install the local fixture so uninstall exercises a real managed plugin record.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- plugins install {kovaRoot}/support/plugins/kova-basic --force",
|
||||
"ocm @{env} -- plugins list"
|
||||
],
|
||||
"evidence": [
|
||||
"install record",
|
||||
"plugin appears before uninstall"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- plugins install {kovaRoot}/support/plugins/kova-basic --force", "ocm @{env} -- plugins list"],
|
||||
"evidence": ["install record", "plugin appears before uninstall"]
|
||||
},
|
||||
{
|
||||
"id": "remove",
|
||||
"title": "Uninstall Plugin",
|
||||
"intent": "Use OpenClaw's real uninstall path and force promptless removal for automation.",
|
||||
"commands": [
|
||||
"ocm @{env} -- plugins uninstall kova-basic --force",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- plugins registry --refresh --json"
|
||||
],
|
||||
"evidence": [
|
||||
"uninstall output",
|
||||
"install index cleanup",
|
||||
"registry after removal"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- plugins uninstall kova-basic --force", "ocm @{env} -- plugins list", "ocm @{env} -- plugins registry --refresh --json"],
|
||||
"evidence": ["uninstall output", "install index cleanup", "registry after removal"]
|
||||
},
|
||||
{
|
||||
"id": "restart",
|
||||
"title": "Restart After Removal",
|
||||
"intent": "Restart after uninstall to verify removed plugin state does not break gateway startup.",
|
||||
"commands": [
|
||||
"ocm service restart {env}",
|
||||
"ocm service status {env} --json",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"restart readiness",
|
||||
"removed plugin not loaded",
|
||||
"missing dependency scan"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm service restart {env}", "ocm service status {env} --json", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["restart readiness", "removed plugin not loaded", "missing dependency scan"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "plugin-update",
|
||||
"title": "Plugin Update",
|
||||
"objective": "Install a managed external plugin and run OpenClaw plugin update paths to verify tracked plugin metadata and dry-run update diagnostics.",
|
||||
"tags": [
|
||||
"plugins",
|
||||
"external-plugin",
|
||||
"update",
|
||||
"lifecycle"
|
||||
],
|
||||
"tags": ["plugins", "external-plugin", "update", "lifecycle"],
|
||||
"timeoutMs": 180000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
@ -23,50 +18,22 @@
|
||||
"id": "install",
|
||||
"title": "Install Managed Plugin",
|
||||
"intent": "Create a tracked plugin install record for update checks.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- plugins install {kovaRoot}/support/plugins/kova-basic --force",
|
||||
"ocm @{env} -- plugins list"
|
||||
],
|
||||
"evidence": [
|
||||
"plugin install record",
|
||||
"plugin appears in list"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- plugins install {kovaRoot}/support/plugins/kova-basic --force", "ocm @{env} -- plugins list"],
|
||||
"evidence": ["plugin install record", "plugin appears in list"]
|
||||
},
|
||||
{
|
||||
"id": "update",
|
||||
"title": "Update Plugin Metadata",
|
||||
"intent": "Run individual and all-plugin update dry-runs so OpenClaw reports update plans without mutating the fixture.",
|
||||
"commands": [
|
||||
"ocm @{env} -- plugins update kova-basic --dry-run",
|
||||
"ocm @{env} -- plugins update --all --dry-run",
|
||||
"ocm @{env} -- plugins registry --refresh --json"
|
||||
],
|
||||
"evidence": [
|
||||
"plugin update dry-run output",
|
||||
"tracked plugin metadata",
|
||||
"registry refresh"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- plugins update kova-basic --dry-run", "ocm @{env} -- plugins update --all --dry-run", "ocm @{env} -- plugins registry --refresh --json"],
|
||||
"evidence": ["plugin update dry-run output", "tracked plugin metadata", "registry refresh"]
|
||||
},
|
||||
{
|
||||
"id": "post-update-health",
|
||||
"title": "Post-Update Health",
|
||||
"intent": "Verify the gateway is still usable after update planning.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"status after update",
|
||||
"plugin lifecycle logs",
|
||||
"dependency errors"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["status after update", "plugin lifecycle logs", "dependency errors"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "provider-models",
|
||||
"title": "Provider And Model Discovery",
|
||||
"objective": "Exercise model/provider discovery paths and verify slow, missing, or unauthenticated providers do not stall the gateway globally.",
|
||||
"tags": [
|
||||
"models",
|
||||
"providers",
|
||||
"timeouts",
|
||||
"gateway"
|
||||
],
|
||||
"tags": ["models", "providers", "timeouts", "gateway"],
|
||||
"thresholds": {
|
||||
"modelsListMs": 20000,
|
||||
"statusAfterModelsMs": 10000,
|
||||
@ -19,34 +14,15 @@
|
||||
"id": "baseline",
|
||||
"title": "Provider Discovery Baseline",
|
||||
"intent": "Start the gateway and run model discovery with the target runtime's default config.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- models list",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"models list duration",
|
||||
"provider timeout warnings",
|
||||
"gateway status after model discovery"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- models list", "ocm @{env} -- status"],
|
||||
"evidence": ["models list duration", "provider timeout warnings", "gateway status after model discovery"]
|
||||
},
|
||||
{
|
||||
"id": "logs",
|
||||
"title": "Provider Logs",
|
||||
"intent": "Capture logs for provider/model discovery stalls, missing auth, and timeout handling.",
|
||||
"commands": [
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"timeout logs",
|
||||
"auth skip logs",
|
||||
"gateway stall logs"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["timeout logs", "auth skip logs", "gateway stall logs"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,14 +3,7 @@
|
||||
"surface": "release-runtime-startup",
|
||||
"title": "Release Runtime Startup",
|
||||
"objective": "Start a release-shaped OpenClaw runtime, measure cold readiness and bundled runtime dependency staging, and fail on degraded plugin startup.",
|
||||
"tags": [
|
||||
"release-runtime",
|
||||
"local-build",
|
||||
"gateway",
|
||||
"plugins",
|
||||
"runtime-deps",
|
||||
"startup"
|
||||
],
|
||||
"tags": ["release-runtime", "local-build", "gateway", "plugins", "runtime-deps", "startup"],
|
||||
"timeoutMs": 180000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
@ -27,52 +20,22 @@
|
||||
"id": "provision",
|
||||
"title": "Provision Release Runtime Env",
|
||||
"intent": "Start OpenClaw from the selected release-shaped runtime and keep probing beyond the expected threshold so slow startups are classified.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"runtime binding",
|
||||
"gateway port",
|
||||
"time to listening",
|
||||
"time to health ready",
|
||||
"readiness classification"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json"],
|
||||
"evidence": ["runtime binding", "gateway port", "time to listening", "time to health ready", "readiness classification"]
|
||||
},
|
||||
{
|
||||
"id": "post-start",
|
||||
"title": "Post-Start Runtime Checks",
|
||||
"intent": "Verify the gateway is usable after cold startup and capture plugin registry behavior.",
|
||||
"commands": [
|
||||
"ocm service status {env} --json",
|
||||
"ocm @{env} -- status",
|
||||
"ocm @{env} -- plugins list"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway state",
|
||||
"status command latency",
|
||||
"plugin list",
|
||||
"plugin startup health"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm service status {env} --json", "ocm @{env} -- status", "ocm @{env} -- plugins list"],
|
||||
"evidence": ["gateway state", "status command latency", "plugin list", "plugin startup health"]
|
||||
},
|
||||
{
|
||||
"id": "startup-logs",
|
||||
"title": "Startup Logs And Runtime Deps",
|
||||
"intent": "Capture dependency staging, missing package/module errors, plugin service failures, and startup timing logs.",
|
||||
"commands": [
|
||||
"ocm logs {env} --tail 400 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"runtime dependency staging duration",
|
||||
"missing dependency errors",
|
||||
"plugin service failures",
|
||||
"startup phase logs"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm logs {env} --tail 400 --raw"],
|
||||
"evidence": ["runtime dependency staging duration", "missing dependency errors", "plugin service failures", "startup phase logs"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "soak",
|
||||
"title": "Gateway Soak And Memory Trend",
|
||||
"objective": "Run a gateway for a longer window with repeated user-facing checks to detect memory growth, event-loop stalls, and degraded responsiveness.",
|
||||
"tags": [
|
||||
"soak",
|
||||
"memory",
|
||||
"latency",
|
||||
"gateway"
|
||||
],
|
||||
"tags": ["soak", "memory", "latency", "gateway"],
|
||||
"timeoutMs": 300000,
|
||||
"thresholds": {
|
||||
"soakMinDurationMs": 60000,
|
||||
@ -16,23 +11,15 @@
|
||||
"soakCommandFailures": 0,
|
||||
"soakHealthFailures": 0,
|
||||
"rssGrowthMb": 300,
|
||||
"postReadyHealthP95Ms": 1000
|
||||
"healthP95Ms": 1000
|
||||
},
|
||||
"phases": [
|
||||
{
|
||||
"id": "start",
|
||||
"title": "Start Gateway",
|
||||
"intent": "Start the target runtime and capture baseline status.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm service status {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"baseline PID",
|
||||
"baseline RSS",
|
||||
"baseline health"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm service status {env} --json"],
|
||||
"evidence": ["baseline PID", "baseline RSS", "baseline health"]
|
||||
},
|
||||
{
|
||||
"id": "loop",
|
||||
@ -41,15 +28,7 @@
|
||||
"commands": [
|
||||
"node support/run-soak-loop.mjs --env {env} --duration-ms 60000 --interval-ms 1000 --timeout-ms 30000"
|
||||
],
|
||||
"evidence": [
|
||||
"latency trend",
|
||||
"RSS trend",
|
||||
"logs during loop"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["latency trend", "RSS trend", "logs during loop"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,13 +3,7 @@
|
||||
"surface": "tui-message-turn",
|
||||
"title": "TUI Message Turn",
|
||||
"objective": "Launch the TUI, send a real user message through stdin, and require the expected assistant response to appear in the TUI output.",
|
||||
"tags": [
|
||||
"agent",
|
||||
"message",
|
||||
"tui",
|
||||
"input",
|
||||
"providers"
|
||||
],
|
||||
"tags": ["agent", "message", "tui", "input", "providers"],
|
||||
"timeoutMs": 180000,
|
||||
"agent": {
|
||||
"expectedText": "KOVA_AGENT_OK"
|
||||
@ -30,30 +24,15 @@
|
||||
"id": "provision",
|
||||
"title": "Provision TUI Env",
|
||||
"intent": "Create a disposable OpenClaw env without starting the gateway yet so Kova auth is applied before the gateway process boots.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --no-service --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway port",
|
||||
"runtime binding",
|
||||
"env created without service"
|
||||
],
|
||||
"healthScope": "none"
|
||||
"commands": ["ocm start {env} {startSelector} --no-service --json"],
|
||||
"evidence": ["gateway port", "runtime binding", "env created without service"]
|
||||
},
|
||||
{
|
||||
"id": "gateway-start",
|
||||
"title": "Start Gateway",
|
||||
"intent": "Start the gateway after auth/provider config is already present in the OpenClaw home.",
|
||||
"commands": [
|
||||
"ocm service install {env} --json",
|
||||
"ocm service start {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway service installed",
|
||||
"gateway service started",
|
||||
"startup readiness"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm service install {env} --json", "ocm service start {env} --json"],
|
||||
"evidence": ["gateway service installed", "gateway service started", "startup readiness"]
|
||||
},
|
||||
{
|
||||
"id": "tui-message-turn",
|
||||
@ -62,33 +41,14 @@
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/run-tui-message-turn.mjs --env {env} --session kova-tui-message --message 'Reply with exact ASCII text KOVA_AGENT_OK only.' --expected-text KOVA_AGENT_OK --timeout 120000"
|
||||
],
|
||||
"evidence": [
|
||||
"TUI input accepted",
|
||||
"final assistant text rendered",
|
||||
"mock provider request timing",
|
||||
"gateway health after turn",
|
||||
"role resource samples"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["TUI input accepted", "final assistant text rendered", "mock provider request timing", "gateway health after turn", "role resource samples"]
|
||||
},
|
||||
{
|
||||
"id": "post-tui-health",
|
||||
"title": "Post-TUI Gateway Health",
|
||||
"intent": "Verify the gateway remains responsive after the TUI message turn.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"provider logs",
|
||||
"plugin errors",
|
||||
"memory after TUI turn"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["gateway status", "provider logs", "plugin errors", "memory after TUI turn"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,12 +3,7 @@
|
||||
"surface": "tui",
|
||||
"title": "TUI Responsiveness",
|
||||
"objective": "Verify OpenClaw's terminal UI can attach to a running gateway, render a connected screen promptly, and shut down cleanly.",
|
||||
"tags": [
|
||||
"tui",
|
||||
"terminal",
|
||||
"gateway",
|
||||
"input"
|
||||
],
|
||||
"tags": ["tui", "terminal", "gateway", "input"],
|
||||
"timeoutMs": 120000,
|
||||
"thresholds": {
|
||||
"gatewayReadyMs": 30000,
|
||||
@ -23,47 +18,22 @@
|
||||
"id": "gateway",
|
||||
"title": "Gateway Start",
|
||||
"intent": "Start the gateway and confirm it is healthy before opening TUI.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway status",
|
||||
"readiness classification"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm @{env} -- status"],
|
||||
"evidence": ["gateway status", "readiness classification"]
|
||||
},
|
||||
{
|
||||
"id": "tui-smoke",
|
||||
"title": "TUI Attach Smoke",
|
||||
"intent": "Spawn the real TUI, wait for a recognizable connected screen, then interrupt it cleanly.",
|
||||
"commands": [
|
||||
"node {kovaRoot}/support/tui-smoke.mjs {env} 15000"
|
||||
],
|
||||
"evidence": [
|
||||
"TUI render time",
|
||||
"connected screen",
|
||||
"clean interrupt"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["node {kovaRoot}/support/tui-smoke.mjs {env} 15000"],
|
||||
"evidence": ["TUI render time", "connected screen", "clean interrupt"]
|
||||
},
|
||||
{
|
||||
"id": "post-tui-health",
|
||||
"title": "Post-TUI Health",
|
||||
"intent": "Verify the gateway remains responsive after a terminal UI session starts and exits.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"status after TUI",
|
||||
"TUI disconnect logs",
|
||||
"gateway errors"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["status after TUI", "TUI disconnect logs", "gateway errors"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,22 +3,9 @@
|
||||
"surface": "upgrade-existing-user",
|
||||
"title": "Durable Clone To Local Build Upgrade",
|
||||
"objective": "Clone a real existing OpenClaw env into disposable state, upgrade the clone to a release-shaped local build, and verify user-state migration, plugin indexes, bundled runtime deps, and gateway health without mutating the source env.",
|
||||
"tags": [
|
||||
"existing-user",
|
||||
"upgrade",
|
||||
"local-build",
|
||||
"plugins",
|
||||
"migration",
|
||||
"gateway"
|
||||
],
|
||||
"states": [
|
||||
"onboarded-user",
|
||||
"plugin-index",
|
||||
"failed-upgrade"
|
||||
],
|
||||
"targetKinds": [
|
||||
"local-build"
|
||||
],
|
||||
"tags": ["existing-user", "upgrade", "local-build", "plugins", "migration", "gateway"],
|
||||
"states": ["onboarded-user", "plugin-index", "failed-upgrade"],
|
||||
"targetKinds": ["local-build"],
|
||||
"timeoutMs": 300000,
|
||||
"thresholds": {
|
||||
"upgradeMs": 180000,
|
||||
@ -32,55 +19,22 @@
|
||||
"id": "clone",
|
||||
"title": "Clone Existing Env",
|
||||
"intent": "Clone a durable existing env into a disposable Kova env before any local-build upgrade action.",
|
||||
"commands": [
|
||||
"ocm env clone {sourceEnv} {env} --json",
|
||||
"ocm service status {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"clone result",
|
||||
"source env",
|
||||
"clone root",
|
||||
"pre-upgrade service status"
|
||||
],
|
||||
"healthScope": "none"
|
||||
"commands": ["ocm env clone {sourceEnv} {env} --json", "ocm service status {env} --json"],
|
||||
"evidence": ["clone result", "source env", "clone root", "pre-upgrade service status"]
|
||||
},
|
||||
{
|
||||
"id": "upgrade",
|
||||
"title": "Upgrade Clone To Local Build",
|
||||
"intent": "Run the real OpenClaw upgrade path against the cloned user state using the release-shaped local build runtime.",
|
||||
"commands": [
|
||||
"ocm upgrade {env} {upgradeSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"local-build upgrade JSON",
|
||||
"snapshot id",
|
||||
"doctor/update output",
|
||||
"rollback status"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm upgrade {env} {upgradeSelector} --json"],
|
||||
"evidence": ["local-build upgrade JSON", "snapshot id", "doctor/update output", "rollback status"]
|
||||
},
|
||||
{
|
||||
"id": "post-upgrade",
|
||||
"title": "Post-Upgrade Clone Checks",
|
||||
"intent": "Verify cloned user state survives local-build upgrade, including plugin index migration and gateway readiness.",
|
||||
"commands": [
|
||||
"ocm service status {env} --json",
|
||||
"ocm @{env} -- status",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- doctor --fix",
|
||||
"ocm logs {env} --tail 400 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway state",
|
||||
"status output",
|
||||
"plugins install index",
|
||||
"doctor output",
|
||||
"gateway logs without missing dependency/plugin load failures"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm service status {env} --json", "ocm @{env} -- status", "ocm @{env} -- plugins list", "ocm @{env} -- doctor --fix", "ocm logs {env} --tail 400 --raw"],
|
||||
"evidence": ["gateway state", "status output", "plugins install index", "doctor output", "gateway logs without missing dependency/plugin load failures"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,13 +3,7 @@
|
||||
"surface": "upgrade-existing-user",
|
||||
"title": "Existing OpenClaw User Upgrade",
|
||||
"objective": "Clone existing OpenClaw state, move the clone through a real upgrade path, and verify migrations, plugin indexes, runtime deps, gateway readiness, and diagnostics.",
|
||||
"tags": [
|
||||
"existing-user",
|
||||
"upgrade",
|
||||
"plugins",
|
||||
"migration",
|
||||
"gateway"
|
||||
],
|
||||
"tags": ["existing-user", "upgrade", "plugins", "migration", "gateway"],
|
||||
"timeoutMs": 180000,
|
||||
"thresholds": {
|
||||
"upgradeMs": 120000,
|
||||
@ -21,65 +15,29 @@
|
||||
"id": "clone",
|
||||
"title": "Clone Existing Env",
|
||||
"intent": "Clone a durable existing env into a disposable Kova env before any upgrade actions.",
|
||||
"commands": [
|
||||
"ocm env clone {sourceEnv} {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"clone result",
|
||||
"source env",
|
||||
"clone root"
|
||||
],
|
||||
"healthScope": "none"
|
||||
"commands": ["ocm env clone {sourceEnv} {env} --json"],
|
||||
"evidence": ["clone result", "source env", "clone root"]
|
||||
},
|
||||
{
|
||||
"id": "source-runtime",
|
||||
"title": "Prepare Source Runtime",
|
||||
"intent": "Optionally bind the clone to the source release and start it once to establish pre-upgrade state.",
|
||||
"commands": [
|
||||
"ocm upgrade {env} {fromUpgradeSelector} --json",
|
||||
"ocm service status {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"pre-upgrade runtime",
|
||||
"pre-upgrade gateway status"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm upgrade {env} {fromUpgradeSelector} --json", "ocm service status {env} --json"],
|
||||
"evidence": ["pre-upgrade runtime", "pre-upgrade gateway status"]
|
||||
},
|
||||
{
|
||||
"id": "upgrade",
|
||||
"title": "Upgrade To Target",
|
||||
"intent": "Run the real OpenClaw upgrade path through OCM and verify post-upgrade service reconciliation.",
|
||||
"commands": [
|
||||
"ocm upgrade {env} {upgradeSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"upgrade JSON",
|
||||
"snapshot id",
|
||||
"doctor/update output",
|
||||
"rollback status"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm upgrade {env} {upgradeSelector} --json"],
|
||||
"evidence": ["upgrade JSON", "snapshot id", "doctor/update output", "rollback status"]
|
||||
},
|
||||
{
|
||||
"id": "post-upgrade",
|
||||
"title": "Post-Upgrade OpenClaw Checks",
|
||||
"intent": "Verify OpenClaw state after upgrade, including plugin install index and gateway readiness.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- doctor --fix",
|
||||
"ocm logs {env} --tail 250 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"status output",
|
||||
"plugins folder/index presence",
|
||||
"doctor output",
|
||||
"gateway logs"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm @{env} -- plugins list", "ocm @{env} -- doctor --fix", "ocm logs {env} --tail 250 --raw"],
|
||||
"evidence": ["status output", "plugins folder/index presence", "doctor output", "gateway logs"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,17 +3,8 @@
|
||||
"surface": "upgrade-existing-user",
|
||||
"title": "Upgrade From OpenClaw 2026.4.20",
|
||||
"objective": "Clone existing OpenClaw user state, move the clone through the real 2026.4.20 runtime, then upgrade the same clone to the target and verify migration, plugin index, runtime deps, and gateway health.",
|
||||
"tags": [
|
||||
"existing-user",
|
||||
"upgrade",
|
||||
"old-release",
|
||||
"plugins",
|
||||
"migration",
|
||||
"gateway"
|
||||
],
|
||||
"states": [
|
||||
"old-release-2026-4-20-user"
|
||||
],
|
||||
"tags": ["existing-user", "upgrade", "old-release", "plugins", "migration", "gateway"],
|
||||
"states": ["old-release-2026-4-20-user"],
|
||||
"timeoutMs": 240000,
|
||||
"thresholds": {
|
||||
"upgradeMs": 180000,
|
||||
@ -25,67 +16,29 @@
|
||||
"id": "clone",
|
||||
"title": "Clone Existing Env",
|
||||
"intent": "Clone a durable existing env into a disposable Kova env before any old-release or upgrade action.",
|
||||
"commands": [
|
||||
"ocm env clone {sourceEnv} {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"clone result",
|
||||
"source env",
|
||||
"clone root"
|
||||
],
|
||||
"healthScope": "none"
|
||||
"commands": ["ocm env clone {sourceEnv} {env} --json"],
|
||||
"evidence": ["clone result", "source env", "clone root"]
|
||||
},
|
||||
{
|
||||
"id": "source-runtime",
|
||||
"title": "Move Clone To 2026.4.20",
|
||||
"intent": "Use the real OCM upgrade path to bind and reconcile the cloned env on OpenClaw 2026.4.20 before the target upgrade.",
|
||||
"commands": [
|
||||
"ocm upgrade {env} --version 2026.4.20 --json",
|
||||
"ocm service status {env} --json",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"2026.4.20 upgrade output",
|
||||
"pre-upgrade service status",
|
||||
"pre-upgrade OpenClaw status"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm upgrade {env} --version 2026.4.20 --json", "ocm service status {env} --json", "ocm @{env} -- status"],
|
||||
"evidence": ["2026.4.20 upgrade output", "pre-upgrade service status", "pre-upgrade OpenClaw status"]
|
||||
},
|
||||
{
|
||||
"id": "upgrade",
|
||||
"title": "Upgrade To Target",
|
||||
"intent": "Run the real target upgrade path through OCM and verify post-upgrade service reconciliation.",
|
||||
"commands": [
|
||||
"ocm upgrade {env} {upgradeSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"target upgrade JSON",
|
||||
"snapshot id",
|
||||
"doctor/update output",
|
||||
"rollback status"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm upgrade {env} {upgradeSelector} --json"],
|
||||
"evidence": ["target upgrade JSON", "snapshot id", "doctor/update output", "rollback status"]
|
||||
},
|
||||
{
|
||||
"id": "post-upgrade",
|
||||
"title": "Post-Upgrade OpenClaw Checks",
|
||||
"intent": "Verify OpenClaw state after upgrade, including plugin install index, plugin command behavior, doctor recovery, and gateway logs.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- doctor --fix",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"status output",
|
||||
"plugins folder/index presence",
|
||||
"doctor output",
|
||||
"gateway logs"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm @{env} -- plugins list", "ocm @{env} -- doctor --fix", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["status output", "plugins folder/index presence", "doctor output", "gateway logs"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,18 +3,8 @@
|
||||
"surface": "upgrade-existing-user",
|
||||
"title": "Upgrade From OpenClaw 2026.4.24",
|
||||
"objective": "Clone existing OpenClaw user state, move the clone through the real 2026.4.24 plugin-architecture release, then upgrade the same clone to the target and verify plugin index migration, bundled plugin runtime deps, doctor recovery, and gateway health.",
|
||||
"tags": [
|
||||
"existing-user",
|
||||
"upgrade",
|
||||
"old-release",
|
||||
"plugin-architecture",
|
||||
"plugins",
|
||||
"migration",
|
||||
"gateway"
|
||||
],
|
||||
"states": [
|
||||
"old-release-2026-4-24-user"
|
||||
],
|
||||
"tags": ["existing-user", "upgrade", "old-release", "plugin-architecture", "plugins", "migration", "gateway"],
|
||||
"states": ["old-release-2026-4-24-user"],
|
||||
"timeoutMs": 240000,
|
||||
"thresholds": {
|
||||
"upgradeMs": 180000,
|
||||
@ -28,69 +18,29 @@
|
||||
"id": "clone",
|
||||
"title": "Clone Existing Env",
|
||||
"intent": "Clone a durable existing env into a disposable Kova env before any old-release or upgrade action.",
|
||||
"commands": [
|
||||
"ocm env clone {sourceEnv} {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"clone result",
|
||||
"source env",
|
||||
"clone root"
|
||||
],
|
||||
"healthScope": "none"
|
||||
"commands": ["ocm env clone {sourceEnv} {env} --json"],
|
||||
"evidence": ["clone result", "source env", "clone root"]
|
||||
},
|
||||
{
|
||||
"id": "source-runtime",
|
||||
"title": "Move Clone To 2026.4.24",
|
||||
"intent": "Use the real OCM upgrade path to bind and reconcile the cloned env on OpenClaw 2026.4.24 before the target upgrade.",
|
||||
"commands": [
|
||||
"ocm upgrade {env} --version 2026.4.24 --json",
|
||||
"ocm service status {env} --json",
|
||||
"ocm @{env} -- status",
|
||||
"ocm logs {env} --tail 300 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"2026.4.24 upgrade output",
|
||||
"pre-upgrade service status",
|
||||
"pre-upgrade OpenClaw status",
|
||||
"known 2026.4.24 plugin/runtime-deps logs"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm upgrade {env} --version 2026.4.24 --json", "ocm service status {env} --json", "ocm @{env} -- status", "ocm logs {env} --tail 300 --raw"],
|
||||
"evidence": ["2026.4.24 upgrade output", "pre-upgrade service status", "pre-upgrade OpenClaw status", "known 2026.4.24 plugin/runtime-deps logs"]
|
||||
},
|
||||
{
|
||||
"id": "upgrade",
|
||||
"title": "Upgrade To Target",
|
||||
"intent": "Run the real target upgrade path through OCM and verify post-upgrade service reconciliation.",
|
||||
"commands": [
|
||||
"ocm upgrade {env} {upgradeSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"target upgrade JSON",
|
||||
"snapshot id",
|
||||
"doctor/update output",
|
||||
"rollback status"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm upgrade {env} {upgradeSelector} --json"],
|
||||
"evidence": ["target upgrade JSON", "snapshot id", "doctor/update output", "rollback status"]
|
||||
},
|
||||
{
|
||||
"id": "post-upgrade",
|
||||
"title": "Post-Upgrade Plugin Checks",
|
||||
"intent": "Verify the target OpenClaw fixes plugin architecture state from 2026.4.24 without missing dependency errors or plugin load failures.",
|
||||
"commands": [
|
||||
"ocm @{env} -- status",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- doctor --fix",
|
||||
"ocm logs {env} --tail 400 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"status output",
|
||||
"plugins install index",
|
||||
"doctor output",
|
||||
"gateway logs without missing dependency/plugin load failures"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm @{env} -- status", "ocm @{env} -- plugins list", "ocm @{env} -- doctor --fix", "ocm logs {env} --tail 400 --raw"],
|
||||
"evidence": ["status output", "plugins install index", "doctor output", "gateway logs without missing dependency/plugin load failures"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,23 +3,10 @@
|
||||
"surface": "upgrade-existing-user",
|
||||
"title": "Stable Channel To Beta Upgrade",
|
||||
"objective": "Start a disposable OpenClaw env on the stable channel, run the real upgrade path to beta, and verify channel upgrade behavior, plugin index migration, bundled runtime deps, and gateway health.",
|
||||
"tags": [
|
||||
"upgrade",
|
||||
"channel",
|
||||
"stable",
|
||||
"beta",
|
||||
"plugins",
|
||||
"gateway"
|
||||
],
|
||||
"states": [
|
||||
"stable-channel-user"
|
||||
],
|
||||
"targetKinds": [
|
||||
"channel"
|
||||
],
|
||||
"targetValues": [
|
||||
"beta"
|
||||
],
|
||||
"tags": ["upgrade", "channel", "stable", "beta", "plugins", "gateway"],
|
||||
"states": ["stable-channel-user"],
|
||||
"targetKinds": ["channel"],
|
||||
"targetValues": ["beta"],
|
||||
"timeoutMs": 240000,
|
||||
"thresholds": {
|
||||
"upgradeMs": 180000,
|
||||
@ -33,55 +20,22 @@
|
||||
"id": "start",
|
||||
"title": "Start Stable Channel Env",
|
||||
"intent": "Create a disposable env through the real stable channel before switching to beta.",
|
||||
"commands": [
|
||||
"ocm start {env} --channel stable --json",
|
||||
"ocm service status {env} --json",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"stable channel start output",
|
||||
"pre-upgrade gateway status",
|
||||
"pre-upgrade OpenClaw status"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} --channel stable --json", "ocm service status {env} --json", "ocm @{env} -- status"],
|
||||
"evidence": ["stable channel start output", "pre-upgrade gateway status", "pre-upgrade OpenClaw status"]
|
||||
},
|
||||
{
|
||||
"id": "upgrade",
|
||||
"title": "Upgrade To Beta",
|
||||
"intent": "Run the real OpenClaw upgrade path to the beta channel through OCM.",
|
||||
"commands": [
|
||||
"ocm upgrade {env} {upgradeSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"beta upgrade JSON",
|
||||
"snapshot id",
|
||||
"doctor/update output",
|
||||
"rollback status"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm upgrade {env} {upgradeSelector} --json"],
|
||||
"evidence": ["beta upgrade JSON", "snapshot id", "doctor/update output", "rollback status"]
|
||||
},
|
||||
{
|
||||
"id": "post-upgrade",
|
||||
"title": "Post-Beta Upgrade Checks",
|
||||
"intent": "Verify OpenClaw is usable after the beta channel upgrade and capture plugin/runtime dependency evidence.",
|
||||
"commands": [
|
||||
"ocm service status {env} --json",
|
||||
"ocm @{env} -- status",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- doctor --fix",
|
||||
"ocm logs {env} --tail 400 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway state",
|
||||
"status output",
|
||||
"plugins install index",
|
||||
"doctor output",
|
||||
"gateway logs without missing dependency/plugin load failures"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm service status {env} --json", "ocm @{env} -- status", "ocm @{env} -- plugins list", "ocm @{env} -- doctor --fix", "ocm logs {env} --tail 400 --raw"],
|
||||
"evidence": ["gateway state", "status output", "plugins install index", "doctor output", "gateway logs without missing dependency/plugin load failures"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,20 +3,9 @@
|
||||
"surface": "upgrade-existing-user",
|
||||
"title": "Stable Channel To Local Build Upgrade",
|
||||
"objective": "Start a disposable OpenClaw env on the stable channel, upgrade it to a release-shaped local build, and verify the same migration/runtime behavior users would see when updating to the upcoming release.",
|
||||
"tags": [
|
||||
"upgrade",
|
||||
"channel",
|
||||
"stable",
|
||||
"local-build",
|
||||
"plugins",
|
||||
"gateway"
|
||||
],
|
||||
"states": [
|
||||
"stable-channel-user"
|
||||
],
|
||||
"targetKinds": [
|
||||
"local-build"
|
||||
],
|
||||
"tags": ["upgrade", "channel", "stable", "local-build", "plugins", "gateway"],
|
||||
"states": ["stable-channel-user"],
|
||||
"targetKinds": ["local-build"],
|
||||
"timeoutMs": 300000,
|
||||
"thresholds": {
|
||||
"upgradeMs": 180000,
|
||||
@ -30,55 +19,22 @@
|
||||
"id": "start",
|
||||
"title": "Start Stable Channel Env",
|
||||
"intent": "Create a disposable env through the real stable channel before upgrading to the local release-shaped runtime.",
|
||||
"commands": [
|
||||
"ocm start {env} --channel stable --json",
|
||||
"ocm service status {env} --json",
|
||||
"ocm @{env} -- status"
|
||||
],
|
||||
"evidence": [
|
||||
"stable channel start output",
|
||||
"pre-upgrade gateway status",
|
||||
"pre-upgrade OpenClaw status"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} --channel stable --json", "ocm service status {env} --json", "ocm @{env} -- status"],
|
||||
"evidence": ["stable channel start output", "pre-upgrade gateway status", "pre-upgrade OpenClaw status"]
|
||||
},
|
||||
{
|
||||
"id": "upgrade",
|
||||
"title": "Upgrade To Local Build",
|
||||
"intent": "Run the real OpenClaw upgrade path to the release-shaped local build runtime.",
|
||||
"commands": [
|
||||
"ocm upgrade {env} {upgradeSelector} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"local-build upgrade JSON",
|
||||
"snapshot id",
|
||||
"doctor/update output",
|
||||
"rollback status"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm upgrade {env} {upgradeSelector} --json"],
|
||||
"evidence": ["local-build upgrade JSON", "snapshot id", "doctor/update output", "rollback status"]
|
||||
},
|
||||
{
|
||||
"id": "post-upgrade",
|
||||
"title": "Post-Local-Build Upgrade Checks",
|
||||
"intent": "Verify the local build handles plugin indexes, runtime deps, doctor recovery, and gateway readiness after upgrade.",
|
||||
"commands": [
|
||||
"ocm service status {env} --json",
|
||||
"ocm @{env} -- status",
|
||||
"ocm @{env} -- plugins list",
|
||||
"ocm @{env} -- doctor --fix",
|
||||
"ocm logs {env} --tail 400 --raw"
|
||||
],
|
||||
"evidence": [
|
||||
"gateway state",
|
||||
"status output",
|
||||
"plugins install index",
|
||||
"doctor output",
|
||||
"gateway logs without missing dependency/plugin load failures"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"commands": ["ocm service status {env} --json", "ocm @{env} -- status", "ocm @{env} -- plugins list", "ocm @{env} -- doctor --fix", "ocm logs {env} --tail 400 --raw"],
|
||||
"evidence": ["gateway state", "status output", "plugins install index", "doctor output", "gateway logs without missing dependency/plugin load failures"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -3,17 +3,8 @@
|
||||
"surface": "workspace-scan",
|
||||
"title": "Workspace Scan Pressure",
|
||||
"objective": "Start OpenClaw, create a large workspace fixture, restart the gateway, and verify user-facing commands stay responsive without memory, CPU, or health degradation.",
|
||||
"tags": [
|
||||
"workspace",
|
||||
"filesystem",
|
||||
"performance",
|
||||
"gateway",
|
||||
"models",
|
||||
"plugins"
|
||||
],
|
||||
"states": [
|
||||
"large-workspace"
|
||||
],
|
||||
"tags": ["workspace", "filesystem", "performance", "gateway", "models", "plugins"],
|
||||
"states": ["large-workspace"],
|
||||
"timeoutMs": 240000,
|
||||
"thresholds": {
|
||||
"warmReadyMs": 30000,
|
||||
@ -25,39 +16,23 @@
|
||||
"soakCommandFailures": 0,
|
||||
"soakHealthFailures": 0,
|
||||
"soakHealthP95Ms": 1000,
|
||||
"peakRssMb": 1000,
|
||||
"postReadyHealthP95Ms": 1000
|
||||
"healthP95Ms": 1000,
|
||||
"peakRssMb": 1000
|
||||
},
|
||||
"phases": [
|
||||
{
|
||||
"id": "start",
|
||||
"title": "Start Gateway",
|
||||
"intent": "Start the target OpenClaw runtime before applying the large workspace fixture.",
|
||||
"commands": [
|
||||
"ocm start {env} {startSelector} --json",
|
||||
"ocm service status {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"startup readiness",
|
||||
"gateway PID",
|
||||
"baseline RSS and CPU"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm start {env} {startSelector} --json", "ocm service status {env} --json"],
|
||||
"evidence": ["startup readiness", "gateway PID", "baseline RSS and CPU"]
|
||||
},
|
||||
{
|
||||
"id": "restart-after-workspace",
|
||||
"title": "Restart With Large Workspace",
|
||||
"intent": "Restart the gateway after the large workspace fixture exists so startup and reload paths see filesystem pressure.",
|
||||
"commands": [
|
||||
"ocm service restart {env}",
|
||||
"ocm service status {env} --json"
|
||||
],
|
||||
"evidence": [
|
||||
"restart readiness",
|
||||
"post-fixture gateway status",
|
||||
"resource samples during restart"
|
||||
],
|
||||
"healthScope": "readiness"
|
||||
"commands": ["ocm service restart {env}", "ocm service status {env} --json"],
|
||||
"evidence": ["restart readiness", "post-fixture gateway status", "resource samples during restart"]
|
||||
},
|
||||
{
|
||||
"id": "user-facing-commands",
|
||||
@ -69,18 +44,7 @@
|
||||
"ocm @{env} -- models list",
|
||||
"node support/run-soak-loop.mjs --env {env} --duration-ms 15000 --interval-ms 1000 --timeout-ms 30000"
|
||||
],
|
||||
"evidence": [
|
||||
"status latency",
|
||||
"plugin list latency",
|
||||
"model list latency",
|
||||
"short repeated command p95",
|
||||
"health p95",
|
||||
"RSS and CPU peaks"
|
||||
],
|
||||
"healthScope": "post-ready"
|
||||
"evidence": ["status latency", "plugin list latency", "model list latency", "short repeated command p95", "health p95", "RSS and CPU peaks"]
|
||||
}
|
||||
],
|
||||
"proves": [
|
||||
"baseline"
|
||||
]
|
||||
}
|
||||
|
||||
@ -39,7 +39,6 @@ Usage:
|
||||
kova setup auth [--provider <id>] [--method <mock|api-key|env-only|external-cli|oauth|skip>] [--env-var <name>] [--value <secret>] [--fallback-policy <mock|external-cli|none>] [--json]
|
||||
kova self-check [--json]
|
||||
kova plan [--scenario <id>] [--json]
|
||||
kova inventory plan [--openclaw-bin <path>] [--openclaw-repo <path>] [--subcommands <a,b>] [--require-modeled <capability[,capability]>] [--script-scope <product|all|none>] [--max-subcommands <n>] [--max-warnings <n>] [--timeout-ms <n>] [--json]
|
||||
kova run --target <selector> [--from <selector>] [--scenario <id>] [--state <id>] [--auth <mock|live|skip>] [--repeat <n>] [--baseline [path]] [--save-baseline [path] --reviewed-good] [--regression-thresholds <json>] [--report-dir <path>] [--health-samples <n>] [--readiness-interval-ms <n>] [--resource-sample-interval-ms <n>] [--deep-profile] [--node-profile] [--heap-snapshot] [--profile-on-failure] [--execute] [--keep-env] [--retain-on-failure] [--json]
|
||||
kova matrix plan --profile <id> --target <selector> [--from <selector>] [--include <filter>] [--exclude <filter>] [--parallel <n>] [--json]
|
||||
kova matrix run --profile <id> --target <selector> [--from <selector>] [--include <filter>] [--exclude <filter>] [--auth <mock|live|skip>] [--parallel <n>] [--repeat <n>] [--baseline [path]] [--save-baseline [path] --reviewed-good] [--regression-thresholds <json>] [--fail-fast] [--gate] [--report-dir <path>] [--health-samples <n>] [--readiness-interval-ms <n>] [--resource-sample-interval-ms <n>] [--deep-profile] [--node-profile] [--heap-snapshot] [--profile-on-failure] [--execute] [--allow-exhaustive] [--keep-env] [--retain-on-failure] [--json]
|
||||
@ -63,8 +62,6 @@ Notes:
|
||||
Kova uses OCM to create isolated OpenClaw envs and runtimes.
|
||||
Kova reports on OpenClaw behavior, not OCM behavior.
|
||||
run is dry-run/report-only unless --execute is passed.
|
||||
inventory is planner-only and reports discovered OpenClaw capabilities that are not mapped to Kova surfaces.
|
||||
inventory package-script discovery defaults to --script-scope product; use all or none to widen or disable it.
|
||||
Executed exhaustive matrix runs require --allow-exhaustive.
|
||||
cleanup artifacts is dry-run by default and only targets Kova-owned run artifact dirs.
|
||||
--repeat records independent samples and computes aggregate performance stats.
|
||||
|
||||
@ -1,57 +0,0 @@
|
||||
import {
|
||||
buildPreProviderAttribution,
|
||||
preProviderMarkdownRows,
|
||||
summarizePreProviderAttributions
|
||||
} from "./pre-provider-attribution.mjs";
|
||||
|
||||
export const AGENT_CLI_PRE_PROVIDER_ATTRIBUTION_SCHEMA = "kova.agentCliPreProviderAttribution.v1";
|
||||
export const AGENT_CLI_PRE_PROVIDER_SUMMARY_SCHEMA = "kova.agentCliPreProviderAttributionSummary.v1";
|
||||
|
||||
export function buildAgentCliPreProviderAttribution({
|
||||
label,
|
||||
phaseId,
|
||||
activeStartedAtEpochMs,
|
||||
activeFinishedAtEpochMs,
|
||||
attribution,
|
||||
timelineSummary
|
||||
}) {
|
||||
return buildPreProviderAttribution({
|
||||
schemaVersion: AGENT_CLI_PRE_PROVIDER_ATTRIBUTION_SCHEMA,
|
||||
label,
|
||||
phaseId,
|
||||
activeStartedAtEpochMs,
|
||||
activeFinishedAtEpochMs,
|
||||
attribution,
|
||||
timelineSummary,
|
||||
isAttributedSpanName: isAgentCliAttributedSpanName,
|
||||
missingEventsError: "timeline contains no agent CLI attribution events"
|
||||
});
|
||||
}
|
||||
|
||||
export function summarizeAgentCliPreProviderAttributions(turns) {
|
||||
return summarizePreProviderAttributions({
|
||||
schemaVersion: AGENT_CLI_PRE_PROVIDER_SUMMARY_SCHEMA,
|
||||
turns,
|
||||
fieldName: "agentCliPreProviderAttribution"
|
||||
});
|
||||
}
|
||||
|
||||
export function agentCliPreProviderMarkdownRows(turns) {
|
||||
return preProviderMarkdownRows({
|
||||
title: "Agent CLI pre-provider attribution",
|
||||
turns,
|
||||
fieldName: "agentCliPreProviderAttribution"
|
||||
});
|
||||
}
|
||||
|
||||
function isAgentCliAttributedSpanName(name) {
|
||||
const text = String(name ?? "");
|
||||
return text === "agent.prepare" ||
|
||||
text === "plugins.metadata.scan" ||
|
||||
text === "runtimeDeps.stage" ||
|
||||
text === "channel.capabilities" ||
|
||||
text === "models.catalog" ||
|
||||
text.startsWith("models.catalog.") ||
|
||||
text.startsWith("models.discovery") ||
|
||||
text.startsWith("channel.plugin.");
|
||||
}
|
||||
@ -1,66 +0,0 @@
|
||||
import {
|
||||
attributedSpanIntervals as collectAttributedSpanIntervals,
|
||||
buildPreProviderAttribution,
|
||||
preProviderMarkdownRows,
|
||||
summarizePreProviderAttributions
|
||||
} from "./pre-provider-attribution.mjs";
|
||||
|
||||
export const GATEWAY_SESSION_PRE_PROVIDER_ATTRIBUTION_SCHEMA = "kova.gatewaySessionPreProviderAttribution.v1";
|
||||
export const GATEWAY_SESSION_PRE_PROVIDER_SUMMARY_SCHEMA = "kova.gatewaySessionPreProviderAttributionSummary.v1";
|
||||
|
||||
export function buildGatewaySessionPreProviderAttribution({
|
||||
label,
|
||||
phaseId,
|
||||
activeStartedAtEpochMs,
|
||||
activeFinishedAtEpochMs,
|
||||
attribution,
|
||||
timelineSummary
|
||||
}) {
|
||||
return buildPreProviderAttribution({
|
||||
schemaVersion: GATEWAY_SESSION_PRE_PROVIDER_ATTRIBUTION_SCHEMA,
|
||||
label,
|
||||
phaseId,
|
||||
activeStartedAtEpochMs,
|
||||
activeFinishedAtEpochMs,
|
||||
attribution,
|
||||
timelineSummary,
|
||||
isAttributedSpanName: isGatewaySessionAttributedSpanName,
|
||||
shouldIncludeSpan: includeGatewaySessionSpanInWindow,
|
||||
missingEventsError: "timeline contains no Gateway session turn attribution events"
|
||||
});
|
||||
}
|
||||
|
||||
export function summarizeGatewaySessionPreProviderAttributions(turns) {
|
||||
return summarizePreProviderAttributions({
|
||||
schemaVersion: GATEWAY_SESSION_PRE_PROVIDER_SUMMARY_SCHEMA,
|
||||
turns,
|
||||
fieldName: "gatewaySessionPreProviderAttribution"
|
||||
});
|
||||
}
|
||||
|
||||
export function gatewaySessionPreProviderMarkdownRows(turns) {
|
||||
return preProviderMarkdownRows({
|
||||
title: "Gateway session pre-provider attribution",
|
||||
turns,
|
||||
fieldName: "gatewaySessionPreProviderAttribution"
|
||||
});
|
||||
}
|
||||
|
||||
export function attributedSpanIntervals(events) {
|
||||
return collectAttributedSpanIntervals(events, isGatewaySessionAttributedSpanName);
|
||||
}
|
||||
|
||||
function isGatewaySessionAttributedSpanName(name) {
|
||||
const text = String(name ?? "");
|
||||
return text === "plugins.metadata.scan" ||
|
||||
text.startsWith("gateway.chat_send") ||
|
||||
text.startsWith("auto_reply") ||
|
||||
text.startsWith("reply.");
|
||||
}
|
||||
|
||||
function includeGatewaySessionSpanInWindow(span, { windowStartEpochMs, windowEndEpochMs }) {
|
||||
if (span.name !== "plugins.metadata.scan") {
|
||||
return true;
|
||||
}
|
||||
return span.endEpochMs >= windowStartEpochMs && span.endEpochMs <= windowEndEpochMs;
|
||||
}
|
||||
@ -1,448 +0,0 @@
|
||||
export function buildPreProviderAttribution({
|
||||
schemaVersion,
|
||||
label,
|
||||
phaseId,
|
||||
activeStartedAtEpochMs,
|
||||
activeFinishedAtEpochMs,
|
||||
attribution,
|
||||
timelineSummary,
|
||||
isAttributedSpanName,
|
||||
shouldIncludeSpan,
|
||||
missingEventsError
|
||||
}) {
|
||||
const artifacts = timelineArtifacts(timelineSummary);
|
||||
const events = attributionEvents(timelineSummary);
|
||||
const providerBoundaryEpochMs = numberOrNull(attribution?.firstProviderRequestAtEpochMs);
|
||||
const windowStartEpochMs = numberOrNull(activeStartedAtEpochMs ?? attribution?.commandStartedAtEpochMs);
|
||||
const activeEndEpochMs = numberOrNull(activeFinishedAtEpochMs ?? attribution?.commandFinishedAtEpochMs);
|
||||
const windowEndEpochMs = providerBoundaryEpochMs;
|
||||
const preProviderMs = numberOrNull(attribution?.preProviderMs) ??
|
||||
durationBetween(windowStartEpochMs, windowEndEpochMs);
|
||||
const base = {
|
||||
schemaVersion,
|
||||
available: false,
|
||||
label: label ?? null,
|
||||
phaseId: phaseId ?? null,
|
||||
timelineAvailable: timelineSummary?.available === true,
|
||||
timelineArtifacts: artifacts,
|
||||
eventCount: events.length,
|
||||
window: {
|
||||
startEpochMs: windowStartEpochMs,
|
||||
startAt: isoOrNull(windowStartEpochMs),
|
||||
endEpochMs: windowEndEpochMs,
|
||||
endAt: isoOrNull(windowEndEpochMs),
|
||||
durationMs: preProviderMs
|
||||
},
|
||||
activeWindow: {
|
||||
startEpochMs: windowStartEpochMs,
|
||||
startAt: isoOrNull(windowStartEpochMs),
|
||||
endEpochMs: activeEndEpochMs,
|
||||
endAt: isoOrNull(activeEndEpochMs),
|
||||
durationMs: durationBetween(windowStartEpochMs, activeEndEpochMs)
|
||||
},
|
||||
providerBoundary: {
|
||||
firstRequestAtEpochMs: providerBoundaryEpochMs,
|
||||
firstRequestAt: isoOrNull(providerBoundaryEpochMs),
|
||||
source: providerBoundaryEpochMs === null ? null : "provider-evidence"
|
||||
},
|
||||
provider: summarizeProviderEvents(events, providerBoundaryEpochMs, activeEndEpochMs, attribution),
|
||||
spanSummaries: [],
|
||||
knownAttributedMs: null,
|
||||
unattributedMs: preProviderMs,
|
||||
coverageRatio: null,
|
||||
error: null
|
||||
};
|
||||
|
||||
if (timelineSummary?.available !== true) {
|
||||
return {
|
||||
...base,
|
||||
error: "OpenClaw diagnostics timeline unavailable"
|
||||
};
|
||||
}
|
||||
if (events.length === 0) {
|
||||
return {
|
||||
...base,
|
||||
error: missingEventsError ?? "timeline contains no pre-provider attribution events"
|
||||
};
|
||||
}
|
||||
if (windowStartEpochMs === null || windowEndEpochMs === null || preProviderMs === null || windowEndEpochMs < windowStartEpochMs) {
|
||||
return {
|
||||
...base,
|
||||
error: "pre-provider window boundary unavailable"
|
||||
};
|
||||
}
|
||||
|
||||
const intervals = attributedSpanIntervals(events, isAttributedSpanName)
|
||||
.filter((span) => shouldIncludeSpan ? shouldIncludeSpan(span, { windowStartEpochMs, windowEndEpochMs }) : true)
|
||||
.map((span) => clipSpanToWindow(span, windowStartEpochMs, windowEndEpochMs))
|
||||
.filter(Boolean);
|
||||
const spanSummaries = summarizeAttributedSpans(intervals);
|
||||
const knownAttributedMs = round(unionDuration(intervals));
|
||||
const unattributedMs = round(Math.max(0, preProviderMs - knownAttributedMs));
|
||||
|
||||
return {
|
||||
...base,
|
||||
available: true,
|
||||
spanSummaries,
|
||||
knownAttributedMs,
|
||||
unattributedMs,
|
||||
coverageRatio: preProviderMs > 0 ? round(knownAttributedMs / preProviderMs) : null,
|
||||
error: null
|
||||
};
|
||||
}
|
||||
|
||||
export function summarizePreProviderAttributions({ schemaVersion, turns, fieldName }) {
|
||||
const entries = (turns ?? [])
|
||||
.map((turn) => turn?.[fieldName])
|
||||
.filter(Boolean);
|
||||
const cold = summarizeLabeledAttribution(entries, "cold");
|
||||
const warm = summarizeLabeledAttribution(entries, "warm");
|
||||
return {
|
||||
schemaVersion,
|
||||
available: entries.some((entry) => entry.available === true),
|
||||
count: entries.length,
|
||||
cold,
|
||||
warm,
|
||||
spanMedians: summarizeSpanMedians(entries),
|
||||
timelineArtifacts: unique(entries.flatMap((entry) => entry.timelineArtifacts ?? []))
|
||||
};
|
||||
}
|
||||
|
||||
export function preProviderMarkdownRows({ title, turns, fieldName }) {
|
||||
const attributions = (turns ?? [])
|
||||
.map((turn) => turn?.[fieldName])
|
||||
.filter(Boolean);
|
||||
if (attributions.length === 0) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const lines = [
|
||||
`- ${title}:`,
|
||||
" - Spans are selected by active turn timestamp window; timeline phase is descriptive, not a startup/turn classifier.",
|
||||
"",
|
||||
" | turn | pre-provider | known | unattributed | provider | timeline |",
|
||||
" |---|---:|---:|---:|---:|---|"
|
||||
];
|
||||
for (const item of attributions) {
|
||||
const timeline = item.timelineArtifacts?.[0] ?? (item.timelineAvailable ? "available" : "missing");
|
||||
lines.push(
|
||||
` | ${item.label ?? "turn"} | ${formatMs(item.window?.durationMs)} | ${formatMs(item.knownAttributedMs)} | ${formatMs(item.unattributedMs)} | ${formatMs(item.provider?.totalDurationMs)} | ${timeline} |`
|
||||
);
|
||||
}
|
||||
|
||||
const spanRows = attributions.flatMap((item) =>
|
||||
(item.spanSummaries ?? []).slice(0, 6).map((span) => ({ turn: item.label ?? "turn", ...span }))
|
||||
);
|
||||
if (spanRows.length > 0) {
|
||||
lines.push("");
|
||||
lines.push(" | turn | span | phase(s) | count | errors | clipped | max |");
|
||||
lines.push(" |---|---|---|---:|---:|---:|---:|");
|
||||
for (const span of spanRows.slice(0, 12)) {
|
||||
lines.push(
|
||||
` | ${span.turn} | \`${span.name}\` | ${formatPhases(span.phases)} | ${span.count} | ${span.errorCount} | ${formatMs(span.totalClippedDurationMs)} | ${formatMs(span.maxClippedDurationMs)} |`
|
||||
);
|
||||
}
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
export function attributedSpanIntervals(events, isAttributedSpanName) {
|
||||
const startsById = new Map();
|
||||
const intervals = [];
|
||||
|
||||
for (const event of events ?? []) {
|
||||
if (event?.type === "span.start" && isAttributedSpanName(event.name)) {
|
||||
const key = spanKey(event);
|
||||
if (key) {
|
||||
startsById.set(key, event);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if ((event?.type === "span.end" || event?.type === "span.error") && isAttributedSpanName(event.name)) {
|
||||
const terminal = spanIntervalFromTerminal(event, startsById.get(spanKey(event)));
|
||||
if (terminal) {
|
||||
intervals.push(terminal);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return intervals;
|
||||
}
|
||||
|
||||
function spanIntervalFromTerminal(event, startEvent) {
|
||||
const endEpochMs = eventEpochMs(event);
|
||||
const durationMs = numberOrNull(event.durationMs) ?? durationBetween(eventEpochMs(startEvent), endEpochMs);
|
||||
const startEpochMs = eventEpochMs(startEvent) ??
|
||||
(endEpochMs !== null && durationMs !== null ? endEpochMs - durationMs : null);
|
||||
if (startEpochMs === null || endEpochMs === null || endEpochMs < startEpochMs) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
name: event.name,
|
||||
type: event.type,
|
||||
startEpochMs,
|
||||
endEpochMs,
|
||||
durationMs: round(endEpochMs - startEpochMs),
|
||||
rawDurationMs: durationMs,
|
||||
spanId: event.spanId ?? null,
|
||||
phase: event.phase ?? startEvent?.phase ?? null,
|
||||
errorName: event.errorName ?? null,
|
||||
errorMessage: event.errorMessage ?? null
|
||||
};
|
||||
}
|
||||
|
||||
function clipSpanToWindow(span, windowStartEpochMs, windowEndEpochMs) {
|
||||
const startEpochMs = Math.max(span.startEpochMs, windowStartEpochMs);
|
||||
const endEpochMs = Math.min(span.endEpochMs, windowEndEpochMs);
|
||||
if (endEpochMs <= startEpochMs) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
...span,
|
||||
clippedStartEpochMs: startEpochMs,
|
||||
clippedEndEpochMs: endEpochMs,
|
||||
clippedDurationMs: round(endEpochMs - startEpochMs)
|
||||
};
|
||||
}
|
||||
|
||||
function summarizeAttributedSpans(intervals) {
|
||||
const byName = new Map();
|
||||
for (const interval of intervals) {
|
||||
const current = byName.get(interval.name) ?? {
|
||||
name: interval.name,
|
||||
count: 0,
|
||||
errorCount: 0,
|
||||
totalClippedDurationMs: 0,
|
||||
maxClippedDurationMs: null,
|
||||
totalRawDurationMs: 0,
|
||||
maxRawDurationMs: null,
|
||||
phases: []
|
||||
};
|
||||
current.count += 1;
|
||||
if (interval.type === "span.error") {
|
||||
current.errorCount += 1;
|
||||
}
|
||||
current.totalClippedDurationMs = round(current.totalClippedDurationMs + interval.clippedDurationMs);
|
||||
current.maxClippedDurationMs = maxNullable(current.maxClippedDurationMs, interval.clippedDurationMs);
|
||||
if (typeof interval.rawDurationMs === "number") {
|
||||
current.totalRawDurationMs = round(current.totalRawDurationMs + interval.rawDurationMs);
|
||||
current.maxRawDurationMs = maxNullable(current.maxRawDurationMs, interval.rawDurationMs);
|
||||
}
|
||||
current.phases = mergePhaseSummary(current.phases, interval.phase, interval.clippedDurationMs);
|
||||
byName.set(interval.name, current);
|
||||
}
|
||||
return [...byName.values()].toSorted((left, right) =>
|
||||
(right.totalClippedDurationMs - left.totalClippedDurationMs) ||
|
||||
left.name.localeCompare(right.name)
|
||||
);
|
||||
}
|
||||
|
||||
function mergePhaseSummary(phases, phase, clippedDurationMs) {
|
||||
const label = String(phase ?? "unknown");
|
||||
const existing = phases.find((item) => item.phase === label);
|
||||
if (existing) {
|
||||
existing.count += 1;
|
||||
existing.totalClippedDurationMs = round(existing.totalClippedDurationMs + clippedDurationMs);
|
||||
return phases;
|
||||
}
|
||||
return [
|
||||
...phases,
|
||||
{
|
||||
phase: label,
|
||||
count: 1,
|
||||
totalClippedDurationMs: round(clippedDurationMs)
|
||||
}
|
||||
].toSorted((left, right) =>
|
||||
(right.totalClippedDurationMs - left.totalClippedDurationMs) ||
|
||||
left.phase.localeCompare(right.phase)
|
||||
);
|
||||
}
|
||||
|
||||
function summarizeProviderEvents(events, providerBoundaryEpochMs, activeFinishedAtEpochMs, attribution) {
|
||||
const providerEvents = (events ?? [])
|
||||
.filter((event) => event?.type === "provider.request" || event?.name === "provider.request")
|
||||
.map((event) => {
|
||||
const startEpochMs = numberOrNull(event.receivedAtEpochMs) ?? eventEpochMs(event);
|
||||
const durationMs = numberOrNull(event.durationMs);
|
||||
const endEpochMs = numberOrNull(event.respondedAtEpochMs) ??
|
||||
(startEpochMs !== null && durationMs !== null ? startEpochMs + durationMs : null);
|
||||
return { event, startEpochMs, endEpochMs, durationMs: durationMs ?? durationBetween(startEpochMs, endEpochMs) };
|
||||
})
|
||||
.filter((event) =>
|
||||
event.startEpochMs !== null &&
|
||||
(providerBoundaryEpochMs === null || event.startEpochMs >= providerBoundaryEpochMs) &&
|
||||
(activeFinishedAtEpochMs === null || event.startEpochMs <= activeFinishedAtEpochMs)
|
||||
);
|
||||
const durations = providerEvents.map((event) => event.durationMs).filter(isNumber);
|
||||
return {
|
||||
requestCount: providerEvents.length,
|
||||
timelineTotalDurationMs: round(durations.reduce((sum, value) => sum + value, 0)),
|
||||
timelineMaxDurationMs: durations.length > 0 ? Math.max(...durations) : null,
|
||||
totalDurationMs: numberOrNull(attribution?.providerFinalMs) ??
|
||||
round(durations.reduce((sum, value) => sum + value, 0)),
|
||||
firstByteLatencyMs: numberOrNull(attribution?.firstByteLatencyMs),
|
||||
firstChunkLatencyMs: numberOrNull(attribution?.firstChunkLatencyMs)
|
||||
};
|
||||
}
|
||||
|
||||
function summarizeLabeledAttribution(entries, label) {
|
||||
const values = entries.filter((entry) => entry.label === label);
|
||||
return {
|
||||
count: values.length,
|
||||
preProviderMs: summarizeValues(values.map((entry) => entry.window?.durationMs)),
|
||||
knownAttributedMs: summarizeValues(values.map((entry) => entry.knownAttributedMs)),
|
||||
unattributedMs: summarizeValues(values.map((entry) => entry.unattributedMs)),
|
||||
coverageRatio: summarizeValues(values.map((entry) => entry.coverageRatio))
|
||||
};
|
||||
}
|
||||
|
||||
function summarizeSpanMedians(entries) {
|
||||
const byName = new Map();
|
||||
for (const entry of entries) {
|
||||
for (const span of entry.spanSummaries ?? []) {
|
||||
const current = byName.get(span.name) ?? [];
|
||||
current.push(span.totalClippedDurationMs);
|
||||
byName.set(span.name, current);
|
||||
}
|
||||
}
|
||||
return [...byName.entries()]
|
||||
.map(([name, values]) => ({
|
||||
name,
|
||||
medianClippedDurationMs: summarizeValues(values).median,
|
||||
sampleCount: values.length
|
||||
}))
|
||||
.toSorted((left, right) => (right.medianClippedDurationMs ?? 0) - (left.medianClippedDurationMs ?? 0));
|
||||
}
|
||||
|
||||
function summarizeValues(values) {
|
||||
const sorted = values.filter(isNumber).toSorted((left, right) => left - right);
|
||||
if (sorted.length === 0) {
|
||||
return { count: 0, median: null, min: null, max: null };
|
||||
}
|
||||
return {
|
||||
count: sorted.length,
|
||||
median: round(percentile(sorted, 50)),
|
||||
min: round(sorted[0]),
|
||||
max: round(sorted.at(-1))
|
||||
};
|
||||
}
|
||||
|
||||
function unionDuration(intervals) {
|
||||
const sorted = intervals
|
||||
.filter((interval) => isNumber(interval.clippedStartEpochMs) && isNumber(interval.clippedEndEpochMs))
|
||||
.toSorted((left, right) => left.clippedStartEpochMs - right.clippedStartEpochMs);
|
||||
let total = 0;
|
||||
let currentStart = null;
|
||||
let currentEnd = null;
|
||||
for (const interval of sorted) {
|
||||
if (currentStart === null) {
|
||||
currentStart = interval.clippedStartEpochMs;
|
||||
currentEnd = interval.clippedEndEpochMs;
|
||||
continue;
|
||||
}
|
||||
if (interval.clippedStartEpochMs <= currentEnd) {
|
||||
currentEnd = Math.max(currentEnd, interval.clippedEndEpochMs);
|
||||
continue;
|
||||
}
|
||||
total += currentEnd - currentStart;
|
||||
currentStart = interval.clippedStartEpochMs;
|
||||
currentEnd = interval.clippedEndEpochMs;
|
||||
}
|
||||
if (currentStart !== null) {
|
||||
total += currentEnd - currentStart;
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
function attributionEvents(timelineSummary) {
|
||||
return Array.isArray(timelineSummary?.turnAttributionEvents) && timelineSummary.turnAttributionEvents.length > 0
|
||||
? timelineSummary.turnAttributionEvents
|
||||
: (Array.isArray(timelineSummary?.events) ? timelineSummary.events : []);
|
||||
}
|
||||
|
||||
function timelineArtifacts(timelineSummary) {
|
||||
return unique([
|
||||
...(timelineSummary?.artifacts ?? []),
|
||||
...(timelineSummary?.timelineArtifacts ?? [])
|
||||
].filter(Boolean));
|
||||
}
|
||||
|
||||
function spanKey(event) {
|
||||
return event?.spanId === undefined || event?.spanId === null || String(event.spanId).length === 0
|
||||
? null
|
||||
: String(event.spanId);
|
||||
}
|
||||
|
||||
function eventEpochMs(event) {
|
||||
const direct = numberOrNull(event?.timestampEpochMs ?? event?.timeEpochMs);
|
||||
if (direct !== null) {
|
||||
return direct;
|
||||
}
|
||||
const parsed = Date.parse(event?.timestamp ?? event?.time ?? "");
|
||||
return Number.isFinite(parsed) ? parsed : null;
|
||||
}
|
||||
|
||||
function durationBetween(startEpochMs, endEpochMs) {
|
||||
return isNumber(startEpochMs) && isNumber(endEpochMs) && endEpochMs >= startEpochMs
|
||||
? round(endEpochMs - startEpochMs)
|
||||
: null;
|
||||
}
|
||||
|
||||
function percentile(sortedValues, percentileValue) {
|
||||
if (sortedValues.length === 1) {
|
||||
return sortedValues[0];
|
||||
}
|
||||
const position = (percentileValue / 100) * (sortedValues.length - 1);
|
||||
const lower = Math.floor(position);
|
||||
const upper = Math.ceil(position);
|
||||
if (lower === upper) {
|
||||
return sortedValues[lower];
|
||||
}
|
||||
const weight = position - lower;
|
||||
return sortedValues[lower] * (1 - weight) + sortedValues[upper] * weight;
|
||||
}
|
||||
|
||||
function maxNullable(left, right) {
|
||||
if (!isNumber(right)) {
|
||||
return left;
|
||||
}
|
||||
return isNumber(left) ? Math.max(left, right) : right;
|
||||
}
|
||||
|
||||
function numberOrNull(value) {
|
||||
if (value === null || value === undefined || value === "") {
|
||||
return null;
|
||||
}
|
||||
const number = Number(value);
|
||||
return Number.isFinite(number) ? round(number) : null;
|
||||
}
|
||||
|
||||
function isoOrNull(epochMs) {
|
||||
return isNumber(epochMs) ? new Date(epochMs).toISOString() : null;
|
||||
}
|
||||
|
||||
function formatMs(value) {
|
||||
return isNumber(value) ? `${value} ms` : "unknown";
|
||||
}
|
||||
|
||||
function formatPhases(phases) {
|
||||
if (!Array.isArray(phases) || phases.length === 0) {
|
||||
return "unknown";
|
||||
}
|
||||
return phases
|
||||
.slice(0, 3)
|
||||
.map((item) => `\`${item.phase}\`${item.count > 1 ? ` x${item.count}` : ""}`)
|
||||
.join(", ");
|
||||
}
|
||||
|
||||
function unique(values) {
|
||||
return [...new Set(values)];
|
||||
}
|
||||
|
||||
function isNumber(value) {
|
||||
return typeof value === "number" && Number.isFinite(value);
|
||||
}
|
||||
|
||||
function round(value) {
|
||||
return Math.round(value * 100) / 100;
|
||||
}
|
||||
@ -103,8 +103,12 @@ export function summarizeResourceSamples(samples) {
|
||||
for (const sample of samples) {
|
||||
const totalRssMb = roundNumber(sample.processes.reduce((total, process) => total + process.rssMb, 0));
|
||||
const totalCpuPercent = roundNumber(sample.processes.reduce((total, process) => total + process.cpuPercent, 0));
|
||||
const commandTreeRssMb = roleRss(sample.processes, "command-tree");
|
||||
const gatewayRssMb = roleRss(sample.processes, "gateway");
|
||||
const commandTreeRssMb = roundNumber(sample.processes
|
||||
.filter((process) => process.roles?.includes("command-tree") || process.role.includes("command-tree"))
|
||||
.reduce((total, process) => total + process.rssMb, 0));
|
||||
const gatewayRssMb = roundNumber(sample.processes
|
||||
.filter((process) => process.roles?.includes("gateway") || process.role.includes("gateway"))
|
||||
.reduce((total, process) => total + process.rssMb, 0));
|
||||
updateRolePeaks(byRole, sample);
|
||||
|
||||
peakTotalRssMb = maxNullable(peakTotalRssMb, totalRssMb);
|
||||
@ -229,7 +233,6 @@ export function captureProcessSnapshot(options = {}) {
|
||||
const gatewayPid = envName ? resolveGatewayPid(envName) : null;
|
||||
const allProcesses = listProcesses();
|
||||
const gatewayTreePids = gatewayPid === null ? new Set() : collectProcessTreePids(allProcesses, gatewayPid);
|
||||
const scopeTokens = snapshotScopeTokens(options);
|
||||
const included = [];
|
||||
|
||||
for (const process of allProcesses) {
|
||||
@ -240,11 +243,7 @@ export function captureProcessSnapshot(options = {}) {
|
||||
if (gatewayTreePids.has(process.pid)) {
|
||||
roles.add("gateway-tree");
|
||||
}
|
||||
for (const role of matchingSnapshotRegistryRoles(process, roleMatchers, {
|
||||
allowGlobalProcessRoleMatches: options.allowGlobalProcessRoleMatches === true,
|
||||
existingRoles: roles,
|
||||
scopeTokens
|
||||
})) {
|
||||
for (const role of matchingRegistryProcessRoles(process, roleMatchers)) {
|
||||
roles.add(role);
|
||||
}
|
||||
if (roles.size === 0) {
|
||||
@ -298,20 +297,6 @@ export function classifyRegistryRolesForProcess(process, options = {}) {
|
||||
return matchingRegistryRoles(process, options.rootCommand, roleMatchers, existingRoles);
|
||||
}
|
||||
|
||||
export function classifySnapshotRolesForProcess(process, options = {}) {
|
||||
const roleMatchers = compileRoleMatchers(options.processRoles ?? []);
|
||||
const existingRoles = new Set(options.existingRoles ?? []);
|
||||
const roles = new Set(existingRoles);
|
||||
for (const role of matchingSnapshotRegistryRoles(process, roleMatchers, {
|
||||
allowGlobalProcessRoleMatches: options.allowGlobalProcessRoleMatches === true,
|
||||
existingRoles,
|
||||
scopeTokens: snapshotScopeTokens(options)
|
||||
})) {
|
||||
roles.add(role);
|
||||
}
|
||||
return [...roles].sort();
|
||||
}
|
||||
|
||||
function compileRoleMatchers(roles) {
|
||||
return roles.map((role) => ({
|
||||
id: role.id,
|
||||
@ -360,30 +345,6 @@ function matchingRegistryProcessRoles(process, roleMatchers) {
|
||||
return roles;
|
||||
}
|
||||
|
||||
function matchingSnapshotRegistryRoles(process, roleMatchers, options = {}) {
|
||||
const existingRoles = options.existingRoles ?? new Set();
|
||||
if (existingRoles.size === 0 && options.allowGlobalProcessRoleMatches !== true && !processMatchesSnapshotScope(process, options.scopeTokens ?? [])) {
|
||||
return [];
|
||||
}
|
||||
return matchingRegistryProcessRoles(process, roleMatchers);
|
||||
}
|
||||
|
||||
function processMatchesSnapshotScope(process, scopeTokens) {
|
||||
const command = String(process?.command ?? "");
|
||||
return scopeTokens.some((token) => token.length > 0 && command.includes(token));
|
||||
}
|
||||
|
||||
function snapshotScopeTokens(options = {}) {
|
||||
const tokens = new Set();
|
||||
if (typeof options.envName === "string" && options.envName.trim().length > 0) {
|
||||
tokens.add(options.envName.trim());
|
||||
}
|
||||
for (const token of String(options.rootCommand ?? "").match(/\bkova-[A-Za-z0-9_.-]+\b/g) ?? []) {
|
||||
tokens.add(token);
|
||||
}
|
||||
return [...tokens].filter((token) => token.length >= 4);
|
||||
}
|
||||
|
||||
function summarizeRoleCounts(processes) {
|
||||
const counts = new Map();
|
||||
for (const process of processes) {
|
||||
|
||||
@ -6,7 +6,6 @@ export const TIMELINE_COLLECTOR_SCHEMA = "kova.timelineCollector.v1";
|
||||
export const KEY_OPENCLAW_SPANS = [
|
||||
"gateway.startup",
|
||||
"gateway.ready",
|
||||
"gateway.chat_send",
|
||||
"config.normalize",
|
||||
"plugins.metadata.scan",
|
||||
"runtimeDeps.stage",
|
||||
@ -20,9 +19,7 @@ export const KEY_OPENCLAW_SPANS = [
|
||||
"channel.plugin.load",
|
||||
"agent.prepare",
|
||||
"agent.turn",
|
||||
"agent.cleanup",
|
||||
"auto_reply",
|
||||
"reply"
|
||||
"agent.cleanup"
|
||||
];
|
||||
|
||||
export async function collectTimelineMetrics(artifactDir) {
|
||||
@ -62,7 +59,6 @@ export async function collectTimelineMetrics(artifactDir) {
|
||||
eventLoop: timeline.eventLoop,
|
||||
providers: timeline.providers,
|
||||
childProcesses: timeline.childProcesses,
|
||||
turnAttributionEvents: timeline.turnAttributionEvents,
|
||||
events: timeline.events,
|
||||
artifacts: timeline.available ? [timelinePath] : [],
|
||||
error: timeline.available ? null : (timeline.error ?? (timeline.missing ? "OpenClaw timeline not emitted" : null))
|
||||
@ -145,7 +141,6 @@ export function summarizeTimeline(events, parseErrors = []) {
|
||||
eventLoop: summarizeEventLoop(eventLoopSamples),
|
||||
providers: summarizeTimedCollection(providerRequests),
|
||||
childProcesses: summarizeChildProcesses(childProcesses),
|
||||
turnAttributionEvents: events.filter(isTurnAttributionEvent).map(compactAttributionEvent),
|
||||
events: events.slice(0, 200)
|
||||
};
|
||||
}
|
||||
@ -189,7 +184,6 @@ function emptyTimeline(extra = {}) {
|
||||
maxDurationMs: null,
|
||||
slowest: null
|
||||
},
|
||||
turnAttributionEvents: [],
|
||||
events: [],
|
||||
...extra
|
||||
};
|
||||
@ -324,8 +318,8 @@ function summarizeOpenSpans({ starts, terminals, events }) {
|
||||
function summarizeKeySpans({ spanEvents, openSpans }) {
|
||||
const byName = {};
|
||||
for (const name of KEY_OPENCLAW_SPANS) {
|
||||
const spans = spanEvents.filter((event) => keySpanMatches(name, event.name));
|
||||
const open = openSpans.filter((event) => keySpanMatches(name, event.name));
|
||||
const spans = spanEvents.filter((event) => event.name === name);
|
||||
const open = openSpans.filter((event) => event.name === name);
|
||||
const durations = spans.map((event) => event.durationMs).filter(isNumber);
|
||||
const slowest = spans
|
||||
.filter((event) => typeof event.durationMs === "number")
|
||||
@ -345,19 +339,6 @@ function summarizeKeySpans({ spanEvents, openSpans }) {
|
||||
return byName;
|
||||
}
|
||||
|
||||
function keySpanMatches(keyName, eventName) {
|
||||
if (keyName === "gateway.chat_send") {
|
||||
return eventName === keyName || eventName.startsWith("gateway.chat_send.");
|
||||
}
|
||||
if (keyName === "auto_reply") {
|
||||
return eventName === keyName || eventName.startsWith("auto_reply.");
|
||||
}
|
||||
if (keyName === "reply") {
|
||||
return eventName === keyName || eventName.startsWith("reply.");
|
||||
}
|
||||
return eventName === keyName;
|
||||
}
|
||||
|
||||
function emptyKeySpans() {
|
||||
return Object.fromEntries(KEY_OPENCLAW_SPANS.map((name) => [name, {
|
||||
name,
|
||||
@ -435,65 +416,6 @@ function compactTimedEvent(event) {
|
||||
};
|
||||
}
|
||||
|
||||
function isTurnAttributionEvent(event) {
|
||||
if (event.type === "eventLoop.sample") {
|
||||
return true;
|
||||
}
|
||||
if (event.type === "provider.request" || event.name === "provider.request") {
|
||||
return true;
|
||||
}
|
||||
if (event.type !== "span.start" && event.type !== "span.end" && event.type !== "span.error") {
|
||||
return false;
|
||||
}
|
||||
return event.name === "plugins.metadata.scan" ||
|
||||
event.name === "provider.request" ||
|
||||
event.name === "agent.prepare" ||
|
||||
event.name === "agent.turn" ||
|
||||
event.name === "agent.cleanup" ||
|
||||
event.name === "runtimeDeps.stage" ||
|
||||
event.name === "channel.capabilities" ||
|
||||
event.name === "models.catalog" ||
|
||||
event.name === "auto_reply" ||
|
||||
event.name.startsWith("auto_reply.") ||
|
||||
event.name.startsWith("gateway.chat_send") ||
|
||||
event.name.startsWith("models.catalog.") ||
|
||||
event.name.startsWith("models.discovery") ||
|
||||
event.name.startsWith("channel.plugin.") ||
|
||||
event.name.startsWith("reply.");
|
||||
}
|
||||
|
||||
function compactAttributionEvent(event) {
|
||||
return {
|
||||
type: event.type,
|
||||
name: event.name,
|
||||
timestamp: event.timestamp ?? null,
|
||||
timestampEpochMs: numberOrNull(event.timestampEpochMs ?? event.timeEpochMs) ?? parsedTimestampMs(event.timestamp ?? event.time),
|
||||
durationMs: event.durationMs ?? null,
|
||||
spanId: event.spanId ?? null,
|
||||
parentSpanId: event.parentSpanId ?? null,
|
||||
phase: event.phase ?? null,
|
||||
pid: event.pid ?? null,
|
||||
provider: event.provider ?? event.attributes?.provider ?? null,
|
||||
operation: event.operation ?? event.attributes?.operation ?? null,
|
||||
pluginId: event.pluginId ?? event.attributes?.pluginId ?? null,
|
||||
errorName: event.errorName ?? event.attributes?.errorName ?? null,
|
||||
errorMessage: event.errorMessage ?? event.attributes?.errorMessage ?? null,
|
||||
maxMs: numberOrNull(event.maxMs ?? event.eventLoopDelayMs),
|
||||
p95Ms: numberOrNull(event.p95Ms),
|
||||
p99Ms: numberOrNull(event.p99Ms),
|
||||
receivedAtEpochMs: numberOrNull(event.receivedAtEpochMs),
|
||||
respondedAtEpochMs: numberOrNull(event.respondedAtEpochMs),
|
||||
status: numberOrNull(event.status),
|
||||
route: event.route ?? event.path ?? null,
|
||||
model: event.model ?? event.modelId ?? event.attributes?.model ?? null
|
||||
};
|
||||
}
|
||||
|
||||
function parsedTimestampMs(value) {
|
||||
const parsed = Date.parse(value ?? "");
|
||||
return Number.isFinite(parsed) ? parsed : null;
|
||||
}
|
||||
|
||||
function spanIdentity(event) {
|
||||
if (event.spanId !== undefined && event.spanId !== null && String(event.spanId).length > 0) {
|
||||
return `id:${event.spanId}`;
|
||||
|
||||
@ -1,165 +0,0 @@
|
||||
import { readdir, rm, stat } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { runCleanupCommand } from "../cleanup.mjs";
|
||||
import { runCommand } from "../commands.mjs";
|
||||
import { artifactsDir } from "../paths.mjs";
|
||||
import { ocmEnvDestroy, ocmEnvListJson } from "../ocm/commands.mjs";
|
||||
import { positiveIntegerFlag } from "./run-support.mjs";
|
||||
|
||||
export async function runCleanupCliCommand(flags) {
|
||||
const [subcommand] = flags._;
|
||||
if (subcommand === "envs") {
|
||||
await cleanupEnvs(flags);
|
||||
return;
|
||||
}
|
||||
if (subcommand === "artifacts") {
|
||||
await cleanupArtifacts(flags);
|
||||
return;
|
||||
}
|
||||
|
||||
throw new Error(`unknown cleanup command: ${subcommand ?? ""}`);
|
||||
}
|
||||
|
||||
async function cleanupEnvs(flags) {
|
||||
const envList = await runCommand(ocmEnvListJson(), { timeoutMs: 30000 });
|
||||
if (envList.status !== 0) {
|
||||
throw new Error(`failed to list OCM envs: ${envList.stderr.trim() || envList.stdout.trim()}`);
|
||||
}
|
||||
|
||||
const summaries = JSON.parse(envList.stdout);
|
||||
if (!Array.isArray(summaries)) {
|
||||
throw new Error("ocm env list --json returned unexpected data");
|
||||
}
|
||||
|
||||
const envs = summaries
|
||||
.map((summary) => summary.name)
|
||||
.filter((name) => /^kova-[a-z0-9-]+$/.test(name));
|
||||
const results = [];
|
||||
|
||||
if (flags.execute) {
|
||||
for (const env of envs) {
|
||||
results.push(await runCleanupCommand(ocmEnvDestroy(env), { timeoutMs: 120000 }));
|
||||
}
|
||||
}
|
||||
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify({
|
||||
schemaVersion: "kova.cleanup.envs.v1",
|
||||
generatedAt: new Date().toISOString(),
|
||||
execute: flags.execute === true,
|
||||
envs,
|
||||
results: results.map((result) => ({
|
||||
command: result.command,
|
||||
status: result.status,
|
||||
durationMs: result.durationMs,
|
||||
timedOut: result.timedOut,
|
||||
attempts: result.attempts ?? []
|
||||
}))
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
if (envs.length === 0) {
|
||||
console.log("No stale Kova envs found.");
|
||||
return;
|
||||
}
|
||||
|
||||
if (!flags.execute) {
|
||||
console.log("Stale Kova envs:");
|
||||
for (const env of envs) {
|
||||
console.log(`- ${env}`);
|
||||
}
|
||||
console.log("Run with --execute to destroy them.");
|
||||
return;
|
||||
}
|
||||
|
||||
for (const result of results) {
|
||||
console.log(`${result.status === 0 ? "PASS" : "FAIL"} ${result.command}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function cleanupArtifacts(flags) {
|
||||
const olderThanDays = positiveIntegerFlag(flags, "older_than_days", 7);
|
||||
const cutoffMs = Date.now() - olderThanDays * 24 * 60 * 60 * 1000;
|
||||
const candidates = [];
|
||||
|
||||
let entries = [];
|
||||
try {
|
||||
entries = await readdir(artifactsDir, { withFileTypes: true });
|
||||
} catch (error) {
|
||||
if (error.code !== "ENOENT") {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory() || !/^kova-\d{4}-\d{2}-\d{2}t/i.test(entry.name)) {
|
||||
continue;
|
||||
}
|
||||
const path = join(artifactsDir, entry.name);
|
||||
const info = await stat(path);
|
||||
if (info.mtimeMs > cutoffMs) {
|
||||
continue;
|
||||
}
|
||||
candidates.push({
|
||||
name: entry.name,
|
||||
path,
|
||||
mtime: info.mtime.toISOString(),
|
||||
ageDays: Math.max(0, Math.floor((Date.now() - info.mtimeMs) / (24 * 60 * 60 * 1000)))
|
||||
});
|
||||
}
|
||||
|
||||
const results = [];
|
||||
if (flags.execute === true) {
|
||||
for (const candidate of candidates) {
|
||||
const started = Date.now();
|
||||
try {
|
||||
await rm(candidate.path, { recursive: true, force: true });
|
||||
results.push({
|
||||
path: candidate.path,
|
||||
status: 0,
|
||||
durationMs: Date.now() - started,
|
||||
error: null
|
||||
});
|
||||
} catch (error) {
|
||||
results.push({
|
||||
path: candidate.path,
|
||||
status: 1,
|
||||
durationMs: Date.now() - started,
|
||||
error: error.message
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify({
|
||||
schemaVersion: "kova.cleanup.artifacts.v1",
|
||||
generatedAt: new Date().toISOString(),
|
||||
execute: flags.execute === true,
|
||||
artifactsDir,
|
||||
olderThanDays,
|
||||
candidates,
|
||||
results
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
if (candidates.length === 0) {
|
||||
console.log(`No Kova run artifact dirs older than ${olderThanDays} day(s) found.`);
|
||||
return;
|
||||
}
|
||||
|
||||
if (flags.execute !== true) {
|
||||
console.log(`Kova run artifact dirs older than ${olderThanDays} day(s):`);
|
||||
for (const candidate of candidates) {
|
||||
console.log(`- ${candidate.path}`);
|
||||
}
|
||||
console.log("Run with --execute to remove them.");
|
||||
return;
|
||||
}
|
||||
|
||||
for (const result of results) {
|
||||
console.log(`${result.status === 0 ? "PASS" : "FAIL"} ${result.path}`);
|
||||
}
|
||||
}
|
||||
@ -1,46 +0,0 @@
|
||||
import { buildOpenClawInventoryPlan } from "../inventory/openclaw.mjs";
|
||||
|
||||
export async function runInventoryCommand(flags) {
|
||||
const [subcommand = "plan"] = flags._;
|
||||
if (subcommand !== "plan") {
|
||||
throw new Error(`unknown inventory command: ${subcommand}`);
|
||||
}
|
||||
|
||||
const plan = await buildOpenClawInventoryPlan(flags);
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify(plan, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log("OpenClaw inventory plan");
|
||||
console.log(`Discovered: ${plan.coverage.discoveredCount}`);
|
||||
console.log(`Matched: ${plan.coverage.matchedCount}`);
|
||||
console.log(`Unmodeled: ${plan.coverage.unmodeledCount}`);
|
||||
for (const source of plan.sources) {
|
||||
console.log(`- ${source.id}: ${source.status}${formatSourceCount(source)}`);
|
||||
}
|
||||
if (plan.coverage.warnings.length > 0) {
|
||||
const warningLimit = positiveIntegerFlag(flags.max_warnings, 25);
|
||||
console.log(`Warnings${plan.coverage.warnings.length > warningLimit ? ` (first ${warningLimit} of ${plan.coverage.warnings.length})` : ""}:`);
|
||||
for (const warning of plan.coverage.warnings.slice(0, warningLimit)) {
|
||||
console.log(`- ${warning.message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function formatSourceCount(source) {
|
||||
if (source.id === "package-scripts" && typeof source.scriptCount === "number") {
|
||||
const included = source.includedScriptCount ?? source.scriptCount;
|
||||
return ` (${included}/${source.scriptCount} scripts, scope=${source.scriptScope ?? "unknown"})`;
|
||||
}
|
||||
const count = source.commandCount ?? source.capabilityCount ?? 0;
|
||||
return count ? ` (${count})` : "";
|
||||
}
|
||||
|
||||
function positiveIntegerFlag(value, fallback) {
|
||||
if (value === undefined || value === null || value === false) {
|
||||
return fallback;
|
||||
}
|
||||
const parsed = Number(value);
|
||||
return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
|
||||
}
|
||||
@ -1,57 +0,0 @@
|
||||
import { required } from "../cli.mjs";
|
||||
import { applyMatrixControls, expandProfile } from "../matrix/expand.mjs";
|
||||
import { matrixControlSummary } from "../matrix/controls.mjs";
|
||||
import { profileSummary, validateProfileTarget } from "../matrix/profile.mjs";
|
||||
import { assertResolvedCoverageIsRunnable, resolveCoverageObligations } from "../matrix/resolver.mjs";
|
||||
import { platformInfo } from "../platform.mjs";
|
||||
import { loadRegistryContext } from "../registries/context.mjs";
|
||||
import { loadProfile } from "../registries/profiles.mjs";
|
||||
import { validateScenarioRun } from "../registries/scenarios.mjs";
|
||||
import { resolveTarget } from "../targets.mjs";
|
||||
|
||||
export async function runMatrixPlan(flags) {
|
||||
const registry = await loadRegistryContext();
|
||||
const profile = await loadProfile(required(flags.profile, "--profile"));
|
||||
const target = required(flags.target, "--target");
|
||||
const targetPlan = resolveTarget(target, "target");
|
||||
validateProfileTarget(profile, targetPlan);
|
||||
const fromPlan = flags.from ? resolveTarget(flags.from, "from") : null;
|
||||
const platform = platformInfo();
|
||||
const entries = applyMatrixControls(await expandProfile(profile), flags, platform);
|
||||
const resolvedCoverage = resolveCoverageObligations({
|
||||
profile,
|
||||
entries,
|
||||
surfaces: registry.surfaces,
|
||||
targetPlan
|
||||
});
|
||||
assertResolvedCoverageIsRunnable(resolvedCoverage);
|
||||
for (const entry of entries.filter((item) => !item.skipReason)) {
|
||||
validateScenarioRun(entry.scenario, flags, { targetPlan, fromPlan });
|
||||
}
|
||||
const response = {
|
||||
schemaVersion: "kova.matrix.plan.v1",
|
||||
generatedAt: new Date().toISOString(),
|
||||
platform,
|
||||
profile: profileSummary(profile),
|
||||
target,
|
||||
from: flags.from ?? null,
|
||||
controls: matrixControlSummary(flags, targetPlan),
|
||||
resolvedCoverage,
|
||||
entries: entries.map((entry) => entry.plan)
|
||||
};
|
||||
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify(response, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`${profile.id}: ${profile.title}`);
|
||||
console.log(`Target: ${target}`);
|
||||
if (flags.from) {
|
||||
console.log(`From: ${flags.from}`);
|
||||
}
|
||||
for (const entry of entries) {
|
||||
const suffix = entry.skipReason ? ` [SKIP: ${entry.skipReason}]` : "";
|
||||
console.log(`- ${entry.scenario.id} / ${entry.state.id}: ${entry.scenario.title}${suffix}`);
|
||||
}
|
||||
}
|
||||
@ -1,324 +0,0 @@
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { join, relative } from "node:path";
|
||||
import { authReportSummary, resolveRunAuthContext } from "../auth.mjs";
|
||||
import { required, resolveFromCwd } from "../cli.mjs";
|
||||
import {
|
||||
cleanupTargetRuntimeIfNeeded,
|
||||
loadRegressionThresholds,
|
||||
positiveIntegerFlag,
|
||||
positiveIntegerValue,
|
||||
profileIntegerFlag,
|
||||
summarizePerformanceReceipt,
|
||||
validateBaselineExecutionFlags
|
||||
} from "./run-support.mjs";
|
||||
import { applyMatrixControls, expandProfile } from "../matrix/expand.mjs";
|
||||
import { evaluateGate, preflightGateRun } from "../matrix/gate.mjs";
|
||||
import { matrixControlSummary } from "../matrix/controls.mjs";
|
||||
import { profileSummary, validateProfileTarget } from "../matrix/profile.mjs";
|
||||
import { assertResolvedCoverageIsRunnable, resolveCoverageObligations } from "../matrix/resolver.mjs";
|
||||
import {
|
||||
comparePerformanceToBaseline,
|
||||
loadBaselineStore,
|
||||
resolveBaselinePath,
|
||||
reviewBaselineUpdate,
|
||||
saveBaselineStore,
|
||||
updateBaselineStore
|
||||
} from "../performance/baselines.mjs";
|
||||
import { buildPerformanceSummary } from "../performance/stats.mjs";
|
||||
import { platformInfo } from "../platform.mjs";
|
||||
import { reportsDir } from "../paths.mjs";
|
||||
import { loadRegistryContext } from "../registries/context.mjs";
|
||||
import { loadProfile } from "../registries/profiles.mjs";
|
||||
import { validateScenarioRun } from "../registries/scenarios.mjs";
|
||||
import { buildReportSummary, renderMarkdownReport, summarizeRecords } from "../reporting/report.mjs";
|
||||
import { bundleReport, retainGateArtifacts } from "../reporting/artifacts.mjs";
|
||||
import { buildDryRunRecord, buildSkippedRecord, createRunId, executeScenario } from "../runner.mjs";
|
||||
import { resolveTarget } from "../targets.mjs";
|
||||
|
||||
const reportSchemaVersion = "kova.report.v1";
|
||||
|
||||
export async function runMatrixRun(flags) {
|
||||
const registry = await loadRegistryContext();
|
||||
const profile = await loadProfile(required(flags.profile, "--profile"));
|
||||
validateProfileExecutionFlags(profile, flags);
|
||||
const target = required(flags.target, "--target");
|
||||
validateBaselineExecutionFlags(flags);
|
||||
const targetPlan = resolveTarget(target, "target");
|
||||
validateProfileTarget(profile, targetPlan);
|
||||
const fromPlan = flags.from ? resolveTarget(flags.from, "from") : null;
|
||||
const entries = applyMatrixControls(await expandProfile(profile), flags, platformInfo());
|
||||
const resolvedCoverage = resolveCoverageObligations({
|
||||
profile,
|
||||
entries,
|
||||
surfaces: registry.surfaces,
|
||||
targetPlan
|
||||
});
|
||||
assertResolvedCoverageIsRunnable(resolvedCoverage);
|
||||
const controls = matrixControlSummary(flags, targetPlan);
|
||||
const auth = await resolveRunAuthContext(flags);
|
||||
const regressionThresholds = await loadRegressionThresholds(flags);
|
||||
const baselinePath = resolveBaselinePath(flags.baseline);
|
||||
const saveBaselinePath = resolveBaselinePath(flags.save_baseline);
|
||||
const baselineStore = baselinePath ? await loadBaselineStore(baselinePath) : null;
|
||||
preflightGateRun({ entries, flags });
|
||||
for (const entry of entries.filter((item) => !item.skipReason)) {
|
||||
validateScenarioRun(entry.scenario, flags, { targetPlan, fromPlan });
|
||||
}
|
||||
const reportRoot = flags.report_dir ? resolveFromCwd(flags.report_dir) : reportsDir;
|
||||
const runId = createRunId();
|
||||
const reportPath = join(reportRoot, `${runId}-${profile.id}.md`);
|
||||
const jsonPath = join(reportRoot, `${runId}-${profile.id}.json`);
|
||||
const summaryPath = join(reportRoot, `${runId}-${profile.id}.summary.json`);
|
||||
const targetSetup = { completed: false };
|
||||
const runEntry = async (entry) => {
|
||||
const context = {
|
||||
target,
|
||||
targetPlan,
|
||||
profile,
|
||||
from: flags.from,
|
||||
fromPlan,
|
||||
state: entry.state,
|
||||
sourceEnv: flags.source_env,
|
||||
runId,
|
||||
controls,
|
||||
execute: flags.execute === true,
|
||||
keepEnv: flags.keep_env === true,
|
||||
retainOnFailure: flags.retain_on_failure === true,
|
||||
timeoutMs: resolveEntryTimeout(entry, flags),
|
||||
healthSamples: profileIntegerFlag(flags, "health_samples", flags.deep_profile === true ? 10 : 3),
|
||||
healthIntervalMs: positiveIntegerFlag(flags, "health_interval_ms", 250),
|
||||
readinessIntervalMs: profileIntegerFlag(flags, "readiness_interval_ms", flags.deep_profile === true ? 100 : 250),
|
||||
heapSnapshot: flags.heap_snapshot === true || flags.deep_profile === true,
|
||||
diagnosticReport: flags.deep_profile === true,
|
||||
nodeProfile: flags.node_profile === true || flags.deep_profile === true,
|
||||
deepProfile: flags.deep_profile === true,
|
||||
profileOnFailure: flags.profile_on_failure === true,
|
||||
resourceSampleIntervalMs: profileIntegerFlag(flags, "resource_sample_interval_ms", flags.deep_profile === true ? 250 : 1000),
|
||||
processRoles: registry.processRoles,
|
||||
surfacesById: Object.fromEntries(registry.surfaces.map((surface) => [surface.id, surface])),
|
||||
targetSetup,
|
||||
auth
|
||||
};
|
||||
|
||||
if (entry.skipReason) {
|
||||
return buildRepeatRecords(entry, context, (iterationContext) => buildSkippedRecord(entry.scenario, iterationContext, entry.skipReason));
|
||||
}
|
||||
|
||||
return buildRepeatRecords(entry, context, async (iterationContext) =>
|
||||
iterationContext.execute
|
||||
? executeScenario(entry.scenario, iterationContext)
|
||||
: buildDryRunRecord(entry.scenario, iterationContext)
|
||||
);
|
||||
};
|
||||
|
||||
const records = flags.execute === true
|
||||
? await runMatrixEntries(entries, runEntry, controls)
|
||||
: (await Promise.all(entries.map((entry) => runEntry(entry)))).flat();
|
||||
const targetCleanup = await cleanupTargetRuntimeIfNeeded(targetPlan, records, {
|
||||
execute: flags.execute === true,
|
||||
timeoutMs: positiveIntegerFlag(flags, "timeout_ms", 120000)
|
||||
});
|
||||
const performance = buildPerformanceSummary(records, {
|
||||
repeat: controls.repeat,
|
||||
parallel: controls.parallel,
|
||||
regressionThresholds
|
||||
});
|
||||
const platform = platformInfo();
|
||||
const reportBase = {
|
||||
schemaVersion: reportSchemaVersion,
|
||||
generatedAt: new Date().toISOString(),
|
||||
runId,
|
||||
outputPaths: {
|
||||
markdown: reportPath,
|
||||
json: jsonPath,
|
||||
summary: summaryPath
|
||||
},
|
||||
mode: flags.execute === true ? "execution" : "dry-run",
|
||||
profile: profileSummary(profile),
|
||||
target,
|
||||
from: flags.from ?? null,
|
||||
controls,
|
||||
auth: authReportSummary(auth),
|
||||
state: null,
|
||||
platform,
|
||||
targetCleanup,
|
||||
performance,
|
||||
baseline: null,
|
||||
gate: null,
|
||||
summary: summarizeRecords(records),
|
||||
records
|
||||
};
|
||||
const baselineComparison = comparePerformanceToBaseline(reportBase, baselineStore, { targetPlan, regressionThresholds });
|
||||
if (baselineComparison) {
|
||||
reportBase.baseline = {
|
||||
path: baselinePath,
|
||||
comparison: baselineComparison
|
||||
};
|
||||
}
|
||||
const gate = flags.gate === true
|
||||
? evaluateGate({
|
||||
mode: flags.execute === true ? "execution" : "dry-run",
|
||||
controls,
|
||||
performance,
|
||||
baseline: reportBase.baseline,
|
||||
platform: reportBase.platform,
|
||||
records
|
||||
}, profile, { resolvedCoverage })
|
||||
: null;
|
||||
|
||||
await mkdir(reportRoot, { recursive: true });
|
||||
const report = {
|
||||
...reportBase,
|
||||
gate
|
||||
};
|
||||
if (saveBaselinePath) {
|
||||
const existingStore = await loadBaselineStore(saveBaselinePath);
|
||||
const review = reviewBaselineUpdate(report, { reviewedGood: flags.reviewed_good === true });
|
||||
const updatedStore = updateBaselineStore(existingStore, report, { targetPlan, reviewedGood: flags.reviewed_good === true });
|
||||
report.baseline = {
|
||||
...(report.baseline ?? {}),
|
||||
review,
|
||||
saved: await saveBaselineStore(saveBaselinePath, updatedStore)
|
||||
};
|
||||
}
|
||||
await writeFile(reportPath, renderMarkdownReport(report), "utf8");
|
||||
await writeFile(jsonPath, `${JSON.stringify(report, null, 2)}\n`, "utf8");
|
||||
await writeFile(summaryPath, `${JSON.stringify(buildReportSummary(report), null, 2)}\n`, "utf8");
|
||||
const bundle = await bundleReport(jsonPath, { outputDir: reportRoot });
|
||||
const retainedGateArtifacts = gate && gate.verdict !== "SHIP"
|
||||
? await retainFailedGateArtifacts(report, reportPath, jsonPath, bundle)
|
||||
: null;
|
||||
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify({
|
||||
schemaVersion: "kova.matrix.run.receipt.v1",
|
||||
generatedAt: new Date().toISOString(),
|
||||
mode: report.mode,
|
||||
runId,
|
||||
profile: profileSummary(profile),
|
||||
reportPath,
|
||||
jsonPath,
|
||||
summaryPath,
|
||||
bundlePath: bundle.outputPath,
|
||||
checksumPath: bundle.checksumPath,
|
||||
retainedGateArtifacts,
|
||||
gate: summarizeGateReceipt(gate),
|
||||
performance: summarizePerformanceReceipt(report.performance, report.baseline),
|
||||
summary: report.summary
|
||||
}, null, 2));
|
||||
failGateIfNeeded(gate);
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`Kova matrix ${report.mode} report written: ${relative(process.cwd(), reportPath)}`);
|
||||
console.log(`Kova matrix ${report.mode} data written: ${relative(process.cwd(), jsonPath)}`);
|
||||
console.log(`Kova matrix bundle written: ${relative(process.cwd(), bundle.outputPath)}`);
|
||||
if (retainedGateArtifacts) {
|
||||
console.log(`Kova failed gate artifacts retained: ${relative(process.cwd(), retainedGateArtifacts.outputDir)}`);
|
||||
}
|
||||
if (gate) {
|
||||
console.log(`Kova gate outcome: ${gate.outcome ?? gate.verdict}`);
|
||||
}
|
||||
failGateIfNeeded(gate);
|
||||
}
|
||||
|
||||
function validateProfileExecutionFlags(profile, flags) {
|
||||
if (flags.execute === true && profile.id === "exhaustive" && flags.allow_exhaustive !== true) {
|
||||
throw new Error("executing profile 'exhaustive' requires --allow-exhaustive");
|
||||
}
|
||||
}
|
||||
|
||||
async function retainFailedGateArtifacts(report, reportPath, jsonPath, bundle) {
|
||||
report.retainedGateArtifacts = {
|
||||
status: "pending"
|
||||
};
|
||||
const summaryPath = report.outputPaths?.summary ?? jsonPath.replace(/\.json$/, ".summary.json");
|
||||
await writeFile(reportPath, renderMarkdownReport(report), "utf8");
|
||||
await writeFile(jsonPath, `${JSON.stringify(report, null, 2)}\n`, "utf8");
|
||||
await writeFile(summaryPath, `${JSON.stringify(buildReportSummary(report), null, 2)}\n`, "utf8");
|
||||
const retained = await retainGateArtifacts(jsonPath, bundle);
|
||||
report.retainedGateArtifacts = retained;
|
||||
await writeFile(reportPath, renderMarkdownReport(report), "utf8");
|
||||
await writeFile(jsonPath, `${JSON.stringify(report, null, 2)}\n`, "utf8");
|
||||
await writeFile(summaryPath, `${JSON.stringify(buildReportSummary(report), null, 2)}\n`, "utf8");
|
||||
await retainGateArtifacts(jsonPath, bundle, { outputDir: retained.outputDir });
|
||||
return retained;
|
||||
}
|
||||
|
||||
function resolveEntryTimeout(entry, flags) {
|
||||
return positiveIntegerValue(flags.timeout_ms ?? entry.timeoutMs ?? entry.scenario.timeoutMs ?? 120000, "--timeout-ms");
|
||||
}
|
||||
|
||||
async function buildRepeatRecords(entry, context, callback) {
|
||||
const total = positiveIntegerValue(context.controls?.repeat ?? 1, "repeat");
|
||||
const records = [];
|
||||
for (let index = 1; index <= total; index += 1) {
|
||||
records.push(await callback({
|
||||
...context,
|
||||
repeat: {
|
||||
index,
|
||||
total
|
||||
}
|
||||
}));
|
||||
}
|
||||
return records;
|
||||
}
|
||||
|
||||
function failGateIfNeeded(gate) {
|
||||
if (gate && gate.verdict !== "SHIP") {
|
||||
throw new Error(`gate outcome: ${gate.outcome ?? gate.verdict}`);
|
||||
}
|
||||
}
|
||||
|
||||
function summarizeGateReceipt(gate) {
|
||||
if (!gate) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
schemaVersion: gate.schemaVersion,
|
||||
enabled: gate.enabled,
|
||||
profileId: gate.profileId,
|
||||
policyId: gate.policyId,
|
||||
purpose: gate.purpose ?? null,
|
||||
verdict: gate.verdict,
|
||||
outcome: gate.outcome ?? null,
|
||||
ok: gate.ok,
|
||||
complete: gate.complete,
|
||||
partial: gate.partial,
|
||||
missingRequiredCount: gate.missingRequiredCount,
|
||||
blockingCount: gate.blockingCount,
|
||||
warningCount: gate.warningCount,
|
||||
infoCount: gate.infoCount,
|
||||
subsystemCount: gate.subsystems?.length ?? 0,
|
||||
fixerSummaryCount: gate.fixerSummaries?.length ?? 0,
|
||||
baselineRegressionCount: gate.baseline?.regressionCount ?? null,
|
||||
missingBaselineCount: gate.baseline?.missingBaselineCount ?? null
|
||||
};
|
||||
}
|
||||
|
||||
async function runMatrixEntries(entries, runEntry, controls) {
|
||||
if (controls.parallel <= 1) {
|
||||
const records = [];
|
||||
for (const entry of entries) {
|
||||
const entryRecords = await runEntry(entry);
|
||||
records.push(...entryRecords);
|
||||
if (controls.failFast && entryRecords.some((record) => record.status === "FAIL" || record.status === "BLOCKED")) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
return records;
|
||||
}
|
||||
|
||||
const records = new Array(entries.length);
|
||||
let nextIndex = 0;
|
||||
async function worker() {
|
||||
while (nextIndex < entries.length) {
|
||||
const index = nextIndex;
|
||||
nextIndex += 1;
|
||||
records[index] = await runEntry(entries[index]);
|
||||
}
|
||||
}
|
||||
|
||||
await Promise.all(Array.from({ length: controls.parallel }, () => worker()));
|
||||
return records.filter(Boolean).flat();
|
||||
}
|
||||
@ -1,18 +0,0 @@
|
||||
import { runMatrixPlan } from "./matrix-plan.mjs";
|
||||
import { runMatrixRun } from "./matrix-run.mjs";
|
||||
|
||||
export async function runMatrixCommand(flags) {
|
||||
const [subcommand = "plan"] = flags._;
|
||||
|
||||
if (subcommand === "plan") {
|
||||
await runMatrixPlan(flags);
|
||||
return;
|
||||
}
|
||||
|
||||
if (subcommand === "run") {
|
||||
await runMatrixRun(flags);
|
||||
return;
|
||||
}
|
||||
|
||||
throw new Error(`unknown matrix command: ${subcommand}`);
|
||||
}
|
||||
@ -1,52 +0,0 @@
|
||||
import { profileSummary } from "../matrix/profile.mjs";
|
||||
import { buildCoverage } from "../matrix/coverage.mjs";
|
||||
import { platformInfo } from "../platform.mjs";
|
||||
import { loadRegistryContext } from "../registries/context.mjs";
|
||||
|
||||
export async function runPlanCommand(flags) {
|
||||
const registry = await loadRegistryContext();
|
||||
const scenarios = filterRegistry(registry.scenarios, flags.scenario, "scenario");
|
||||
const states = filterRegistry(registry.states, flags.state, "state");
|
||||
const profiles = flags.profile ? filterRegistry(registry.profiles, flags.profile, "profile") : registry.profiles;
|
||||
const platform = platformInfo();
|
||||
const coverage = buildCoverage({ ...registry, platform });
|
||||
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify({
|
||||
schemaVersion: "kova.plan.v1",
|
||||
generatedAt: new Date().toISOString(),
|
||||
platform,
|
||||
surfaces: registry.surfaces,
|
||||
processRoles: registry.processRoles,
|
||||
metrics: registry.metrics,
|
||||
scenarios,
|
||||
states,
|
||||
profiles: profiles.map(profileSummary),
|
||||
coverage
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
for (const scenario of scenarios) {
|
||||
console.log(`${scenario.id}: ${scenario.title}`);
|
||||
console.log(` Surface: ${scenario.surface}`);
|
||||
console.log(` Objective: ${scenario.objective}`);
|
||||
console.log(` Tags: ${scenario.tags.join(", ")}`);
|
||||
console.log(" Phases:");
|
||||
for (const phase of scenario.phases) {
|
||||
console.log(` - ${phase.id}: ${phase.title}`);
|
||||
}
|
||||
console.log("");
|
||||
}
|
||||
}
|
||||
|
||||
function filterRegistry(items, selectedId, kind) {
|
||||
if (!selectedId) {
|
||||
return items;
|
||||
}
|
||||
const filtered = items.filter((item) => item.id === selectedId);
|
||||
if (filtered.length === 0) {
|
||||
throw new Error(`no ${kind} found for ${selectedId}`);
|
||||
}
|
||||
return filtered;
|
||||
}
|
||||
@ -1,70 +0,0 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { relative } from "node:path";
|
||||
import { required, resolveFromCwd } from "../cli.mjs";
|
||||
import { bundleReport } from "../reporting/artifacts.mjs";
|
||||
import { compareReports, renderCompareFixerSummary, renderCompareSummary } from "../reporting/compare.mjs";
|
||||
import { buildReportSummary, renderPasteSummary, renderReportSummary } from "../reporting/report.mjs";
|
||||
|
||||
export async function runReportCommand(flags) {
|
||||
const [subcommand, firstPath, secondPath] = flags._;
|
||||
|
||||
if (subcommand === "summarize") {
|
||||
const report = await readReport(required(firstPath, "report path"));
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify(buildReportSummary(report), null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(renderReportSummary(report));
|
||||
return;
|
||||
}
|
||||
|
||||
if (subcommand === "paste") {
|
||||
const report = await readReport(required(firstPath, "report path"));
|
||||
console.log(renderPasteSummary(report));
|
||||
return;
|
||||
}
|
||||
|
||||
if (subcommand === "compare") {
|
||||
await compareReportsCommand(required(firstPath, "baseline report path"), required(secondPath, "current report path"), flags);
|
||||
return;
|
||||
}
|
||||
|
||||
if (subcommand === "bundle") {
|
||||
const receipt = await bundleReport(required(firstPath, "report path"), {
|
||||
outputDir: flags.output_dir
|
||||
});
|
||||
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify(receipt, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`Bundle: ${relative(process.cwd(), receipt.outputPath)}`);
|
||||
console.log(`SHA256: ${relative(process.cwd(), receipt.checksumPath)}`);
|
||||
return;
|
||||
}
|
||||
|
||||
throw new Error(`unknown report command: ${subcommand ?? ""}`);
|
||||
}
|
||||
|
||||
async function compareReportsCommand(baselinePath, currentPath, flags) {
|
||||
const baseline = await readReport(baselinePath);
|
||||
const current = await readReport(currentPath);
|
||||
const thresholds = flags.thresholds ? await readReport(flags.thresholds) : null;
|
||||
const comparison = compareReports(baseline, current, { thresholds });
|
||||
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify(comparison, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(flags.fixer ? renderCompareFixerSummary(comparison) : renderCompareSummary(comparison));
|
||||
if (!comparison.ok) {
|
||||
throw new Error("comparison found regressions");
|
||||
}
|
||||
}
|
||||
|
||||
async function readReport(path) {
|
||||
return JSON.parse(await readFile(resolveFromCwd(path), "utf8"));
|
||||
}
|
||||
@ -1,122 +0,0 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { runCleanupCommand } from "../cleanup.mjs";
|
||||
import { resolveFromCwd } from "../cli.mjs";
|
||||
import { ocmRuntimeRemoveJson } from "../ocm/commands.mjs";
|
||||
|
||||
export async function loadRegressionThresholds(flags) {
|
||||
if (!flags.regression_thresholds) {
|
||||
return null;
|
||||
}
|
||||
if (flags.regression_thresholds === true) {
|
||||
throw new Error("--regression-thresholds requires a JSON file path");
|
||||
}
|
||||
return JSON.parse(await readFile(resolveFromCwd(String(flags.regression_thresholds)), "utf8"));
|
||||
}
|
||||
|
||||
export function validateBaselineExecutionFlags(flags) {
|
||||
if ((flags.baseline || flags.save_baseline) && flags.execute !== true) {
|
||||
throw new Error("--baseline and --save-baseline require --execute so baseline evidence comes from real OpenClaw runs");
|
||||
}
|
||||
if (flags.save_baseline && flags.reviewed_good !== true) {
|
||||
throw new Error("--save-baseline requires --reviewed-good after reviewing a passing, stable execution report");
|
||||
}
|
||||
}
|
||||
|
||||
export async function cleanupTargetRuntimeIfNeeded(targetPlan, records, options) {
|
||||
if (targetPlan.kind !== "local-build") {
|
||||
return null;
|
||||
}
|
||||
|
||||
const command = ocmRuntimeRemoveJson(targetPlan.runtimeName);
|
||||
if (!options.execute) {
|
||||
return {
|
||||
status: "planned",
|
||||
runtimeName: targetPlan.runtimeName,
|
||||
command
|
||||
};
|
||||
}
|
||||
|
||||
if (records.some((record) => record.cleanup === "retained")) {
|
||||
return {
|
||||
status: "retained",
|
||||
runtimeName: targetPlan.runtimeName,
|
||||
command,
|
||||
reason: "one or more envs were retained"
|
||||
};
|
||||
}
|
||||
|
||||
const result = await runCleanupCommand(command, { timeoutMs: options.timeoutMs });
|
||||
const cleanupStatus = classifyTargetRuntimeCleanup(result);
|
||||
return {
|
||||
status: cleanupStatus.status,
|
||||
runtimeName: targetPlan.runtimeName,
|
||||
command,
|
||||
reason: cleanupStatus.reason,
|
||||
result: {
|
||||
status: result.status,
|
||||
durationMs: result.durationMs,
|
||||
timedOut: result.timedOut,
|
||||
stdout: result.stdout,
|
||||
stderr: result.stderr,
|
||||
attempts: result.attempts ?? []
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
function classifyTargetRuntimeCleanup(result) {
|
||||
if (result.status === 0) {
|
||||
return { status: "removed" };
|
||||
}
|
||||
|
||||
const output = `${result.stdout}\n${result.stderr}`;
|
||||
if (/\bruntime\b[\s\S]*\bdoes not exist\b/i.test(output) || /\bnot found\b/i.test(output)) {
|
||||
return {
|
||||
status: "already-absent",
|
||||
reason: "target runtime was not present when cleanup ran"
|
||||
};
|
||||
}
|
||||
|
||||
return { status: "remove-failed" };
|
||||
}
|
||||
|
||||
export function positiveIntegerFlag(flags, key, defaultValue) {
|
||||
if (flags[key] === undefined) {
|
||||
return defaultValue;
|
||||
}
|
||||
return positiveIntegerValue(flags[key], `--${key.replaceAll("_", "-")}`);
|
||||
}
|
||||
|
||||
export function profileIntegerFlag(flags, key, defaultValue) {
|
||||
return positiveIntegerFlag(flags, key, defaultValue);
|
||||
}
|
||||
|
||||
export function positiveIntegerValue(raw, label) {
|
||||
if (raw === true) {
|
||||
throw new Error(`${label} requires a positive integer value`);
|
||||
}
|
||||
const value = Number(raw);
|
||||
if (!Number.isInteger(value) || value < 1) {
|
||||
throw new Error(`${label} must be a positive integer, got ${JSON.stringify(raw)}`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function summarizePerformanceReceipt(performance, baseline) {
|
||||
if (!performance) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
schemaVersion: performance.schemaVersion,
|
||||
repeat: performance.repeat,
|
||||
parallel: performance.parallel ?? null,
|
||||
parallelContaminated: performance.parallelContaminated === true,
|
||||
groupCount: performance.groupCount,
|
||||
unstableGroupCount: performance.unstableGroupCount,
|
||||
profiledRunCount: performance.profiledRunCount ?? 0,
|
||||
baselineRegressionCount: baseline?.comparison?.regressionCount ?? null,
|
||||
missingBaselineCount: baseline?.comparison?.missingBaselineCount ?? null,
|
||||
baselineReviewOk: baseline?.review?.ok ?? null,
|
||||
baselineReviewBlockerCount: baseline?.review?.blockerCount ?? null,
|
||||
savedBaselinePath: baseline?.saved?.path ?? null
|
||||
};
|
||||
}
|
||||
@ -1,191 +0,0 @@
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { join, relative } from "node:path";
|
||||
import { authReportSummary, resolveRunAuthContext } from "../auth.mjs";
|
||||
import { required, resolveFromCwd } from "../cli.mjs";
|
||||
import {
|
||||
cleanupTargetRuntimeIfNeeded,
|
||||
loadRegressionThresholds,
|
||||
positiveIntegerFlag,
|
||||
profileIntegerFlag,
|
||||
summarizePerformanceReceipt,
|
||||
validateBaselineExecutionFlags
|
||||
} from "./run-support.mjs";
|
||||
import {
|
||||
comparePerformanceToBaseline,
|
||||
loadBaselineStore,
|
||||
resolveBaselinePath,
|
||||
reviewBaselineUpdate,
|
||||
saveBaselineStore,
|
||||
updateBaselineStore
|
||||
} from "../performance/baselines.mjs";
|
||||
import { buildPerformanceSummary } from "../performance/stats.mjs";
|
||||
import { platformInfo } from "../platform.mjs";
|
||||
import { reportsDir } from "../paths.mjs";
|
||||
import { loadRegistryContext } from "../registries/context.mjs";
|
||||
import { loadScenarios, validateScenarioRun } from "../registries/scenarios.mjs";
|
||||
import { loadState } from "../registries/states.mjs";
|
||||
import { buildReportSummary, renderMarkdownReport, summarizeRecords } from "../reporting/report.mjs";
|
||||
import { buildDryRunRecord, createRunId, executeScenario } from "../runner.mjs";
|
||||
import { resolveTarget } from "../targets.mjs";
|
||||
|
||||
const reportSchemaVersion = "kova.report.v1";
|
||||
|
||||
export async function runScenarioCommand(flags) {
|
||||
const registry = await loadRegistryContext();
|
||||
const target = required(flags.target, "--target");
|
||||
if (flags.execute === true && !flags.scenario) {
|
||||
throw new Error("--execute requires --scenario so real runs stay deliberate");
|
||||
}
|
||||
validateBaselineExecutionFlags(flags);
|
||||
|
||||
const targetPlan = resolveTarget(target, "target");
|
||||
const fromPlan = flags.from ? resolveTarget(flags.from, "from") : null;
|
||||
const state = await loadState(flags.state ?? "fresh");
|
||||
const scenarios = await loadScenarios(flags.scenario);
|
||||
for (const scenario of scenarios) {
|
||||
validateScenarioRun(scenario, flags, { targetPlan, fromPlan });
|
||||
}
|
||||
|
||||
const reportRoot = flags.report_dir ? resolveFromCwd(flags.report_dir) : reportsDir;
|
||||
const runId = createRunId();
|
||||
const reportPath = join(reportRoot, `${runId}.md`);
|
||||
const jsonPath = join(reportRoot, `${runId}.json`);
|
||||
const summaryPath = join(reportRoot, `${runId}.summary.json`);
|
||||
const repeat = positiveIntegerFlag(flags, "repeat", 1);
|
||||
const auth = await resolveRunAuthContext(flags);
|
||||
const regressionThresholds = await loadRegressionThresholds(flags);
|
||||
const baselinePath = resolveBaselinePath(flags.baseline);
|
||||
const saveBaselinePath = resolveBaselinePath(flags.save_baseline);
|
||||
const baselineStore = baselinePath ? await loadBaselineStore(baselinePath) : null;
|
||||
const context = {
|
||||
target,
|
||||
targetPlan,
|
||||
from: flags.from,
|
||||
fromPlan,
|
||||
state,
|
||||
sourceEnv: flags.source_env,
|
||||
runId,
|
||||
execute: flags.execute === true,
|
||||
keepEnv: flags.keep_env === true,
|
||||
retainOnFailure: flags.retain_on_failure === true,
|
||||
timeoutMs: resolveRunTimeout(scenarios, flags),
|
||||
healthSamples: profileIntegerFlag(flags, "health_samples", flags.deep_profile === true ? 10 : 3),
|
||||
healthIntervalMs: positiveIntegerFlag(flags, "health_interval_ms", 250),
|
||||
readinessIntervalMs: profileIntegerFlag(flags, "readiness_interval_ms", flags.deep_profile === true ? 100 : 250),
|
||||
heapSnapshot: flags.heap_snapshot === true || flags.deep_profile === true,
|
||||
diagnosticReport: flags.deep_profile === true,
|
||||
nodeProfile: flags.node_profile === true || flags.deep_profile === true,
|
||||
deepProfile: flags.deep_profile === true,
|
||||
profileOnFailure: flags.profile_on_failure === true,
|
||||
resourceSampleIntervalMs: profileIntegerFlag(flags, "resource_sample_interval_ms", flags.deep_profile === true ? 250 : 1000),
|
||||
processRoles: registry.processRoles,
|
||||
surfacesById: Object.fromEntries(registry.surfaces.map((surface) => [surface.id, surface])),
|
||||
targetSetup: { completed: false },
|
||||
auth
|
||||
};
|
||||
const records = [];
|
||||
|
||||
for (const scenario of scenarios) {
|
||||
for (let index = 1; index <= repeat; index += 1) {
|
||||
const iterationContext = {
|
||||
...context,
|
||||
repeat: {
|
||||
index,
|
||||
total: repeat
|
||||
}
|
||||
};
|
||||
if (iterationContext.execute) {
|
||||
records.push(await executeScenario(scenario, iterationContext));
|
||||
} else {
|
||||
records.push(buildDryRunRecord(scenario, iterationContext));
|
||||
}
|
||||
}
|
||||
}
|
||||
const targetCleanup = await cleanupTargetRuntimeIfNeeded(targetPlan, records, {
|
||||
execute: context.execute,
|
||||
timeoutMs: context.timeoutMs
|
||||
});
|
||||
const performance = buildPerformanceSummary(records, { repeat, regressionThresholds });
|
||||
|
||||
await mkdir(reportRoot, { recursive: true });
|
||||
const report = {
|
||||
schemaVersion: reportSchemaVersion,
|
||||
generatedAt: new Date().toISOString(),
|
||||
runId,
|
||||
outputPaths: {
|
||||
markdown: reportPath,
|
||||
json: jsonPath,
|
||||
summary: summaryPath
|
||||
},
|
||||
mode: context.execute ? "execution" : "dry-run",
|
||||
target,
|
||||
from: flags.from ?? null,
|
||||
state: {
|
||||
id: state.id,
|
||||
title: state.title,
|
||||
objective: state.objective
|
||||
},
|
||||
platform: platformInfo(),
|
||||
targetCleanup,
|
||||
auth: authReportSummary(auth),
|
||||
controls: {
|
||||
repeat,
|
||||
baseline: baselinePath,
|
||||
saveBaseline: saveBaselinePath,
|
||||
auth: auth.requestedMode
|
||||
},
|
||||
performance,
|
||||
baseline: null,
|
||||
summary: summarizeRecords(records),
|
||||
records
|
||||
};
|
||||
const baselineComparison = comparePerformanceToBaseline(report, baselineStore, { targetPlan, regressionThresholds });
|
||||
if (baselineComparison) {
|
||||
report.baseline = {
|
||||
path: baselinePath,
|
||||
comparison: baselineComparison
|
||||
};
|
||||
}
|
||||
if (saveBaselinePath) {
|
||||
const existingStore = await loadBaselineStore(saveBaselinePath);
|
||||
const review = reviewBaselineUpdate(report, { reviewedGood: flags.reviewed_good === true });
|
||||
const updatedStore = updateBaselineStore(existingStore, report, { targetPlan, reviewedGood: flags.reviewed_good === true });
|
||||
report.baseline = {
|
||||
...(report.baseline ?? {}),
|
||||
review,
|
||||
saved: await saveBaselineStore(saveBaselinePath, updatedStore)
|
||||
};
|
||||
}
|
||||
await writeFile(reportPath, renderMarkdownReport(report), "utf8");
|
||||
await writeFile(jsonPath, `${JSON.stringify(report, null, 2)}\n`, "utf8");
|
||||
await writeFile(summaryPath, `${JSON.stringify(buildReportSummary(report), null, 2)}\n`, "utf8");
|
||||
|
||||
const mode = context.execute ? "execution" : "dry-run";
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify({
|
||||
schemaVersion: "kova.run.receipt.v1",
|
||||
generatedAt: new Date().toISOString(),
|
||||
mode,
|
||||
runId,
|
||||
reportPath,
|
||||
jsonPath,
|
||||
summaryPath,
|
||||
performance: summarizePerformanceReceipt(report.performance, report.baseline),
|
||||
summary: report.summary
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`Kova ${mode} report written: ${relative(process.cwd(), reportPath)}`);
|
||||
console.log(`Kova ${mode} data written: ${relative(process.cwd(), jsonPath)}`);
|
||||
}
|
||||
|
||||
function resolveRunTimeout(scenarios, flags) {
|
||||
if (flags.timeout_ms !== undefined) {
|
||||
return positiveIntegerFlag(flags, "timeout_ms", 120000);
|
||||
}
|
||||
const scenarioTimeouts = scenarios
|
||||
.map((scenario) => scenario.timeoutMs)
|
||||
.filter((timeout) => typeof timeout === "number");
|
||||
return scenarioTimeouts.length === 0 ? 120000 : Math.max(...scenarioTimeouts);
|
||||
}
|
||||
@ -1,17 +0,0 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { repoRoot } from "../paths.mjs";
|
||||
|
||||
export async function runVersionCommand(flags = {}) {
|
||||
const packageJson = JSON.parse(await readFile(join(repoRoot, "package.json"), "utf8"));
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify({
|
||||
schemaVersion: "kova.version.v1",
|
||||
name: packageJson.name,
|
||||
version: packageJson.version
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(packageJson.version);
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
318
src/health.mjs
318
src/health.mjs
@ -1,318 +0,0 @@
|
||||
export const HEALTH_SCHEMA = "kova.health.v1";
|
||||
export const HEALTH_SCOPES = ["readiness", "startup-sample", "post-ready", "final", "none", "unknown"];
|
||||
|
||||
const startupScopes = new Set(["readiness", "startup-sample"]);
|
||||
|
||||
export function buildHealthMeasurement(record, scenario = null) {
|
||||
const phaseContracts = new Map((scenario?.phases ?? []).map((phase) => [phase.id, phase]));
|
||||
const entries = [];
|
||||
for (const phase of record.phases ?? []) {
|
||||
entries.push({
|
||||
source: "phase",
|
||||
phaseId: phase.id ?? null,
|
||||
scope: normalizeHealthScope(phase.healthScope ?? phaseContracts.get(phase.id)?.healthScope),
|
||||
metrics: phase.metrics ?? null
|
||||
});
|
||||
}
|
||||
|
||||
const finalEntry = {
|
||||
source: "final",
|
||||
phaseId: "final",
|
||||
scope: "final",
|
||||
metrics: record.finalMetrics ?? null
|
||||
};
|
||||
entries.push(finalEntry);
|
||||
|
||||
const readiness = selectReadiness(entries);
|
||||
const startupSamples = summarizeScopedSamples(
|
||||
entries.filter((entry) => startupScopes.has(entry.scope)),
|
||||
"startup-sample",
|
||||
startupSamplesForEntry
|
||||
);
|
||||
const postReadySamples = summarizeScopedSamples(
|
||||
entries.filter((entry) => entry.scope === "post-ready"),
|
||||
"post-ready",
|
||||
postReadySamplesForEntry
|
||||
);
|
||||
const unknownSamples = summarizeScopedSamples(
|
||||
entries.filter((entry) => entry.scope === "unknown"),
|
||||
"unknown",
|
||||
postReadySamplesForEntry
|
||||
);
|
||||
const final = summarizeFinalHealth(finalEntry.metrics);
|
||||
const slowestSample = selectSlowestSample([startupSamples, postReadySamples, final]);
|
||||
|
||||
return {
|
||||
schemaVersion: HEALTH_SCHEMA,
|
||||
readiness,
|
||||
startupSamples,
|
||||
postReadySamples,
|
||||
unknownSamples,
|
||||
final,
|
||||
slowestSample
|
||||
};
|
||||
}
|
||||
|
||||
export function healthReadinessClassification(health) {
|
||||
if (!health?.readiness) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
phaseId: health.readiness.phaseId,
|
||||
state: health.readiness.classification,
|
||||
severity: health.readiness.severity,
|
||||
reason: health.readiness.reason,
|
||||
thresholdMs: health.readiness.thresholdMs,
|
||||
deadlineMs: health.readiness.deadlineMs,
|
||||
listeningReadyAtMs: health.readiness.listeningReadyAtMs,
|
||||
healthReadyAtMs: health.readiness.healthReadyAtMs
|
||||
};
|
||||
}
|
||||
|
||||
export function healthTotalFailures(health) {
|
||||
return (health?.startupSamples?.failureCount ?? 0) +
|
||||
(health?.postReadySamples?.failureCount ?? 0) +
|
||||
(health?.unknownSamples?.failureCount ?? 0) +
|
||||
(health?.final?.failureCount ?? 0);
|
||||
}
|
||||
|
||||
export function measurementMetricValue(measurements, metric) {
|
||||
if (!measurements) {
|
||||
return null;
|
||||
}
|
||||
switch (metric) {
|
||||
case "readinessListeningMs":
|
||||
return measurements.health?.readiness?.listeningReadyAtMs ?? null;
|
||||
case "readinessHealthReadyMs":
|
||||
return measurements.health?.readiness?.healthReadyAtMs ?? null;
|
||||
case "startupHealthP95Ms":
|
||||
return measurements.health?.startupSamples?.p95Ms ?? null;
|
||||
case "postReadyHealthP95Ms":
|
||||
return measurements.health?.postReadySamples?.p95Ms ?? null;
|
||||
case "startupHealthFailures":
|
||||
return measurements.health?.startupSamples?.failureCount ?? null;
|
||||
case "postReadyHealthFailures":
|
||||
return measurements.health?.postReadySamples?.failureCount ?? null;
|
||||
case "finalHealthFailures":
|
||||
return measurements.health?.final?.failureCount ?? null;
|
||||
default:
|
||||
return measurements[metric] ?? null;
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeHealthScope(scope) {
|
||||
return typeof scope === "string" && HEALTH_SCOPES.includes(scope) ? scope : "unknown";
|
||||
}
|
||||
|
||||
function selectReadiness(entries) {
|
||||
const scoped = entries
|
||||
.filter((entry) => startupScopes.has(entry.scope))
|
||||
.map((entry) => readinessValue(entry.metrics?.readiness, entry.phaseId))
|
||||
.filter(Boolean);
|
||||
const candidates = scoped.length > 0
|
||||
? scoped
|
||||
: entries.map((entry) => readinessValue(entry.metrics?.readiness, entry.phaseId)).filter(Boolean);
|
||||
if (candidates.length === 0) {
|
||||
return null;
|
||||
}
|
||||
candidates.sort((left, right) => {
|
||||
const rankDelta = readinessRank(right.classification) - readinessRank(left.classification);
|
||||
if (rankDelta !== 0) {
|
||||
return rankDelta;
|
||||
}
|
||||
return (right.healthReadyAtMs ?? 0) - (left.healthReadyAtMs ?? 0);
|
||||
});
|
||||
return candidates[0];
|
||||
}
|
||||
|
||||
function readinessValue(readiness, phaseId) {
|
||||
if (!readiness?.classification || !(readiness.deadlineMs > 0)) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
phaseId,
|
||||
listeningReadyAtMs: readiness.listeningReadyAtMs,
|
||||
healthReadyAtMs: readiness.healthReadyAtMs,
|
||||
classification: readiness.classification.state,
|
||||
severity: readiness.classification.severity,
|
||||
reason: readiness.classification.reason,
|
||||
thresholdMs: readiness.thresholdMs,
|
||||
deadlineMs: readiness.deadlineMs,
|
||||
attempts: readiness.attempts ?? null
|
||||
};
|
||||
}
|
||||
|
||||
function readinessRank(state) {
|
||||
if (state === "hard-failure") {
|
||||
return 4;
|
||||
}
|
||||
if (state === "unhealthy") {
|
||||
return 3;
|
||||
}
|
||||
if (state === "slow-startup") {
|
||||
return 2;
|
||||
}
|
||||
if (state === "ready") {
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
function startupSamplesForEntry(entry) {
|
||||
const attempts = entry.metrics?.readiness?.healthAttempts;
|
||||
if (Array.isArray(attempts) && attempts.length > 0) {
|
||||
return attempts;
|
||||
}
|
||||
return entry.metrics?.healthSamples ?? [];
|
||||
}
|
||||
|
||||
function postReadySamplesForEntry(entry) {
|
||||
return entry.metrics?.healthSamples ?? [];
|
||||
}
|
||||
|
||||
function summarizeScopedSamples(entries, scope, sampleSelector) {
|
||||
const samples = [];
|
||||
for (const entry of entries) {
|
||||
for (const sample of sampleSelector(entry)) {
|
||||
samples.push({ ...sample, phaseId: entry.phaseId });
|
||||
}
|
||||
}
|
||||
if (samples.length > 0) {
|
||||
return summarizeSamples(samples, scope);
|
||||
}
|
||||
|
||||
const summaries = entries
|
||||
.map((entry) => ({ phaseId: entry.phaseId, summary: entry.metrics?.healthSummary }))
|
||||
.filter((entry) => entry.summary);
|
||||
if (summaries.length === 0) {
|
||||
return emptyHealthSummary(scope);
|
||||
}
|
||||
|
||||
let slowestPhaseId = null;
|
||||
let maxMs = null;
|
||||
for (const { phaseId, summary } of summaries) {
|
||||
if (typeof summary.maxMs === "number" && (maxMs === null || summary.maxMs > maxMs)) {
|
||||
maxMs = summary.maxMs;
|
||||
slowestPhaseId = phaseId;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
scope,
|
||||
count: sum(summaries, "count"),
|
||||
okCount: sum(summaries, "okCount"),
|
||||
failureCount: sum(summaries, "failureCount"),
|
||||
minMs: minNullable(...summaries.map(({ summary }) => summary.minMs)),
|
||||
p50Ms: maxNullable(...summaries.map(({ summary }) => summary.p50Ms)),
|
||||
p95Ms: maxNullable(...summaries.map(({ summary }) => summary.p95Ms)),
|
||||
maxMs,
|
||||
slowestPhaseId
|
||||
};
|
||||
}
|
||||
|
||||
function summarizeSamples(samples, scope) {
|
||||
const durations = samples
|
||||
.map((sample) => sample.durationMs)
|
||||
.filter((duration) => typeof duration === "number")
|
||||
.sort((left, right) => left - right);
|
||||
let slowestPhaseId = null;
|
||||
let slowestMs = null;
|
||||
for (const sample of samples) {
|
||||
if (typeof sample.durationMs === "number" && (slowestMs === null || sample.durationMs > slowestMs)) {
|
||||
slowestMs = sample.durationMs;
|
||||
slowestPhaseId = sample.phaseId ?? null;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
scope,
|
||||
count: samples.length,
|
||||
okCount: samples.filter((sample) => sample.ok === true).length,
|
||||
failureCount: samples.filter((sample) => sample.ok !== true).length,
|
||||
minMs: durations.at(0) ?? null,
|
||||
p50Ms: percentile(durations, 0.5),
|
||||
p95Ms: percentile(durations, 0.95),
|
||||
maxMs: durations.at(-1) ?? null,
|
||||
slowestPhaseId
|
||||
};
|
||||
}
|
||||
|
||||
function emptyHealthSummary(scope) {
|
||||
return {
|
||||
scope,
|
||||
count: 0,
|
||||
okCount: 0,
|
||||
failureCount: 0,
|
||||
minMs: null,
|
||||
p50Ms: null,
|
||||
p95Ms: null,
|
||||
maxMs: null,
|
||||
slowestPhaseId: null
|
||||
};
|
||||
}
|
||||
|
||||
function summarizeFinalHealth(metrics) {
|
||||
const samples = Array.isArray(metrics?.healthSamples) ? metrics.healthSamples : [];
|
||||
const summary = samples.length > 0 ? summarizeSamples(samples.map((sample) => ({ ...sample, phaseId: "final" })), "final") : null;
|
||||
const fallbackFailureCount = healthFailureCount([metrics?.health]);
|
||||
const failureCount = summary?.failureCount ?? metrics?.healthSummary?.failureCount ?? fallbackFailureCount;
|
||||
const maxMs = summary?.maxMs ?? metrics?.healthSummary?.maxMs ?? metrics?.health?.durationMs ?? null;
|
||||
const p95Ms = summary?.p95Ms ?? metrics?.healthSummary?.p95Ms ?? null;
|
||||
const gatewayState = metrics?.service?.gatewayState ?? null;
|
||||
const ok = metrics
|
||||
? (gatewayState === null ? failureCount === 0 : gatewayState === "running" && failureCount === 0)
|
||||
: null;
|
||||
return {
|
||||
scope: "final",
|
||||
gatewayState,
|
||||
ok,
|
||||
healthOk: metrics?.health?.ok ?? null,
|
||||
failureCount,
|
||||
p95Ms,
|
||||
maxMs,
|
||||
slowestPhaseId: maxMs === null ? null : "final"
|
||||
};
|
||||
}
|
||||
|
||||
function selectSlowestSample(summaries) {
|
||||
let slowest = null;
|
||||
for (const summary of summaries) {
|
||||
if (!summary || typeof summary.maxMs !== "number") {
|
||||
continue;
|
||||
}
|
||||
if (!slowest || summary.maxMs > slowest.durationMs) {
|
||||
slowest = {
|
||||
scope: summary.scope,
|
||||
phaseId: summary.slowestPhaseId ?? null,
|
||||
durationMs: summary.maxMs
|
||||
};
|
||||
}
|
||||
}
|
||||
return slowest;
|
||||
}
|
||||
|
||||
function healthFailureCount(samples) {
|
||||
return samples.filter((sample) => sample && sample.ok === false).length;
|
||||
}
|
||||
|
||||
function sum(entries, key) {
|
||||
return entries.reduce((total, entry) => total + (entry.summary?.[key] ?? 0), 0);
|
||||
}
|
||||
|
||||
function maxNullable(...values) {
|
||||
const numeric = values.filter((value) => typeof value === "number");
|
||||
return numeric.length === 0 ? null : Math.max(...numeric);
|
||||
}
|
||||
|
||||
function minNullable(...values) {
|
||||
const numeric = values.filter((value) => typeof value === "number");
|
||||
return numeric.length === 0 ? null : Math.min(...numeric);
|
||||
}
|
||||
|
||||
function percentile(values, percentileValue) {
|
||||
if (values.length === 0) {
|
||||
return null;
|
||||
}
|
||||
const index = Math.ceil(values.length * percentileValue) - 1;
|
||||
return values[Math.min(Math.max(index, 0), values.length - 1)];
|
||||
}
|
||||
@ -1,646 +0,0 @@
|
||||
import { readFile, readdir } from "node:fs/promises";
|
||||
import { basename, join, relative } from "node:path";
|
||||
import { quoteShell, runCommand } from "../commands.mjs";
|
||||
import { positiveIntegerFlag } from "../commands/run-support.mjs";
|
||||
import { resolveFromCwd } from "../cli.mjs";
|
||||
import { loadRegistryContext } from "../registries/context.mjs";
|
||||
|
||||
const inventorySchemaVersion = "kova.inventory.plan.v1";
|
||||
const manifestSearchDirs = ["apps", "extensions", "packages", "plugins", "src"];
|
||||
const ignoredDirs = new Set([
|
||||
".git",
|
||||
".next",
|
||||
".turbo",
|
||||
"build",
|
||||
"coverage",
|
||||
"dist",
|
||||
"node_modules",
|
||||
"out",
|
||||
"target",
|
||||
"tmp"
|
||||
]);
|
||||
|
||||
const commandSurfaceMap = new Map([
|
||||
["acp", ["agent-cli-local-turn", "agent-gateway-rpc-turn"]],
|
||||
["agent", ["agent-cli-local-turn", "agent-gateway-rpc-turn"]],
|
||||
["agents", ["agent-cli-local-turn", "agent-gateway-rpc-turn", "workspace-scan"]],
|
||||
["browser", ["browser-automation"]],
|
||||
["capability", ["openai-compatible-turn", "provider-models"]],
|
||||
["chat", ["gateway-session-send-turn", "tui", "tui-message-turn"]],
|
||||
["configure", ["fresh-install", "provider-models"]],
|
||||
["daemon", ["release-runtime-startup", "gateway-performance"]],
|
||||
["dashboard", ["dashboard", "gateway-session-send-turn"]],
|
||||
["doctor", ["failure-containment", "release-runtime-startup"]],
|
||||
["gateway", ["release-runtime-startup", "gateway-performance", "gateway-session-send-turn"]],
|
||||
["health", ["release-runtime-startup", "gateway-performance"]],
|
||||
["infer", ["openai-compatible-turn", "provider-models"]],
|
||||
["logs", ["release-runtime-startup", "gateway-performance"]],
|
||||
["mcp", ["mcp-runtime"]],
|
||||
["media", ["media-understanding"]],
|
||||
["model", ["provider-models"]],
|
||||
["models", ["provider-models"]],
|
||||
["node", ["release-runtime-startup", "gateway-performance"]],
|
||||
["nodes", ["release-runtime-startup", "gateway-performance"]],
|
||||
["onboard", ["fresh-install", "provider-models"]],
|
||||
["plugin", ["plugin-lifecycle"]],
|
||||
["plugins", ["plugin-lifecycle", "official-plugin-install"]],
|
||||
["provider", ["provider-models"]],
|
||||
["providers", ["provider-models"]],
|
||||
["setup", ["fresh-install", "release-runtime-startup"]],
|
||||
["start", ["release-runtime-startup", "gateway-performance"]],
|
||||
["status", ["release-runtime-startup"]],
|
||||
["stop", ["release-runtime-startup"]],
|
||||
["terminal", ["tui", "tui-message-turn"]],
|
||||
["tui", ["tui", "tui-message-turn"]],
|
||||
["uninstall", ["fresh-install", "failure-containment"]],
|
||||
["update", ["upgrade-existing-user"]],
|
||||
["upgrade", ["upgrade-existing-user"]],
|
||||
["workspace", ["workspace-scan"]]
|
||||
]);
|
||||
const packageScriptScopes = new Set(["product", "all", "none"]);
|
||||
const productScriptNames = new Set([
|
||||
"gateway:dev",
|
||||
"gateway:dev:reset",
|
||||
"gateway:watch",
|
||||
"gateway:watch:raw",
|
||||
"openclaw",
|
||||
"openclaw:rpc",
|
||||
"plugin-sdk:api:check",
|
||||
"plugin-sdk:check-exports",
|
||||
"plugins:inventory:check",
|
||||
"plugins:inventory:gen",
|
||||
"plugins:sync",
|
||||
"plugins:sync:check",
|
||||
"release-metadata:check",
|
||||
"release:check",
|
||||
"release:openclaw:npm:check",
|
||||
"release:openclaw:npm:verify-published",
|
||||
"start",
|
||||
"test:docker:live-gateway",
|
||||
"test:docker:live-models",
|
||||
"test:docker:npm-onboard-channel-agent",
|
||||
"test:docker:plugin-update",
|
||||
"test:docker:plugins",
|
||||
"test:docker:published-upgrade-survivor",
|
||||
"test:docker:update-migration",
|
||||
"test:docker:upgrade-survivor",
|
||||
"test:gateway",
|
||||
"test:install:e2e",
|
||||
"test:install:smoke",
|
||||
"test:live:gateway-profiles",
|
||||
"test:live:media",
|
||||
"test:live:models-profiles",
|
||||
"test:perf:budget",
|
||||
"test:perf:groups",
|
||||
"test:plugins:gateway-gauntlet",
|
||||
"test:stability:gateway",
|
||||
"test:startup:bench",
|
||||
"test:startup:bench:check",
|
||||
"test:startup:bench:smoke",
|
||||
"test:startup:gateway",
|
||||
"test:startup:memory",
|
||||
"tui",
|
||||
"tui:dev",
|
||||
"ui:build",
|
||||
"ui:dev",
|
||||
"ui:install"
|
||||
]);
|
||||
const productScriptPrefixes = [
|
||||
"release:plugins:",
|
||||
"test:docker:live-acp-bind:",
|
||||
"test:docker:live-cli-backend:",
|
||||
"test:docker:live-gateway:",
|
||||
"test:docker:live-models:",
|
||||
"test:live:media:"
|
||||
];
|
||||
|
||||
export async function buildOpenClawInventoryPlan(flags = {}) {
|
||||
const registry = await loadRegistryContext();
|
||||
const timeoutMs = positiveIntegerFlag(flags, "timeout_ms", 10000);
|
||||
const maxSubcommands = positiveIntegerFlag(flags, "max_subcommands", 40);
|
||||
const openclawBin = normalizeOptionalCommand(flags.openclaw_bin);
|
||||
const repoPath = normalizeOptionalPath(flags.openclaw_repo);
|
||||
const scriptScope = normalizeScriptScope(flags.script_scope);
|
||||
const requestedSubcommands = parseList(flags.subcommands);
|
||||
const requiredModeled = parseList(flags.require_modeled);
|
||||
const sources = [];
|
||||
const capabilities = [];
|
||||
|
||||
const helpInventory = await discoverCliHelp({
|
||||
openclawBin,
|
||||
requestedSubcommands,
|
||||
maxSubcommands,
|
||||
timeoutMs
|
||||
});
|
||||
sources.push(helpInventory.source);
|
||||
capabilities.push(...helpInventory.capabilities);
|
||||
|
||||
const repoInventory = await discoverRepoInventory({ repoPath, scriptScope });
|
||||
sources.push(...repoInventory.sources);
|
||||
capabilities.push(...repoInventory.capabilities);
|
||||
|
||||
const modeledSurfaces = registry.surfaces.map((surface) => ({
|
||||
id: surface.id,
|
||||
title: surface.title,
|
||||
ownerArea: surface.ownerArea,
|
||||
purposes: surface.purposes ?? []
|
||||
}));
|
||||
const classifiedCapabilities = capabilities.map((capability) =>
|
||||
classifyCapability(capability, registry.surfaces)
|
||||
);
|
||||
|
||||
return {
|
||||
schemaVersion: inventorySchemaVersion,
|
||||
generatedAt: new Date().toISOString(),
|
||||
openclaw: {
|
||||
bin: openclawBin,
|
||||
repoPath,
|
||||
scriptScope
|
||||
},
|
||||
sources,
|
||||
modeledSurfaces,
|
||||
capabilities: classifiedCapabilities,
|
||||
coverage: summarizeCoverage(classifiedCapabilities, modeledSurfaces, {
|
||||
requiredModeled
|
||||
})
|
||||
};
|
||||
}
|
||||
|
||||
async function discoverCliHelp({ openclawBin, requestedSubcommands, maxSubcommands, timeoutMs }) {
|
||||
if (!openclawBin) {
|
||||
return {
|
||||
source: {
|
||||
id: "openclaw-help",
|
||||
kind: "cli-help",
|
||||
status: "skipped",
|
||||
reason: "--openclaw-bin was not provided"
|
||||
},
|
||||
capabilities: []
|
||||
};
|
||||
}
|
||||
|
||||
const helpCommand = `${quoteShell(openclawBin)} --help`;
|
||||
const topLevel = await runCommand(helpCommand, {
|
||||
timeoutMs,
|
||||
maxOutputChars: 50000
|
||||
});
|
||||
if (topLevel.status !== 0) {
|
||||
return {
|
||||
source: {
|
||||
id: "openclaw-help",
|
||||
kind: "cli-help",
|
||||
status: "failed",
|
||||
command: topLevel.command,
|
||||
statusCode: topLevel.status,
|
||||
timedOut: topLevel.timedOut,
|
||||
error: topLevel.stderr.trim() || topLevel.stdout.trim() || "openclaw --help failed"
|
||||
},
|
||||
capabilities: []
|
||||
};
|
||||
}
|
||||
|
||||
const parsedCommands = requestedSubcommands.length > 0
|
||||
? requestedSubcommands
|
||||
: parseHelpCommands(topLevel.stdout);
|
||||
const allUniqueCommands = [...new Set(parsedCommands)].sort();
|
||||
const uniqueCommands = allUniqueCommands.slice(0, maxSubcommands);
|
||||
const capabilities = uniqueCommands.map((command) => ({
|
||||
id: `cli:${command}`,
|
||||
kind: "cli-command",
|
||||
name: command,
|
||||
source: "openclaw-help",
|
||||
path: null,
|
||||
summary: null,
|
||||
evidence: {
|
||||
command: helpCommand
|
||||
}
|
||||
}));
|
||||
|
||||
for (const capability of capabilities) {
|
||||
const result = await runCommand(`${quoteShell(openclawBin)} ${quoteShell(capability.name)} --help`, {
|
||||
timeoutMs,
|
||||
maxOutputChars: 30000
|
||||
});
|
||||
capability.evidence.subcommandHelp = {
|
||||
command: result.command,
|
||||
status: result.status,
|
||||
timedOut: result.timedOut
|
||||
};
|
||||
if (result.status === 0) {
|
||||
capability.summary = firstUsefulHelpLine(result.stdout);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
source: {
|
||||
id: "openclaw-help",
|
||||
kind: "cli-help",
|
||||
status: "scanned",
|
||||
command: topLevel.command,
|
||||
commandCount: capabilities.length,
|
||||
discoveredCommandCount: allUniqueCommands.length,
|
||||
truncated: allUniqueCommands.length > uniqueCommands.length,
|
||||
requestedSubcommands
|
||||
},
|
||||
capabilities
|
||||
};
|
||||
}
|
||||
|
||||
async function discoverRepoInventory({ repoPath, scriptScope }) {
|
||||
if (!repoPath) {
|
||||
return {
|
||||
sources: [
|
||||
{
|
||||
id: "package-scripts",
|
||||
kind: "package-json",
|
||||
status: "skipped",
|
||||
reason: "--openclaw-repo was not provided"
|
||||
},
|
||||
{
|
||||
id: "manifests",
|
||||
kind: "manifest-scan",
|
||||
status: "skipped",
|
||||
reason: "--openclaw-repo was not provided"
|
||||
}
|
||||
],
|
||||
capabilities: []
|
||||
};
|
||||
}
|
||||
|
||||
const sources = [];
|
||||
const capabilities = [];
|
||||
const packagePath = join(repoPath, "package.json");
|
||||
try {
|
||||
const packageJson = JSON.parse(await readFile(packagePath, "utf8"));
|
||||
const scripts = Object.keys(packageJson.scripts ?? {}).sort();
|
||||
const includedScripts = filterPackageScripts(scripts, scriptScope);
|
||||
for (const script of includedScripts) {
|
||||
capabilities.push({
|
||||
id: `script:${script}`,
|
||||
kind: "package-script",
|
||||
name: script,
|
||||
source: "package-scripts",
|
||||
path: relative(repoPath, packagePath),
|
||||
summary: packageJson.scripts[script],
|
||||
evidence: {
|
||||
packageName: packageJson.name ?? null
|
||||
}
|
||||
});
|
||||
}
|
||||
sources.push({
|
||||
id: "package-scripts",
|
||||
kind: "package-json",
|
||||
status: "scanned",
|
||||
path: packagePath,
|
||||
scriptScope,
|
||||
scriptCount: scripts.length,
|
||||
includedScriptCount: includedScripts.length,
|
||||
excludedScriptCount: scripts.length - includedScripts.length
|
||||
});
|
||||
} catch (error) {
|
||||
sources.push({
|
||||
id: "package-scripts",
|
||||
kind: "package-json",
|
||||
status: error.code === "ENOENT" ? "missing" : "failed",
|
||||
path: packagePath,
|
||||
error: error.code === "ENOENT" ? null : error.message
|
||||
});
|
||||
}
|
||||
|
||||
const manifestResult = await discoverManifests(repoPath);
|
||||
sources.push(manifestResult.source);
|
||||
capabilities.push(...manifestResult.capabilities);
|
||||
return { sources, capabilities };
|
||||
}
|
||||
|
||||
async function discoverManifests(repoPath) {
|
||||
const candidates = [];
|
||||
const roots = manifestSearchDirs.map((dir) => join(repoPath, dir));
|
||||
for (const root of roots) {
|
||||
await collectManifestCandidates(root, repoPath, candidates);
|
||||
}
|
||||
|
||||
const capabilities = [];
|
||||
for (const path of candidates.slice(0, 300)) {
|
||||
try {
|
||||
const manifest = JSON.parse(await readFile(path, "utf8"));
|
||||
const kind = classifyManifest(path, manifest);
|
||||
if (!kind) {
|
||||
continue;
|
||||
}
|
||||
const name = manifest.name ?? manifest.id ?? manifest.displayName ?? basename(path);
|
||||
capabilities.push({
|
||||
id: `${kind}:${normalizeToken(name) || normalizeToken(relative(repoPath, path))}`,
|
||||
kind,
|
||||
name,
|
||||
source: "manifests",
|
||||
path: relative(repoPath, path),
|
||||
summary: manifest.description ?? manifest.title ?? null,
|
||||
evidence: {
|
||||
manifestId: manifest.id ?? null,
|
||||
packageName: manifest.name ?? null
|
||||
}
|
||||
});
|
||||
} catch {
|
||||
// Non-JSON manifest-looking files are ignored; registry validation catches Kova contracts.
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
source: {
|
||||
id: "manifests",
|
||||
kind: "manifest-scan",
|
||||
status: "scanned",
|
||||
roots: roots.map((root) => relative(repoPath, root)),
|
||||
candidateCount: candidates.length,
|
||||
capabilityCount: capabilities.length,
|
||||
truncated: candidates.length > 300
|
||||
},
|
||||
capabilities
|
||||
};
|
||||
}
|
||||
|
||||
async function collectManifestCandidates(root, repoPath, candidates, depth = 0) {
|
||||
if (depth > 8 || candidates.length > 300) {
|
||||
return;
|
||||
}
|
||||
|
||||
let entries;
|
||||
try {
|
||||
entries = await readdir(root, { withFileTypes: true });
|
||||
} catch (error) {
|
||||
if (error.code === "ENOENT") {
|
||||
return;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
if (entry.isDirectory()) {
|
||||
if (!ignoredDirs.has(entry.name)) {
|
||||
await collectManifestCandidates(join(root, entry.name), repoPath, candidates, depth + 1);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (!entry.isFile()) {
|
||||
continue;
|
||||
}
|
||||
const name = entry.name.toLowerCase();
|
||||
if (name === "plugin.json" || name === "manifest.json" || name.endsWith(".manifest.json")) {
|
||||
candidates.push(join(root, entry.name));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function classifyManifest(path, manifest) {
|
||||
const lowerPath = path.toLowerCase();
|
||||
if (manifest.openclawPlugin === true || manifest.plugin === true || lowerPath.endsWith("/plugin.json")) {
|
||||
return "plugin-manifest";
|
||||
}
|
||||
if (manifest.openclawExtension === true || manifest.extension === true || lowerPath.includes("/extensions/")) {
|
||||
return "extension-manifest";
|
||||
}
|
||||
if (Array.isArray(manifest.contributes) || manifest.activationEvents || manifest.main) {
|
||||
return lowerPath.includes("plugin") ? "plugin-manifest" : "extension-manifest";
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function classifyCapability(capability, surfaces) {
|
||||
const matchedSurfaceIds = matchSurfaceIds(capability, surfaces);
|
||||
return {
|
||||
...capability,
|
||||
modeled: matchedSurfaceIds.length > 0,
|
||||
matchedSurfaceIds,
|
||||
matchStatus: matchedSurfaceIds.length === 0
|
||||
? "unmodeled"
|
||||
: matchedSurfaceIds.length === 1 ? "matched" : "ambiguous"
|
||||
};
|
||||
}
|
||||
|
||||
function matchSurfaceIds(capability, surfaces) {
|
||||
const normalizedName = normalizeToken(capability.name);
|
||||
const mapped = [
|
||||
...(commandSurfaceMap.get(normalizedName) ?? []),
|
||||
...matchPackageScriptSurfaceIds(capability)
|
||||
];
|
||||
const surfaceIds = new Set(surfaces.map((surface) => surface.id));
|
||||
const matches = new Set(mapped.filter((id) => surfaceIds.has(id)));
|
||||
|
||||
for (const surface of surfaces) {
|
||||
const haystack = [
|
||||
surface.id,
|
||||
surface.title,
|
||||
surface.ownerArea,
|
||||
...(surface.purposes ?? [])
|
||||
].map(normalizeToken);
|
||||
if (haystack.includes(normalizedName)) {
|
||||
matches.add(surface.id);
|
||||
}
|
||||
}
|
||||
|
||||
return [...matches].sort();
|
||||
}
|
||||
|
||||
function summarizeCoverage(capabilities, modeledSurfaces, options = {}) {
|
||||
const unmodeled = capabilities.filter((capability) => !capability.modeled);
|
||||
const matched = capabilities.filter((capability) => capability.modeled);
|
||||
const ambiguous = matched.filter((capability) => capability.matchStatus === "ambiguous");
|
||||
const requiredModeled = options.requiredModeled ?? [];
|
||||
const capabilitiesById = new Map(capabilities.map((capability) => [capability.id, capability]));
|
||||
const blockers = requiredModeled.flatMap((id) => {
|
||||
const capability = capabilitiesById.get(id);
|
||||
if (!capability) {
|
||||
return [{
|
||||
kind: "required-capability-missing",
|
||||
capability: id,
|
||||
message: `required inventory capability ${id} was not discovered`
|
||||
}];
|
||||
}
|
||||
if (!capability.modeled) {
|
||||
return [{
|
||||
kind: "required-capability-unmodeled",
|
||||
capability: id,
|
||||
name: capability.name,
|
||||
source: capability.source,
|
||||
message: `required inventory capability ${id} is not mapped to a Kova surface`
|
||||
}];
|
||||
}
|
||||
return [];
|
||||
});
|
||||
return {
|
||||
discoveredCount: capabilities.length,
|
||||
modeledSurfaceCount: modeledSurfaces.length,
|
||||
matchedCount: matched.length,
|
||||
ambiguousCount: ambiguous.length,
|
||||
unmodeledCount: unmodeled.length,
|
||||
requiredModeled,
|
||||
ok: blockers.length === 0,
|
||||
blockers,
|
||||
warnings: unmodeled.map((capability) => ({
|
||||
kind: "unmodeled-capability",
|
||||
capability: capability.id,
|
||||
name: capability.name,
|
||||
source: capability.source,
|
||||
message: `${capability.kind} ${capability.name} is not mapped to a Kova surface`
|
||||
})),
|
||||
ambiguous: ambiguous.map((capability) => ({
|
||||
id: capability.id,
|
||||
kind: capability.kind,
|
||||
name: capability.name,
|
||||
source: capability.source,
|
||||
matchedSurfaceIds: capability.matchedSurfaceIds
|
||||
})),
|
||||
unmodeled: unmodeled.map((capability) => ({
|
||||
id: capability.id,
|
||||
kind: capability.kind,
|
||||
name: capability.name,
|
||||
source: capability.source,
|
||||
path: capability.path
|
||||
}))
|
||||
};
|
||||
}
|
||||
|
||||
function parseHelpCommands(text) {
|
||||
const commands = [];
|
||||
let inCommands = false;
|
||||
for (const rawLine of String(text ?? "").split(/\r?\n/)) {
|
||||
const line = rawLine.replace(/\u001b\[[0-9;]*m/g, "");
|
||||
if (/^\s*(commands|available commands):\s*$/i.test(line)) {
|
||||
inCommands = true;
|
||||
continue;
|
||||
}
|
||||
if (inCommands && /^\S/.test(line) && !/^\s/.test(rawLine)) {
|
||||
inCommands = false;
|
||||
}
|
||||
if (!inCommands) {
|
||||
continue;
|
||||
}
|
||||
const match = line.match(/^\s{2,}([a-z][a-z0-9-]*)\s*(?:\*)?\s{2,}\S/);
|
||||
if (match) {
|
||||
commands.push(match[1]);
|
||||
}
|
||||
}
|
||||
return commands.filter((command) => !["help", "completion"].includes(command));
|
||||
}
|
||||
|
||||
function filterPackageScripts(scripts, scriptScope) {
|
||||
if (scriptScope === "none") {
|
||||
return [];
|
||||
}
|
||||
if (scriptScope === "all") {
|
||||
return scripts;
|
||||
}
|
||||
return scripts.filter(isProductScript);
|
||||
}
|
||||
|
||||
function isProductScript(script) {
|
||||
return productScriptNames.has(script) ||
|
||||
productScriptPrefixes.some((prefix) => script.startsWith(prefix));
|
||||
}
|
||||
|
||||
function matchPackageScriptSurfaceIds(capability) {
|
||||
if (capability.kind !== "package-script") {
|
||||
return [];
|
||||
}
|
||||
const name = String(capability.name ?? "");
|
||||
const normalizedName = normalizeToken(name);
|
||||
const searchableName = `${name} ${normalizedName}`;
|
||||
const surfaceIds = new Set();
|
||||
const add = (...ids) => {
|
||||
for (const id of ids) {
|
||||
surfaceIds.add(id);
|
||||
}
|
||||
};
|
||||
|
||||
if (/gateway|(^|[: -])start($|[: -])|openclaw|startup|stability/.test(searchableName)) {
|
||||
add("release-runtime-startup", "gateway-performance");
|
||||
}
|
||||
if (/tui|chat|terminal/.test(searchableName)) {
|
||||
add("tui", "tui-message-turn");
|
||||
}
|
||||
if (/chat|session|channel/.test(searchableName)) {
|
||||
add("gateway-session-send-turn");
|
||||
}
|
||||
if (/dashboard|(^|[: -])ui[: -]/.test(searchableName)) {
|
||||
add("dashboard", "gateway-session-send-turn");
|
||||
}
|
||||
if (/plugin-sdk|plugins?|plugin-update/.test(searchableName)) {
|
||||
add("plugin-lifecycle", "official-plugin-install");
|
||||
}
|
||||
if (/release|install|onboard|published-upgrade|update-migration|upgrade-survivor/.test(searchableName)) {
|
||||
add("fresh-install", "upgrade-existing-user");
|
||||
}
|
||||
if (/models?|provider|capability|infer|openai/.test(searchableName)) {
|
||||
add("provider-models", "openai-compatible-turn");
|
||||
}
|
||||
if (/agent|acp|cli-backend/.test(searchableName)) {
|
||||
add("agent-cli-local-turn", "agent-gateway-rpc-turn");
|
||||
}
|
||||
if (/mcp/.test(searchableName)) {
|
||||
add("mcp-runtime");
|
||||
}
|
||||
if (/browser/.test(searchableName)) {
|
||||
add("browser-automation");
|
||||
}
|
||||
if (/media/.test(searchableName)) {
|
||||
add("media-understanding");
|
||||
}
|
||||
if (/workspace/.test(searchableName)) {
|
||||
add("workspace-scan");
|
||||
}
|
||||
if (/perf|soak/.test(searchableName)) {
|
||||
add("gateway-performance", "soak");
|
||||
}
|
||||
|
||||
return [...surfaceIds];
|
||||
}
|
||||
|
||||
function firstUsefulHelpLine(text) {
|
||||
return String(text ?? "")
|
||||
.split(/\r?\n/)
|
||||
.map((line) => line.trim())
|
||||
.find((line) => line && !/^usage:/i.test(line) && !/^commands:/i.test(line)) ?? null;
|
||||
}
|
||||
|
||||
function parseList(raw) {
|
||||
if (!raw || raw === true) {
|
||||
return [];
|
||||
}
|
||||
const values = Array.isArray(raw) ? raw : String(raw).split(",");
|
||||
return values.map((value) => value.trim()).filter(Boolean);
|
||||
}
|
||||
|
||||
function normalizeOptionalPath(value) {
|
||||
if (!value || value === true) {
|
||||
return null;
|
||||
}
|
||||
return resolveFromCwd(String(value));
|
||||
}
|
||||
|
||||
function normalizeOptionalCommand(value) {
|
||||
if (!value || value === true) {
|
||||
return null;
|
||||
}
|
||||
const command = String(value);
|
||||
return command.includes("/") || command.startsWith(".") ? resolveFromCwd(command) : command;
|
||||
}
|
||||
|
||||
function normalizeScriptScope(value) {
|
||||
if (!value || value === true) {
|
||||
return "product";
|
||||
}
|
||||
const scope = String(value).trim().toLowerCase();
|
||||
if (!packageScriptScopes.has(scope)) {
|
||||
throw new Error(`invalid --script-scope ${scope}; expected product, all, or none`);
|
||||
}
|
||||
return scope;
|
||||
}
|
||||
|
||||
function normalizeToken(value) {
|
||||
return String(value ?? "")
|
||||
.toLowerCase()
|
||||
.replace(/openclaw/g, "")
|
||||
.replace(/[^a-z0-9]+/g, "-")
|
||||
.replace(/^-+|-+$/g, "");
|
||||
}
|
||||
1121
src/main.mjs
1121
src/main.mjs
File diff suppressed because it is too large
Load Diff
@ -1,38 +0,0 @@
|
||||
import { resolveBaselinePath } from "../performance/baselines.mjs";
|
||||
import { parseFilterList } from "./expand.mjs";
|
||||
|
||||
export function matrixControlSummary(flags, targetPlan) {
|
||||
const requestedParallel = positiveIntegerFlag(flags, "parallel", 1);
|
||||
const repeat = positiveIntegerFlag(flags, "repeat", 1);
|
||||
const failFast = flags.fail_fast === true;
|
||||
const parallel = failFast || targetPlan.kind === "local-build" ? 1 : requestedParallel;
|
||||
return {
|
||||
include: parseFilterList(flags.include),
|
||||
exclude: parseFilterList(flags.exclude),
|
||||
failFast,
|
||||
continueOnFailure: !failFast,
|
||||
requestedParallel,
|
||||
parallel,
|
||||
parallelAdjusted: parallel !== requestedParallel,
|
||||
repeat,
|
||||
baseline: flags.baseline ? resolveBaselinePath(flags.baseline) : null,
|
||||
saveBaseline: flags.save_baseline ? resolveBaselinePath(flags.save_baseline) : null,
|
||||
gate: flags.gate === true,
|
||||
reviewedGood: flags.reviewed_good === true,
|
||||
bundle: true
|
||||
};
|
||||
}
|
||||
|
||||
function positiveIntegerFlag(flags, key, defaultValue) {
|
||||
if (flags[key] === undefined) {
|
||||
return defaultValue;
|
||||
}
|
||||
if (flags[key] === true) {
|
||||
throw new Error(`--${key.replaceAll("_", "-")} requires a positive integer value`);
|
||||
}
|
||||
const value = Number(flags[key]);
|
||||
if (!Number.isInteger(value) || value < 1) {
|
||||
throw new Error(`--${key.replaceAll("_", "-")} must be a positive integer, got ${JSON.stringify(flags[key])}`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
@ -1,114 +0,0 @@
|
||||
export const coveragePolicyKeys = [
|
||||
"surfaces",
|
||||
"platforms",
|
||||
"states",
|
||||
"traits",
|
||||
"scenarios",
|
||||
"stateSurfaces",
|
||||
"requirements"
|
||||
];
|
||||
|
||||
export const derivedCoveragePolicyKeys = ["surfaces", "scenarios", "states", "traits", "stateSurfaces"];
|
||||
|
||||
export function normalizeCoveragePolicy(coverage) {
|
||||
const input = coverage && typeof coverage === "object" ? coverage : {};
|
||||
return Object.fromEntries(coveragePolicyKeys.map((key) => [key, normalizeCoverageSet(input[key])]));
|
||||
}
|
||||
|
||||
export function deriveCoveragePolicy(coverage, obligations = []) {
|
||||
const policy = normalizeCoveragePolicy(coverage);
|
||||
const requirementSeverity = requirementSeverityByKey(policy.requirements);
|
||||
|
||||
for (const obligation of obligations ?? []) {
|
||||
const key = requirementKey(obligation.surface, obligation.requirement);
|
||||
const severity = requirementSeverity.get(key);
|
||||
if (!severity) {
|
||||
continue;
|
||||
}
|
||||
add(policy.surfaces, severity, obligation.surface);
|
||||
add(policy.scenarios, severity, obligation.scenario);
|
||||
add(policy.states, severity, obligation.state);
|
||||
add(policy.stateSurfaces, severity, obligation.surface && obligation.state ? `${obligation.surface}:${obligation.state}` : null);
|
||||
for (const trait of obligation.stateTraits ?? []) {
|
||||
add(policy.traits, severity, trait);
|
||||
}
|
||||
}
|
||||
|
||||
return sortCoveragePolicy(policy);
|
||||
}
|
||||
|
||||
export function buildEntryCoverageObligations(profile, { scenarios, states }) {
|
||||
const scenarioById = new Map((scenarios ?? []).map((scenario) => [scenario.id, scenario]));
|
||||
const stateById = new Map((states ?? []).map((state) => [state.id, state]));
|
||||
const obligations = [];
|
||||
|
||||
for (const entry of profile?.entries ?? []) {
|
||||
const scenario = scenarioById.get(entry.scenario);
|
||||
const state = stateById.get(entry.state);
|
||||
if (!scenario) {
|
||||
continue;
|
||||
}
|
||||
for (const requirement of scenario.proves ?? []) {
|
||||
obligations.push({
|
||||
surface: scenario.surface,
|
||||
requirement,
|
||||
scenario: scenario.id,
|
||||
state: entry.state,
|
||||
stateTraits: state?.traits ?? [],
|
||||
status: "planned"
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return obligations;
|
||||
}
|
||||
|
||||
export function coverageIdsFromSet(set) {
|
||||
return [...new Set([...(set?.blocking ?? []), ...(set?.warning ?? [])])].sort();
|
||||
}
|
||||
|
||||
function normalizeCoverageSet(value) {
|
||||
const input = value && typeof value === "object" ? value : {};
|
||||
return {
|
||||
blocking: normalizeStringList(input.blocking),
|
||||
warning: normalizeStringList(input.warning)
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeStringList(value) {
|
||||
return Array.isArray(value) ? value.filter((item) => typeof item === "string" && item.length > 0) : [];
|
||||
}
|
||||
|
||||
function requirementSeverityByKey(requirements) {
|
||||
const severities = new Map();
|
||||
for (const value of requirements.warning) {
|
||||
severities.set(value, "warning");
|
||||
}
|
||||
for (const value of requirements.blocking) {
|
||||
severities.set(value, "blocking");
|
||||
}
|
||||
return severities;
|
||||
}
|
||||
|
||||
function requirementKey(surface, requirement) {
|
||||
return `${surface}:${requirement}`;
|
||||
}
|
||||
|
||||
function add(set, severity, value) {
|
||||
if (typeof value !== "string" || value.length === 0) {
|
||||
return;
|
||||
}
|
||||
if (!set[severity].includes(value)) {
|
||||
set[severity].push(value);
|
||||
}
|
||||
}
|
||||
|
||||
function sortCoveragePolicy(policy) {
|
||||
return Object.fromEntries(Object.entries(policy).map(([key, value]) => [
|
||||
key,
|
||||
{
|
||||
blocking: derivedCoveragePolicyKeys.includes(key) ? [...value.blocking].sort() : value.blocking,
|
||||
warning: derivedCoveragePolicyKeys.includes(key) ? [...value.warning].sort() : value.warning
|
||||
}
|
||||
]));
|
||||
}
|
||||
@ -1,9 +1,4 @@
|
||||
import { platformCoverageKeys } from "../platform.mjs";
|
||||
import {
|
||||
buildEntryCoverageObligations,
|
||||
coverageIdsFromSet,
|
||||
deriveCoveragePolicy
|
||||
} from "./coverage-policy.mjs";
|
||||
|
||||
export function buildCoverage({ surfaces, scenarios, states, profiles, platform }) {
|
||||
const scenarioSurfaceMap = scenarios
|
||||
@ -41,7 +36,6 @@ function profileCoverage(profile, { scenarios, states, platform }) {
|
||||
const entryScenarios = new Set();
|
||||
const entryStates = new Set();
|
||||
const entrySurfaces = new Set();
|
||||
const entryRequirements = new Set();
|
||||
const entryStateSurfaces = new Set();
|
||||
|
||||
for (const entry of profile.entries ?? []) {
|
||||
@ -51,23 +45,15 @@ function profileCoverage(profile, { scenarios, states, platform }) {
|
||||
if (scenario?.surface) {
|
||||
entrySurfaces.add(scenario.surface);
|
||||
entryStateSurfaces.add(`${scenario.surface}:${entry.state}`);
|
||||
for (const requirement of scenario.proves ?? []) {
|
||||
entryRequirements.add(`${scenario.surface}:${requirement}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const derivedPolicy = deriveCoveragePolicy(
|
||||
profile.gate?.coverage,
|
||||
buildEntryCoverageObligations(profile, { scenarios, states })
|
||||
);
|
||||
const requiredSurfaces = coverageIdsFromSet(derivedPolicy.surfaces);
|
||||
const requiredScenarios = coverageIdsFromSet(derivedPolicy.scenarios);
|
||||
const requiredStates = coverageIdsFromSet(derivedPolicy.states);
|
||||
const requiredTraits = coverageIdsFromSet(derivedPolicy.traits);
|
||||
const requiredStateSurfaces = coverageIdsFromSet(derivedPolicy.stateSurfaces);
|
||||
const requiredRequirements = coverageIdsFromSet(derivedPolicy.requirements);
|
||||
const requiredPlatforms = coverageIdsFromSet(derivedPolicy.platforms);
|
||||
const requiredSurfaces = coverageIds(profile, "surfaces");
|
||||
const requiredScenarios = coverageIds(profile, "scenarios");
|
||||
const requiredStates = coverageIds(profile, "states");
|
||||
const requiredTraits = coverageIds(profile, "traits");
|
||||
const requiredStateSurfaces = coverageIds(profile, "stateSurfaces");
|
||||
const requiredPlatforms = coverageIds(profile, "platforms");
|
||||
const coveredTraits = coveredStateTraits(profile, states);
|
||||
const currentPlatformKeys = platformCoverageKeys(platform);
|
||||
|
||||
@ -77,7 +63,6 @@ function profileCoverage(profile, { scenarios, states, platform }) {
|
||||
surfaces: [...entrySurfaces].sort(),
|
||||
states: [...entryStates].sort(),
|
||||
scenarios: [...entryScenarios].sort(),
|
||||
requirements: [...entryRequirements].sort(),
|
||||
stateSurfaces: [...entryStateSurfaces].sort(),
|
||||
required: {
|
||||
surfaces: requiredSurfaces,
|
||||
@ -85,7 +70,6 @@ function profileCoverage(profile, { scenarios, states, platform }) {
|
||||
states: requiredStates,
|
||||
traits: requiredTraits,
|
||||
platforms: requiredPlatforms,
|
||||
requirements: requiredRequirements,
|
||||
stateSurfaces: requiredStateSurfaces
|
||||
},
|
||||
gaps: {
|
||||
@ -94,7 +78,6 @@ function profileCoverage(profile, { scenarios, states, platform }) {
|
||||
states: requiredStates.filter((id) => !entryStates.has(id)),
|
||||
traits: requiredTraits.filter((id) => !coveredTraits.has(id)),
|
||||
platforms: requiredPlatforms.filter((id) => !currentPlatformKeys.has(id)),
|
||||
requirements: requiredRequirements.filter((id) => !entryRequirements.has(id)),
|
||||
stateSurfaces: requiredStateSurfaces.filter((id) => !entryStateSurfaces.has(id))
|
||||
},
|
||||
currentPlatformKeys: [...currentPlatformKeys].sort(),
|
||||
@ -170,6 +153,14 @@ function traitSurfaceCoverage(profile, { scenarios, states }) {
|
||||
.map(([trait, surfaces]) => [trait, [...surfaces].sort()]));
|
||||
}
|
||||
|
||||
function coverageIds(profile, key) {
|
||||
const coverage = profile.gate?.coverage?.[key];
|
||||
if (!coverage) {
|
||||
return [];
|
||||
}
|
||||
return [...new Set([...(coverage.blocking ?? []), ...(coverage.warning ?? [])])].sort();
|
||||
}
|
||||
|
||||
function byScenario(left, right) {
|
||||
return left.scenario.localeCompare(right.scenario);
|
||||
}
|
||||
|
||||
@ -1,123 +0,0 @@
|
||||
import { loadScenarios } from "../registries/scenarios.mjs";
|
||||
import { loadState } from "../registries/states.mjs";
|
||||
|
||||
export async function expandProfile(profile) {
|
||||
const entries = [];
|
||||
for (const entry of profile.entries) {
|
||||
const [scenario] = await loadScenarios(entry.scenario);
|
||||
const state = await loadState(entry.state);
|
||||
entries.push({
|
||||
scenario: {
|
||||
id: scenario.id,
|
||||
surface: scenario.surface,
|
||||
title: scenario.title,
|
||||
objective: scenario.objective,
|
||||
tags: scenario.tags
|
||||
},
|
||||
state: {
|
||||
id: state.id,
|
||||
title: state.title,
|
||||
objective: state.objective,
|
||||
tags: state.tags
|
||||
},
|
||||
entry: {
|
||||
timeoutMs: entry.timeoutMs ?? null,
|
||||
platforms: entry.platforms ?? null
|
||||
},
|
||||
fullScenario: scenario,
|
||||
fullState: state
|
||||
});
|
||||
}
|
||||
|
||||
return entries.map((entry) => ({
|
||||
scenario: entry.fullScenario,
|
||||
state: entry.fullState,
|
||||
timeoutMs: entry.entry.timeoutMs,
|
||||
platforms: entry.entry.platforms,
|
||||
plan: {
|
||||
scenario: entry.scenario,
|
||||
state: entry.state,
|
||||
surface: entry.fullScenario.surface,
|
||||
timeoutMs: entry.entry.timeoutMs ?? entry.fullScenario.timeoutMs ?? null,
|
||||
platforms: entry.entry.platforms ?? entry.fullScenario.platforms ?? null
|
||||
}
|
||||
}));
|
||||
}
|
||||
|
||||
export function applyMatrixControls(entries, flags, platform) {
|
||||
const included = parseFilterList(flags.include);
|
||||
const excluded = parseFilterList(flags.exclude);
|
||||
return entries
|
||||
.filter((entry) => included.length === 0 || included.some((filter) => entryMatchesFilter(entry, filter)))
|
||||
.filter((entry) => !excluded.some((filter) => entryMatchesFilter(entry, filter)))
|
||||
.map((entry) => {
|
||||
const skipReason = platformSkipReason(entry, platform);
|
||||
return {
|
||||
...entry,
|
||||
skipReason,
|
||||
plan: {
|
||||
...entry.plan,
|
||||
status: skipReason ? "SKIPPED" : "SELECTED",
|
||||
skipReason
|
||||
}
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
export function parseFilterList(value) {
|
||||
if (!value) {
|
||||
return [];
|
||||
}
|
||||
return String(value)
|
||||
.split(",")
|
||||
.map((item) => item.trim())
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
function entryMatchesFilter(entry, filter) {
|
||||
const [kind, value] = filter.includes(":") ? filter.split(":", 2) : ["any", filter];
|
||||
if (kind === "scenario") {
|
||||
return entry.scenario.id === value;
|
||||
}
|
||||
if (kind === "state") {
|
||||
return entry.state.id === value;
|
||||
}
|
||||
if (kind === "tag") {
|
||||
return [...(entry.scenario.tags ?? []), ...(entry.state.tags ?? [])].includes(value);
|
||||
}
|
||||
return entry.scenario.id === value || entry.state.id === value ||
|
||||
(entry.scenario.tags ?? []).includes(value) || (entry.state.tags ?? []).includes(value);
|
||||
}
|
||||
|
||||
function platformSkipReason(entry, platform) {
|
||||
for (const policy of [entry.scenario.platforms, entry.platforms]) {
|
||||
const reason = platformPolicySkipReason(policy, platform);
|
||||
if (reason) {
|
||||
return reason;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function platformPolicySkipReason(policy, platform) {
|
||||
if (!policy) {
|
||||
return null;
|
||||
}
|
||||
const keys = platformKeys(platform);
|
||||
if (Array.isArray(policy.include) && policy.include.length > 0 && !policy.include.some((item) => keys.includes(item))) {
|
||||
return `platform ${platform.os}/${platform.arch} not included`;
|
||||
}
|
||||
if (Array.isArray(policy.exclude) && policy.exclude.some((item) => keys.includes(item))) {
|
||||
return `platform ${platform.os}/${platform.arch} excluded`;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function platformKeys(platform) {
|
||||
return [
|
||||
platform.os,
|
||||
platform.arch,
|
||||
`${platform.os}-${platform.arch}`,
|
||||
platform.release
|
||||
].filter(Boolean);
|
||||
}
|
||||
@ -1,5 +1,4 @@
|
||||
import { platformCoverageKeys } from "../platform.mjs";
|
||||
import { deriveCoveragePolicy } from "./coverage-policy.mjs";
|
||||
|
||||
export function preflightGateRun({ entries, flags }) {
|
||||
if (flags?.gate !== true || flags?.execute !== true) {
|
||||
@ -22,9 +21,8 @@ function scenarioUsesSourceEnv(scenario) {
|
||||
);
|
||||
}
|
||||
|
||||
export function evaluateGate(report, profile, options = {}) {
|
||||
const policy = normalizeGatePolicy(profile, options);
|
||||
const purpose = profile?.purpose ?? "release";
|
||||
export function evaluateGate(report, profile) {
|
||||
const policy = normalizeGatePolicy(profile);
|
||||
const records = report.records ?? [];
|
||||
const cards = [];
|
||||
const partial = isPartialGate(report);
|
||||
@ -70,7 +68,7 @@ export function evaluateGate(report, profile, options = {}) {
|
||||
}
|
||||
}
|
||||
|
||||
for (const card of buildCoverageCards(report, policy, partial, options)) {
|
||||
for (const card of buildCoverageCards(report, policy, partial)) {
|
||||
cards.push(card);
|
||||
if (card.severity === "blocking" || card.severity === "info") {
|
||||
missingRequired.push({
|
||||
@ -116,11 +114,9 @@ export function evaluateGate(report, profile, options = {}) {
|
||||
return {
|
||||
schemaVersion: "kova.gate.v1",
|
||||
enabled: true,
|
||||
purpose,
|
||||
profileId: profile?.id ?? null,
|
||||
policyId: policy.id,
|
||||
verdict,
|
||||
outcome: outcomeForVerdict(verdict, purpose),
|
||||
ok: verdict === "SHIP",
|
||||
complete: !incomplete,
|
||||
partial,
|
||||
@ -138,25 +134,12 @@ export function evaluateGate(report, profile, options = {}) {
|
||||
};
|
||||
}
|
||||
|
||||
function outcomeForVerdict(verdict, purpose) {
|
||||
if (purpose === "release") {
|
||||
return verdict;
|
||||
}
|
||||
if (verdict === "SHIP") {
|
||||
return "PASS";
|
||||
}
|
||||
if (verdict === "DO_NOT_SHIP") {
|
||||
return "FAIL";
|
||||
}
|
||||
return verdict;
|
||||
}
|
||||
|
||||
function isPartialGate(report) {
|
||||
const controls = report.controls;
|
||||
return (controls?.include?.length ?? 0) > 0 || (controls?.exclude?.length ?? 0) > 0;
|
||||
}
|
||||
|
||||
function normalizeGatePolicy(profile, options = {}) {
|
||||
function normalizeGatePolicy(profile) {
|
||||
const gate = profile?.gate && typeof profile.gate === "object" ? profile.gate : {};
|
||||
const entries = Array.isArray(profile?.entries) ? profile.entries : [];
|
||||
const warning = normalizePolicyEntries(gate.warning ?? []);
|
||||
@ -167,19 +150,57 @@ function normalizeGatePolicy(profile, options = {}) {
|
||||
id: typeof gate.id === "string" && gate.id ? gate.id : `${profile?.id ?? "matrix"}-gate`,
|
||||
blocking,
|
||||
warning,
|
||||
coverage: deriveCoveragePolicy(gate.coverage, options.resolvedCoverage?.obligations ?? [])
|
||||
coverage: normalizeCoveragePolicy(gate.coverage)
|
||||
};
|
||||
}
|
||||
|
||||
function buildCoverageCards(report, policy, partial, options = {}) {
|
||||
const cards = [];
|
||||
const resolvedCoverage = options.resolvedCoverage ?? null;
|
||||
const platformKeys = platformCoverageKeys(report.platform);
|
||||
const requirementKeys = new Set((resolvedCoverage?.obligations ?? [])
|
||||
.filter((obligation) => obligation.status === "planned")
|
||||
.map((obligation) => `${obligation.surface}:${obligation.requirement}`)
|
||||
.filter((value) => !value.endsWith(":null")));
|
||||
function normalizeCoveragePolicy(coverage) {
|
||||
const input = coverage && typeof coverage === "object" ? coverage : {};
|
||||
return {
|
||||
surfaces: normalizeCoverageSet(input.surfaces),
|
||||
platforms: normalizeCoverageSet(input.platforms),
|
||||
states: normalizeCoverageSet(input.states),
|
||||
traits: normalizeCoverageSet(input.traits),
|
||||
scenarios: normalizeCoverageSet(input.scenarios),
|
||||
stateSurfaces: normalizeCoverageSet(input.stateSurfaces)
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeCoverageSet(value) {
|
||||
const input = value && typeof value === "object" ? value : {};
|
||||
return {
|
||||
blocking: normalizeStringList(input.blocking),
|
||||
warning: normalizeStringList(input.warning)
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeStringList(value) {
|
||||
return Array.isArray(value) ? value.filter((item) => typeof item === "string" && item.length > 0) : [];
|
||||
}
|
||||
|
||||
function buildCoverageCards(report, policy, partial) {
|
||||
const cards = [];
|
||||
const records = report.records ?? [];
|
||||
const platformKeys = platformCoverageKeys(report.platform);
|
||||
const scenarioKeys = new Set(records.map((record) => record.scenario).filter(Boolean));
|
||||
const stateKeys = new Set(records.map((record) => record.state?.id).filter(Boolean));
|
||||
const surfaceKeys = new Set(records.map((record) => record.surface ?? record.measurements?.surface).filter(Boolean));
|
||||
const traitKeys = new Set(records.flatMap((record) => record.state?.traits ?? []).filter(Boolean));
|
||||
const stateSurfaceKeys = new Set(records
|
||||
.map((record) => {
|
||||
const surface = record.surface ?? record.measurements?.surface;
|
||||
const state = record.state?.id;
|
||||
return surface && state ? `${surface}:${state}` : null;
|
||||
})
|
||||
.filter(Boolean));
|
||||
|
||||
addCoverageCards(cards, {
|
||||
kind: "surface",
|
||||
expected: policy.coverage.surfaces,
|
||||
observed: surfaceKeys,
|
||||
partial,
|
||||
statusText: `${surfaceKeys.size} surface(s) present`
|
||||
});
|
||||
addCoverageCards(cards, {
|
||||
kind: "platform",
|
||||
expected: policy.coverage.platforms,
|
||||
@ -188,11 +209,32 @@ function buildCoverageCards(report, policy, partial, options = {}) {
|
||||
statusText: report.platform ? `${report.platform.os}/${report.platform.arch}` : "unknown platform"
|
||||
});
|
||||
addCoverageCards(cards, {
|
||||
kind: "requirement",
|
||||
expected: policy.coverage.requirements,
|
||||
observed: requirementKeys,
|
||||
kind: "scenario",
|
||||
expected: policy.coverage.scenarios,
|
||||
observed: scenarioKeys,
|
||||
partial,
|
||||
statusText: `${requirementKeys.size} requirement obligation(s) present`
|
||||
statusText: `${scenarioKeys.size} scenario(s) present`
|
||||
});
|
||||
addCoverageCards(cards, {
|
||||
kind: "state",
|
||||
expected: policy.coverage.states,
|
||||
observed: stateKeys,
|
||||
partial,
|
||||
statusText: `${stateKeys.size} state(s) present`
|
||||
});
|
||||
addCoverageCards(cards, {
|
||||
kind: "trait",
|
||||
expected: policy.coverage.traits,
|
||||
observed: traitKeys,
|
||||
partial,
|
||||
statusText: `${traitKeys.size} state trait(s) present`
|
||||
});
|
||||
addCoverageCards(cards, {
|
||||
kind: "state-surface",
|
||||
expected: policy.coverage.stateSurfaces,
|
||||
observed: stateSurfaceKeys,
|
||||
partial,
|
||||
statusText: `${stateSurfaceKeys.size} state/surface pair(s) present`
|
||||
});
|
||||
|
||||
return cards;
|
||||
@ -221,7 +263,6 @@ function coverageCard({ severity, kind, value, partial, statusText }) {
|
||||
coverage: kind,
|
||||
scenario: kind === "scenario" ? value : null,
|
||||
state: kind === "state" ? value : stateFromCoverage(kind, value),
|
||||
requirement: kind === "requirement" ? value : null,
|
||||
status: "MISSING",
|
||||
title: `Required ${kind} Coverage Missing`,
|
||||
summary: filtered
|
||||
@ -363,12 +404,9 @@ function summarizeGateMeasurements(measurements) {
|
||||
if (!measurements) {
|
||||
return null;
|
||||
}
|
||||
const readiness = measurements.health?.readiness ?? null;
|
||||
return {
|
||||
readiness: readiness ? {
|
||||
classification: readiness.classification ?? null,
|
||||
healthReadyAtMs: readiness.healthReadyAtMs ?? null
|
||||
} : null,
|
||||
readinessClassification: measurements.readinessClassification ?? null,
|
||||
timeToHealthReadyMs: measurements.timeToHealthReadyMs ?? null,
|
||||
peakRssMb: measurements.peakRssMb ?? null,
|
||||
cpuPercentMax: measurements.cpuPercentMax ?? null,
|
||||
missingDependencyErrors: measurements.missingDependencyErrors ?? null,
|
||||
|
||||
@ -1,30 +0,0 @@
|
||||
export function profileSummary(profile) {
|
||||
return {
|
||||
id: profile.id,
|
||||
title: profile.title,
|
||||
objective: profile.objective,
|
||||
purpose: profile.purpose ?? null,
|
||||
entryCount: profile.entries.length,
|
||||
targetKinds: profile.targetKinds ?? null,
|
||||
diagnostics: profile.diagnostics ?? null,
|
||||
calibration: profile.calibration ? {
|
||||
surfaceCount: Object.keys(profile.calibration.surfaces ?? {}).length,
|
||||
roleCount: Object.keys(profile.calibration.roles ?? {}).length
|
||||
} : null,
|
||||
gate: profile.gate ? {
|
||||
id: profile.gate.id ?? `${profile.id}-gate`,
|
||||
blockingCount: Array.isArray(profile.gate.blocking) ? profile.gate.blocking.length : profile.entries.length,
|
||||
warningCount: Array.isArray(profile.gate.warning) ? profile.gate.warning.length : 0
|
||||
} : null
|
||||
};
|
||||
}
|
||||
|
||||
export function validateProfileTarget(profile, targetPlan) {
|
||||
const targetKinds = profile.targetKinds ?? [];
|
||||
if (targetKinds.length === 0) {
|
||||
return;
|
||||
}
|
||||
if (!targetKinds.includes(targetPlan.kind)) {
|
||||
throw new Error(`profile '${profile.id}' requires target kind ${targetKinds.join(", ")}, got ${targetPlan.kind}`);
|
||||
}
|
||||
}
|
||||
@ -1,178 +0,0 @@
|
||||
import { stateSatisfiesRequirement } from "../registries/surface-requirements.mjs";
|
||||
|
||||
export const RESOLVED_COVERAGE_SCHEMA = "kova.resolvedCoverage.v1";
|
||||
|
||||
export function resolveCoverageObligations({ profile, entries, surfaces, targetPlan }) {
|
||||
const surfaceById = new Map((surfaces ?? []).map((surface) => [surface.id, surface]));
|
||||
const obligations = [];
|
||||
const warnings = [];
|
||||
|
||||
for (const entry of entries ?? []) {
|
||||
const scenario = entry.scenario;
|
||||
const state = entry.state;
|
||||
const surface = surfaceById.get(scenario?.surface);
|
||||
if (!surface) {
|
||||
obligations.push(obligationFor(entry, {
|
||||
surface: scenario?.surface ?? null,
|
||||
requirement: null,
|
||||
targetPlan,
|
||||
status: "invalid",
|
||||
reason: `scenario '${scenario?.id ?? "unknown"}' references missing surface '${scenario?.surface ?? ""}'`
|
||||
}));
|
||||
continue;
|
||||
}
|
||||
|
||||
const requirements = new Map((surface.requirements ?? []).map((requirement) => [requirement.id, requirement]));
|
||||
const proves = scenario.proves ?? [];
|
||||
if (proves.length === 0) {
|
||||
obligations.push(obligationFor(entry, {
|
||||
surface: surface.id,
|
||||
requirement: null,
|
||||
targetPlan,
|
||||
status: "missing-proof",
|
||||
reason: `scenario '${scenario.id}' does not declare proved requirements for surface '${surface.id}'`
|
||||
}));
|
||||
continue;
|
||||
}
|
||||
|
||||
for (const requirementId of proves) {
|
||||
const requirement = requirements.get(requirementId);
|
||||
if (!requirement) {
|
||||
obligations.push(obligationFor(entry, {
|
||||
surface: surface.id,
|
||||
requirement: requirementId,
|
||||
targetPlan,
|
||||
status: "invalid",
|
||||
reason: `scenario '${scenario.id}' proves unknown requirement '${surface.id}:${requirementId}'`
|
||||
}));
|
||||
continue;
|
||||
}
|
||||
|
||||
const stateResult = stateSatisfiesRequirement(state, requirement);
|
||||
const targetResult = targetSatisfiesRequirement(targetPlan, requirement);
|
||||
const status = entry.skipReason
|
||||
? "skipped"
|
||||
: !stateResult.ok
|
||||
? "unsupported-state"
|
||||
: !targetResult.ok
|
||||
? "unsupported-target"
|
||||
: "planned";
|
||||
obligations.push(obligationFor(entry, {
|
||||
surface: surface.id,
|
||||
requirement: requirement.id,
|
||||
requirementContract: requirement,
|
||||
targetPlan,
|
||||
status,
|
||||
reason: entry.skipReason ?? stateResult.reason ?? targetResult.reason ?? null
|
||||
}));
|
||||
}
|
||||
}
|
||||
|
||||
const gaps = buildRequirementGaps(profile, obligations);
|
||||
|
||||
return {
|
||||
schemaVersion: RESOLVED_COVERAGE_SCHEMA,
|
||||
purpose: profile?.purpose ?? null,
|
||||
targetKind: targetPlan?.kind ?? null,
|
||||
total: obligations.length,
|
||||
statuses: countStatuses(obligations),
|
||||
obligations,
|
||||
gaps,
|
||||
warnings
|
||||
};
|
||||
}
|
||||
|
||||
export function assertResolvedCoverageIsRunnable(resolved) {
|
||||
const invalid = (resolved?.obligations ?? []).filter((obligation) =>
|
||||
["invalid", "missing-proof", "unsupported-state", "unsupported-target"].includes(obligation.status)
|
||||
);
|
||||
if (invalid.length === 0) {
|
||||
return;
|
||||
}
|
||||
const messages = invalid.slice(0, 5).map((obligation) =>
|
||||
`${obligation.scenario}/${obligation.state ?? "no-state"} -> ${obligation.surface}:${obligation.requirement ?? "none"} ${obligation.status}${obligation.reason ? ` (${obligation.reason})` : ""}`
|
||||
);
|
||||
throw new Error(`resolved coverage contains invalid obligation(s):\n- ${messages.join("\n- ")}`);
|
||||
}
|
||||
|
||||
function obligationFor(entry, options) {
|
||||
const requirement = options.requirementContract ?? {};
|
||||
return {
|
||||
surface: options.surface,
|
||||
requirement: options.requirement,
|
||||
scenario: entry.scenario?.id ?? null,
|
||||
state: entry.state?.id ?? null,
|
||||
stateTraits: entry.state?.traits ?? [],
|
||||
targetKind: options.targetPlan?.kind ?? null,
|
||||
status: options.status,
|
||||
reason: options.reason,
|
||||
requiredStates: requirement.states ?? [],
|
||||
requiredStateTraits: requirement.stateTraits ?? [],
|
||||
requiredTargetKinds: requirement.targetKinds ?? [],
|
||||
requiredMetrics: requirement.metrics ?? []
|
||||
};
|
||||
}
|
||||
|
||||
function targetSatisfiesRequirement(targetPlan, requirement) {
|
||||
const targetKinds = requirement.targetKinds ?? [];
|
||||
if (targetKinds.length === 0 || targetKinds.includes(targetPlan?.kind)) {
|
||||
return { ok: true, reason: null };
|
||||
}
|
||||
return {
|
||||
ok: false,
|
||||
reason: `target kind '${targetPlan?.kind ?? "unknown"}' is not supported by requirement`
|
||||
};
|
||||
}
|
||||
|
||||
function buildRequirementGaps(profile, obligations) {
|
||||
const required = profileRequirementCoverage(profile);
|
||||
if (required.length === 0) {
|
||||
return [];
|
||||
}
|
||||
const planned = new Set(obligations
|
||||
.filter((obligation) => obligation.status === "planned")
|
||||
.map((obligation) => requirementKey(obligation.surface, obligation.requirement)));
|
||||
return required
|
||||
.filter((item) => !planned.has(item.key))
|
||||
.map((item) => ({
|
||||
surface: item.surface,
|
||||
requirement: item.requirement,
|
||||
severity: item.severity,
|
||||
reason: "no selected runnable scenario/state/target obligation proves this requirement"
|
||||
}));
|
||||
}
|
||||
|
||||
function profileRequirementCoverage(profile) {
|
||||
const coverage = profile?.gate?.coverage?.requirements;
|
||||
if (!coverage) {
|
||||
return [];
|
||||
}
|
||||
return [
|
||||
...coverageEntries(coverage.blocking, "blocking"),
|
||||
...coverageEntries(coverage.warning, "warning")
|
||||
];
|
||||
}
|
||||
|
||||
function coverageEntries(values, severity) {
|
||||
return (values ?? []).map((value) => {
|
||||
const [surface, requirement] = String(value).split(":");
|
||||
return {
|
||||
surface,
|
||||
requirement,
|
||||
severity,
|
||||
key: requirementKey(surface, requirement)
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
function requirementKey(surface, requirement) {
|
||||
return `${surface}:${requirement}`;
|
||||
}
|
||||
|
||||
function countStatuses(obligations) {
|
||||
const counts = {};
|
||||
for (const obligation of obligations) {
|
||||
counts[obligation.status] = (counts[obligation.status] ?? 0) + 1;
|
||||
}
|
||||
return counts;
|
||||
}
|
||||
@ -1,68 +0,0 @@
|
||||
export const MEASUREMENT_SCOPES = new Set(["product", "harness", "cleanup"]);
|
||||
|
||||
export function normalizeMeasurementScope(value, phaseId = null) {
|
||||
if (MEASUREMENT_SCOPES.has(value)) {
|
||||
return value;
|
||||
}
|
||||
if (phaseId === "target-setup" || phaseId === "auth-prepare" || phaseId === "auth-setup" || phaseId === "prepare" || phaseId?.startsWith("state-")) {
|
||||
return "harness";
|
||||
}
|
||||
if (phaseId === "cleanup" || phaseId === "auth-cleanup" || phaseId === "env-cleanup") {
|
||||
return "cleanup";
|
||||
}
|
||||
return "product";
|
||||
}
|
||||
|
||||
export function measuredProductPhase(phase) {
|
||||
return measurementScopeForPhase(phase) === "product";
|
||||
}
|
||||
|
||||
export function measurementScopeForPhase(phase) {
|
||||
if (MEASUREMENT_SCOPES.has(phase?.measurementScope)) {
|
||||
return phase.measurementScope;
|
||||
}
|
||||
if (phase?.id === "provision" && (phase.commands ?? []).some((command) => /(?:^|\s)--no-service(?:\s|$)/.test(command))) {
|
||||
return "harness";
|
||||
}
|
||||
return normalizeMeasurementScope(phase?.measurementScope, phase?.id);
|
||||
}
|
||||
|
||||
export function driverKindForCommand(command) {
|
||||
const text = String(command ?? "");
|
||||
if (text.includes("run-gateway-session-send-turn.mjs")) {
|
||||
return "gateway-rpc";
|
||||
}
|
||||
if (text.includes("run-openai-compatible-turn.mjs")) {
|
||||
return "gateway-http";
|
||||
}
|
||||
if (text.includes("run-tui-message-turn.mjs")) {
|
||||
return "gateway-rpc";
|
||||
}
|
||||
if (/\bocm\s+@[^ ]+\s+--\s+agent\b/.test(text)) {
|
||||
return text.includes("--local") ? "openclaw-cli-local" : "openclaw-cli-gateway";
|
||||
}
|
||||
if (/\bocm\s+@[^ ]+\s+--\s+gateway\s+call\b/.test(text)) {
|
||||
return "gateway-rpc-via-cli";
|
||||
}
|
||||
if (/\bocm\b/.test(text)) {
|
||||
return "ocm";
|
||||
}
|
||||
if (/\bnode\b/.test(text)) {
|
||||
return "kova-helper";
|
||||
}
|
||||
return "unknown";
|
||||
}
|
||||
|
||||
export function phaseDriverKind(phase, commands = phase?.commands ?? []) {
|
||||
if (phase?.driverKind) {
|
||||
return phase.driverKind;
|
||||
}
|
||||
const kinds = new Set(commands.map(driverKindForCommand));
|
||||
if (kinds.size === 1) {
|
||||
return [...kinds][0];
|
||||
}
|
||||
if (kinds.size === 0) {
|
||||
return "none";
|
||||
}
|
||||
return "mixed";
|
||||
}
|
||||
@ -208,15 +208,6 @@ export function reviewBaselineUpdate(report, options = {}) {
|
||||
});
|
||||
}
|
||||
|
||||
const parallel = Number(report?.controls?.parallel ?? performance?.parallel ?? 1);
|
||||
if (Number.isFinite(parallel) && parallel > 1) {
|
||||
blockers.push({
|
||||
kind: "parallel-performance",
|
||||
parallel,
|
||||
message: `baseline updates require sequential runs; parallel=${parallel} can contaminate RSS/CPU/timing samples`
|
||||
});
|
||||
}
|
||||
|
||||
const comparison = report?.baseline?.comparison;
|
||||
if (comparison && comparison.ok !== true) {
|
||||
blockers.push({
|
||||
|
||||
@ -1,10 +1,8 @@
|
||||
import { measurementMetricValue } from "../health.mjs";
|
||||
|
||||
export const PERFORMANCE_SCHEMA = "kova.performance.v1";
|
||||
|
||||
export const PERFORMANCE_METRICS = [
|
||||
{ id: "readinessHealthReadyMs", title: "Health Ready", unit: "ms", regressionKey: "startupRegressionPercent" },
|
||||
{ id: "readinessListeningMs", title: "TCP Listening", unit: "ms", regressionKey: "startupRegressionPercent" },
|
||||
{ id: "timeToHealthReadyMs", title: "Health Ready", unit: "ms", regressionKey: "startupRegressionPercent" },
|
||||
{ id: "timeToListeningMs", title: "TCP Listening", unit: "ms", regressionKey: "startupRegressionPercent" },
|
||||
{ id: "peakRssMb", title: "Peak RSS", unit: "MB", regressionKey: "rssRegressionPercent" },
|
||||
{ id: "resourcePeakGatewayRssMb", title: "Gateway RSS", unit: "MB", regressionKey: "rssRegressionPercent" },
|
||||
{ id: "cpuPercentMax", title: "Max CPU", unit: "%", regressionKey: "cpuRegressionPercent" },
|
||||
@ -21,16 +19,9 @@ export const PERFORMANCE_METRICS = [
|
||||
{ id: "agentCleanupMaxMs", title: "Agent Cleanup Max", unit: "ms", regressionKey: "agentLatencyRegressionPercent" },
|
||||
{ id: "coldPreProviderMs", title: "Cold Pre-Provider", unit: "ms", regressionKey: "agentLatencyRegressionPercent" },
|
||||
{ id: "warmPreProviderMs", title: "Warm Pre-Provider", unit: "ms", regressionKey: "agentLatencyRegressionPercent" },
|
||||
{ id: "coldPreProviderAttributedMs", title: "Cold Pre-Provider Attributed", unit: "ms", regressionKey: "agentLatencyRegressionPercent" },
|
||||
{ id: "warmPreProviderAttributedMs", title: "Warm Pre-Provider Attributed", unit: "ms", regressionKey: "agentLatencyRegressionPercent" },
|
||||
{ id: "coldPreProviderUnattributedMs", title: "Cold Pre-Provider Unattributed", unit: "ms", regressionKey: "agentLatencyRegressionPercent" },
|
||||
{ id: "warmPreProviderUnattributedMs", title: "Warm Pre-Provider Unattributed", unit: "ms", regressionKey: "agentLatencyRegressionPercent" },
|
||||
{ id: "agentMetadataScanCount", title: "Agent Metadata Scans", unit: "count", regressionKey: "agentLatencyRegressionPercent" },
|
||||
{ id: "agentMetadataScanTotalMs", title: "Agent Metadata Scan Total", unit: "ms", regressionKey: "agentLatencyRegressionPercent" },
|
||||
{ id: "agentEventLoopMaxMs", title: "Agent Event Loop Max", unit: "ms", regressionKey: "eventLoopRegressionPercent" },
|
||||
{ id: "agentSessionPollCount", title: "Agent Session Polls", unit: "count", regressionKey: "agentLatencyRegressionPercent" },
|
||||
{ id: "healthP95Ms", title: "Phase Health p95", unit: "ms", regressionKey: "startupRegressionPercent" },
|
||||
{ id: "startupHealthP95Ms", title: "Startup Health p95", unit: "ms", regressionKey: "startupRegressionPercent" },
|
||||
{ id: "postReadyHealthP95Ms", title: "Post-Ready Health p95", unit: "ms", regressionKey: "startupRegressionPercent" },
|
||||
{ id: "postReadyHealthP95Ms", title: "Post-Ready Health p95", unit: "ms", regressionKey: "eventLoopRegressionPercent" },
|
||||
{ id: "runtimeDepsStagingMs", title: "Runtime Deps Staging", unit: "ms", regressionKey: "startupRegressionPercent" }
|
||||
];
|
||||
|
||||
@ -60,8 +51,6 @@ export function buildPerformanceSummary(records, options = {}) {
|
||||
schemaVersion: PERFORMANCE_SCHEMA,
|
||||
generatedAt: new Date().toISOString(),
|
||||
repeat: options.repeat ?? null,
|
||||
parallel: options.parallel ?? null,
|
||||
parallelContaminated: Number(options.parallel ?? 1) > 1,
|
||||
metricCatalog: PERFORMANCE_METRICS.map(({ id, title, unit }) => ({ id, title, unit })),
|
||||
groupCount: groups.length,
|
||||
unstableGroupCount: unstableGroups.length,
|
||||
@ -76,7 +65,6 @@ export function performanceRecordKey(record, platform, targetPlan) {
|
||||
key.platform.os,
|
||||
key.platform.arch,
|
||||
key.targetKind,
|
||||
key.targetValue,
|
||||
key.surface,
|
||||
key.state,
|
||||
key.scenario
|
||||
@ -89,7 +77,6 @@ export function performanceIdentity(record, platform, targetPlan) {
|
||||
surface: record.surface ?? null,
|
||||
state: record.state?.id ?? null,
|
||||
targetKind: targetPlan?.kind ?? targetKindFromSelector(record.target),
|
||||
targetValue: targetPlan?.value ?? targetValueFromSelector(record.target),
|
||||
platform: {
|
||||
os: platform?.os ?? null,
|
||||
arch: platform?.arch ?? null
|
||||
@ -154,7 +141,7 @@ function summarizeGroup(records, options) {
|
||||
const metrics = {};
|
||||
for (const metric of PERFORMANCE_METRICS) {
|
||||
const values = records
|
||||
.map((record) => measurementMetricValue(record.measurements, metric.id))
|
||||
.map((record) => record.measurements?.[metric.id])
|
||||
.filter(isFiniteNumber);
|
||||
const summary = summarizeMetricValues(values, options.regressionThresholds);
|
||||
if (summary) {
|
||||
@ -214,14 +201,6 @@ function targetKindFromSelector(selector) {
|
||||
return string.includes(":") ? string.split(":", 1)[0] : null;
|
||||
}
|
||||
|
||||
function targetValueFromSelector(selector) {
|
||||
const string = String(selector ?? "");
|
||||
if (!string.includes(":")) {
|
||||
return null;
|
||||
}
|
||||
return string.slice(string.indexOf(":") + 1) || null;
|
||||
}
|
||||
|
||||
function isFiniteNumber(value) {
|
||||
return typeof value === "number" && Number.isFinite(value);
|
||||
}
|
||||
|
||||
@ -1,20 +0,0 @@
|
||||
import { loadMetrics } from "./metrics.mjs";
|
||||
import { loadProcessRoles } from "./process-roles.mjs";
|
||||
import { loadProfiles } from "./profiles.mjs";
|
||||
import { loadScenarios } from "./scenarios.mjs";
|
||||
import { loadStates } from "./states.mjs";
|
||||
import { loadSurfaces } from "./surfaces.mjs";
|
||||
import { validateRegistryReferences } from "./validate.mjs";
|
||||
|
||||
export async function loadRegistryContext() {
|
||||
const [surfaces, processRoles, metrics, scenarios, states, profiles] = await Promise.all([
|
||||
loadSurfaces(),
|
||||
loadProcessRoles(),
|
||||
loadMetrics(),
|
||||
loadScenarios(),
|
||||
loadStates(),
|
||||
loadProfiles()
|
||||
]);
|
||||
validateRegistryReferences({ scenarios, states, profiles, surfaces, processRoles, metrics });
|
||||
return { surfaces, processRoles, metrics, scenarios, states, profiles };
|
||||
}
|
||||
@ -1,5 +1,4 @@
|
||||
import { profilesDir } from "../paths.mjs";
|
||||
import { validatePurpose } from "./purposes.mjs";
|
||||
import { assertNoShapeErrors, loadJsonRegistry, requireArray, requireKebabId, requireString } from "./validate.mjs";
|
||||
|
||||
export async function loadProfiles(selectedId) {
|
||||
@ -23,7 +22,6 @@ export function validateProfileShape(profile, sourceName = "profile") {
|
||||
requireString(profile, "title", errors);
|
||||
requireString(profile, "objective", errors);
|
||||
requireArray(profile, "entries", errors);
|
||||
validatePurpose(profile.purpose, "purpose", errors, { optional: true });
|
||||
validateStringArray(profile.targetKinds, "targetKinds", errors, { optional: true });
|
||||
validateDiagnostics(profile.diagnostics, "diagnostics", errors);
|
||||
validateCalibration(profile.calibration, "calibration", errors);
|
||||
@ -153,12 +151,7 @@ function validateCoverage(coverage, prefix, errors) {
|
||||
errors.push(`${prefix} must be an object`);
|
||||
return;
|
||||
}
|
||||
for (const key of ["surfaces", "scenarios", "states", "traits", "stateSurfaces"]) {
|
||||
if (coverage[key] !== undefined) {
|
||||
errors.push(`${prefix}.${key} is derived from entries and requirements; use ${prefix}.requirements instead`);
|
||||
}
|
||||
}
|
||||
for (const key of ["platforms", "requirements"]) {
|
||||
for (const key of ["surfaces", "scenarios", "states", "traits", "stateSurfaces", "platforms"]) {
|
||||
const value = coverage[key];
|
||||
if (value === undefined) {
|
||||
continue;
|
||||
|
||||
@ -1,42 +0,0 @@
|
||||
export const knownPurposes = [
|
||||
"diagnostic",
|
||||
"performance",
|
||||
"plugin",
|
||||
"provider",
|
||||
"regression",
|
||||
"release",
|
||||
"soak",
|
||||
"upgrade"
|
||||
];
|
||||
|
||||
export function validatePurposes(values, key, errors, options = {}) {
|
||||
if (values === undefined && options.optional) {
|
||||
return;
|
||||
}
|
||||
if (!Array.isArray(values)) {
|
||||
errors.push(`${key} must be an array`);
|
||||
return;
|
||||
}
|
||||
for (const [index, value] of values.entries()) {
|
||||
if (typeof value !== "string" || value.length === 0) {
|
||||
errors.push(`${key}[${index}] must be a non-empty string`);
|
||||
continue;
|
||||
}
|
||||
if (!knownPurposes.includes(value)) {
|
||||
errors.push(`${key}[${index}] references unknown purpose '${value}'`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function validatePurpose(value, key, errors, options = {}) {
|
||||
if (value === undefined && options.optional) {
|
||||
return;
|
||||
}
|
||||
if (typeof value !== "string" || value.length === 0) {
|
||||
errors.push(`${key} must be a non-empty string`);
|
||||
return;
|
||||
}
|
||||
if (!knownPurposes.includes(value)) {
|
||||
errors.push(`${key} references unknown purpose '${value}'`);
|
||||
}
|
||||
}
|
||||
@ -1,8 +1,6 @@
|
||||
import { scenariosDir } from "../paths.mjs";
|
||||
import { assertNoShapeErrors, loadJsonRegistry, requireArray, requireKebabId, requireObject, requireString } from "./validate.mjs";
|
||||
|
||||
export const HEALTH_SCOPES = ["readiness", "startup-sample", "post-ready", "final", "none"];
|
||||
|
||||
export async function loadScenarios(selectedId) {
|
||||
return loadJsonRegistry({
|
||||
dir: scenariosDir,
|
||||
@ -44,7 +42,6 @@ export function validateScenarioShape(scenario, sourceName = "scenario") {
|
||||
validateStringArray(scenario.targetValues, "targetValues", errors, { optional: true });
|
||||
validateStringArray(scenario.fromKinds, "fromKinds", errors, { optional: true });
|
||||
validateStringArray(scenario.fromValues, "fromValues", errors, { optional: true });
|
||||
validateStringArray(scenario.proves, "proves", errors);
|
||||
if (scenario.requiresFrom !== undefined && typeof scenario.requiresFrom !== "boolean") {
|
||||
errors.push("requiresFrom must be a boolean when set");
|
||||
}
|
||||
@ -110,7 +107,6 @@ function validatePhases(phases, errors) {
|
||||
requireKebabId(phase, "id", errors, prefix);
|
||||
requireString(phase, "title", errors, prefix);
|
||||
requireString(phase, "intent", errors, prefix);
|
||||
requireString(phase, "healthScope", errors, prefix);
|
||||
requireArray(phase, "commands", errors, prefix);
|
||||
requireArray(phase, "evidence", errors, prefix);
|
||||
|
||||
@ -123,9 +119,6 @@ function validatePhases(phases, errors) {
|
||||
|
||||
validateStringArray(phase.commands, `${prefix}.commands`, errors);
|
||||
validateStringArray(phase.evidence, `${prefix}.evidence`, errors);
|
||||
if (typeof phase.healthScope === "string" && !HEALTH_SCOPES.includes(phase.healthScope)) {
|
||||
errors.push(`${prefix}.healthScope must be one of ${HEALTH_SCOPES.join(", ")}`);
|
||||
}
|
||||
if (phase.expectedAgentFailure !== undefined && typeof phase.expectedAgentFailure !== "boolean") {
|
||||
errors.push(`${prefix}.expectedAgentFailure must be a boolean when set`);
|
||||
}
|
||||
|
||||
@ -18,7 +18,6 @@ export const knownStateTraits = [
|
||||
"mock-provider",
|
||||
"old-release",
|
||||
"onboarded-user",
|
||||
"official-plugin",
|
||||
"performance-pressure",
|
||||
"platform-specific",
|
||||
"plugin-pressure",
|
||||
@ -52,6 +51,8 @@ export function validateStateShape(state, sourceName = "state") {
|
||||
requireString(state, "objective", errors);
|
||||
requireArray(state, "tags", errors);
|
||||
requireArray(state, "traits", errors);
|
||||
requireArray(state, "compatibleSurfaces", errors);
|
||||
requireArray(state, "incompatibleSurfaces", errors);
|
||||
requireString(state, "riskArea", errors);
|
||||
requireString(state, "ownerArea", errors);
|
||||
requireArray(state, "setupEvidence", errors);
|
||||
@ -67,9 +68,7 @@ export function validateStateShape(state, sourceName = "state") {
|
||||
validateSteps(state.prepare, "prepare", errors, { phaseBinding: false });
|
||||
validateSteps(state.setup, "setup", errors, { phaseBinding: true });
|
||||
validateSteps(state.cleanup, "cleanup", errors, { phaseBinding: false });
|
||||
if (state.compatibleSurfaces !== undefined) {
|
||||
errors.push("compatibleSurfaces is not supported; surface requirements own positive state compatibility");
|
||||
}
|
||||
validateStringArray(state.compatibleSurfaces, "compatibleSurfaces", errors, { optional: true });
|
||||
validateStringArray(state.incompatibleSurfaces, "incompatibleSurfaces", errors, { optional: true });
|
||||
validateStringArray(state.traits, "traits", errors);
|
||||
validateStringArray(state.setupEvidence, "setupEvidence", errors, { nonEmpty: true });
|
||||
@ -81,49 +80,10 @@ export function validateStateShape(state, sourceName = "state") {
|
||||
if (state.auth !== undefined) {
|
||||
validateAuth(state.auth, errors);
|
||||
}
|
||||
if (state.officialPlugins !== undefined) {
|
||||
validateOfficialPlugins(state.officialPlugins, errors);
|
||||
}
|
||||
|
||||
assertNoShapeErrors(errors, sourceName);
|
||||
}
|
||||
|
||||
function validateOfficialPlugins(plugins, errors) {
|
||||
if (!Array.isArray(plugins)) {
|
||||
errors.push("officialPlugins must be an array");
|
||||
return;
|
||||
}
|
||||
if (plugins.length === 0) {
|
||||
errors.push("officialPlugins must not be empty");
|
||||
}
|
||||
const ids = new Set();
|
||||
for (const [index, plugin] of plugins.entries()) {
|
||||
const prefix = `officialPlugins[${index}]`;
|
||||
requireObject({ plugin }, "plugin", errors, prefix);
|
||||
if (!plugin || typeof plugin !== "object" || Array.isArray(plugin)) {
|
||||
continue;
|
||||
}
|
||||
requireKebabId(plugin, "id", errors, prefix);
|
||||
requireString(plugin, "package", errors, prefix);
|
||||
requireString(plugin, "title", errors, prefix);
|
||||
if (typeof plugin.id === "string") {
|
||||
if (ids.has(plugin.id)) {
|
||||
errors.push(`${prefix}.id duplicates official plugin id '${plugin.id}'`);
|
||||
}
|
||||
ids.add(plugin.id);
|
||||
}
|
||||
if (typeof plugin.package === "string" && !/^@openclaw\/[a-z0-9][a-z0-9-]*$/.test(plugin.package)) {
|
||||
errors.push(`${prefix}.package must be a scoped @openclaw/<name> package`);
|
||||
}
|
||||
if (plugin.required !== undefined && typeof plugin.required !== "boolean") {
|
||||
errors.push(`${prefix}.required must be a boolean when set`);
|
||||
}
|
||||
if (plugin.riskArea !== undefined && (typeof plugin.riskArea !== "string" || plugin.riskArea.length === 0)) {
|
||||
errors.push(`${prefix}.riskArea must be a non-empty string when set`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function validateAuth(auth, errors) {
|
||||
requireObject({ auth }, "auth", errors);
|
||||
if (!auth || typeof auth !== "object" || Array.isArray(auth)) {
|
||||
|
||||
@ -1,63 +0,0 @@
|
||||
export const knownTargetKinds = ["npm", "channel", "runtime", "local-build"];
|
||||
|
||||
export function requirementsForScenario(surface, scenario) {
|
||||
return requirementsForIds(surface, scenario?.proves ?? []);
|
||||
}
|
||||
|
||||
export function requirementsForIds(surface, ids) {
|
||||
const requirements = surface?.requirements ?? [];
|
||||
if (!Array.isArray(ids) || ids.length === 0) {
|
||||
return [];
|
||||
}
|
||||
const byId = new Map(requirements.map((requirement) => [requirement.id, requirement]));
|
||||
return ids.map((id) => byId.get(id)).filter(Boolean);
|
||||
}
|
||||
|
||||
export function targetKindsForRequirements(requirements) {
|
||||
return [...new Set((requirements ?? []).flatMap((requirement) => requirement.targetKinds ?? []))].sort();
|
||||
}
|
||||
|
||||
export function stateSatisfiesRequirement(state, requirement) {
|
||||
const states = requirement?.states ?? [];
|
||||
const traits = requirement?.stateTraits ?? [];
|
||||
if (states.length === 0 && traits.length === 0) {
|
||||
return { ok: true, reason: null };
|
||||
}
|
||||
if (state?.id && states.includes(state.id)) {
|
||||
return { ok: true, reason: null };
|
||||
}
|
||||
const stateTraits = new Set(state?.traits ?? []);
|
||||
if (traits.some((trait) => stateTraits.has(trait))) {
|
||||
return { ok: true, reason: null };
|
||||
}
|
||||
return {
|
||||
ok: false,
|
||||
reason: `state '${state?.id ?? "unknown"}' does not satisfy requirement state ids or traits`
|
||||
};
|
||||
}
|
||||
|
||||
export function scenarioSupportsState({ scenario, surface, state }) {
|
||||
if ((scenario?.states ?? []).length > 0) {
|
||||
return {
|
||||
ok: scenario.states.includes(state?.id),
|
||||
reason: scenario.states.includes(state?.id)
|
||||
? null
|
||||
: `scenario '${scenario.id}' supports only states: ${scenario.states.join(", ")}`
|
||||
};
|
||||
}
|
||||
|
||||
const requirements = requirementsForScenario(surface, scenario);
|
||||
if (requirements.length === 0) {
|
||||
return {
|
||||
ok: false,
|
||||
reason: `scenario '${scenario?.id ?? "unknown"}' has no known requirements for surface '${surface?.id ?? "unknown"}'`
|
||||
};
|
||||
}
|
||||
if (requirements.some((requirement) => stateSatisfiesRequirement(state, requirement).ok)) {
|
||||
return { ok: true, reason: null };
|
||||
}
|
||||
return {
|
||||
ok: false,
|
||||
reason: `state '${state?.id ?? "unknown"}' does not satisfy scenario '${scenario.id}' requirement state ids or traits`
|
||||
};
|
||||
}
|
||||
@ -1,7 +1,5 @@
|
||||
import { surfacesDir } from "../paths.mjs";
|
||||
import { validatePurposes } from "./purposes.mjs";
|
||||
import { assertNoShapeErrors, loadJsonRegistry, requireArray, requireKebabId, requireObject, requireString } from "./validate.mjs";
|
||||
import { knownTargetKinds } from "./surface-requirements.mjs";
|
||||
|
||||
export async function loadSurfaces(selectedId) {
|
||||
return loadJsonRegistry({
|
||||
@ -18,19 +16,12 @@ export function validateSurfaceShape(surface, sourceName = "surface") {
|
||||
requireString(surface, "title", errors);
|
||||
requireString(surface, "ownerArea", errors);
|
||||
requireString(surface, "description", errors);
|
||||
requireArray(surface, "requiredMetrics", errors);
|
||||
requireArray(surface, "processRoles", errors);
|
||||
requireObject(surface, "thresholds", errors);
|
||||
requireObject(surface, "diagnostics", errors);
|
||||
requireArray(surface, "requirements", errors);
|
||||
validatePurposes(surface.purposes, "purposes", errors, { optional: true });
|
||||
|
||||
for (const key of ["requiredStates", "targetKinds", "requiredMetrics"]) {
|
||||
if (surface[key] !== undefined) {
|
||||
errors.push(`${key} is not supported on surfaces; put requirement-specific contract data in requirements[]`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const key of ["processRoles"]) {
|
||||
for (const key of ["requiredMetrics", "processRoles", "requiredStates", "targetKinds"]) {
|
||||
if (surface[key] === undefined) {
|
||||
continue;
|
||||
}
|
||||
@ -55,52 +46,10 @@ export function validateSurfaceShape(surface, sourceName = "surface") {
|
||||
}
|
||||
}
|
||||
validateRoleThresholds(surface.roleThresholds, "roleThresholds", errors);
|
||||
validateRequirements(surface.requirements, errors);
|
||||
|
||||
assertNoShapeErrors(errors, sourceName);
|
||||
}
|
||||
|
||||
function validateRequirements(requirements, errors) {
|
||||
if (!Array.isArray(requirements)) {
|
||||
return;
|
||||
}
|
||||
if (requirements.length === 0) {
|
||||
errors.push("requirements must not be empty");
|
||||
}
|
||||
|
||||
const ids = new Set();
|
||||
for (const [index, requirement] of requirements.entries()) {
|
||||
const prefix = `requirements[${index}]`;
|
||||
requireKebabId(requirement, "id", errors, prefix);
|
||||
if (typeof requirement?.id === "string") {
|
||||
if (ids.has(requirement.id)) {
|
||||
errors.push(`requirements duplicate id '${requirement.id}'`);
|
||||
}
|
||||
ids.add(requirement.id);
|
||||
}
|
||||
validateStringArray(requirement?.states, `${prefix}.states`, errors, { optional: true });
|
||||
validateStringArray(requirement?.stateTraits, `${prefix}.stateTraits`, errors, { optional: true });
|
||||
if (!Array.isArray(requirement?.states) && !Array.isArray(requirement?.stateTraits)) {
|
||||
errors.push(`${prefix} must define states or stateTraits`);
|
||||
}
|
||||
validateStringArray(requirement?.targetKinds, `${prefix}.targetKinds`, errors);
|
||||
validateKnownTargetKinds(requirement?.targetKinds, `${prefix}.targetKinds`, errors);
|
||||
validateStringArray(requirement?.metrics, `${prefix}.metrics`, errors);
|
||||
validatePurposes(requirement?.purposes, `${prefix}.purposes`, errors, { optional: true });
|
||||
}
|
||||
}
|
||||
|
||||
function validateKnownTargetKinds(values, prefix, errors) {
|
||||
if (!Array.isArray(values)) {
|
||||
return;
|
||||
}
|
||||
for (const [index, value] of values.entries()) {
|
||||
if (typeof value === "string" && !knownTargetKinds.includes(value)) {
|
||||
errors.push(`${prefix}[${index}] references unknown target kind '${value}'`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function validateRoleThresholds(value, prefix, errors) {
|
||||
if (value === undefined) {
|
||||
return;
|
||||
@ -124,18 +73,3 @@ function validateRoleThresholds(value, prefix, errors) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function validateStringArray(values, key, errors, options = {}) {
|
||||
if (values === undefined && options.optional) {
|
||||
return;
|
||||
}
|
||||
if (!Array.isArray(values)) {
|
||||
errors.push(`${key} must be an array`);
|
||||
return;
|
||||
}
|
||||
for (const [index, value] of values.entries()) {
|
||||
if (typeof value !== "string" || value.length === 0) {
|
||||
errors.push(`${key}[${index}] must be a non-empty string`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,12 +1,6 @@
|
||||
import { readFile, readdir } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { isKnownPlatformCoverageKey } from "../platform.mjs";
|
||||
import {
|
||||
knownTargetKinds,
|
||||
requirementsForScenario,
|
||||
scenarioSupportsState,
|
||||
targetKindsForRequirements
|
||||
} from "./surface-requirements.mjs";
|
||||
|
||||
export async function loadJsonRegistry({ dir, kind, selectedId, validate }) {
|
||||
const names = await readdir(dir);
|
||||
@ -49,10 +43,15 @@ export function validateRegistryReferences({ scenarios, states, profiles, surfac
|
||||
errors.push(`scenario '${scenario.id}' references unknown surface '${scenario.surface}'`);
|
||||
continue;
|
||||
}
|
||||
validateScenarioContract(scenario, surfaceById.get(scenario.surface), { stateIds, processRoleIds, metricIds, traitIds }, errors);
|
||||
validateScenarioContract(scenario, surfaceById.get(scenario.surface), { stateIds, processRoleIds, metricIds }, errors);
|
||||
}
|
||||
|
||||
for (const state of states) {
|
||||
for (const surface of state.compatibleSurfaces ?? []) {
|
||||
if (!surfaceIds.has(surface)) {
|
||||
errors.push(`state '${state.id}' compatibleSurfaces references unknown surface '${surface}'`);
|
||||
}
|
||||
}
|
||||
for (const surface of state.incompatibleSurfaces ?? []) {
|
||||
if (!surfaceIds.has(surface)) {
|
||||
errors.push(`state '${state.id}' incompatibleSurfaces references unknown surface '${surface}'`);
|
||||
@ -71,7 +70,12 @@ export function validateRegistryReferences({ scenarios, states, profiles, surfac
|
||||
errors.push(`surface '${surface.id}' roleThresholds references unknown process role '${role}'`);
|
||||
}
|
||||
}
|
||||
validateSurfaceRequirements(surface, { stateIds, traitIds, metricIds }, errors);
|
||||
for (const state of surface.requiredStates ?? []) {
|
||||
if (!stateIds.has(state)) {
|
||||
errors.push(`surface '${surface.id}' references unknown required state '${state}'`);
|
||||
}
|
||||
}
|
||||
validateMetricList(surface.requiredMetrics ?? [], metricIds, errors, `surface '${surface.id}' requiredMetrics`);
|
||||
validateThresholdMetrics(surface.thresholds ?? {}, metricIds, errors, `surface '${surface.id}' thresholds`);
|
||||
for (const [role, thresholds] of Object.entries(surface.roleThresholds ?? {})) {
|
||||
validateThresholdMetrics(thresholds, metricIds, errors, `surface '${surface.id}' roleThresholds.${role}`);
|
||||
@ -98,47 +102,15 @@ function validateScenarioContract(scenario, surface, refs, errors) {
|
||||
errors.push(`scenario '${scenario.id}' processRoles references unknown process role '${role}'`);
|
||||
}
|
||||
}
|
||||
const scenarioRequirements = requirementsForScenario(surface, scenario);
|
||||
const surfaceTargetKinds = new Set(targetKindsForRequirements(scenarioRequirements));
|
||||
const surfaceTargetKinds = new Set(surface.targetKinds ?? []);
|
||||
for (const targetKind of scenario.targetKinds ?? []) {
|
||||
if (surfaceTargetKinds.size > 0 && !surfaceTargetKinds.has(targetKind)) {
|
||||
errors.push(`scenario '${scenario.id}' targetKinds references '${targetKind}' which is not supported by proved requirements on surface '${surface.id}'`);
|
||||
}
|
||||
}
|
||||
const requirementIds = new Set((surface.requirements ?? []).map((requirement) => requirement.id));
|
||||
if ((scenario.proves ?? []).length === 0) {
|
||||
errors.push(`scenario '${scenario.id}' must prove at least one requirement for surface '${surface.id}'`);
|
||||
}
|
||||
for (const requirement of scenario.proves ?? []) {
|
||||
if (requirementIds.size > 0 && !requirementIds.has(requirement)) {
|
||||
errors.push(`scenario '${scenario.id}' proves unknown surface requirement '${surface.id}.${requirement}'`);
|
||||
errors.push(`scenario '${scenario.id}' targetKinds references '${targetKind}' which is not supported by surface '${surface.id}'`);
|
||||
}
|
||||
}
|
||||
validateThresholdMetrics(scenario.thresholds ?? {}, refs.metricIds, errors, `scenario '${scenario.id}' thresholds`);
|
||||
}
|
||||
|
||||
function validateSurfaceRequirements(surface, refs, errors) {
|
||||
for (const requirement of surface.requirements ?? []) {
|
||||
const prefix = `surface '${surface.id}' requirement '${requirement.id}'`;
|
||||
for (const state of requirement.states ?? []) {
|
||||
if (!refs.stateIds.has(state)) {
|
||||
errors.push(`${prefix} references unknown state '${state}'`);
|
||||
}
|
||||
}
|
||||
for (const trait of requirement.stateTraits ?? []) {
|
||||
if (!refs.traitIds.has(trait)) {
|
||||
errors.push(`${prefix} references unknown state trait '${trait}'`);
|
||||
}
|
||||
}
|
||||
for (const targetKind of requirement.targetKinds ?? []) {
|
||||
if (!knownTargetKinds.includes(targetKind)) {
|
||||
errors.push(`${prefix} targetKinds references unknown target kind '${targetKind}'`);
|
||||
}
|
||||
}
|
||||
validateMetricList(requirement.metrics ?? [], refs.metricIds, errors, `${prefix} metrics`);
|
||||
}
|
||||
}
|
||||
|
||||
function validateMetricList(metrics, metricIds, errors, prefix) {
|
||||
for (const metric of metrics) {
|
||||
if (!metricIds.has(metric)) {
|
||||
@ -200,8 +172,12 @@ function validateProfileReferences(profile, refs, errors) {
|
||||
}
|
||||
}
|
||||
|
||||
validateCoverageRefs(profile, refs, errors, "surfaces", refs.surfaceIds);
|
||||
validateCoverageRefs(profile, refs, errors, "scenarios", refs.scenarioIds);
|
||||
validateCoverageRefs(profile, refs, errors, "states", refs.stateIds);
|
||||
validateCoverageRefs(profile, refs, errors, "traits", refs.traitIds);
|
||||
validatePlatformCoverageRefs(profile, errors);
|
||||
validateRequirementCoverageRefs(profile, refs, errors);
|
||||
validateStateSurfaceCoverageRefs(profile, refs, errors);
|
||||
validateCalibrationRefs(profile, refs, errors);
|
||||
}
|
||||
|
||||
@ -257,35 +233,74 @@ function validateScenarioStatePair({ profileId, location, scenarioId, stateId, r
|
||||
if (!surface) {
|
||||
return;
|
||||
}
|
||||
const stateResult = scenarioSupportsState({ scenario, surface, state });
|
||||
if (!stateResult.ok) {
|
||||
errors.push(`profile '${profileId}' ${location} pairs scenario '${scenario.id}' with state '${state.id}', but ${stateResult.reason}`);
|
||||
const allowedStates = scenario.states?.length > 0 ? scenario.states : surface.requiredStates ?? [];
|
||||
if (allowedStates.length > 0 && !allowedStates.includes(state.id)) {
|
||||
errors.push(`profile '${profileId}' ${location} pairs scenario '${scenario.id}' with state '${state.id}', but surface/scenario allows only: ${allowedStates.join(", ")}`);
|
||||
}
|
||||
if ((state.compatibleSurfaces ?? []).length > 0 && !state.compatibleSurfaces.includes(scenario.surface)) {
|
||||
errors.push(`profile '${profileId}' ${location} pairs state '${state.id}' with incompatible surface '${scenario.surface}'; compatible surfaces: ${state.compatibleSurfaces.join(", ")}`);
|
||||
}
|
||||
if ((state.incompatibleSurfaces ?? []).includes(scenario.surface)) {
|
||||
errors.push(`profile '${profileId}' ${location} pairs state '${state.id}' with explicitly incompatible surface '${scenario.surface}'`);
|
||||
}
|
||||
}
|
||||
|
||||
function validateRequirementCoverageRefs(profile, refs, errors) {
|
||||
const coverage = profile.gate?.coverage?.requirements;
|
||||
function validateStateSurfaceCoverageRefs(profile, refs, errors) {
|
||||
const coverage = profile.gate?.coverage?.stateSurfaces;
|
||||
if (!coverage) {
|
||||
return;
|
||||
}
|
||||
for (const level of ["blocking", "warning"]) {
|
||||
for (const value of coverage[level] ?? []) {
|
||||
const [surface, requirement, extra] = String(value).split(":");
|
||||
if (!surface || !requirement || extra !== undefined) {
|
||||
errors.push(`profile '${profile.id}' gate.coverage.requirements.${level} must use surface:requirement, got '${value}'`);
|
||||
const [surface, state, extra] = String(value).split(":");
|
||||
if (!surface || !state || extra !== undefined) {
|
||||
errors.push(`profile '${profile.id}' gate.coverage.stateSurfaces.${level} must use surface:state, got '${value}'`);
|
||||
continue;
|
||||
}
|
||||
const surfaceContract = refs.surfaceById.get(surface);
|
||||
if (!surfaceContract) {
|
||||
errors.push(`profile '${profile.id}' gate.coverage.requirements.${level} references unknown surface '${surface}'`);
|
||||
continue;
|
||||
if (!refs.surfaceIds.has(surface)) {
|
||||
errors.push(`profile '${profile.id}' gate.coverage.stateSurfaces.${level} references unknown surface '${surface}'`);
|
||||
}
|
||||
const requirementIds = new Set((surfaceContract.requirements ?? []).map((item) => item.id));
|
||||
if (!requirementIds.has(requirement)) {
|
||||
errors.push(`profile '${profile.id}' gate.coverage.requirements.${level} references unknown requirement '${surface}:${requirement}'`);
|
||||
if (!refs.stateIds.has(state)) {
|
||||
errors.push(`profile '${profile.id}' gate.coverage.stateSurfaces.${level} references unknown state '${state}'`);
|
||||
}
|
||||
validateStateSurfacePair({
|
||||
profileId: profile.id,
|
||||
location: `gate.coverage.stateSurfaces.${level}`,
|
||||
surfaceId: surface,
|
||||
stateId: state,
|
||||
refs,
|
||||
errors
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function validateStateSurfacePair({ profileId, location, surfaceId, stateId, refs, errors }) {
|
||||
const surface = refs.surfaceById.get(surfaceId);
|
||||
const state = refs.stateById.get(stateId);
|
||||
if (!surface || !state) {
|
||||
return;
|
||||
}
|
||||
if ((surface.requiredStates ?? []).length > 0 && !surface.requiredStates.includes(state.id)) {
|
||||
errors.push(`profile '${profileId}' ${location} requires '${surface.id}:${state.id}', but surface allows only: ${surface.requiredStates.join(", ")}`);
|
||||
}
|
||||
if ((state.compatibleSurfaces ?? []).length > 0 && !state.compatibleSurfaces.includes(surface.id)) {
|
||||
errors.push(`profile '${profileId}' ${location} requires '${surface.id}:${state.id}', but state compatible surfaces are: ${state.compatibleSurfaces.join(", ")}`);
|
||||
}
|
||||
if ((state.incompatibleSurfaces ?? []).includes(surface.id)) {
|
||||
errors.push(`profile '${profileId}' ${location} requires explicitly incompatible state/surface pair '${surface.id}:${state.id}'`);
|
||||
}
|
||||
}
|
||||
|
||||
function validateCoverageRefs(profile, _refs, errors, key, allowedIds) {
|
||||
const coverage = profile.gate?.coverage?.[key];
|
||||
if (!coverage) {
|
||||
return;
|
||||
}
|
||||
for (const level of ["blocking", "warning"]) {
|
||||
for (const id of coverage[level] ?? []) {
|
||||
if (!allowedIds.has(id)) {
|
||||
errors.push(`profile '${profile.id}' gate.coverage.${key}.${level} references unknown ${key.slice(0, -1)} '${id}'`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,9 +1,7 @@
|
||||
import { measurementMetricValue } from "../health.mjs";
|
||||
import { buildReportSummary } from "./report.mjs";
|
||||
|
||||
const defaultThresholds = {
|
||||
missingDependencyErrors: 0,
|
||||
pluginLoadFailures: 0,
|
||||
healthFailures: 0,
|
||||
peakRssMb: 100,
|
||||
cpuPercentMax: 25,
|
||||
coldReadyMs: 5000,
|
||||
@ -18,19 +16,11 @@ const defaultThresholds = {
|
||||
agentColdWarmDeltaMs: 10000,
|
||||
coldPreProviderMs: 5000,
|
||||
warmPreProviderMs: 2500,
|
||||
agentMetadataScanCount: 5,
|
||||
agentMetadataScanTotalMs: 1000,
|
||||
agentEventLoopMaxMs: 250,
|
||||
agentSessionPollCount: 10,
|
||||
tcpConnectMaxMs: 250,
|
||||
readinessListeningMs: 3000,
|
||||
readinessHealthReadyMs: 5000,
|
||||
timeToListeningMs: 3000,
|
||||
timeToHealthReadyMs: 5000,
|
||||
readinessFailures: 0,
|
||||
startupHealthFailures: 0,
|
||||
postReadyHealthFailures: 0,
|
||||
finalHealthFailures: 0,
|
||||
startupHealthP95Ms: 1000,
|
||||
postReadyHealthP95Ms: 1000,
|
||||
healthP95Ms: 1000,
|
||||
gatewayRestartCount: 0,
|
||||
providerTimeoutMentions: 0,
|
||||
eventLoopDelayMentions: 0,
|
||||
@ -45,8 +35,6 @@ const defaultThresholds = {
|
||||
heapSnapshotBytes: 50 * 1024 * 1024,
|
||||
resourcePeakCommandTreeRssMb: 100,
|
||||
resourcePeakGatewayRssMb: 100,
|
||||
resourcePeakTrackedRssMb: 100,
|
||||
resourceCpuPercentMaxTracked: 25,
|
||||
openclawTimelineParseErrors: 0,
|
||||
openclawSlowestSpanMs: 5000,
|
||||
openclawEventLoopMaxMs: 250,
|
||||
@ -58,8 +46,6 @@ const defaultThresholds = {
|
||||
|
||||
export function compareReports(baseline, current, options = {}) {
|
||||
const thresholds = resolveThresholds(options.thresholds);
|
||||
const baselineSummary = buildReportSummary(baseline);
|
||||
const currentSummary = buildReportSummary(current);
|
||||
const baselineRecords = indexRecords(baseline.records ?? []);
|
||||
const currentRecords = current.records ?? [];
|
||||
const scenarios = [];
|
||||
@ -128,26 +114,18 @@ export function compareReports(baseline, current, options = {}) {
|
||||
});
|
||||
}
|
||||
|
||||
const scenarioRegressionCount = scenarios.reduce((count, scenario) => count + scenario.regressions.length, 0);
|
||||
const statusChanges = compareGroupStatuses(baselineSummary.groups, currentSummary.groups);
|
||||
const findingChanges = compareFindings(baselineSummary.findings, currentSummary.findings);
|
||||
const newBlockingFindingCount = findingChanges.new.filter(isBlockingFinding).length;
|
||||
const regressionCount = scenarioRegressionCount + statusChanges.regressions.length + newBlockingFindingCount;
|
||||
const regressionCount = scenarios.reduce((count, scenario) => count + scenario.regressions.length, 0);
|
||||
const sourceRelease = compareSourceReleaseDiagnostics(baseline, current);
|
||||
const sourceReleaseBlockingCount = sourceRelease?.blockingCount ?? 0;
|
||||
return {
|
||||
schemaVersion: "kova.compare.v1",
|
||||
generatedAt: new Date().toISOString(),
|
||||
baseline: reportSummary(baseline, baselineSummary),
|
||||
current: reportSummary(current, currentSummary),
|
||||
baseline: reportSummary(baseline),
|
||||
current: reportSummary(current),
|
||||
thresholds,
|
||||
sourceRelease,
|
||||
ok: regressionCount === 0 && sourceReleaseBlockingCount === 0,
|
||||
regressionCount,
|
||||
scenarioRegressionCount,
|
||||
statusChanges,
|
||||
findingChanges,
|
||||
improvementCount: statusChanges.improvements.length + findingChanges.resolved.length,
|
||||
scenarios
|
||||
};
|
||||
}
|
||||
@ -175,22 +153,6 @@ export function renderCompareFixerSummary(comparison) {
|
||||
lines.push("");
|
||||
}
|
||||
|
||||
if (comparison.statusChanges?.regressions?.length > 0) {
|
||||
lines.push("Status regressions:");
|
||||
for (const change of comparison.statusChanges.regressions) {
|
||||
lines.push(`- ${change.key}: ${change.baselineLabel} -> ${change.currentLabel}`);
|
||||
}
|
||||
lines.push("");
|
||||
}
|
||||
|
||||
if (comparison.findingChanges?.new?.some(isBlockingFinding)) {
|
||||
lines.push("New findings:");
|
||||
for (const finding of comparison.findingChanges.new.filter(isBlockingFinding).slice(0, 8)) {
|
||||
lines.push(`- ${finding.scenario ?? "run"}${finding.state ? `/${finding.state}` : ""}: ${finding.summary}`);
|
||||
}
|
||||
lines.push("");
|
||||
}
|
||||
|
||||
for (const scenario of comparison.scenarios.filter((item) => item.regressions.length > 0)) {
|
||||
lines.push(`Scenario: ${scenario.key}`);
|
||||
lines.push(`Status: ${scenario.baselineStatus ?? "missing"} -> ${scenario.currentStatus ?? "missing"}`);
|
||||
@ -210,39 +172,11 @@ export function renderCompareSummary(comparison) {
|
||||
`Current: ${comparison.current.runId ?? "unknown"} (${comparison.current.target ?? "unknown"})`,
|
||||
`Result: ${comparison.ok ? "OK" : "REGRESSED"}`,
|
||||
`Regressions: ${comparison.regressionCount}`,
|
||||
`Improvements: ${comparison.improvementCount ?? 0}`,
|
||||
"",
|
||||
"Status changes:"
|
||||
"Scenarios:"
|
||||
];
|
||||
|
||||
for (const change of comparison.statusChanges?.changes ?? []) {
|
||||
lines.push(`- ${change.direction.toUpperCase()} ${change.key}: ${change.baselineLabel} -> ${change.currentLabel}`);
|
||||
}
|
||||
if ((comparison.statusChanges?.changes ?? []).length === 0) {
|
||||
lines.push("- none");
|
||||
}
|
||||
|
||||
if (comparison.findingChanges) {
|
||||
lines.push("");
|
||||
lines.push("Findings:");
|
||||
if (comparison.findingChanges.new.length === 0 && comparison.findingChanges.resolved.length === 0) {
|
||||
lines.push("- no finding changes");
|
||||
}
|
||||
for (const finding of comparison.findingChanges.new.slice(0, 8)) {
|
||||
lines.push(`- NEW ${finding.severity.toUpperCase()} ${finding.scenario ?? "run"}${finding.state ? `/${finding.state}` : ""}: ${finding.summary}`);
|
||||
}
|
||||
for (const finding of comparison.findingChanges.resolved.slice(0, 8)) {
|
||||
lines.push(`- RESOLVED ${finding.severity.toUpperCase()} ${finding.scenario ?? "run"}${finding.state ? `/${finding.state}` : ""}: ${finding.summary}`);
|
||||
}
|
||||
}
|
||||
|
||||
lines.push("");
|
||||
lines.push("Metric regressions:");
|
||||
const regressedScenarios = comparison.scenarios.filter((item) => item.regressions.length > 0);
|
||||
if (regressedScenarios.length === 0) {
|
||||
lines.push("- none");
|
||||
}
|
||||
for (const scenario of regressedScenarios) {
|
||||
for (const scenario of comparison.scenarios) {
|
||||
lines.push(`- ${scenario.status} ${scenario.key}`);
|
||||
for (const regression of scenario.regressions) {
|
||||
lines.push(` ${regression.message}`);
|
||||
@ -275,7 +209,7 @@ function recordKey(record) {
|
||||
return `${record.scenario}:${record.state?.id ?? "none"}`;
|
||||
}
|
||||
|
||||
function reportSummary(report, summary) {
|
||||
function reportSummary(report) {
|
||||
return {
|
||||
runId: report.runId ?? null,
|
||||
mode: report.mode ?? null,
|
||||
@ -283,98 +217,10 @@ function reportSummary(report, summary) {
|
||||
target: report.target ?? null,
|
||||
targetKind: targetKind(report.target),
|
||||
generatedAt: report.generatedAt ?? null,
|
||||
statuses: report.summary?.statuses ?? {},
|
||||
decision: summary.decision,
|
||||
findingCount: summary.findings.length,
|
||||
groupCount: summary.groups.length,
|
||||
sampleCount: summary.samples.length
|
||||
statuses: report.summary?.statuses ?? {}
|
||||
};
|
||||
}
|
||||
|
||||
function compareGroupStatuses(baselineGroups = [], currentGroups = []) {
|
||||
const baselineByKey = new Map(baselineGroups.map((group) => [group.key, group]));
|
||||
const currentByKey = new Map(currentGroups.map((group) => [group.key, group]));
|
||||
const changes = [];
|
||||
for (const [key, currentGroup] of currentByKey.entries()) {
|
||||
const baselineGroup = baselineByKey.get(key);
|
||||
if (!baselineGroup) {
|
||||
continue;
|
||||
}
|
||||
const baselineWorst = worstGroupStatus(baselineGroup.statuses);
|
||||
const currentWorst = worstGroupStatus(currentGroup.statuses);
|
||||
if (baselineWorst.rank === currentWorst.rank && statusCountsText(baselineGroup.statuses) === statusCountsText(currentGroup.statuses)) {
|
||||
continue;
|
||||
}
|
||||
const direction = currentWorst.rank > baselineWorst.rank
|
||||
? "regressed"
|
||||
: currentWorst.rank < baselineWorst.rank
|
||||
? "improved"
|
||||
: "changed";
|
||||
changes.push({
|
||||
key,
|
||||
scenario: currentGroup.scenario ?? baselineGroup.scenario ?? null,
|
||||
state: currentGroup.state ?? baselineGroup.state ?? null,
|
||||
direction,
|
||||
baseline: baselineGroup.statuses ?? {},
|
||||
current: currentGroup.statuses ?? {},
|
||||
baselineLabel: statusCountsText(baselineGroup.statuses),
|
||||
currentLabel: statusCountsText(currentGroup.statuses)
|
||||
});
|
||||
}
|
||||
return {
|
||||
changes,
|
||||
improvements: changes.filter((change) => change.direction === "improved"),
|
||||
regressions: changes.filter((change) => change.direction === "regressed")
|
||||
};
|
||||
}
|
||||
|
||||
function compareFindings(baselineFindings = [], currentFindings = []) {
|
||||
const baselineByKey = new Map(baselineFindings.map((finding) => [findingKey(finding), finding]));
|
||||
const currentByKey = new Map(currentFindings.map((finding) => [findingKey(finding), finding]));
|
||||
return {
|
||||
new: [...currentByKey.entries()]
|
||||
.filter(([key]) => !baselineByKey.has(key))
|
||||
.map(([, finding]) => finding),
|
||||
resolved: [...baselineByKey.entries()]
|
||||
.filter(([key]) => !currentByKey.has(key))
|
||||
.map(([, finding]) => finding),
|
||||
unchangedCount: [...currentByKey.keys()].filter((key) => baselineByKey.has(key)).length
|
||||
};
|
||||
}
|
||||
|
||||
function worstGroupStatus(statuses = {}) {
|
||||
let worst = { status: "PASS", rank: 0 };
|
||||
for (const [status, count] of Object.entries(statuses)) {
|
||||
if (!count) {
|
||||
continue;
|
||||
}
|
||||
const rank = statusRank(status);
|
||||
if (rank > worst.rank) {
|
||||
worst = { status, rank };
|
||||
}
|
||||
}
|
||||
return worst;
|
||||
}
|
||||
|
||||
function statusCountsText(statuses = {}) {
|
||||
return Object.entries(statuses).map(([status, count]) => `${status}:${count}`).join(", ") || "none";
|
||||
}
|
||||
|
||||
function findingKey(finding) {
|
||||
return [
|
||||
finding.severity ?? "unknown",
|
||||
finding.kind ?? "finding",
|
||||
finding.scenario ?? "run",
|
||||
finding.state ?? "none",
|
||||
finding.metric ?? "none",
|
||||
finding.summary ?? ""
|
||||
].join("|");
|
||||
}
|
||||
|
||||
function isBlockingFinding(finding) {
|
||||
return ["blocking", "blocked", "fail"].includes(finding?.severity);
|
||||
}
|
||||
|
||||
function compareSourceReleaseDiagnostics(leftReport, rightReport) {
|
||||
const leftLane = targetLane(leftReport.target);
|
||||
const rightLane = targetLane(rightReport.target);
|
||||
@ -469,14 +315,8 @@ function diagnosticRecordSummary(record) {
|
||||
agentTurnMs: measurements.agentTurnMs ?? measurements.coldAgentTurnMs ?? null,
|
||||
agentPreProviderMs: measurements.agentPreProviderMs ?? measurements.coldPreProviderMs ?? null,
|
||||
providerFinalMs: measurements.agentProviderFinalMs ?? measurements.coldProviderFinalMs ?? null,
|
||||
agentMetadataScanCount: measurements.agentMetadataScanCount ?? null,
|
||||
agentMetadataScanTotalMs: measurements.agentMetadataScanTotalMs ?? null,
|
||||
agentEventLoopMaxMs: measurements.agentEventLoopMaxMs ?? null,
|
||||
agentSessionPollCount: measurements.agentSessionPollCount ?? null,
|
||||
runtimeDepsStagingMs: measurements.runtimeDepsStagingMs ?? null,
|
||||
readinessHealthReadyMs: measurementMetricValue(measurements, "readinessHealthReadyMs"),
|
||||
startupHealthP95Ms: measurementMetricValue(measurements, "startupHealthP95Ms"),
|
||||
postReadyHealthP95Ms: measurementMetricValue(measurements, "postReadyHealthP95Ms"),
|
||||
timeToHealthReadyMs: measurements.timeToHealthReadyMs ?? null,
|
||||
peakRssMb: measurements.peakRssMb ?? null
|
||||
};
|
||||
}
|
||||
@ -519,8 +359,8 @@ function metricRegressions(baseline, current, thresholds) {
|
||||
}
|
||||
|
||||
function addIncreaseRegression(regressions, baseline, current, metric, tolerance) {
|
||||
const baselineValue = measurementMetricValue(baseline, metric);
|
||||
const currentValue = measurementMetricValue(current, metric);
|
||||
const baselineValue = baseline[metric];
|
||||
const currentValue = current[metric];
|
||||
if (typeof baselineValue !== "number" || typeof currentValue !== "number") {
|
||||
return;
|
||||
}
|
||||
@ -559,29 +399,15 @@ function metricDeltas(baseline, current) {
|
||||
"coldPreProviderMs",
|
||||
"warmPreProviderMs",
|
||||
"agentColdWarmPreProviderDeltaMs",
|
||||
"coldPreProviderAttributedMs",
|
||||
"warmPreProviderAttributedMs",
|
||||
"coldPreProviderUnattributedMs",
|
||||
"warmPreProviderUnattributedMs",
|
||||
"coldPreProviderAttributionCoverage",
|
||||
"warmPreProviderAttributionCoverage",
|
||||
"coldProviderFinalMs",
|
||||
"warmProviderFinalMs",
|
||||
"agentMetadataScanCount",
|
||||
"agentMetadataScanTotalMs",
|
||||
"agentMetadataScanMaxMs",
|
||||
"agentEventLoopMaxMs",
|
||||
"agentEventLoopSampleCount",
|
||||
"agentSessionPollCount",
|
||||
"agentSessionPollErrorCount",
|
||||
"tcpConnectMaxMs",
|
||||
"readinessListeningMs",
|
||||
"readinessHealthReadyMs",
|
||||
"timeToListeningMs",
|
||||
"timeToHealthReadyMs",
|
||||
"healthP95Ms",
|
||||
"startupHealthP95Ms",
|
||||
"postReadyHealthP95Ms",
|
||||
"startupHealthFailures",
|
||||
"postReadyHealthFailures",
|
||||
"finalHealthFailures",
|
||||
"healthFailures",
|
||||
"readinessFailures",
|
||||
"missingDependencyErrors",
|
||||
"pluginLoadFailures",
|
||||
@ -602,8 +428,6 @@ function metricDeltas(baseline, current) {
|
||||
"nodeProfileTopFunctionMs",
|
||||
"heapSnapshotBytes",
|
||||
"resourceSampleCount",
|
||||
"resourcePeakTrackedRssMb",
|
||||
"resourceCpuPercentMaxTracked",
|
||||
"resourcePeakCommandTreeRssMb",
|
||||
"resourcePeakGatewayRssMb",
|
||||
"openclawTimelineEventCount",
|
||||
@ -619,8 +443,8 @@ function metricDeltas(baseline, current) {
|
||||
"eventLoopDelayMs",
|
||||
"providerModelTimingMs"
|
||||
]) {
|
||||
const currentValue = measurementMetricValue(current, metric);
|
||||
const baselineValue = measurementMetricValue(baseline, metric);
|
||||
const currentValue = current?.[metric] ?? null;
|
||||
const baselineValue = baseline?.[metric] ?? null;
|
||||
metrics[metric] = {
|
||||
baseline: baselineValue,
|
||||
current: currentValue,
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user