docs: rewrite plugin-inspector README

This commit is contained in:
Vincent Koc 2026-04-28 00:45:47 -07:00
parent 85f360548e
commit 5ea07a8fca
No known key found for this signature in database

409
README.md
View File

@ -1,46 +1,42 @@
<img src="docs/plugin-inspector-banner.jpg" alt="OpenClaw Plugin Inspector banner">
# OpenClaw Plugin Inspector
`@openclaw/plugin-inspector` checks an OpenClaw plugin package before it reaches
users. It is meant to answer practical compatibility questions:
`@openclaw/plugin-inspector` is the offline compatibility checker for OpenClaw
plugin packages and plugin fixture suites.
- Can OpenClaw discover this plugin package and manifest?
- Which SDK imports, hooks, and registration calls does the plugin use?
- Will the plugin still load in clean CI without local OpenClaw internals?
- What report artifacts should CI upload when compatibility breaks?
It answers the questions that matter before a plugin reaches users:
The default check is static, offline, and credential-free. Runtime capture is
available, but it is always explicit because it imports plugin code.
- can OpenClaw discover the package metadata and `openclaw.plugin.json`
manifest?
- which hooks, registration calls, manifest contracts, and SDK imports does the
plugin use?
- does the plugin still look compatible without local OpenClaw internals?
- if CI finds a breakage, which JSON, Markdown, SARIF, JUnit, and summary
artifacts should downstream automation read?
- when a fixture-suite harness such as Crabpot runs many plugins, which findings
are hard breakages, known warnings, live issues, deprecations, or inspector
proof gaps?
## When To Use It
Use the plugin-root workflow for normal plugin repositories:
```bash
npx @openclaw/plugin-inspector inspect --no-openclaw
```
Use fixture suites only when one repository is intentionally checking many
plugins, such as Crabpot:
```bash
plugin-inspector report --config crabpot.config.json --out reports
```
Most plugin authors should start with `inspect --no-openclaw`, then add
`init --ci --scripts` once the local check makes sense.
The default path is static, offline, and credential-free. Runtime capture exists,
but it is opt-in because it imports plugin code.
## Requirements
- Node.js 22 or newer.
- A plugin package root with `package.json`.
- An `openclaw.plugin.json` manifest when the plugin uses the manifest contract.
- `openclaw.plugin.json` when the plugin uses the OpenClaw manifest contract.
- No OpenClaw checkout, credentials, network service, or live provider access for
default inspection.
OpenClaw itself is optional. Pass `--no-openclaw` when CI should not compare
against a local OpenClaw checkout.
Pass `--no-openclaw` when CI should not compare against a local OpenClaw
checkout. If an OpenClaw checkout is supplied with `--openclaw <path>`, the
inspector only reads public compatibility surfaces such as compat records, SDK
exports, hook names, manifest fields, and registrar metadata.
## First Check
## Quick Start
From a plugin package directory:
Run this from a plugin package root:
```bash
npx @openclaw/plugin-inspector inspect --no-openclaw
@ -60,12 +56,13 @@ The command writes:
- `reports/plugin-inspector-report.md`
- `reports/plugin-inspector-issues.md`
It exits non-zero when compatibility breakages are found. Warnings and
suggestions stay visible in the reports without necessarily failing the command.
It exits non-zero when hard compatibility breakages are found. Warnings,
suggestions, issue classifications, and logs stay visible in the report without
necessarily failing the command.
## Add It To A Plugin Repo
## Install In A Plugin Repo
Install the package when you want local scripts and repeatable CI:
Install the package when you want repeatable local scripts and CI:
```bash
npm install --save-dev @openclaw/plugin-inspector
@ -88,7 +85,7 @@ Then run:
npm run plugin:check
```
Or let the inspector write the starting config, package scripts, and GitHub
The initializer can write the starter config, package scripts, and GitHub
Actions workflow:
```bash
@ -96,8 +93,9 @@ npx @openclaw/plugin-inspector init --ci --scripts --dry-run
npx @openclaw/plugin-inspector init --ci --scripts
```
`init` detects `packageManager` and common lockfiles. Override that detection
with `--package-manager npm`, `pnpm`, `yarn`, or `bun`.
`init` detects `packageManager` and common lockfiles. Override that with
`--package-manager npm`, `--package-manager pnpm`, `--package-manager yarn`, or
`--package-manager bun`. Existing files are protected unless you pass `--force`.
## Configuration
@ -106,7 +104,8 @@ Small plugin repos can keep configuration in `package.json`:
```json
{
"scripts": {
"plugin:check": "plugin-inspector inspect --no-openclaw"
"plugin:check": "plugin-inspector inspect --no-openclaw",
"plugin:ci": "plugin-inspector ci --no-openclaw --runtime --mock-sdk --allow-execute"
},
"pluginInspector": {
"version": 1,
@ -114,7 +113,10 @@ Small plugin repos can keep configuration in `package.json`:
"id": "weather",
"priority": "high",
"seams": ["dynamic-tool"],
"sourceRoot": "src"
"sourceRoot": "src",
"expect": {
"registrations": ["registerTool"]
}
},
"capture": {
"mockSdk": true
@ -123,7 +125,7 @@ Small plugin repos can keep configuration in `package.json`:
}
```
Use `plugin-inspector.config.json` when you want a standalone config file:
Use `plugin-inspector.config.json` for a standalone config file:
```json
{
@ -146,18 +148,94 @@ Use `plugin-inspector.config.json` when you want a standalone config file:
}
```
Check what the inspector resolved before wiring CI:
Inspect the resolved config before wiring CI:
```bash
plugin-inspector config --json
```
Copy-ready config examples live in:
Copy-ready examples live in:
- `examples/plugin-inspector.config.json`
- `examples/package-json-plugin-inspector.json`
## CI Setup
## Commands
| Command | Purpose |
| --- | --- |
| `plugin-inspector` | Default alias for `check`. |
| `plugin-inspector check` | Script-friendly plugin-root check. |
| `plugin-inspector inspect` | Plugin-root check unless `--config` is supplied; with `--config`, runs a fixture report. |
| `plugin-inspector ci` | Compatibility report plus CI summary, SARIF, and JUnit outputs. |
| `plugin-inspector config` | Print resolved plugin-root config as text or JSON. |
| `plugin-inspector init` | Write starter config, scripts, and optional GitHub Actions workflow. |
| `plugin-inspector report` | Run a fixture-suite config with many plugins. |
| `plugin-inspector capture` | Runtime-capture one entrypoint directly. |
Common options:
| Option | Meaning |
| --- | --- |
| `--plugin-root <path>` / `--root <path>` | Check a plugin somewhere other than the current directory. |
| `--config <path>` | Read a standalone config file. Required for fixture-suite `report`. |
| `--out <dir>` | Write reports somewhere other than `reports/`. |
| `--openclaw <path>` | Compare against a local OpenClaw checkout. |
| `--no-openclaw` | Disable OpenClaw checkout comparison. |
| `--runtime` / `--capture` | Add opt-in runtime registration capture. |
| `--no-runtime` / `--no-capture` | Disable runtime capture even when config enables it. |
| `--mock-sdk` / `--sdk mock` | Use generated SDK and external-package mocks for runtime capture. |
| `--real-sdk` / `--sdk real` | Use installed real SDK dependencies instead of mocks. |
| `--allow-execute` | Permit commands that import plugin code. |
| `--json` | Print machine-readable JSON to stdout. |
| `--sarif [path]` | Write SARIF from `check` or `inspect`; `ci` enables this by default. |
| `--junit [path]` | Write JUnit XML from `check` or `inspect`; `ci` enables this by default. |
| `--no-sarif` / `--no-junit` | Disable default `ci` outputs. |
Run the built-in help for the exact CLI surface:
```bash
plugin-inspector --help
```
## Runtime Capture
Runtime capture imports plugin entrypoints in an isolated subprocess and records
what `register(api)` does. Use it when static inspection cannot prove the actual
registrations made at runtime.
```bash
plugin-inspector inspect --no-openclaw --runtime --mock-sdk --allow-execute
```
`--allow-execute` is the deliberate safety switch. Without it, modes that import
plugin code fail closed. The older environment guard still works for custom
harnesses:
```bash
PLUGIN_INSPECTOR_EXECUTE_ISOLATED=1 plugin-inspector inspect --no-openclaw --runtime --mock-sdk
```
By default, runtime capture uses generated mocks for `openclaw/plugin-sdk`
subpaths and unresolved external packages discovered in the plugin import graph.
That keeps compatibility CI offline and credential-free. It does not call live
services, launch OpenClaw, run provider SDKs, or emulate service lifecycle side
effects.
Use `--real-sdk` only when the plugin workspace already has real SDK
dependencies installed and you intentionally want that path.
Runtime capture writes:
- `reports/plugin-inspector-runtime-capture.json`
- `reports/plugin-inspector-runtime-capture.md`
Capture one entrypoint directly:
```bash
plugin-inspector capture ./dist/index.js --mock-sdk --allow-execute
```
## CI
`plugin-inspector ci` writes the normal compatibility report plus CI-native
summary, SARIF, and JUnit artifacts.
@ -190,7 +268,7 @@ jobs:
path: reports/plugin-inspector-*
```
Generated CI artifacts include:
Generated `ci` artifacts:
- `reports/plugin-inspector-report.json`
- `reports/plugin-inspector-report.md`
@ -200,83 +278,109 @@ Generated CI artifacts include:
- `reports/plugin-inspector.sarif`
- `reports/plugin-inspector.junit.xml`
Use `--no-sarif` or `--no-junit` only if your CI surface cannot consume those
formats.
More examples:
CI examples:
- `examples/github-actions-plugin-inspector.yml`
- `examples/github-actions-code-scanning.yml`
- `examples/gitlab-ci-plugin-inspector.yml`
- `examples/circleci-plugin-inspector.yml`
## Runtime Capture
## Report Surfaces
Runtime capture imports plugin entrypoints in an isolated subprocess and records
what `register(api)` does. Use it when static inspection cannot show the actual
registrations your plugin makes at runtime.
The compatibility report is the primary contract. Preserve field names and
finding codes because downstream CI and Crabpot reports may consume them.
```bash
plugin-inspector inspect --no-openclaw --runtime --mock-sdk --allow-execute
```
Important report sections:
`--allow-execute` is the deliberate safety switch. Without it, modes that import
plugin code fail closed. The older `PLUGIN_INSPECTOR_EXECUTE_ISOLATED=1`
environment guard still works for custom harnesses.
By default, runtime capture uses generated mocks for `openclaw/plugin-sdk`
subpaths and unresolved external packages discovered in the plugin import graph.
That lets compatibility CI run without OpenClaw installed.
Use `--real-sdk` only when the plugin workspace already has real SDK
dependencies installed and you intentionally want that path.
Runtime capture writes:
- `reports/plugin-inspector-runtime-capture.json`
- `reports/plugin-inspector-runtime-capture.md`
Capture one entrypoint directly:
```bash
plugin-inspector capture ./dist/index.js --mock-sdk --allow-execute
```
## Command Reference
| Command | Use it when |
| Field | Meaning |
| --- | --- |
| `plugin-inspector inspect` | You are checking one plugin package. |
| `plugin-inspector check` | You need the older script-friendly alias for `inspect`. |
| `plugin-inspector ci` | You want reports plus CI summary, SARIF, and JUnit outputs. |
| `plugin-inspector config` | You want to inspect resolved config before running CI. |
| `plugin-inspector init` | You want starter config, scripts, and CI workflow files. |
| `plugin-inspector report` | You are running a fixture-suite config with many plugins. |
| `plugin-inspector capture` | You want runtime capture for one entrypoint. |
| `status` | `pass` unless hard breakages exist. |
| `summary` | Counts for fixtures, breakages, warnings, suggestions, issues, issue classes, and contract probes. |
| `targetOpenClaw` | Status and public compatibility data read from the optional OpenClaw checkout. |
| `fixtures` | Per-plugin metadata, hooks, registrations, manifest contracts, package data, and SDK imports. |
| `breakages` | Blocking compatibility failures. |
| `warnings` / `suggestions` | Non-blocking compatibility findings. |
| `issues` | Normalized issue rows with severity and class. |
| `contractProbes` | Suggested synthetic probes derived from observed contracts. |
| `logs` | Informational inventory and coverage rows. |
| `decisions` | Maintainer-facing follow-up or compatibility-policy decisions. |
Useful options:
Issue classes currently flow through the reports as live issues, compat gaps,
deprecation warnings, inspector gaps, upstream metadata, and fixture regressions.
| Option | Meaning |
| --- | --- |
| `--plugin-root <path>` | Check a plugin somewhere other than the current directory. |
| `--config <path>` | Read a standalone config file. |
| `--out <dir>` | Write reports somewhere other than `reports/`. |
| `--no-openclaw` | Skip comparison against a local OpenClaw checkout. |
| `--runtime` | Add opt-in runtime registration capture. |
| `--mock-sdk` | Use generated SDK and external-package mocks for runtime capture. |
| `--real-sdk` | Use installed real SDK dependencies instead of mocks. |
| `--allow-execute` | Permit commands that import plugin code. |
## CI Policy And Shared Reporting Primitives
Run the built-in help for the complete flag list:
`plugin-inspector` owns the shared CI policy and report rendering primitives.
Fixture-suite harnesses such as Crabpot should call these exports instead of
reimplementing scoring, summaries, Markdown, SARIF, or JUnit handling.
```bash
plugin-inspector --help
The root API exposes grouped helpers:
```js
import { ci } from "@openclaw/plugin-inspector";
const policyReport = ci.buildPolicyReport({
policy,
compatibilityReport,
executionResults,
strict: false,
});
await ci.writePolicyReport(policyReport);
```
## Embedding In A Harness
CI policy reports default to:
Most plugin repos should use the CLI. Test harnesses can import grouped helpers
from the root package when they need to compose inspector workflows directly:
- `reports/plugin-inspector-ci-policy.json`
- `reports/plugin-inspector-ci-policy.md`
A policy must use `version: 1` and define:
- `allowedBlocked`
- `expectedWarnings`
- `thresholds`
- `fixtureSets`
Policy scoring fails hard breakages, unknown blocked synthetic probes, hard ref
diff regressions, failed execution results, strict live P0 issues, and strict
classified blockers. Non-strict mode keeps classified blocked probes and live P0
issues visible as warnings.
CI summary helpers read the known report set from `reports/` and render one
machine-readable and one Markdown rollup:
- compatibility
- runtime capture
- synthetic probes
- cold import readiness
- workspace plan
- platform probes
- import-loop profile
- execution results
- runtime profile
- ref diff
- profile diff
- CI policy
## Fixture Suites
Most plugin authors should use the plugin-root workflow. Use fixture suites when
one repository intentionally checks many plugins or packages, as Crabpot does.
```bash
plugin-inspector report --config crabpot.config.json --out reports
plugin-inspector report --config crabpot.config.json --out reports --check
plugin-inspector ci --config crabpot.config.json --out reports --no-openclaw
```
Fixture-suite configs are loaded through the explicit fixture helpers. That keeps
normal plugin-root configuration simple while still supporting bulk compatibility
harnesses.
## Public API
Prefer the CLI for normal plugin repositories. Import the public API when a test
harness needs to compose workflows directly:
```js
import { pluginRoot } from "@openclaw/plugin-inspector";
@ -290,31 +394,88 @@ const { report, paths } = await pluginRoot.runCheck({
console.log(report.status, paths.jsonPath);
```
The root package groups stable workflows as `pluginRoot`, `fixtureSuites`,
`staticInspection`, `reports`, `contracts`, `ci`, `runtime`, and `synthetic`.
Named exports remain available for existing automation.
Stable grouped facades:
## What The Mock SDK Does
| Facade | Use |
| --- | --- |
| `pluginRoot` | Load config, inspect, run checks, capture entrypoints, or set up a plugin repo. |
| `fixtureSuites` | Load fixture-suite configs, run reports, and build fixture-suite readiness plans. |
| `staticInspection` | Inspect source text or fixture sets without the compatibility report layer. |
| `reports` | Render/write reports and classify issue findings. |
| `contracts` | Build, render, validate, and write contract captures and coverage. |
| `ci` | Build summaries, policy reports, execution results, SARIF, and JUnit outputs. |
| `runtime` | Build runtime profiles, profile diffs, ref diffs, and import-loop profiles. |
| `synthetic` | Build and run synthetic probe plans. |
The mock lane is for compatibility CI, not live service testing.
Named exports remain available for existing automation. Prefer the grouped
facades for new code because they show ownership and keep downstream wrappers
thin.
When `--mock-sdk` is enabled, the inspector generates temporary modules for
`openclaw/plugin-sdk` subpaths and unresolved external packages. The mock SDK
captures registrations; it does not call network services, launch OpenClaw, run
provider SDKs, or emulate service lifecycle side effects.
## Development
Keep live provider/service tests in the plugin repository behind credentials
and explicit opt-in flags.
Repository checks are intentionally small and offline:
## Scope
```bash
npm test
npm run release:contents
npm run check
```
Default inspection reads manifests, package metadata, and source files, then
reports observed `api.on(...)`, `api.register*`, `define*`, SDK imports, and
manifest contracts.
`npm run check` runs the Node test suite and the package-contents guard. The
contents guard shells through `npm pack --dry-run --json` and verifies the npm
tarball includes package entrypoints, examples, README assets, and no private
`test/`, `scripts/`, or `.github/` paths.
OpenClaw target checkout parsing is limited to public compatibility registries,
SDK package exports, manifest types, hooks, and captured registrar metadata.
Useful release-prep commands:
Cold import capture, synthetic contract probes, runtime capture, and live lanes
are separate opt-in modes. Live lanes must stay credential-gated and should not
run in default CI.
```bash
npm run release:local
npm run release:readiness
npm run release:notes
npm run release:plan
npm run release:crabpot -- --crabpot ../crabpot
```
`release:readiness` proves the local package and verifies Crabpot follow-through.
It does not publish.
Keep this package dependency-light. Do not add runtime dependencies unless they
remove real complexity. Default checks must stay offline and credential-free.
## Release Notes
The package publishes from annotated `v*` tags through GitHub Actions. The
release workflow runs the test suite, verifies the npm tarball, publishes the
GitHub release, and publishes the public npm package through npm trusted
publishing.
Before tagging a release:
1. Move `CHANGELOG.md` `Unreleased` notes into a versioned section.
2. Update `package.json` to the same version.
3. Update Crabpot's `pluginInspectorRef` to the release commit.
4. Run `npm run release:readiness`.
5. Run the Crabpot plugin-inspector smoke commands printed by
`npm run release:crabpot -- --crabpot ../crabpot`.
After npm publish, update Crabpot's package pin and run:
```bash
npm run release:crabpot -- --crabpot ../crabpot --published
```
Do not publish npm packages without explicit owner approval.
## Contribution Notes
There is no `CONTRIBUTING.md` in this repository. Until one exists, use the repo
scripts above as the local contract and follow these project rules:
- preserve stable report field names and finding codes;
- prefer public OpenClaw plugin contracts over core internals;
- isolate any OpenClaw source parsing behind explicit helpers;
- keep runtime execution behind `--allow-execute` or
`PLUGIN_INSPECTOR_EXECUTE_ISOLATED=1`;
- when behavior, entrypoints, release metadata, or the npm package version
change, update Crabpot's `@openclaw/plugin-inspector` pin/docs/smoke path and
run the Crabpot plugin-inspector smoke before calling the work done.