Compare commits

...

83 Commits
v3.2.2 ... main

Author SHA1 Message Date
Peter Steinberger
dda07c245f
fix: document element IDs as opaque (#202)
Some checks failed
macOS CI / PeekabooCore build & tests (push) Has been cancelled
Website (GitHub Pages) / build (push) Has been cancelled
macOS CI / Peekaboo CLI build & tests (push) Has been cancelled
macOS CI / Tachikoma build & tests (push) Has been cancelled
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Has been cancelled
macOS CI / SwiftLint (core + CLI) (push) Has been cancelled
Website (GitHub Pages) / deploy (push) Has been cancelled
2026-06-24 09:11:52 +01:00
Brandon Charleson
efde5b18ca
fix(release): remove appcast entry for unpublished 3.5.3 release (#199)
* fix(release): remove appcast entry for unpublished 3.5.3 release

The appcast advertised v3.5.3 but no GitHub release asset exists,
causing Sparkle updates to fail with a 404 when downloading the zip.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs: credit appcast rollback contributor

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-06-24 08:42:00 +01:00
Peter Steinberger
cde3b04991
test(cli): stabilize timeout watermark checks (#201) 2026-06-24 08:18:58 +01:00
Peter Steinberger
4085f18ddc
fix(cli): keep implicit see screenshots private (#200)
Some checks are pending
macOS CI / PeekabooCore build & tests (push) Waiting to run
macOS CI / Peekaboo CLI build & tests (push) Blocked by required conditions
macOS CI / Tachikoma build & tests (push) Blocked by required conditions
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Blocked by required conditions
macOS CI / SwiftLint (core + CLI) (push) Blocked by required conditions
Website (GitHub Pages) / build (push) Waiting to run
Website (GitHub Pages) / deploy (push) Blocked by required conditions
2026-06-24 00:58:16 +01:00
Sebastien Tardif
db5192bb37
fix(capture): honor stop during watch transient backoff (#193)
* fix(capture): honor stop during watch transient backoff

* test(capture): tighten transient stop proof and add proof script

* test(capture): tighten transient stop regression

* chore: complete main merge

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-06-24 00:09:05 +01:00
Coy Geek
1771d7db34
docs: refresh Peekaboo agent skill guidance (#197)
* docs(skill): refresh Peekaboo agent skill

* docs(skill): polish Peekaboo guidance

* docs: note refreshed agent skill guidance

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-06-23 15:41:58 -07:00
Peter Steinberger
ab0d96e6d5
chore(deps): update TauTUI and Tachikoma 2026-06-23 22:10:01 +01:00
Peter Steinberger
9b9c5de43b
chore(release): update appcast for 3.5.3
Some checks failed
macOS CI / PeekabooCore build & tests (push) Has been cancelled
Website (GitHub Pages) / build (push) Has been cancelled
macOS CI / Peekaboo CLI build & tests (push) Has been cancelled
macOS CI / Tachikoma build & tests (push) Has been cancelled
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Has been cancelled
macOS CI / SwiftLint (core + CLI) (push) Has been cancelled
Website (GitHub Pages) / deploy (push) Has been cancelled
2026-06-13 23:09:35 -07:00
Peter Steinberger
f22e46cc1a
chore(release): prepare 3.5.3 2026-06-13 22:30:12 -07:00
Peter Steinberger
4f2b0e9cb4
fix: harden background computer automation 2026-06-13 21:44:58 -07:00
Peter Steinberger
5fba9b79de
docs: refresh project banner (#190)
Some checks are pending
macOS CI / PeekabooCore build & tests (push) Waiting to run
macOS CI / Peekaboo CLI build & tests (push) Blocked by required conditions
macOS CI / Tachikoma build & tests (push) Blocked by required conditions
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Blocked by required conditions
macOS CI / SwiftLint (core + CLI) (push) Blocked by required conditions
2026-06-13 14:59:28 -07:00
Peter Steinberger
3aaa96bfa4
fix: filter background apps from app list (#189) 2026-06-13 14:28:47 -07:00
Peter Steinberger
8cf0796692
Fix clicks on hidden menu extras (#188)
Some checks failed
macOS CI / PeekabooCore build & tests (push) Waiting to run
macOS CI / Peekaboo CLI build & tests (push) Blocked by required conditions
macOS CI / Tachikoma build & tests (push) Blocked by required conditions
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Blocked by required conditions
macOS CI / SwiftLint (core + CLI) (push) Blocked by required conditions
Website (GitHub Pages) / build (push) Has been cancelled
Website (GitHub Pages) / deploy (push) Has been cancelled
* fix: reject hidden menu extras

* docs: note hidden menu extra safety
2026-06-13 02:59:25 -07:00
Peter Steinberger
26c7291292
docs(changelog): open 3.5.3 2026-06-13 02:06:20 -07:00
Peter Steinberger
84100e4cc1
chore(release): update appcast for 3.5.2 2026-06-13 02:04:38 -07:00
Peter Steinberger
1fa8eead7e
chore(release): prepare 3.5.2 2026-06-13 01:15:58 -07:00
Peter Steinberger
b0f5086ad4 chore(deps): synchronize Tachikoma 2026-06-13 02:43:13 -04:00
Peter Steinberger
aabea1550e fix(type): remove default keystroke delay 2026-06-13 02:20:30 -04:00
Peter Steinberger
1c6273c017 fix(type): default to fast linear typing 2026-06-13 02:17:10 -04:00
Peter Steinberger
131be20a69
docs(release): harden release workflow 2026-06-12 22:10:28 -07:00
Peter Steinberger
8656242865
docs(changelog): open 3.5.2 2026-06-12 21:30:34 -07:00
Peter Steinberger
e123fa5bc1
chore(release): update appcast for 3.5.1 2026-06-12 21:30:18 -07:00
Peter Steinberger
6a932d0004
chore(release): prepare 3.5.1 2026-06-12 20:51:04 -07:00
Peter Steinberger
ee3f90c404
fix(cli): enforce suspended observation deadlines 2026-06-12 15:23:45 -07:00
Peter Steinberger
a5bbd1ebdc
docs(changelog): open 3.5.1
Some checks are pending
macOS CI / PeekabooCore build & tests (push) Waiting to run
macOS CI / Peekaboo CLI build & tests (push) Blocked by required conditions
macOS CI / Tachikoma build & tests (push) Blocked by required conditions
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Blocked by required conditions
macOS CI / SwiftLint (core + CLI) (push) Blocked by required conditions
Website (GitHub Pages) / build (push) Waiting to run
Website (GitHub Pages) / deploy (push) Blocked by required conditions
2026-06-12 11:52:29 -07:00
Peter Steinberger
371bed775b
chore(release): update appcast for 3.5.0 2026-06-12 11:52:04 -07:00
Peter Steinberger
231fa48370
fix(release): publish verified npm tarballs from dirty trees 2026-06-12 01:36:34 -07:00
Peter Steinberger
0f66ff5c24
fix(release): build app before Developer ID signing 2026-06-12 01:12:50 -07:00
Peter Steinberger
e183cd15fb
chore(release): prepare 3.5.0 2026-06-12 00:45:49 -07:00
Peter Steinberger
64a4bd6184
docs: refresh runtime and provider guidance 2026-06-12 00:45:45 -07:00
Peter Steinberger
d50472e5a3
fix(bridge): enforce exclusive socket ownership (#187)
Give Bridge listeners exclusive lease-backed socket ownership, bound transport and shutdown drains, isolate reusable daemon hosting, preserve healthy Peekaboo.app fallback, and safely migrate legacy daemons.

Fixes #184.
2026-06-12 00:17:34 -07:00
Vishal Jain
b873daf790
fix(capture): avoid false-success screen captures (#185)
Some checks are pending
macOS CI / PeekabooCore build & tests (push) Waiting to run
macOS CI / Peekaboo CLI build & tests (push) Blocked by required conditions
macOS CI / Tachikoma build & tests (push) Blocked by required conditions
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Blocked by required conditions
macOS CI / SwiftLint (core + CLI) (push) Blocked by required conditions
Website (GitHub Pages) / build (push) Waiting to run
Website (GitHub Pages) / deploy (push) Blocked by required conditions
* fix(capture): avoid false-success screen captures

* fix(capture): fail closed when screen capture fallback is unsafe

* fix(capture): translate screencapture display regions

* fix(capture): harden legacy screen capture

* fix(cli): keep screen permission requests local

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-06-11 19:24:39 -07:00
Peter Steinberger
e4cd616e19
chore(release): prepare 3.4.2 2026-06-11 18:01:03 -07:00
Peter Steinberger
e44486ff16
feat: add Claude Fable 5 support (#186) 2026-06-11 17:59:43 -07:00
Peter Steinberger
7c3862b032
docs(changelog): open 3.4.2
Some checks failed
macOS CI / PeekabooCore build & tests (push) Has been cancelled
macOS CI / Peekaboo CLI build & tests (push) Has been cancelled
macOS CI / Tachikoma build & tests (push) Has been cancelled
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Has been cancelled
macOS CI / SwiftLint (core + CLI) (push) Has been cancelled
2026-06-10 05:56:30 +01:00
Peter Steinberger
ee0e318543
chore(release): update appcast for 3.4.1 2026-06-10 05:38:14 +01:00
Peter Steinberger
56ec6d24a9
chore(deps): update chrome-devtools-mcp 2026-06-10 05:33:56 +01:00
Peter Steinberger
689013808f
chore(release): prepare 3.4.1 2026-06-10 05:00:23 +01:00
Peter Steinberger
660e6f35c9
test: clear release gate warnings 2026-06-10 05:00:23 +01:00
Peter Steinberger
3c15f57652
docs: clarify autonomous maintenance rules 2026-06-10 04:55:57 +01:00
Peter Steinberger
e75db3a7aa
fix(agent): support latest provider models
Some checks failed
macOS CI / PeekabooCore build & tests (push) Has been cancelled
Website (GitHub Pages) / build (push) Has been cancelled
macOS CI / Peekaboo CLI build & tests (push) Has been cancelled
macOS CI / Tachikoma build & tests (push) Has been cancelled
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Has been cancelled
macOS CI / SwiftLint (core + CLI) (push) Has been cancelled
Website (GitHub Pages) / deploy (push) Has been cancelled
Co-authored-by: Ulrich Diedrichsen <uli@moinsen.dev>
2026-06-09 01:23:32 +01:00
Peter Steinberger
8c51fefb66
docs(changelog): open 3.4.1
Some checks failed
macOS CI / PeekabooCore build & tests (push) Has been cancelled
Website (GitHub Pages) / build (push) Has been cancelled
macOS CI / Peekaboo CLI build & tests (push) Has been cancelled
macOS CI / Tachikoma build & tests (push) Has been cancelled
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Has been cancelled
macOS CI / SwiftLint (core + CLI) (push) Has been cancelled
Website (GitHub Pages) / deploy (push) Has been cancelled
2026-06-07 08:50:21 +01:00
Peter Steinberger
3a56ed2aa7
chore(release): update appcast for 3.4.0 2026-06-07 08:17:13 +01:00
Peter Steinberger
613f0435a8
chore(release): prepare 3.4.0 2026-06-07 07:40:24 +01:00
Peter Steinberger
7e61018019
feat(cli): expose MCP wrappers and capture action 2026-06-07 07:35:00 +01:00
Peter Steinberger
3608d9c782
feat(mcp): expose capture tool 2026-06-07 05:46:39 +01:00
Peter Steinberger
87d4721e29
test(cli): cover default background interactions 2026-06-07 04:32:26 +01:00
Peter Steinberger
1665ca8061
docs: update changelog for bridge error fix 2026-06-06 18:05:44 -07:00
Coy Geek
01fcfba877
fix(cli): map bridge permission errors (#181)
Use bridge envelope messages and details when emitting JSON command errors so bridge failures do not collapse into opaque Swift localized descriptions. Map bridge permission-denied envelopes to the existing permission-specific CLI error codes, including screen recording for capture live failures.

Refs: #170
2026-06-06 18:05:11 -07:00
Peter Steinberger
0ce0895cb8
docs: update changelog for clipboard fix 2026-06-06 17:42:32 -07:00
Coy Geek
01d10db9b0
fix(clipboard): validate alternate text payloads (#180)
* fix(clipboard): validate alternate text payloads

* docs: clarify clipboard size guard accounting

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-06-06 17:42:04 -07:00
Peter Steinberger
6d29dae6cb
docs: update changelog for tools help clarification 2026-06-06 17:16:51 -07:00
Shubhankar Tripathy
a6ee79a89b
docs(cli): clarify peekaboo tools lists MCP/agent catalog not CLI commands (#174)
* docs(cli): clarify peekaboo tools lists MCP/agent catalog not CLI commands

* docs: remove contributor changelog entry

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-06-06 17:16:28 -07:00
Peter Steinberger
f7a1fc9707
docs: update changelog for automation fixes 2026-06-06 16:20:54 -07:00
Coy Geek
0e05e3acf5
fix(process): preserve generic interaction aliases (#179)
* fix(process): preserve generic interaction aliases

* fix(automation): preserve generic menu paths
2026-06-06 16:20:32 -07:00
Coy Geek
11baff0b59
fix(scroll): report unsupported targetless action scroll (#178) 2026-06-06 16:19:57 -07:00
Peter Steinberger
3be1dc7ef2
docs: document input delivery modes
Some checks failed
macOS CI / PeekabooCore build & tests (push) Has been cancelled
Website (GitHub Pages) / build (push) Has been cancelled
macOS CI / Peekaboo CLI build & tests (push) Has been cancelled
macOS CI / Tachikoma build & tests (push) Has been cancelled
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Has been cancelled
macOS CI / SwiftLint (core + CLI) (push) Has been cancelled
Website (GitHub Pages) / deploy (push) Has been cancelled
2026-06-01 01:46:31 +01:00
Peter Steinberger
8af8b90d07
docs(changelog): open 3.3.1 2026-06-01 00:43:19 +01:00
Peter Steinberger
faf8430327
chore(release): update appcast for 3.3.0 2026-06-01 00:37:34 +01:00
Peter Steinberger
6feffe58b1
chore(release): prepare 3.3.0 2026-06-01 00:08:26 +01:00
Peter Steinberger
697acdd0cf
chore: update Tachikoma submodule 2026-06-01 00:03:50 +01:00
Peter Steinberger
6609a43415 fix(cli): improve background text input 2026-05-31 18:16:53 -04:00
Peter Steinberger
619a033f89 fix(cli): harden background text input 2026-05-31 17:16:25 -04:00
Peter Steinberger
122c96da0b feat(cli): use background keyboard delivery by default 2026-05-31 16:24:09 -04:00
Peter Steinberger
abb4e87a50
fix(cli): default clicks to background delivery (#168) 2026-05-31 20:24:18 +01:00
Peter Steinberger
c15ff187b7
docs: update AXorcist submodule
Some checks failed
macOS CI / PeekabooCore build & tests (push) Has been cancelled
Website (GitHub Pages) / build (push) Has been cancelled
Commander Multiplatform / macos-host (push) Has been cancelled
Commander Multiplatform / apple-simulators (iOS, iphonesimulator, arm64-apple-ios17.0-simulator) (push) Has been cancelled
Commander Multiplatform / apple-simulators (tvOS, appletvsimulator, arm64-apple-tvos17.0-simulator) (push) Has been cancelled
Commander Multiplatform / apple-simulators (watchOS, watchsimulator, arm64-apple-watchos10.0-simulator) (push) Has been cancelled
Commander Multiplatform / linux (push) Has been cancelled
macOS CI / Peekaboo CLI build & tests (push) Has been cancelled
macOS CI / Tachikoma build & tests (push) Has been cancelled
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Has been cancelled
macOS CI / SwiftLint (core + CLI) (push) Has been cancelled
Website (GitHub Pages) / deploy (push) Has been cancelled
2026-05-28 20:51:07 +01:00
Peter Steinberger
28a47f9967
docs: position README banner 2026-05-28 20:48:37 +01:00
Vincent Koc
9d32a65e4a
ci: pin macOS runner labels 2026-05-28 20:53:35 +02:00
Peter Steinberger
4144b9bdc0
docs: update AXorcist submodule 2026-05-28 19:47:14 +01:00
Peter Steinberger
3059d96764
docs: add README banner 2026-05-28 19:43:50 +01:00
Peter Steinberger
871e132a37
fix: update capture command references 2026-05-28 18:46:53 +01:00
Peter Steinberger
2f8113706a
docs: fix skill frontmatter and MCP capture docs 2026-05-28 16:04:37 +01:00
Peter Steinberger
1add96f214
docs(skills): defer private release locators
Some checks failed
macOS CI / PeekabooCore build & tests (push) Has been cancelled
macOS CI / Peekaboo CLI build & tests (push) Has been cancelled
macOS CI / Tachikoma build & tests (push) Has been cancelled
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Has been cancelled
macOS CI / SwiftLint (core + CLI) (push) Has been cancelled
2026-05-25 15:04:46 +01:00
Peter Steinberger
86f541184f
feat(agent): add MiniMax China provider support (#162)
Some checks failed
macOS CI / PeekabooCore build & tests (push) Waiting to run
macOS CI / Peekaboo CLI build & tests (push) Blocked by required conditions
macOS CI / Tachikoma build & tests (push) Blocked by required conditions
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Blocked by required conditions
macOS CI / SwiftLint (core + CLI) (push) Blocked by required conditions
Website (GitHub Pages) / build (push) Has been cancelled
Website (GitHub Pages) / deploy (push) Has been cancelled
2026-05-25 11:05:05 +01:00
Peter Steinberger
1d92128af3
chore(skills): add repo release skill 2026-05-25 10:34:48 +01:00
Peter Steinberger
4a66f9352d
docs(changelog): open 3.2.4
Some checks are pending
macOS CI / PeekabooCore build & tests (push) Waiting to run
macOS CI / Peekaboo CLI build & tests (push) Blocked by required conditions
macOS CI / Tachikoma build & tests (push) Blocked by required conditions
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Blocked by required conditions
macOS CI / SwiftLint (core + CLI) (push) Blocked by required conditions
2026-05-24 05:35:01 +01:00
Peter Steinberger
e008e763cf
docs: update appcast for 3.2.3 2026-05-24 05:31:07 +01:00
Peter Steinberger
a16d700479
fix(release): handle empty Sparkle key args 2026-05-24 05:14:59 +01:00
Peter Steinberger
3078d65b8d
chore(release): prepare 3.2.3 2026-05-24 04:41:14 +01:00
Peter Steinberger
e1726f1259
fix: honor explicit latest snapshot aliases (#160)
Some checks are pending
macOS CI / PeekabooCore build & tests (push) Waiting to run
macOS CI / Peekaboo CLI build & tests (push) Blocked by required conditions
macOS CI / Tachikoma build & tests (push) Blocked by required conditions
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Blocked by required conditions
macOS CI / SwiftLint (core + CLI) (push) Blocked by required conditions
2026-05-24 04:02:23 +01:00
Peter Steinberger
e2e04e2eb6
fix: improve cli capture ergonomics (#159) 2026-05-24 03:04:41 +01:00
Vincent Koc
264ea28311
chore: add constrained Crabbox setup
Some checks failed
macOS CI / PeekabooCore build & tests (push) Has been cancelled
macOS CI / Peekaboo CLI build & tests (push) Has been cancelled
macOS CI / Tachikoma build & tests (push) Has been cancelled
macOS CI / Build macOS apps (Peekaboo + Inspector) (push) Has been cancelled
macOS CI / SwiftLint (core + CLI) (push) Has been cancelled
Adds constrained Crabbox setup and the exact OpenClaw Crabbox skill for maintainer validation.
2026-05-23 05:59:39 +08:00
Peter Steinberger
4a4bd3b060
docs(changelog): open 3.2.3 2026-05-22 14:58:47 +01:00
469 changed files with 40195 additions and 4207 deletions

View File

@ -0,0 +1,711 @@
---
name: crabbox
description: Use the Crabbox wrapper for OpenClaw remote validation across Linux, macOS, Windows, and WSL2, including delegated Blacksmith Testbox proof. Report the actual provider and id.
---
# Crabbox
Use the Crabbox wrapper when OpenClaw needs remote Linux proof for broad tests,
CI-parity checks, secrets, hosted services, Docker/E2E/package lanes, warmed
reusable boxes, sync timing, logs/results, cache inspection, or lease cleanup.
Crabbox is the transport/orchestration surface. The actual backend can be:
- brokered AWS Crabbox: direct provider, `provider=aws`, lease ids like
`cbx_...`, `syncDelegated=false`
- Blacksmith Testbox through Crabbox: delegated provider,
`provider=blacksmith-testbox`, ids like `tbx_...`, `syncDelegated=true`
For OpenClaw maintainer broad `pnpm` gates, Blacksmith Testbox through the
Crabbox wrapper is acceptable and often preferred when the standing Testbox
rules apply. Do not describe those runs as "AWS Crabbox"; report them as
Testbox-through-Crabbox with the `tbx_...` id and Actions run.
Use the repo `.crabbox.yaml` brokered AWS path when the task specifically needs
direct AWS Crabbox behavior, persistent direct-provider leases, `--fresh-pr`,
`--full-resync`, environment forwarding, capture/download support, or provider
comparison. Use `--provider blacksmith-testbox` when the task needs OpenClaw
maintainer Testbox proof, prepared CI environment, broad/heavy pnpm gates, or
the user asks for Testbox/Blacksmith.
## First Checks
- Run from the repo root. Crabbox sync mirrors the current checkout.
- Check the wrapper and providers before remote work:
```sh
command -v crabbox
../crabbox/bin/crabbox --version
pnpm crabbox:run -- --help | sed -n '1,120p'
../crabbox/bin/crabbox desktop launch --help
../crabbox/bin/crabbox webvnc --help
```
- OpenClaw scripts prefer `../crabbox/bin/crabbox` when present. The user PATH
shim can be stale.
- Check `.crabbox.yaml` for direct-provider defaults. Omitting `--provider`
means brokered AWS today.
- The brokered AWS default is a Linux developer image in `eu-west-1`; the repo
config pins hot `eu-west-1a/b/c` placement so Fast Snapshot Restore can apply.
If warmup drifts well past the minute-scale path, verify image promotion,
region/AZ placement, and FSR state before blaming OpenClaw.
- For broad OpenClaw maintainer `pnpm` gates, prefer the repo wrapper with
`--provider blacksmith-testbox` or the repo Testbox helpers when the standing
Testbox policy applies.
- Always report the actual provider and id. `cbx_...` means AWS Crabbox;
`tbx_...` means Blacksmith Testbox through Crabbox. If the output only says
`blacksmith testbox list`, use `blacksmith testbox list --all` before
concluding no box exists.
- If a warm direct-provider lease smells stale, retry with `--full-resync`
(alias `--fresh-sync`) before replacing the lease. This resets the remote
workdir, skips the fingerprint fast path, reseeds Git when possible, and
uploads the checkout from scratch.
- For live/provider bugs, use the configured secret workflow before downgrading
to mocks. Copy only the exact needed key into the remote process environment
for that one command. Do not print it, do not sync it as a repo file, and do
not leave it in remote shell history or logs. If no secret-safe injection path
is available, say true live provider auth is blocked instead of silently using
a fake key.
- Prefer local targeted tests for tight edit loops. Broad gates belong remote.
- Do not treat inherited shell env as operator intent. In particular,
`OPENCLAW_LOCAL_CHECK_MODE=throttled` from the local shell is not permission
to move broad `pnpm check:changed`, `pnpm test:changed`, full `pnpm test`, or
lint/typecheck fan-out onto the laptop.
- Only use `OPENCLAW_LOCAL_CHECK_MODE=throttled|full` when the user explicitly
asks for local proof in the current task. If Testbox is queued or capacity is
constrained, report the blocker and keep only targeted local edit-loop checks
running.
## macOS And Windows Targets
Use these only when the task needs an existing non-Linux host. OpenClaw broad
Linux validation uses the repo Crabbox config unless a provider is explicitly
requested.
Native brokered Windows is available for Windows-specific proof. Use the AWS
developer image in `us-west-2` on demand; it has the expected OpenClaw developer
toolchain and Docker image cache. Keep broad Linux gates on Linux/Testbox unless
the bug is Windows-specific:
```sh
../crabbox/bin/crabbox warmup \
--provider aws \
--target windows \
--windows-mode normal \
--region us-west-2 \
--market on-demand \
--timing-json
```
The hydrate workflow assumes Docker should already be baked into Linux images
and only installs it as a fallback. Do not add per-run Docker installs to proof
commands unless the image probe shows Docker is actually missing.
When the user explicitly asks for brokered macOS runners, use Crabbox AWS
macOS only after confirming the deployed coordinator supports EC2 Mac host
lifecycle/image routes and the operator has AWS EC2 Mac Dedicated Host quota
and IAM. Prefer `CRABBOX_HOST_ID` for a known Crabbox-managed Dedicated Host,
or run the no-spend preflight first:
```sh
crabbox admin hosts quota --provider aws --target macos --region eu-west-1 --type mac2.metal --json
crabbox admin hosts allocate --provider aws --target macos --region eu-west-1 --type mac2.metal --dry-run --json
CRABBOX_MACOS_TYPES=all scripts/macos-host-region-preflight.sh
```
Do not silently substitute AWS macOS for normal OpenClaw Linux proof. Report
paid-host blockers as quota, IAM, coordinator deployment, or host availability
instead of falling back to local macOS.
Crabbox supports static SSH targets:
```sh
../crabbox/bin/crabbox run --provider ssh --target macos --static-host mac-studio.local -- xcodebuild test
../crabbox/bin/crabbox run --provider ssh --target windows --windows-mode normal --static-host win-dev.local -- pwsh -NoProfile -Command "dotnet test"
../crabbox/bin/crabbox run --provider ssh --target windows --windows-mode wsl2 --static-host win-dev.local -- pnpm test
```
- `target=macos` and `target=windows --windows-mode wsl2` use the POSIX SSH,
bash, Git, rsync, and tar contract.
- Native Windows uses OpenSSH, PowerShell, Git, and tar; sync is manifest tar
archive transfer into `static.workRoot`. Direct native Windows runs support
`--script*`, `--env-from-profile`, `--preflight`, and PowerShell `--shell`.
- `crabbox actions hydrate/register` are Linux-only today; use plain
`crabbox run` loops for static macOS and Windows hosts.
- Live proof needs a reachable, operator-managed SSH host. Without one, verify
with `../crabbox/bin/crabbox run --help`, config/flag tests, and the Crabbox
Go test suite.
## Direct Brokered AWS Backend
Use this when the task needs direct AWS Crabbox semantics rather than the
prepared Blacksmith Testbox CI environment.
Changed gate:
```sh
pnpm crabbox:run -- \
--idle-timeout 90m \
--ttl 240m \
--timing-json \
--shell -- \
"env CI=1 NODE_OPTIONS=--max-old-space-size=4096 OPENCLAW_TEST_PROJECTS_PARALLEL=6 OPENCLAW_VITEST_MAX_WORKERS=1 OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=900000 pnpm test:changed"
```
Full suite:
```sh
pnpm crabbox:run -- \
--idle-timeout 90m \
--ttl 240m \
--timing-json \
--shell -- \
"env CI=1 NODE_OPTIONS=--max-old-space-size=4096 OPENCLAW_TEST_PROJECTS_PARALLEL=6 OPENCLAW_VITEST_MAX_WORKERS=1 OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=900000 pnpm test"
```
Focused rerun:
```sh
pnpm crabbox:run -- \
--idle-timeout 90m \
--ttl 240m \
--timing-json \
--shell -- \
"env CI=1 NODE_OPTIONS=--max-old-space-size=4096 OPENCLAW_VITEST_MAX_WORKERS=1 OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=900000 pnpm test <path-or-filter>"
```
Read the JSON summary. Useful fields:
- `provider`: `aws`
- `leaseId`: `cbx_...`
- `syncDelegated`: `false`
- `commandPhases`: populated when the command prints `CRABBOX_PHASE:<name>`
- `commandMs` / `totalMs`
- `exitCode`
Crabbox should stop one-shot AWS leases automatically after the run. Verify
cleanup when a run fails, is interrupted, or the command output is unclear:
```sh
../crabbox/bin/crabbox list --provider aws
```
## Blacksmith Testbox Through Crabbox
Use this for OpenClaw maintainer broad/heavy `pnpm` gates when the prepared CI
environment is the right proof surface:
```sh
node scripts/crabbox-wrapper.mjs run \
--provider blacksmith-testbox \
--blacksmith-org openclaw \
--blacksmith-workflow .github/workflows/ci-check-testbox.yml \
--blacksmith-job check \
--blacksmith-ref main \
--idle-timeout 90m \
--ttl 240m \
--timing-json \
-- \
CI=1 NODE_OPTIONS=--max-old-space-size=4096 OPENCLAW_TEST_PROJECTS_PARALLEL=6 OPENCLAW_VITEST_MAX_WORKERS=1 OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=900000 OPENCLAW_TESTBOX=1 OPENCLAW_TESTBOX_REMOTE_RUN=1 pnpm check:changed
```
Read the JSON summary and the Testbox line. Useful fields:
- `provider`: `blacksmith-testbox`
- `leaseId`: `tbx_...`
- `syncDelegated`: `true`
- `syncPhases`: delegated/skipped because Blacksmith owns checkout/sync
- Actions run URL/id from the Testbox output
- `exitCode`
`blacksmith testbox list` may hide hydrating or ready boxes. Use:
```sh
blacksmith testbox list --all
blacksmith testbox status <tbx_id>
```
## Observability Flags
Use these on debugging runs before inventing ad hoc logging:
- `--preflight`: prints run context, workspace mode, SSH target, remote user/cwd,
and target-specific tool probes. Defaults cover `git`, `tar`, `node`, `npm`,
`corepack`, `pnpm`, `yarn`, `bun`, `docker`, plus POSIX
`sudo`/`apt`/`bubblewrap` and native Windows
`powershell`/`execution_policy`/`longpaths`/`temp`/`pwsh`. Add
`--preflight-tools node,bun,docker`, `CRABBOX_PREFLIGHT_TOOLS`, or repo
`run.preflightTools` to replace the list. `default` expands built-ins; `none`
prints only the workspace summary. Preflight is diagnostic only; install
toolchains through Actions hydration, images, devcontainer/Nix/mise/asdf, or
the run script. On `blacksmith-testbox`, this prints a delegated-unsupported
note because the workflow owns setup.
- `CRABBOX_ENV_ALLOW=NAME,...`: forwards only listed local env vars for direct
providers and prints `set len=N secret=true` style summaries. On
`blacksmith-testbox`, env forwarding is unsupported; put secrets in the
Testbox workflow instead.
- `--env-from-profile <file>` plus `--allow-env NAME`: loads simple
`export NAME=value` / `NAME=value` lines from a local profile without
executing it, then forwards only allowlisted names. `--allow-env` is
repeatable and comma-separated. Profile values override ambient allowlisted
env values for that run. Direct POSIX, WSL2, and native Windows runs are
supported; delegated providers are not. Crabbox probes the uploaded profile
remotely and prints redacted presence/length metadata before the command.
- `--env-helper <name>`: with `--env-from-profile` on POSIX SSH targets,
persists `.crabbox/env/<name>` and `.crabbox/env/<name>.env` so follow-up
commands on the same lease can run through `./.crabbox/env/<name> <command>`.
Use only on leases you control; the profile stays until cleanup, lease reset,
or `--full-resync`.
- `--script <file>` / `--script-stdin`: upload a local script into
`.crabbox/scripts/` and execute it on the remote box. Shebang scripts execute
directly on POSIX; scripts without a shebang run through `bash`. Native
Windows uploads run through Windows PowerShell, and Crabbox appends `.ps1`
when needed. Arguments after `--` become script args.
- `--fresh-pr owner/repo#123|URL|number`: skip dirty local sync and create a
fresh remote checkout of the GitHub PR. Bare numbers use the current repo's
GitHub origin. Add `--apply-local-patch` only when the current local
`git diff --binary HEAD` should be applied on top of that PR checkout.
- `--full-resync` / `--fresh-sync`: reset a stale direct-provider workdir
before syncing. Use after sync fingerprints look wrong, SSH times out before
sync, or rsync watchdog output suggests it. It is redundant with
`--fresh-pr`, incompatible with `--no-sync`, and unsupported by delegated
providers.
- `--capture-stdout <path>` / `--capture-stderr <path>`: write remote streams to
local files and keep binary/noisy output out of retained logs. Parent
directories must already exist. These are direct-provider only.
- `--capture-on-fail`: on non-zero direct-provider exits, downloads
`.crabbox/captures/*.tar.gz` with `test-results`, `playwright-report`,
`coverage`, JUnit XML, and nearby logs. Treat as secret-bearing until reviewed.
- `--keep-on-failure`: leave a failed one-shot lease alive for live debugging
until idle/TTL expiry. Useful on direct providers and delegated one-shots.
- `--timing-json`: final machine-readable timing. Add
`echo CRABBOX_PHASE:install`, `CRABBOX_PHASE:test`, etc. in long shell
commands; direct providers and Blacksmith Testbox both report them as
`commandPhases`.
Live-provider debug template for direct AWS/Hetzner leases:
```sh
mkdir -p .crabbox/logs
pnpm crabbox:run -- --provider aws \
--preflight \
--allow-env OPENAI_API_KEY,OPENAI_BASE_URL \
--timing-json \
--capture-stdout .crabbox/logs/live-provider.stdout.log \
--capture-stderr .crabbox/logs/live-provider.stderr.log \
--capture-on-fail \
--shell -- \
"echo CRABBOX_PHASE:install; pnpm install --frozen-lockfile; echo CRABBOX_PHASE:test; pnpm test:live"
```
Do not pass `--capture-*`, `--download`, `--checksum`, `--force-sync-large`, or
`--sync-only` to delegated providers. Also do not pass `--script*`,
`--fresh-pr`, `--full-resync`, or `--env-helper` there. Crabbox rejects these
because the provider owns sync or command transport. `--keep-on-failure` is OK
for delegated one-shots when you need to inspect a failed lease.
## Efficient Bug E2E Verification
Use the smallest Crabbox lane that proves the reported user path, not just the
touched code. Aim for one after-fix E2E proof before commenting, closing, or
opening a PR for a user-visible bug.
When the user says "test in Crabbox", do not simply copy tests to the remote
box and run them there. Crabbox is for remote real-scenario proof: copy or
install OpenClaw as the user would, run the same setup/update/CLI/Gateway/API
call that failed, and capture behavior from that entrypoint. For regressions or
bug reports, prove the broken state first when feasible, then run the same
scenario after the fix.
Pick the lane by symptom:
- Docker/setup/install bug: build a package tarball and run the matching
`scripts/e2e/*-docker.sh` or package script. This proves npm packaging,
install paths, runtime deps, config writes, and container behavior.
- Provider/model/auth bug: prefer true live E2E. Use the configured secret
workflow, then inject the single needed key into Crabbox if needed. Scrub
unrelated provider env vars in the child command so interactive defaults do
not drift to another provider. If only a dummy key is used, label the proof
narrowly, e.g. "UI/install path only; live provider auth not exercised."
- Channel delivery bug: use the channel Docker/live lane when available; include
setup, config, gateway start, send/receive or agent-turn proof, and redacted
logs.
- Gateway/session/tool bug: prefer an end-to-end CLI or Gateway RPC command that
creates real state and inspects the resulting files/API output.
- Pure parser/config bug: targeted tests may be enough, but still run a
Crabbox command when OS, package, Docker, secrets, or service lifecycle could
change behavior.
Efficient flow:
1. Reproduce or prove the pre-fix symptom from the real user-facing entrypoint
when feasible. If the issue cannot be reproduced, capture the exact command
and observed behavior instead.
2. Patch locally and run narrow local tests for edit speed.
3. Run one Crabbox E2E command that starts from the user-facing entrypoint:
package install, Docker setup, onboarding, channel add, gateway start, or
agent turn as appropriate.
4. Record proof as: Testbox id, command, environment shape, redacted secret
source, and copied success/failure output.
5. If the issue says "cannot reproduce", ask for the missing config/log fields
that would distinguish the tested path from the reporter's path.
Keep it efficient:
- Reuse existing E2E scripts and helper assertions before writing ad hoc shell.
- Use `--script <file>` or `--script-stdin` for multi-line E2E commands instead
of quote-heavy `--shell` strings on direct SSH providers.
- Use `--fresh-pr <pr>` when validating an upstream PR in isolation from the
local dirty tree. Add `--apply-local-patch` only when testing a local fixup on
top of that PR.
- Use `--full-resync` before replacing a warmed direct-provider lease when the
remote workdir or sync fingerprint appears stale.
- Use one-shot Crabbox for a single proof; use a reusable Testbox only when
several commands must share built images, installed packages, or live state.
- Prefer `OPENCLAW_CURRENT_PACKAGE_TGZ` with Docker/package lanes when testing a
candidate tarball; prefer the repo's package helper instead of direct source
execution when the bug might be packaging/install related.
- Keep secrets redacted. It is fine to report key presence, source, and length;
never print secret values.
- Include `--timing-json` on broad or flaky runs when command duration or sync
behavior matters.
Before/after PR proof on delegated Testbox:
- For PRs that should prove "broken before, fixed after", compare base and PR
on the same Testbox when practical. Fetch both refs, create detached temp
worktrees under `/tmp`, install in each, then run the same harness twice.
- Do not checkout base/PR refs in the synced repo root. Delegated Testbox sync
may leave the root dirty with local files; `git checkout` can abort or mix
proof state.
- Temp harness files under `/tmp` do not resolve repo packages by default. Put
the harness inside the worktree, or in ESM use
`createRequire(path.join(process.cwd(), "package.json"))` before requiring
workspace deps such as `@lydell/node-pty`.
- For full-screen TUI/CLI bugs, a PTY harness is stronger than helper-only
assertions. Use a real PTY, wait for visible lifecycle markers, send input,
then send control keys and assert process exit/stuck behavior.
- When validating a rebased local branch before push, remember delegated sync
usually validates synced file content on a detached dirty checkout, not a
remote commit object. Record the local head SHA, changed files, Testbox id,
and final success markers; after pushing, ensure the pushed SHA has the same
file content.
- If GitHub CI is still queued but the exact changed content passed Testbox
`pnpm check:changed`, `pnpm check:test-types`, and the real E2E proof, it is
reasonable to merge once required checks allow it. Note any still-running
unrelated shards in the proof comment instead of waiting forever.
Interactive CLI/onboarding:
- For full-screen or prompt-heavy CLI flows, run the target command inside tmux
on the Crabbox and drive it with `tmux send-keys`; capture proof with
`tmux capture-pane`, redacted through `sed`.
- Prefer deterministic arrow navigation over search typing for Clack-style
searchable selects. Raw `send-keys -l openai` may not trigger filtering in a
tmux pane; inspect option order locally or on-box and send exact Down/Enter
sequences.
- Isolate mutable state with `OPENCLAW_STATE_DIR=$(mktemp -d)`. Plugin npm
installs live under that state dir (`npm/node_modules/...`), not under
`OPENCLAW_CONFIG_DIR`. Verify downloads by checking the state dir, package
lock, and installed package metadata.
- To test automatic setup installs against local package artifacts, use
`OPENCLAW_ALLOW_PLUGIN_INSTALL_OVERRIDES=1` plus
`OPENCLAW_PLUGIN_INSTALL_OVERRIDES='{"plugin-id":"npm-pack:/tmp/plugin.tgz"}'`.
Pack with `npm pack`, set an isolated `OPENCLAW_STATE_DIR`, and verify the
package under `npm/node_modules`. Overrides are test-only and must not be
treated as official/trusted-source installs.
- For OpenAI/Codex onboarding proof, the useful markers are the UI line
`Installed Codex plugin`, `npm/node_modules/@openclaw/codex`, and the
package-lock entry showing the bundled `@openai/codex` dependency. A dummy
OpenAI-shaped key can prove only UI/install behavior; it is not live auth.
## Reuse And Keepalive
For most Crabbox calls, one-shot is enough. Use reuse only when you need
multiple manual commands on the same hydrated box.
If Crabbox returns a reusable id or you intentionally keep a lease:
```sh
pnpm crabbox:run -- --id <cbx_id-or-slug> --no-sync --timing-json --shell -- "pnpm test <path>"
```
Stop boxes you created before handoff:
```sh
pnpm crabbox:stop -- <id-or-slug>
blacksmith testbox stop --id <tbx_id>
```
## Interactive Desktop And WebVNC
Prefer WebVNC for human inspection because the browser portal can preload the
lease VNC password and avoids a native VNC client's copy/paste/password dance.
Use native `crabbox vnc` only when WebVNC is unavailable, the browser portal is
broken, or the user explicitly wants a local VNC client.
Common desktop flow:
```sh
../crabbox/bin/crabbox warmup --provider hetzner --desktop --browser --class standard --idle-timeout 60m --ttl 240m
../crabbox/bin/crabbox desktop launch --provider hetzner --id <cbx_id-or-slug> --browser --url https://example.com --webvnc --open --take-control
```
Useful WebVNC commands:
```sh
../crabbox/bin/crabbox webvnc --provider hetzner --id <cbx_id-or-slug> --open --take-control
../crabbox/bin/crabbox webvnc daemon start --provider hetzner --id <cbx_id-or-slug> --open --take-control
../crabbox/bin/crabbox webvnc daemon status --provider hetzner --id <cbx_id-or-slug>
../crabbox/bin/crabbox webvnc daemon stop --provider hetzner --id <cbx_id-or-slug>
../crabbox/bin/crabbox webvnc status --provider hetzner --id <cbx_id-or-slug>
../crabbox/bin/crabbox webvnc reset --provider hetzner --id <cbx_id-or-slug> --open --take-control
../crabbox/bin/crabbox desktop doctor --provider hetzner --id <cbx_id-or-slug>
../crabbox/bin/crabbox desktop click --provider hetzner --id <cbx_id-or-slug> --x 640 --y 420
../crabbox/bin/crabbox desktop paste --provider hetzner --id <cbx_id-or-slug> --text "user@example.com"
../crabbox/bin/crabbox desktop key --provider hetzner --id <cbx_id-or-slug> ctrl+l
../crabbox/bin/crabbox artifacts collect --id <cbx_id-or-slug> --all --output artifacts/<slug>
../crabbox/bin/crabbox artifacts publish --dir artifacts/<slug> --pr <number>
```
`desktop launch --webvnc --open` is usually the nicest one-shot: it starts the
browser/app inside the visible session, bridges the lease into the authenticated
WebVNC portal, and opens the portal. Keep browsers windowed for human QA; use
`--fullscreen` only for capture/video workflows.
For human handoff, include `--take-control` so the opened portal viewer gets
keyboard/mouse control automatically instead of landing as an observer.
Human handoff preflight:
- Do not assume a visible desktop or launched browser means the repo CLI/app is
installed, built, or on the interactive terminal's `PATH`.
- Before handing WebVNC to a human tester, prove the expected command from the
same kept lease and from a neutral directory such as `~`.
- If the handoff needs repo-local code, sync/build/link it explicitly on that
lease. Source-tree CLIs often need build output before a symlink works.
- Prefer a real `command -v <expected-command> && <expected-command> --version`
check over a repo-root-only `pnpm ...` command.
Generic handoff repair pattern:
```sh
../crabbox/bin/crabbox run --id <cbx_id-or-slug> --full-resync --shell -- \
"set -euo pipefail
pnpm install --frozen-lockfile
pnpm build
sudo ln -sf \"\$PWD/<cli-entry>\" /usr/local/bin/<expected-command>
cd ~
command -v <expected-command>
<expected-command> --version"
```
## If Crabbox Fails
Keep the fallback narrow. First decide whether the failure is Crabbox itself,
the brokered AWS lease, Blacksmith/Testbox, repo hydration, sync, or the test
command.
Fast checks:
```sh
command -v crabbox
../crabbox/bin/crabbox --version
pnpm crabbox:run -- --help | sed -n '1,140p'
../crabbox/bin/crabbox doctor
command -v blacksmith
blacksmith --version
blacksmith testbox list
```
Common Crabbox-only failures:
- Provider missing or old CLI: use `../crabbox/bin/crabbox` from the sibling
repo, or update/install Crabbox before retrying.
- Bad local config: inspect `.crabbox.yaml`, `crabbox config show`, and
`crabbox whoami`; normal OpenClaw proof should use brokered AWS without
asking for cloud keys.
- Slug/claim confusion: use the raw `cbx_...` / `tbx_...` id, or run one-shot
without `--id`.
- Sync/timing bug: add `--debug --timing-json`; capture the final JSON and the
printed Actions URL. Large sync warnings now include top source directories
by file count and a hint to update `.crabboxignore` / `sync.exclude`; inspect
those before reaching for `--force-sync-large`. Quiet rsync watchdogs and SSH
timeouts now print `next_action=` hints; follow them, usually `--full-resync`
first and a fresh lease second.
- Cleanup uncertainty: run `crabbox list --provider aws`; for explicit
Blacksmith runs, use `blacksmith testbox list` and stop only boxes you
created.
- Testbox queued/capacity pressure: do not retry Blacksmith repeatedly. Rerun
once without `--provider` so `.crabbox.yaml` routes to brokered AWS, or report
the Blacksmith blocker if Testbox itself is the requested proof.
If brokered AWS cannot dispatch, sync, attach, or stop, retry once with
`--debug` and `--timing-json`:
```sh
pnpm crabbox:run -- --debug --timing-json -- \
CI=1 NODE_OPTIONS=--max-old-space-size=4096 OPENCLAW_TEST_PROJECTS_PARALLEL=6 OPENCLAW_VITEST_MAX_WORKERS=1 OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=900000 pnpm test:changed
```
Full suite:
```sh
pnpm crabbox:run -- --debug --timing-json -- \
CI=1 NODE_OPTIONS=--max-old-space-size=4096 OPENCLAW_TEST_PROJECTS_PARALLEL=6 OPENCLAW_VITEST_MAX_WORKERS=1 OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=900000 pnpm test
```
Auth fallback, only when `blacksmith` says auth is missing:
```sh
blacksmith auth login --non-interactive --organization openclaw
```
Raw Blacksmith footguns:
- Run from repo root. The CLI syncs the current directory.
- Save the returned `tbx_...` id in the session.
- Reuse that id for focused reruns; stop it before handoff.
- Raw commit SHAs are not reliable `warmup --ref` refs; use a branch or tag.
- Treat `blacksmith testbox list` as cleanup diagnostics, not a shared reusable
queue.
Use Blacksmith only when the task is specifically about Testbox, brokered AWS
is unavailable, or an explicit comparison is needed. If Blacksmith is down or
quota-limited, do not keep probing it; stay on brokered AWS and note the
delegated-provider outage.
## Blacksmith Backend Notes
Crabbox Blacksmith backend delegates setup to:
- org: `openclaw`
- workflow: `.github/workflows/ci-check-testbox.yml`
- job: `check`
- ref: `main` unless testing a branch/tag intentionally
The hydration workflow owns checkout, Node/pnpm setup, dependency install,
secrets, ready marker, and keepalive. Crabbox owns dispatch, sync, SSH command
execution, timing, logs/results, and cleanup.
Minimal Blacksmith-backed Crabbox run, from repo root:
```sh
pnpm crabbox:run -- --provider blacksmith-testbox --timing-json -- \
CI=1 NODE_OPTIONS=--max-old-space-size=4096 OPENCLAW_TEST_PROJECTS_PARALLEL=6 OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test:changed
```
Use direct Blacksmith only when Crabbox is the broken layer and you are
isolating a Crabbox bug. Prefer direct `blacksmith testbox list` for cleanup
diagnostics, not as a reusable work queue.
Important Blacksmith footguns:
- Always run from repo root. The CLI syncs the current directory.
- Raw commit SHAs are not reliable `warmup --ref` refs; use a branch or tag.
- If auth is missing and browser auth is acceptable:
```sh
blacksmith auth login --non-interactive --organization openclaw
```
## Brokered AWS
Use AWS for normal OpenClaw remote proof. The repo `.crabbox.yaml` already
selects brokered AWS, so omit `--provider` unless you are testing a different
provider deliberately.
```sh
pnpm crabbox:warmup -- --class beast --market on-demand --idle-timeout 90m
pnpm crabbox:hydrate -- --id <cbx_id-or-slug>
pnpm crabbox:run -- --id <cbx_id-or-slug> --timing-json --shell -- "env NODE_OPTIONS=--max-old-space-size=4096 OPENCLAW_TEST_PROJECTS_PARALLEL=6 OPENCLAW_VITEST_MAX_WORKERS=1 OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=900000 pnpm test:changed"
pnpm crabbox:stop -- <cbx_id-or-slug>
```
Install/auth for owned Crabbox if needed:
```sh
brew install openclaw/tap/crabbox
crabbox login --url https://crabbox.openclaw.ai --provider aws
```
New users should self-resolve broker auth before anyone asks for AWS keys:
```sh
crabbox config show
crabbox doctor
crabbox whoami
```
- If broker auth is missing, run `crabbox login --url https://crabbox.openclaw.ai --provider aws`.
- If the CLI asks for `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, or AWS
profile setup during normal OpenClaw validation, assume the agent selected
the wrong path. Use brokered `crabbox login` or an existing brokered lease
before asking the user for cloud credentials.
- Ask for AWS keys only for explicit direct-provider/account administration,
not for normal brokered OpenClaw proof.
- Trusted automation may still use
`printf '%s' "$CRABBOX_COORDINATOR_TOKEN" | crabbox login --url https://crabbox.openclaw.ai --provider aws --token-stdin`.
macOS config lives at:
```text
~/Library/Application Support/crabbox/config.yaml
```
It should include `broker.url`, `broker.token`, and usually `provider: aws`
for OpenClaw lanes. Let that config drive normal validation.
### Interactive Desktop / WebVNC
For human desktop demos, prefer `webvnc` over native `vnc` and keep the remote
desktop visible/windowed. Do not fullscreen the remote browser or hide the XFCE
panel/window chrome unless the explicit goal is video/capture output. After
launch, verify a screenshot shows the desktop panel plus browser title bar. If
Chrome is fullscreen, toggle it back with:
```sh
crabbox run --id <lease> --shell -- 'DISPLAY=:99 xdotool search --onlyvisible --class google-chrome windowactivate key F11'
```
## Diagnostics
```sh
crabbox status --id <id-or-slug> --wait
crabbox inspect --id <id-or-slug> --json
crabbox sync-plan
crabbox history --limit 20
crabbox history --lease <id-or-slug>
crabbox attach <run_id>
crabbox events <run_id> --json
crabbox logs <run_id>
crabbox results <run_id>
crabbox cache stats --id <id-or-slug>
crabbox ssh --id <id-or-slug>
blacksmith testbox list
```
Use `--debug` on `run` when measuring sync timing.
Use `--timing-json` on warmup, hydrate, and run when comparing backends.
Use `--market spot|on-demand` only on AWS warmup/one-shot runs.
## Failure Triage
- Crabbox cannot find provider: verify `../crabbox/bin/crabbox --help` lists
the provider selected by `.crabbox.yaml`; update Crabbox before falling back.
- Hydration stuck or failed: open the printed GitHub Actions run URL and inspect
the hydration step.
- Sync failed: rerun with `--debug`; check changed-file count and whether the
checkout is dirty.
- Command failed: rerun only the failing shard/file first. Do not rerun a full
suite until the focused failure is understood.
- Cleanup uncertain: `crabbox list --provider aws`; for explicit Blacksmith
runs, use `blacksmith testbox list` and stop owned `tbx_...` leases you
created.
- Crabbox broken but Blacksmith works: use the direct Blacksmith fallback above,
then file/fix the Crabbox issue.
## Boundary
Do not add OpenClaw-specific setup to Crabbox itself. Put repo setup in the
hydration workflow and keep Crabbox generic around lease, sync, command
execution, logs/results, timing, and cleanup.

View File

@ -0,0 +1,140 @@
---
name: release-peekaboo
description: "Peekaboo release: notarization, npm/GitHub release, appcast, verify, closeout."
metadata: {"clawdbot":{"emoji":"👁️","requires":{"bins":["pnpm","op","tmux","gh","xcrun","jq","node","npm"]}}}
---
# Peekaboo Release
Release `~/Projects/Peekaboo` as the npm package `@steipete/peekaboo` plus signed/notarized macOS app assets.
Use `$one-password`, `$browser-use`, `$npm`, `$autoreview`, and repo `AGENTS.md` rules. Load `$release-private` if it exists before resolving Peter-owned credential locators. Read `$npm` before any npm auth, token, or publish recovery work. Keep all `op` secret work inside one persistent tmux session. Never print `.p8`, npm tokens, passwords, or OTPs.
## Current Secrets
- Peter-owned credential item names, key ids, issuer ids, keychain paths, and npm token locators live in `$release-private`.
- Required ASC fields: `key_id`, `issuer_id`, `private_key_p8`.
- Stale/revoked key symptom: `xcrun notarytool submit` fails with `HTTP status code: 401. Unauthenticated`.
- All ASC fields must come from the same current item; do not mix profile values with 1Password refs.
Sparkle key:
- Repo `.mac-release.env` has the current fallback.
- Do not set `SPARKLE_PRIVATE_KEY_FILE` for normal releases.
Developer ID release keychain:
- Resolve the release keychain item/path from `$release-private`.
- If macOS shows `codesign wants to use the release keychain`, enter the keychain item password, not the Developer ID `.p12` password.
- The Developer ID certificate password is only for importing the `.p12` while creating the keychain.
- After setup/import, run `security unlock-keychain` and `security set-key-partition-list -S apple-tool:,apple:,codesign: -s -k "$KEYCHAIN_PASSWORD" "$KEYCHAIN_PATH"` so `codesign` can use the identity without GUI prompts.
npm publish token:
- Resolve token/TOTP locators from `$release-private`.
- Use `$npm` rules. Run inside the same tmux session, write only a temp npmrc, delete it immediately, and use the `npmjs` TOTP item for web auth if npm prompts.
- Do not create short-lived/granular bypass tokens for a normal Peekaboo publish. They add cleanup risk and did not help the 3.2.1 slow-upload/web-auth path.
## Notary Credential Check
Use the service account from `$release-private` first. Put the token in the tmux environment without printing it:
```bash
# Resolve SERVICE_ACCOUNT_TOKEN from $release-private first.
tmux -S "$SOCKET" set-environment -t "$SESSION" OP_SERVICE_ACCOUNT_TOKEN "$SERVICE_ACCOUNT_TOKEN"
```
Create a temp env file with service-account refs from `$release-private`:
```text
APP_STORE_CONNECT_API_KEY_P8=<1Password ref from release-private>
APP_STORE_CONNECT_KEY_ID=<1Password ref from release-private>
APP_STORE_CONNECT_ISSUER_ID=<1Password ref from release-private>
```
Before a release, verify shape and Apple auth without printing values:
```bash
op run --env-file "$ENVFILE" -- bash -c '
set -euo pipefail
KEY_FILE="/tmp/AuthKey_${APP_STORE_CONNECT_KEY_ID}.p8"
printf "%s\n" "$APP_STORE_CONNECT_API_KEY_P8" > "$KEY_FILE"
chmod 600 "$KEY_FILE"
xcrun notarytool history \
--key "$KEY_FILE" \
--key-id "$APP_STORE_CONNECT_KEY_ID" \
--issuer "$APP_STORE_CONNECT_ISSUER_ID" \
--output-format json >/dev/null
rm -f "$KEY_FILE"
'
```
Peekaboo forces `notarytool submit --no-s3-acceleration`; the default S3 accelerated upload path can return a misleading `401` even when `history` auth succeeds.
If both `history` and non-S3 `submit` fail, suspect wrong access level or stale key. Browser route:
1. Use `$browser-use` real Chrome profile.
2. Open `https://appstoreconnect.apple.com/access/integrations/api`.
3. Generate Team Key named `Peekaboo Release <version>` with `Admin` access.
4. Download `.p8` once from the key row.
5. Store immediately into the private credential map; verify `notarytool history`; delete `~/Downloads/AuthKey_<key_id>.p8`.
6. Revoke the older Peekaboo release key after the new key validates.
## Release Flow
1. Start on clean `main`; pull ff-only if needed.
2. Set version in:
- `package.json`
- `version.json`
- `Apps/CLI/Sources/Resources/version.json`
- README npm badge
- `Core/PeekabooCore/Sources/PeekabooAgentRuntime/MCP/PeekabooMCPVersion.swift`
- Xcode marketing versions under `Apps/*`
3. Date `CHANGELOG.md` and `Apps/CLI/CHANGELOG.md` for the release.
4. Run focused proof or release script preflight. Release gates must be warning-free.
5. Use `$autoreview` before commit unless the change is trivial/docs-only.
6. Commit release prep with `committer`.
7. Push `main`.
8. Run:
```bash
op run --env-file "$ENVFILE" -- \
bash -c 'printf "y\n" | ./scripts/release-binaries.sh --create-github-release --publish-npm'
```
The script builds universal CLI, npm package, signed/notarized app zip, appcast, checksums, draft GitHub release, and npm publish.
Use a non-login shell: profile exports can replace current 1Password ASC IDs with stale values while leaving the current `.p8`, producing a misleading `401`.
Notarized releases must sign with `Developer ID Application: Peter Steinberger (Y5PE65HELJ)`, not `Apple Development`. If your shell has `SIGN_IDENTITY` exported for CLI builds, override it for the release command.
If npm upload is slow and TOTP expires, use the stored npm token through a temp npmrc and complete npm web auth immediately when prompted with the configured TOTP. Do not create granular bypass tokens for this; if one was created by mistake, delete it before closeout.
## Verify
Required before closeout:
```bash
npm view @steipete/peekaboo@<version> version dist-tags dist.tarball dist.integrity time --json
(cd /tmp && npm exec --yes --package=@steipete/peekaboo@<version> -- peekaboo --version)
gh release view v<version> --repo openclaw/Peekaboo --json tagName,isDraft,isPrerelease,url,assets,body
xmllint --noout appcast.xml
git status --short --branch
```
Confirm:
- npm version exists and `latest` points to it.
- npm-downloaded CLI reports the release version from a neutral cwd.
- GitHub release/tag/assets exist; release body is from changelog.
- app zip asset exists and appcast points at `v<version>`.
- `appcast.xml` changes are committed and pushed.
- Publish draft release if the script leaves it draft.
## Closeout
1. Add next patch `Unreleased` section to root and CLI changelogs.
2. Commit with `committer "docs(changelog): open <next-version>" CHANGELOG.md Apps/CLI/CHANGELOG.md`.
3. Push.
4. Watch release/homebrew/CI workflows if triggered.
5. `git checkout main && git pull --ff-only && git status --short --branch`.
6. Clear tmux `OP_SERVICE_ACCOUNT_TOKEN`, remove temp env/key files, and final with what landed.

50
.crabbox.yaml Normal file
View File

@ -0,0 +1,50 @@
profile: peekaboo-check
provider: aws
class: standard
capacity:
market: spot
strategy: most-available
fallback: on-demand-after-120s
hints: true
regions:
- eu-west-1
- eu-west-2
- eu-central-1
- us-east-1
- us-west-2
actions:
workflow: .github/workflows/crabbox-hydrate.yml
job: hydrate
ref: main
runnerLabels:
- crabbox
- openclaw
- peekaboo
runnerVersion: latest
ephemeral: true
aws:
region: eu-west-1
rootGB: 160
sync:
delete: true
checksum: false
gitSeed: true
fingerprint: true
baseRef: main
exclude:
- .artifacts
- .codex
- .DS_Store
- coverage
- dist
- node_modules
- .build
env:
allow:
- CI
- NODE_OPTIONS
- PNPM_*
- NPM_CONFIG_*
ssh:
user: crabbox
port: "2222"

9
.github/CODEOWNERS vendored Normal file
View File

@ -0,0 +1,9 @@
# Protect ownership and automation rules.
/.github/CODEOWNERS @openclaw/openclaw-secops
/.github/workflows/ @openclaw/openclaw-secops
/package.json @openclaw/openclaw-secops
/pnpm-lock.yaml @openclaw/openclaw-secops
/Package.swift @openclaw/openclaw-secops
/.github/actionlint.yaml @openclaw/openclaw-secops
/.agents/skills/ @openclaw/openclaw-secops
/.crabbox.yaml @openclaw/openclaw-secops

5
.github/actionlint.yaml vendored Normal file
View File

@ -0,0 +1,5 @@
self-hosted-runner:
labels:
- crabbox
- openclaw
- peekaboo

View File

@ -13,7 +13,7 @@ on:
jobs:
macos-host:
runs-on: macos-latest
runs-on: macos-15
steps:
- uses: actions/checkout@v6
with:
@ -26,7 +26,7 @@ jobs:
run: swift test
apple-simulators:
runs-on: macos-latest
runs-on: macos-15
needs: macos-host
strategy:
matrix:

126
.github/workflows/crabbox-hydrate.yml vendored Normal file
View File

@ -0,0 +1,126 @@
name: Crabbox Hydrate
on:
workflow_dispatch:
inputs:
crabbox_id:
description: "Crabbox lease ID"
required: true
type: string
ref:
description: "Git ref to hydrate"
required: false
type: string
crabbox_runner_label:
description: "Dynamic Crabbox runner label"
required: true
type: string
crabbox_job:
description: "Hydration job identifier expected by Crabbox"
required: false
default: "hydrate"
type: string
crabbox_keep_alive_minutes:
description: "Minutes to keep the hydrated job alive"
required: false
default: "90"
type: string
permissions:
contents: read
env:
NODE_VERSION: "24"
PNPM_VERSION: "11.1.2"
jobs:
hydrate:
name: hydrate
runs-on: [self-hosted, crabbox, openclaw, peekaboo, "${{ inputs.crabbox_runner_label }}"]
timeout-minutes: 120
steps:
- uses: actions/checkout@v6
with:
ref: ${{ inputs.ref || github.ref }}
- uses: pnpm/action-setup@v6.0.8
with:
version: ${{ env.PNPM_VERSION }}
- uses: actions/setup-node@v6
with:
node-version: ${{ env.NODE_VERSION }}
cache: pnpm
- name: Prepare pnpm and Swift workspace
shell: bash
run: |
set -euo pipefail
git fetch --no-tags --depth=50 origin "+refs/heads/main:refs/remotes/origin/main"
pnpm install --frozen-lockfile
node --version
pnpm --version
swift --version
- name: Mark Crabbox ready
shell: bash
env:
CRABBOX_ID: ${{ inputs.crabbox_id }}
CRABBOX_JOB: ${{ inputs.crabbox_job }}
run: |
set -euo pipefail
job="${CRABBOX_JOB}"
if [ -z "$job" ]; then job=hydrate; fi
case "$CRABBOX_ID" in
''|*[!A-Za-z0-9._-]*)
echo "Invalid crabbox_id" >&2
exit 2
;;
esac
mkdir -p "$HOME/.crabbox/actions"
state="$HOME/.crabbox/actions/${CRABBOX_ID}.env"
env_file="$HOME/.crabbox/actions/${CRABBOX_ID}.env.sh"
{
for key in CI GITHUB_ACTIONS GITHUB_WORKSPACE GITHUB_REPOSITORY GITHUB_RUN_ID GITHUB_RUN_NUMBER GITHUB_RUN_ATTEMPT GITHUB_REF GITHUB_REF_NAME GITHUB_SHA GITHUB_EVENT_NAME GITHUB_ACTOR RUNNER_OS RUNNER_ARCH RUNNER_TEMP RUNNER_TOOL_CACHE PATH; do
value="${!key-}"
if [ -n "$value" ]; then
printf 'export %s=%q\n' "$key" "$value"
fi
done
} > "${env_file}.tmp"
mv "${env_file}.tmp" "$env_file"
tmp="${state}.tmp"
{
echo "WORKSPACE=${GITHUB_WORKSPACE}"
echo "RUN_ID=${GITHUB_RUN_ID}"
echo "JOB=${job}"
echo "ENV_FILE=${env_file}"
echo "READY_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
} > "$tmp"
mv "$tmp" "$state"
- name: Keep Crabbox job alive
shell: bash
env:
CRABBOX_ID: ${{ inputs.crabbox_id }}
CRABBOX_KEEP_ALIVE_MINUTES: ${{ inputs.crabbox_keep_alive_minutes }}
run: |
set -euo pipefail
case "$CRABBOX_ID" in
''|*[!A-Za-z0-9._-]*)
echo "Invalid crabbox_id" >&2
exit 2
;;
esac
minutes="${CRABBOX_KEEP_ALIVE_MINUTES}"
case "$minutes" in
''|*[!0-9]*) minutes=90 ;;
esac
stop="$HOME/.crabbox/actions/${CRABBOX_ID}.stop"
deadline=$(( $(date +%s) + minutes * 60 ))
while [ "$(date +%s)" -lt "$deadline" ]; do
if [ -f "$stop" ]; then
exit 0
fi
sleep 15
done

View File

@ -13,7 +13,7 @@ concurrency:
jobs:
peekaboo-core:
name: PeekabooCore build & tests
runs-on: macos-latest
runs-on: macos-15
env:
PEEKABOO_INCLUDE_AUTOMATION_TESTS: "false"
RUN_AUTOMATION_TESTS: "false"
@ -127,7 +127,7 @@ jobs:
peekaboo-cli:
name: Peekaboo CLI build & tests
runs-on: macos-latest
runs-on: macos-15
needs: peekaboo-core
env:
PEEKABOO_INCLUDE_AUTOMATION_TESTS: "false"
@ -236,7 +236,7 @@ jobs:
tachikoma:
name: Tachikoma build & tests
runs-on: macos-latest
runs-on: macos-15
needs: peekaboo-cli
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
@ -347,7 +347,7 @@ jobs:
mac-apps:
name: Build macOS apps (Peekaboo + Inspector)
runs-on: macos-latest
runs-on: macos-15
needs: [peekaboo-cli, tachikoma]
steps:
- uses: actions/checkout@v6
@ -407,7 +407,7 @@ jobs:
lint:
name: SwiftLint (core + CLI)
runs-on: macos-latest
runs-on: macos-15
needs: [peekaboo-cli, tachikoma, mac-apps]
steps:
- uses: actions/checkout@v6

View File

@ -34,6 +34,12 @@
- Use `./scripts/committer "type(scope): summary" <paths…>` to stage and create commits; avoid raw `git add`.
- Batch git network ops in groups: commit related repo changes first, then push/pull repos together so submodule gitlinks stay coherent.
- PRs should summarize intent, list test commands executed, mention doc updates, and include screenshots or terminal snippets when behavior changes.
- Never release or publish without an explicit release command.
- Peekaboo releases: follow `$release-peekaboo`; current Mac + existing 1Password credentials first. App Store Connect changes last resort, only after same-item `notarytool history` and non-S3 `submit` both fail.
- Credentialed release wrappers: `bash -c`, never login shells; profile exports can override ASC IDs and mix credentials.
- Published CLI proof: run `npm exec` from `/tmp`; repo cwd may shadow the downloaded package with a local binary.
- During PR triage, keep moving autonomously: fix defects, add obvious scoped features, and rewrite or land what makes sense.
- Before landing every PR, run autoreview until no actionable findings remain and fix or rerun CI until green.
## Security & Configuration Tips
- Secrets and provider tokens live under `~/.peekaboo` (managed by Tachikoma); never commit credentials or sample keys.

@ -1 +1 @@
Subproject commit fbb2a577c98015cbfcefb606eefdd2369ce99de5
Subproject commit c276ac88a0ebddb2a618b31092715d6df87456e0

View File

@ -5,6 +5,52 @@ All notable changes to Peekaboo CLI will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [3.5.3] - 2026-06-13
### Fixed
- Public CLI, agent, MCP, and API guidance now treats runtime element IDs as opaque strings to copy exactly instead of implying role-specific ID shapes. Thanks @coygeek for #194.
- JSON-only `peekaboo see` runs without `--path` now keep required screenshots in snapshot storage instead of leaving files on Desktop or exposing their temporary paths. Thanks @coygeek for #196.
- Background element/query/coordinate clicks now pin actions to the requested process and exact window, reject mismatched window/PID selectors and unverifiable snapshots, invalidate implicit latest snapshots without deleting history, and no longer require Event Synthesizing when Accessibility completes the click.
- App launch, open, and inventory commands now use the selected runtime host, fixing sandboxed LaunchServices failures; launch/open preserve `--no-focus` and caller-relative app paths, relaunch preflights and keeps quit/wait/launch in one daemon-held transaction, build-scoped fallback daemons remain reusable and controllable across native/Rosetta execution and executable upgrades, incompatible legacy hosts no longer force sandboxed local fallback, and inventory ignores unrelated input overrides.
- Agent, MCP, script, CLI, and bridge mutations now advance implicit-snapshot watermarks at host-confirmed completion or observation boundaries, keep durable pending barriers across client timeouts/disconnects without hiding the acting command's own snapshot, carry remote script observation certificates, recover safely from PID reuse, ignore unavailable alternate hosts after protecting the selected/local stores, and preserve explicit snapshot history.
## [3.5.2] - 2026-06-13
### Changed
- `peekaboo type` and the MCP `type` tool now default to zero-delay linear typing; supplying `--wpm`/`wpm` still opts into human cadence.
### Fixed
- Synchronized Tachikoma's OpenAI `gpt-5-chat-latest` catalog metadata so configured models apply the correct GPT-5 parameter filtering.
## [3.5.1] - 2026-06-12
### Fixed
- `peekaboo see` now returns at its configured wall-clock deadline when suspended capture or detection work ignores task cancellation, while preserving explicit command cancellation.
## [3.5.0] - 2026-06-12
### Added
- `peekaboo agent` now supports explicit Claude Fable 5 (`claude-fable-5`) selection with 1M context and 128K max output while keeping Anthropic defaults on Opus 4.8 for zero-retention compatibility.
### Changed
- Agent runs now honor the saved `agent.temperature` and `agent.maxTokens` values shared by the CLI and macOS Settings UI, clamp them to each provider's capabilities, infer Fable limits through compatible providers, and omit unsupported sampling parameters for GPT-5 and current Anthropic reasoning models.
- Project, issue, build, release, and app About links now use the canonical `openclaw/Peekaboo` repository.
### Fixed
- Bridge hosts now use atomic lease-backed socket ownership and bounded nonblocking transport, keep Peekaboo.app and the reusable daemon on distinct paths while preserving the healthy app's TCC-backed fallback, preserve lifecycle settings while migrating legacy daemons, prevent MCP from hosting a bridge listener, safely recover stale sockets, and release abandoned client connections instead of wedging. Thanks @Artifact-LV for #184.
- Legacy screen and area capture now fails with a permission or native capture error instead of returning wallpaper-only/redacted pixels from background sessions. Thanks @VishalJ99 for #185.
## [3.4.1] - 2026-06-10
### Fixed
- `peekaboo agent` now resolves saved custom providers, xAI/Grok, Gemini 3.5 Flash, Claude Opus 4.8, and GPT-5.5 model selections before falling back to unavailable built-in defaults. Thanks @udiedrichsen for #182.
## [3.4.0] - 2026-06-07
## [3.3.0] - 2026-06-01
## [3.2.3] - 2026-05-24
## [3.2.2] - 2026-05-22
### Fixed

View File

@ -1,11 +1,34 @@
import Commander
import Foundation
import PeekabooAutomationKit
/// Commands or runtime contexts that can specify a preferred capture engine.
protocol CaptureEngineConfigurable: AnyObject {
var captureEngine: String? { get }
}
enum CommanderRuntimeExecutorMessage {
static let snapshotInvalidationWarning =
"Warning: The requested action succeeded, but stale UI snapshots could not be invalidated after retry. " +
"Do not retry the action."
}
enum CommanderRuntimeExecutorError: LocalizedError {
case snapshotCatchUpFailed(any Error)
case mutationBarrierFailed(any Error)
var errorDescription: String? {
switch self {
case let .snapshotCatchUpFailed(error):
"Could not synchronize the selected host's UI snapshot watermark before execution: " +
"the requested command was not executed, so retrying later is safe. " + error.localizedDescription
case let .mutationBarrierFailed(error):
"Could not establish the desktop mutation barrier before execution: " +
"the requested command was not executed, so retrying later is safe. " + error.localizedDescription
}
}
}
@MainActor
enum CommanderRuntimeExecutor {
static func resolveAndRun(arguments: [String]) async throws {
@ -30,11 +53,204 @@ enum CommanderRuntimeExecutor {
setenv("PEEKABOO_CAPTURE_ENGINE", capturePreference, 1)
}
let runtime = await CommandRuntime.makeDefaultAsync(options: runtimeOptions)
try await runtimeCommand.run(using: runtime)
try await self.catchUpSelectedHostIfNeeded(
using: runtime,
required: runtimeOptions.requiresImplicitSnapshotInvalidation ||
runtimeOptions.usesPerToolSnapshotInvalidation
)
try await DeferredCommandOutput.run(
bufferingOutput: runtimeOptions.requiresImplicitSnapshotInvalidation
) {
try await self.runWithImplicitSnapshotInvalidation(
using: runtime,
required: runtimeOptions.requiresImplicitSnapshotInvalidation,
requiresCallerBarrier: runtimeOptions.requiresCallerDesktopMutationBarrier
) {
try await runtimeCommand.run(using: runtime)
}
}
return
}
var plainCommand = command
try await plainCommand.run()
}
static func catchUpSelectedHostIfNeeded(
using runtime: CommandRuntime,
required: Bool
) async throws {
guard required else { return }
try Task.checkCancellation()
let cutoff = runtime.services.snapshots.effectiveImplicitLatestInvalidationWatermark
try Task.checkCancellation()
guard let cutoff else { return }
do {
_ = try await runtime.services.snapshots.invalidateImplicitLatestSnapshot(
through: cutoff,
preserving: nil,
preservedAt: nil
)
try Task.checkCancellation()
} catch let error as CancellationError {
throw error
} catch {
throw CommanderRuntimeExecutorError.snapshotCatchUpFailed(error)
}
}
static func runWithImplicitSnapshotInvalidation<T>(
using runtime: CommandRuntime,
required: Bool,
requiresCallerBarrier: Bool = false,
operation: () async throws -> T
) async throws -> T {
let mutationSequenceAtStart = runtime.interactionMutationTracker.mutationSequence
let needsCallerBarrier = required &&
(runtime.selectedRemoteSocketPath == nil || requiresCallerBarrier)
let createdDurableMutation: Bool
if needsCallerBarrier {
do {
createdDurableMutation = try runtime.interactionMutationTracker.beginDurableMutation()
} catch {
throw CommanderRuntimeExecutorError.mutationBarrierFailed(error)
}
} else {
createdDurableMutation = false
}
let result: T
do {
result = try await runtime.interactionMutationTracker.withPendingDurableMutationVisible(
createdByCurrentCommand: createdDurableMutation,
operation: operation
)
try Task.checkCancellation()
} catch {
_ = await self.invalidateSnapshotsAfterCommandIfNeeded(
using: runtime,
required: required,
succeeded: false,
mutationSequenceAtStart: mutationSequenceAtStart,
createdDurableMutation: createdDurableMutation
)
throw error
}
let hadPendingMutation = required && runtime.interactionMutationTracker.mutationStartedAt != nil
let invalidated = await invalidateSnapshotsAfterCommandIfNeeded(
using: runtime,
required: required,
succeeded: true,
mutationSequenceAtStart: mutationSequenceAtStart,
createdDurableMutation: createdDurableMutation
)
do {
try Task.checkCancellation()
} catch {
if hadPendingMutation {
_ = await self.invalidateSnapshots(
using: runtime,
reason: "command cancellation",
through: Date(),
preserving: nil,
preservedAt: nil
)
}
throw error
}
if !invalidated {
fputs("\(CommanderRuntimeExecutorMessage.snapshotInvalidationWarning)\n", stderr)
}
return result
}
private static func invalidateSnapshotsAfterCommandIfNeeded(
using runtime: CommandRuntime,
required: Bool,
succeeded: Bool,
mutationSequenceAtStart: UInt64,
createdDurableMutation: Bool
) async -> Bool {
let completion = Date()
guard required else { return true }
guard runtime.interactionMutationTracker.mutationStartedAt != nil else {
guard createdDurableMutation else {
return !runtime.interactionMutationTracker.hasPendingDurableMutation
}
do {
try runtime.interactionMutationTracker.cancelDurableMutation()
return true
} catch {
return false
}
}
guard let requestedCutoff = runtime.interactionMutationTracker.invalidationCutoff(
commandCompletedAt: completion,
succeeded: succeeded
)
else { return true }
let durableCompletion: DesktopMutationWatermarkStore.MutationCompletion?
do {
if createdDurableMutation,
runtime.interactionMutationTracker.mutationSequence == mutationSequenceAtStart {
try runtime.interactionMutationTracker.cancelDurableMutation()
durableCompletion = nil
} else {
durableCompletion = try runtime.interactionMutationTracker.completeDurableMutation(
through: succeeded ? requestedCutoff : completion
)
}
} catch {
runtime.interactionMutationTracker.markInvalidationFailed(through: completion)
return false
}
let cutoff = max(requestedCutoff, durableCompletion?.cutoff ?? requestedCutoff)
let preservationAllowed = durableCompletion?.allowsObservationPreservation ?? true
let preservedSnapshotID = succeeded && preservationAllowed
? runtime.interactionMutationTracker.preservedSnapshotID
: nil
let preservedAt = preservedSnapshotID == nil
? nil
: runtime.interactionMutationTracker.preservedAt
return await self.invalidateSnapshots(
using: runtime,
reason: "command execution",
through: cutoff,
preserving: preservedSnapshotID,
preservedAt: preservedAt
)
}
private static func invalidateSnapshots(
using runtime: CommandRuntime,
reason: String,
through cutoff: Date,
preserving preservedSnapshotID: String?,
preservedAt: Date?
) async -> Bool {
let targets = runtime.interactionMutationTargets
let isRetry = runtime.interactionMutationTracker.hasFailedInvalidationAttempt
let invalidated = await InteractionObservationInvalidator.invalidateAfterMutation(
targets: targets,
logger: runtime.logger,
reason: reason,
through: cutoff,
preserving: preservedSnapshotID,
preservedAt: preservedAt
)
if invalidated {
return true
}
if isRetry {
return false
}
return await InteractionObservationInvalidator.invalidateAfterMutation(
targets: targets,
logger: runtime.logger,
reason: "\(reason) retry",
through: cutoff,
preserving: preservedSnapshotID,
preservedAt: preservedAt
)
}
}

View File

@ -80,8 +80,9 @@ enum CommanderRuntimeRouter {
return true
}
if let index = arguments.firstIndex(where: { self.isHelpToken($0) }) {
let tokens = Array(arguments.prefix(index))
let helpSearchArguments = Array(arguments.prefix { $0 != "--" })
if let index = helpSearchArguments.firstIndex(where: { self.isHelpToken($0) }) {
let tokens = Array(helpSearchArguments.prefix(index))
if self.handleAgentPermissionHelp(tokens: tokens) {
return true
}

View File

@ -69,6 +69,8 @@ enum CommandRegistry {
.init(type: CompletionsCommand.self, category: .core),
.init(type: CommanderCommand.self, category: .core),
.init(type: AgentCommand.self, category: .ai),
.init(type: BrowserCommand.self, category: .mcp),
.init(type: InspectUICommand.self, category: .mcp),
.init(type: MCPCommand.self, category: .mcp),
]

View File

@ -0,0 +1,248 @@
import Darwin
import Foundation
enum DeferredCommandOutput {
static func run<T>(
bufferingOutput: Bool,
operation: () async throws -> T
) async throws -> T {
guard bufferingOutput else {
return try await operation()
}
let inheritedTerminalOutput = TerminalDetector.standardOutputFileDescriptor
let capture = try FileDescriptorOutputCapture()
let terminalOutput = inheritedTerminalOutput ?? capture.originalStandardOutputDescriptor
let result: T
do {
result = try await TerminalDetector.$standardOutputFileDescriptor.withValue(terminalOutput) {
try await operation()
}
} catch {
let shouldReplay = !(error is CancellationError)
// Preserve the command's primary error even if restoring or replaying output fails.
Logger.shared.flush()
try? capture.finish(replayingOutput: shouldReplay)
throw error
}
Logger.shared.flush()
try capture.finish(replayingOutput: true)
return result
}
}
private nonisolated enum DeferredCommandOutputError: LocalizedError {
case posix(operation: String, code: Int32)
var errorDescription: String? {
switch self {
case let .posix(operation, code):
"Failed to \(operation): \(String(cString: strerror(code)))"
}
}
}
private final nonisolated class FileDescriptorOutputCapture {
private var stdoutCapture: Int32 = -1
private var stderrCapture: Int32 = -1
private var originalStdout: Int32 = -1
private var originalStderr: Int32 = -1
private var stdoutRedirected = false
private var stderrRedirected = false
private var finished = false
var originalStandardOutputDescriptor: Int32 {
self.originalStdout
}
init() throws {
// Keep output emitted before this command outside its deferred transaction.
_ = fflush(nil)
do {
self.stdoutCapture = try Self.makeTemporaryFile(named: "stdout")
self.stderrCapture = try Self.makeTemporaryFile(named: "stderr")
self.originalStdout = try Self.duplicate(STDOUT_FILENO, named: "stdout")
self.originalStderr = try Self.duplicate(STDERR_FILENO, named: "stderr")
try Self.redirect(self.stdoutCapture, to: STDOUT_FILENO, named: "stdout")
self.stdoutRedirected = true
try Self.redirect(self.stderrCapture, to: STDERR_FILENO, named: "stderr")
self.stderrRedirected = true
} catch {
self.restoreIgnoringErrors()
self.closeDescriptors()
throw error
}
}
deinit {
guard !self.finished else { return }
_ = fflush(nil)
self.restoreIgnoringErrors()
self.closeDescriptors()
}
func finish(replayingOutput: Bool) throws {
guard !self.finished else { return }
_ = fflush(nil)
try self.restore()
defer {
self.finished = true
self.closeDescriptors()
}
if replayingOutput {
try Self.replay(from: self.stdoutCapture, to: self.originalStdout, named: "stdout")
try Self.replay(from: self.stderrCapture, to: self.originalStderr, named: "stderr")
}
}
private func restore() throws {
var firstError: (any Error)?
if self.stdoutRedirected {
do {
try Self.redirect(self.originalStdout, to: STDOUT_FILENO, named: "stdout")
self.stdoutRedirected = false
} catch {
firstError = error
}
}
if self.stderrRedirected {
do {
try Self.redirect(self.originalStderr, to: STDERR_FILENO, named: "stderr")
self.stderrRedirected = false
} catch {
firstError = firstError ?? error
}
}
if let firstError {
throw firstError
}
}
private func restoreIgnoringErrors() {
if self.stdoutRedirected, dup2(self.originalStdout, STDOUT_FILENO) != -1 {
self.stdoutRedirected = false
}
if self.stderrRedirected, dup2(self.originalStderr, STDERR_FILENO) != -1 {
self.stderrRedirected = false
}
}
private func closeDescriptors() {
Self.close(&self.stdoutCapture)
Self.close(&self.stderrCapture)
Self.close(&self.originalStdout)
Self.close(&self.originalStderr)
}
private static func makeTemporaryFile(named stream: String) throws -> Int32 {
var template = Array("\(NSTemporaryDirectory())peekaboo-\(stream).XXXXXX".utf8CString)
let descriptor = template.withUnsafeMutableBufferPointer { buffer -> Int32 in
guard let baseAddress = buffer.baseAddress else { return -1 }
let descriptor = mkstemp(baseAddress)
if descriptor != -1 {
_ = unlink(baseAddress)
}
return descriptor
}
guard descriptor != -1 else {
throw DeferredCommandOutputError.posix(
operation: "create deferred \(stream) output",
code: errno
)
}
Self.setCloseOnExec(descriptor)
return descriptor
}
private static func duplicate(_ descriptor: Int32, named stream: String) throws -> Int32 {
let duplicate = dup(descriptor)
guard duplicate != -1 else {
throw DeferredCommandOutputError.posix(
operation: "duplicate \(stream)",
code: errno
)
}
Self.setCloseOnExec(duplicate)
return duplicate
}
private static func redirect(_ source: Int32, to destination: Int32, named stream: String) throws {
guard dup2(source, destination) != -1 else {
throw DeferredCommandOutputError.posix(
operation: "redirect \(stream)",
code: errno
)
}
}
private static func replay(from source: Int32, to destination: Int32, named stream: String) throws {
guard lseek(source, 0, SEEK_SET) != -1 else {
throw DeferredCommandOutputError.posix(
operation: "rewind deferred \(stream) output",
code: errno
)
}
var buffer = [UInt8](repeating: 0, count: 64 * 1024)
while true {
let bytesRead = buffer.withUnsafeMutableBytes { bytes in
Darwin.read(source, bytes.baseAddress, bytes.count)
}
if bytesRead == 0 {
return
}
if bytesRead == -1 {
if errno == EINTR {
continue
}
throw DeferredCommandOutputError.posix(
operation: "read deferred \(stream) output",
code: errno
)
}
var offset = 0
while offset < bytesRead {
let bytesWritten = buffer.withUnsafeBytes { bytes in
Darwin.write(
destination,
bytes.baseAddress?.advanced(by: offset),
bytesRead - offset
)
}
if bytesWritten == -1 {
if errno == EINTR {
continue
}
throw DeferredCommandOutputError.posix(
operation: "replay deferred \(stream) output",
code: errno
)
}
offset += bytesWritten
}
}
}
private static func setCloseOnExec(_ descriptor: Int32) {
let flags = fcntl(descriptor, F_GETFD)
if flags != -1 {
_ = fcntl(descriptor, F_SETFD, flags | FD_CLOEXEC)
}
}
private static func close(_ descriptor: inout Int32) {
guard descriptor != -1 else { return }
_ = Darwin.close(descriptor)
descriptor = -1
}
}

View File

@ -28,14 +28,18 @@ struct TerminalCapabilities {
/// Terminal detection utilities following modern CLI best practices
enum TerminalDetector {
@TaskLocal
static var standardOutputFileDescriptor: Int32?
/// Detect comprehensive terminal capabilities
static func detectCapabilities() -> TerminalCapabilities {
// Detect comprehensive terminal capabilities
let isInteractive = self.isInteractiveTerminal()
let (width, height) = self.getTerminalDimensions()
let outputFileDescriptor = self.standardOutputFileDescriptor ?? STDOUT_FILENO
let isInteractive = self.isInteractiveTerminal(outputFileDescriptor)
let (width, height) = self.getTerminalDimensions(outputFileDescriptor)
let termType = ProcessInfo.processInfo.environment["TERM"]
let isCI = self.isCIEnvironment()
let isPiped = self.isPipedOutput()
let isPiped = self.isPipedOutput(outputFileDescriptor)
let supportsColors = self.detectColorSupport(termType: termType, isInteractive: isInteractive)
let supportsTrueColor = self.detectTrueColorSupport()
@ -54,15 +58,15 @@ enum TerminalDetector {
// MARK: - Core Detection Methods
/// Check if stdout is connected to an interactive terminal
private static func isInteractiveTerminal() -> Bool {
private static func isInteractiveTerminal(_ outputFileDescriptor: Int32) -> Bool {
// Check if stdout is connected to an interactive terminal
isatty(STDOUT_FILENO) != 0
isatty(outputFileDescriptor) != 0
}
/// Check if output is being piped or redirected
private static func isPipedOutput() -> Bool {
private static func isPipedOutput(_ outputFileDescriptor: Int32) -> Bool {
// Check if output is being piped or redirected
isatty(STDOUT_FILENO) == 0
isatty(outputFileDescriptor) == 0
}
/// Detect CI/automation environments
@ -79,7 +83,7 @@ enum TerminalDetector {
"AZURE_PIPELINES", "TF_BUILD",
"BITBUCKET_COMMIT", "BITBUCKET_BUILD_NUMBER",
"DRONE", "DRONE_BUILD_NUMBER",
"SEMAPHORE", "SEMAPHORE_BUILD_NUMBER"
"SEMAPHORE", "SEMAPHORE_BUILD_NUMBER",
]
let env = ProcessInfo.processInfo.environment
@ -87,11 +91,11 @@ enum TerminalDetector {
}
/// Get terminal dimensions using ioctl
private static func getTerminalDimensions() -> (width: Int, height: Int) {
private static func getTerminalDimensions(_ outputFileDescriptor: Int32) -> (width: Int, height: Int) {
// Get terminal dimensions using ioctl
var windowSize = winsize()
guard ioctl(STDOUT_FILENO, TIOCGWINSZ, &windowSize) == 0 else {
guard ioctl(outputFileDescriptor, TIOCGWINSZ, &windowSize) == 0 else {
// Fallback to environment variables
let width = Int(ProcessInfo.processInfo.environment["COLUMNS"] ?? "80") ?? 80
let height = Int(ProcessInfo.processInfo.environment["LINES"] ?? "24") ?? 24
@ -120,7 +124,7 @@ enum TerminalDetector {
if let term = termType {
let colorTermPatterns = [
"color", "256color", "truecolor", "24bit",
"xterm-256", "screen-256", "tmux-256"
"xterm-256", "screen-256", "tmux-256",
]
if colorTermPatterns.contains(where: term.contains) {
@ -131,7 +135,7 @@ enum TerminalDetector {
let colorTerminals = [
"xterm", "screen", "tmux", "rxvt", "konsole",
"gnome", "mate", "xfce", "terminology", "kitty",
"alacritty", "iterm", "hyper", "vscode"
"alacritty", "iterm", "hyper", "vscode",
]
if colorTerminals.contains(where: term.contains) {
@ -163,7 +167,7 @@ enum TerminalDetector {
if let term = env["TERM"] {
let trueColorTerminals = [
"iterm", "kitty", "alacritty", "wezterm",
"hyper", "vscode", "gnome-terminal"
"hyper", "vscode", "gnome-terminal",
]
return trueColorTerminals.contains(where: term.contains)
}

View File

@ -9,27 +9,22 @@ import TauTUI
@available(macOS 14.0, *)
extension AgentCommand {
func ensureAgentHasCredentials(
_ peekabooAgent: PeekabooAgentService,
requestedModel: LanguageModel?
) async -> Bool {
if let requestedModel {
if self.hasCredentials(for: requestedModel) {
return true
}
let providerName = self.providerDisplayName(for: requestedModel)
let envVar = self.providerEnvironmentVariable(for: requestedModel)
self.printAgentExecutionError(
"Missing API key for \(providerName). Set \(envVar) and retry."
)
return false
selectedModel: LanguageModel
) -> Bool {
if self.isLocalModel(selectedModel) {
return true
}
let hasCredential = await peekabooAgent.maskedApiKey != nil
if !hasCredential {
self.emitAgentUnavailableMessage()
if self.hasCredentials(for: selectedModel) {
return true
}
return hasCredential
let providerName = self.providerDisplayName(for: selectedModel)
let envVar = self.providerEnvironmentVariable(for: selectedModel)
self.printAgentExecutionError(
"Missing API key for \(providerName). Set \(envVar) and retry."
)
return false
}
/// Render the agent execution result using either JSON output or a rich CLI transcript.

View File

@ -5,14 +5,46 @@ import Tachikoma
@available(macOS 14.0, *)
extension AgentCommand {
func parseModelString(_ modelString: String) -> LanguageModel? {
@MainActor
func parseModelString(
_ modelString: String,
configuration: PeekabooCore.ConfigurationManager? = nil
) -> LanguageModel? {
let trimmed = modelString.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { return nil }
let explicitProvider = trimmed
.split(separator: "/", maxSplits: 1)
.first
.map { String($0).lowercased() }
if trimmed.caseInsensitiveCompare("claude") == .orderedSame ||
trimmed.caseInsensitiveCompare("anthropic") == .orderedSame {
return .anthropic(.opus48)
}
if let configuration {
switch self.parseConfiguredCustomModel(
trimmed,
explicitProvider: explicitProvider,
configuration: configuration
) {
case let .resolved(model):
return model
case .unresolved:
break
}
}
guard let parsed = LanguageModel.parse(from: trimmed) else {
return nil
}
return self.supportedParsedModel(parsed, explicitProvider: explicitProvider)
}
@MainActor
private func supportedParsedModel(_ parsed: LanguageModel, explicitProvider: String?) -> LanguageModel? {
switch parsed {
case let .openai(model):
if Self.supportedOpenAIInputs.contains(model) {
@ -20,17 +52,28 @@ extension AgentCommand {
}
case let .anthropic(model):
if Self.supportedAnthropicInputs.contains(model) {
return .anthropic(.opus47)
return .anthropic(model)
}
case let .google(model):
if Self.supportedGoogleInputs.contains(model) {
return .google(model)
}
case .grok:
return parsed.supportsTools ? parsed : nil
case let .minimax(model):
if Self.supportedMiniMaxInputs.contains(model) {
return .minimax(model)
}
case .ollama, .lmstudio, .openRouter:
case let .minimaxCN(model):
if Self.supportedMiniMaxInputs.contains(model) {
return .minimaxCN(model)
}
case .ollama, .lmstudio:
return parsed.supportsTools ? parsed : nil
case .openRouter:
if let explicitProvider, Self.reservedProviderInputs.contains(explicitProvider) {
return nil
}
return parsed.supportsTools ? parsed : nil
default:
break
@ -39,9 +82,36 @@ extension AgentCommand {
return nil
}
func validatedModelSelection() throws -> LanguageModel? {
@MainActor
private func parseConfiguredCustomModel(
_ modelString: String,
explicitProvider: String?,
configuration: PeekabooCore.ConfigurationManager
) -> ConfiguredModelResolution {
if let configuredModel = PeekabooAIService(configuration: configuration).resolveConfiguredModel(modelString),
case .custom = configuredModel {
return .resolved(configuredModel.supportsTools ? configuredModel : nil)
}
if let explicitProvider,
configuration.listCustomProviders().contains(where: { providerID, provider in
provider.enabled && providerID.caseInsensitiveCompare(explicitProvider) == .orderedSame
}) {
return .resolved(nil)
}
return .unresolved
}
private enum ConfiguredModelResolution {
case resolved(LanguageModel?)
case unresolved
}
@MainActor
func validatedModelSelection(configuration: PeekabooCore.ConfigurationManager? = nil) throws -> LanguageModel? {
guard let modelString = self.model else { return nil }
guard let parsed = self.parseModelString(modelString) else {
guard let parsed = self.parseModelString(modelString, configuration: configuration) else {
throw PeekabooError.invalidInput(
"Unsupported model '\(modelString)'. Allowed values: \(Self.allowedModelList)"
)
@ -61,6 +131,8 @@ extension AgentCommand {
]
private static let supportedAnthropicInputs: Set<LanguageModel.Anthropic> = [
.fable5,
.opus48,
.opus47,
.opus45,
.opus4,
@ -70,6 +142,7 @@ extension AgentCommand {
]
private static let supportedGoogleInputs: Set<LanguageModel.Google> = [
.gemini35Flash,
.gemini31ProPreview,
.gemini31FlashLite,
.gemini3Flash,
@ -83,20 +156,80 @@ extension AgentCommand {
.m27Highspeed,
]
private static let reservedProviderInputs: Set<String> = [
"openai",
"anthropic",
"google",
"gemini",
"grok",
"xai",
"minimax",
"minimax-cn",
"minimax_cn",
"minimaxi",
"ollama",
"lmstudio",
"lm-studio",
]
private static var allowedModelList: String {
let openAIModels = Self.supportedOpenAIInputs.map(\.modelId)
let anthropicModels = Self.supportedAnthropicInputs.map(\.modelId)
let googleModels = Self.supportedGoogleInputs.map(\.userFacingModelId)
let miniMaxModels = Self.supportedMiniMaxInputs.map(\.modelId)
return (openAIModels + anthropicModels + googleModels + miniMaxModels + [
"grok/<model>",
"xai/<model>",
"minimax-cn/<model>",
"ollama/<model>",
"lmstudio/<model>",
"openrouter/<provider>/<model>",
"<custom-provider>/<model>",
])
.sorted()
.joined(separator: ", ")
}
@MainActor
func firstAvailableToolModel(from service: PeekabooAIService) -> LanguageModel? {
service.availableModels().first { model in
model.supportsTools && service.isModelAvailable(model)
}
}
@MainActor
func configuredDefaultToolModel(
from service: PeekabooAIService,
configuration: PeekabooCore.ConfigurationManager
) -> LanguageModel? {
guard let defaultModel = configuration.getAgentModel(),
let model = service.resolveConfiguredModel(defaultModel),
model.supportsTools,
service.isModelAvailable(model)
else {
return nil
}
return model
}
@MainActor
func implicitToolModel(
from service: PeekabooAIService,
configuration: PeekabooCore.ConfigurationManager,
existingAgentModel: LanguageModel?
) -> LanguageModel? {
if let existingAgentModel {
return existingAgentModel
}
if configuration.hasExplicitAIProviderList() {
return nil
}
return self.configuredDefaultToolModel(from: service, configuration: configuration) ??
self.firstAvailableToolModel(from: service)
}
@MainActor
func hasCredentials(for model: LanguageModel) -> Bool {
let configuration = self.services.configuration
@ -111,8 +244,14 @@ extension AgentCommand {
return configuration.getGeminiAPIKey()?.isEmpty == false
case .minimax:
return configuration.getMiniMaxAPIKey()?.isEmpty == false
case .minimaxCN:
return configuration.getMiniMaxChinaAPIKey()?.isEmpty == false
case .grok:
return configuration.getGrokAPIKey()?.isEmpty == false
case .openRouter:
return configuration.getOpenRouterAPIKey()?.isEmpty == false
case let .custom(provider):
return provider.apiKey?.isEmpty == false
default:
return false
}
@ -128,12 +267,18 @@ extension AgentCommand {
"Google"
case .minimax:
"MiniMax"
case .minimaxCN:
"MiniMax China"
case .ollama:
"Ollama"
case .lmstudio:
"LM Studio"
case .openRouter:
"OpenRouter"
case .grok:
"xAI"
case let .custom(provider):
"custom provider \(provider.modelId)"
default:
"the selected provider"
}
@ -149,12 +294,18 @@ extension AgentCommand {
"GEMINI_API_KEY"
case .minimax:
"MINIMAX_API_KEY"
case .minimaxCN:
"MINIMAX_CN_API_KEY or MINIMAX_API_KEY"
case .ollama:
"OLLAMA_BASE_URL or PEEKABOO_OLLAMA_BASE_URL"
case .lmstudio:
"LM Studio local server URL"
case .openRouter:
"OPENROUTER_API_KEY"
case .grok:
"X_AI_API_KEY, XAI_API_KEY, or GROK_API_KEY"
case .custom:
"the custom provider API key reference"
default:
"provider API key"
}

View File

@ -89,8 +89,9 @@ struct AgentCommand: RuntimeOptionsConfigurable {
@Option(
name: .long,
help: """
AI model to use (for example: gpt-5.5, claude-opus-4-7, \
gemini-3-flash, minimax-m2.7, ollama/<model>, or lmstudio/<model>)
AI model to use (for example: gpt-5.5, claude-fable-5, \
gemini-3.5-flash, grok-4.3, minimax-m2.7, minimax-cn/m2.7, \
ollama/<model>, lmstudio/<model>, or <custom-provider>/<model>)
"""
)
var model: String?
@ -212,38 +213,74 @@ extension AgentCommand {
let requestedModel: LanguageModel?
do {
requestedModel = try self.validatedModelSelection()
requestedModel = try self.validatedModelSelection(configuration: services.configuration)
} catch {
self.printAgentExecutionError(error.localizedDescription)
throw ExitCode.failure
}
let agentService: any AgentServiceProtocol
if let existing = services.agent {
agentService = existing
} else if let requestedModel {
agentService = try PeekabooAgentService(services: services, defaultModel: requestedModel)
} else {
let configuredAIService = PeekabooAIService(configuration: services.configuration)
let existingAgent = services.agent as? PeekabooAgentService
let mutationCoordinator = runtime.toolSnapshotMutationCoordinator
existingAgent?.configureSnapshotMutationCoordinator(mutationCoordinator)
let existingAgentModel = existingAgent.flatMap {
configuredAIService.resolveConfiguredModel($0.defaultModelSelection) ??
LanguageModel.parse(from: $0.defaultModelSelection)
}
let selectedModel = requestedModel ??
self.implicitToolModel(
from: configuredAIService,
configuration: services.configuration,
existingAgentModel: existingAgentModel
)
if self.listSessions {
let listingModel = selectedModel ?? existingAgentModel ?? .anthropic(.opus48)
let agentService: any AgentServiceProtocol = if let existing = existingAgent {
existing
} else {
try PeekabooAgentService(
services: services,
defaultModel: listingModel,
snapshotMutationCoordinator: mutationCoordinator
)
}
try await self.showSessions(agentService)
return
}
guard let selectedModel else {
self.emitAgentUnavailableMessage()
return
}
guard self.hasCredentials(for: selectedModel) || self.isLocalModel(selectedModel) else {
if requestedModel != nil {
let providerName = self.providerDisplayName(for: selectedModel)
let envVar = self.providerEnvironmentVariable(for: selectedModel)
self.printAgentExecutionError(
"Missing API key for \(providerName). Set \(envVar) and retry."
)
} else {
self.emitAgentUnavailableMessage()
}
return
}
let agentService: any AgentServiceProtocol = if let existing = existingAgent {
existing
} else {
try PeekabooAgentService(
services: services,
defaultModel: selectedModel,
snapshotMutationCoordinator: mutationCoordinator
)
}
let terminalCapabilities = TerminalDetector.detectCapabilities()
if self.debugTerminal {
self.printTerminalDetectionDebug(terminalCapabilities, actualMode: self.outputMode)
}
if self.listSessions {
try await self.showSessions(agentService)
return
}
guard self.hasConfiguredAIProvider(configuration: services.configuration) || self.isLocalModel(requestedModel)
else {
self.emitAgentUnavailableMessage()
return
}
let shouldSuppressMCPLogs = !self.verbose && !self.debugTerminal
self.configureLogging(suppressingMCPLogs: shouldSuppressMCPLogs)
@ -251,7 +288,7 @@ extension AgentCommand {
throw PeekabooError.commandFailed("Agent service not properly initialized")
}
guard await self.ensureAgentHasCredentials(peekabooAgent, requestedModel: requestedModel) else {
guard self.ensureAgentHasCredentials(selectedModel: selectedModel) else {
return
}
@ -332,29 +369,11 @@ extension AgentCommand {
}
}
private func hasConfiguredAIProvider(configuration: PeekabooCore.ConfigurationManager) -> Bool {
let hasOpenAI = configuration.hasOpenAIAuth()
let hasAnthropic = configuration.hasAnthropicAuth()
let hasGemini = configuration.getGeminiAPIKey()?.isEmpty == false
let hasMiniMax = configuration.getMiniMaxAPIKey()?.isEmpty == false
let hasOpenRouter = configuration.getOpenRouterAPIKey()?.isEmpty == false
let hasLocalProvider = configuration.getAIProviders()
.split(separator: ",")
.contains { entry in
let provider = entry
.trimmingCharacters(in: .whitespacesAndNewlines)
.split(separator: "/", maxSplits: 1)
.first?
.lowercased()
return provider == "ollama" || provider == "lmstudio" || provider == "lm-studio"
}
return hasOpenAI || hasAnthropic || hasGemini || hasMiniMax || hasOpenRouter || hasLocalProvider
}
func emitAgentUnavailableMessage() {
if self.jsonOutput {
let message = "Agent service not available. Please set OPENAI_API_KEY, ANTHROPIC_API_KEY, " +
"GEMINI_API_KEY, MINIMAX_API_KEY, OPENROUTER_API_KEY, or configure ollama/<model> or lmstudio/<model>."
"GEMINI_API_KEY, X_AI_API_KEY, MINIMAX_API_KEY, MINIMAX_CN_API_KEY, OPENROUTER_API_KEY, " +
"or configure ollama/<model>, lmstudio/<model>, or a custom provider."
let error = [
"success": false,
"error": message
@ -368,8 +387,9 @@ extension AgentCommand {
} else {
let errorPrefix = [
"\(TerminalColor.red)Error: Agent service not available.",
" Please set OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, MINIMAX_API_KEY, OPENROUTER_API_KEY,",
" or configure ollama/<model> or lmstudio/<model>."
" Please set OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, X_AI_API_KEY,",
" MINIMAX_API_KEY, MINIMAX_CN_API_KEY, OPENROUTER_API_KEY,",
" or configure ollama/<model>, lmstudio/<model>, or a custom provider."
].joined()
let errorMessageLine = [errorPrefix, "\(TerminalColor.reset)"].joined()
print(errorMessageLine)

View File

@ -7,7 +7,8 @@ import PeekabooFoundation
extension SeeCommand {
func detectElements(
imageData: Data,
windowContext: WindowContext?
windowContext: WindowContext?,
snapshotID: String? = nil
) async throws -> ElementDetectionResult {
self.logger.operationStart("element_detection")
defer { self.logger.operationComplete("element_detection") }
@ -22,7 +23,9 @@ extension SeeCommand {
automation: self.services.automation,
imageData: imageData,
windowContext: windowContext,
timeoutSeconds: timeoutSeconds
timeoutSeconds: timeoutSeconds,
snapshotID: snapshotID,
interactionMutationTracker: self.resolvedRuntime.observationTimeoutMutationTracker
)
} catch is TimeoutError {
throw CaptureError.detectionTimedOut(timeoutSeconds)
@ -44,13 +47,18 @@ extension SeeCommand {
automation: any UIAutomationServiceProtocol,
imageData: Data,
windowContext: WindowContext?,
timeoutSeconds: TimeInterval
timeoutSeconds: TimeInterval,
snapshotID: String? = nil,
interactionMutationTracker: InteractionMutationTracker? = nil
) async throws -> ElementDetectionResult {
try await withWallClockTimeout(seconds: timeoutSeconds) {
try await withWallClockTimeout(
seconds: timeoutSeconds,
interactionMutationTracker: interactionMutationTracker
) {
if let timeoutAdjustingAutomation = automation as? any DetectElementsRequestTimeoutAdjusting {
return try await timeoutAdjustingAutomation.detectElements(
in: imageData,
snapshotId: nil,
snapshotId: snapshotID,
windowContext: windowContext,
requestTimeoutSec: Self.remoteDetectionRequestTimeoutSeconds(for: timeoutSeconds)
)
@ -58,7 +66,7 @@ extension SeeCommand {
return try await AutomationServiceBridge.detectElements(
automation: automation,
imageData: imageData,
snapshotId: nil,
snapshotId: snapshotID,
windowContext: windowContext
)
}

View File

@ -5,7 +5,17 @@ import PeekabooFoundation
@MainActor
extension SeeCommand {
func screenshotOutputPath() -> String {
var usesTemporaryScreenshotOutput: Bool {
self.jsonOutput && self.path == nil
}
func screenshotOutputPath(snapshotID: String? = nil) -> String {
if self.usesTemporaryScreenshotOutput {
return self.temporaryScreenshotDirectory(snapshotID: snapshotID)
.appendingPathComponent("raw.png")
.path
}
let timestamp = Date().timeIntervalSince1970
let filename = "peekaboo_see_\(Int(timestamp)).png"
return ObservationCommandSupport.outputPath(
@ -16,8 +26,8 @@ extension SeeCommand {
)
}
func saveScreenshot(_ imageData: Data) throws -> String {
let outputPath = self.screenshotOutputPath()
func saveScreenshot(_ imageData: Data, snapshotID: String) throws -> String {
let outputPath = self.screenshotOutputPath(snapshotID: snapshotID)
let directory = (outputPath as NSString).deletingLastPathComponent
try FileManager.default.createDirectory(
@ -31,6 +41,17 @@ extension SeeCommand {
return outputPath
}
func cleanupTemporaryScreenshotOutput(snapshotID: String) {
guard self.usesTemporaryScreenshotOutput else { return }
try? FileManager.default.removeItem(at: self.temporaryScreenshotDirectory(snapshotID: snapshotID))
}
private func temporaryScreenshotDirectory(snapshotID: String?) -> URL {
FileManager.default.temporaryDirectory
.appendingPathComponent("peekaboo-see", isDirectory: true)
.appendingPathComponent(snapshotID ?? UUID().uuidString, isDirectory: true)
}
func resolveSeeWindowIndex(appIdentifier: String, titleFragment: String?) async throws -> Int? {
guard let fragment = titleFragment, !fragment.isEmpty else {
return nil

View File

@ -4,8 +4,10 @@ import PeekabooCore
@available(macOS 14.0, *)
@MainActor
extension SeeCommand {
func performCaptureWithDetection() async throws -> CaptureAndDetectionResult {
if let observationResult = try await self.performObservationCaptureWithDetectionIfPossible() {
func performCaptureWithDetection(snapshotID: String) async throws -> CaptureAndDetectionResult {
if let observationResult = try await self.performObservationCaptureWithDetectionIfPossible(
snapshotID: snapshotID
) {
return observationResult
}
@ -13,7 +15,7 @@ extension SeeCommand {
let captureResult = captureContext.captureResult
self.logger.startTimer("file_write")
let outputPath = try saveScreenshot(captureResult.imageData)
let outputPath = try saveScreenshot(captureResult.imageData, snapshotID: snapshotID)
self.logger.stopTimer("file_write")
let windowContext = WindowContext(
@ -27,10 +29,14 @@ extension SeeCommand {
traversalBudget: self.axTraversalBudget()
)
let detectionResult = try await self.detectElements(for: captureContext, windowContext: windowContext)
let detectionResult = try await self.detectElements(
for: captureContext,
windowContext: windowContext,
snapshotID: snapshotID
)
let resultWithPath = ElementDetectionResult(
snapshotId: detectionResult.snapshotId,
snapshotId: snapshotID,
screenshotPath: outputPath,
elements: detectionResult.elements,
metadata: detectionResult.metadata
@ -38,7 +44,7 @@ extension SeeCommand {
try await self.services.snapshots.storeScreenshot(
SnapshotScreenshotRequest(
snapshotId: detectionResult.snapshotId,
snapshotId: snapshotID,
screenshotPath: outputPath,
applicationBundleId: captureResult.metadata.applicationInfo?.bundleIdentifier,
applicationProcessId: captureResult.metadata.applicationInfo.map { Int32($0.processIdentifier) },
@ -49,12 +55,12 @@ extension SeeCommand {
)
try await self.services.snapshots.storeDetectionResult(
snapshotId: detectionResult.snapshotId,
snapshotId: snapshotID,
result: resultWithPath
)
return CaptureAndDetectionResult(
snapshotId: detectionResult.snapshotId,
snapshotId: snapshotID,
screenshotPath: outputPath,
annotatedPath: nil,
elements: detectionResult.elements,
@ -65,7 +71,8 @@ extension SeeCommand {
private func detectElements(
for captureContext: CaptureContext,
windowContext: WindowContext
windowContext: WindowContext,
snapshotID: String
) async throws -> ElementDetectionResult {
let captureResult = captureContext.captureResult
let detectionStart = Date()
@ -87,20 +94,29 @@ extension SeeCommand {
isDialog: false
)
return ElementDetectionResult(
snapshotId: UUID().uuidString,
snapshotId: snapshotID,
screenshotPath: "",
elements: DetectedElements(other: ocrElements),
metadata: metadata
)
}
return try await self.detectElements(
let detectionResult = try await self.detectElements(
imageData: captureResult.imageData,
windowContext: windowContext
windowContext: windowContext,
snapshotID: snapshotID
)
return ElementDetectionResult(
snapshotId: snapshotID,
screenshotPath: detectionResult.screenshotPath,
elements: detectionResult.elements,
metadata: detectionResult.metadata
)
}
private func performObservationCaptureWithDetectionIfPossible() async throws -> CaptureAndDetectionResult? {
private func performObservationCaptureWithDetectionIfPossible(
snapshotID: String
) async throws -> CaptureAndDetectionResult? {
guard let target = try self.observationTargetForCaptureWithDetectionIfPossible() else {
return nil
}
@ -114,7 +130,7 @@ extension SeeCommand {
let observation: DesktopObservationResult
do {
observation = try await self.services.desktopObservation
.observe(self.makeObservationRequest(target: target))
.observe(self.makeObservationRequest(target: target, snapshotID: snapshotID))
} catch DesktopObservationError.targetNotFound(_) where self.menubar {
self.logger.verbose("No observation-backed menu bar popover found; falling back", category: "Capture")
self.logger.operationComplete("capture_phase", success: false, metadata: [
@ -138,7 +154,7 @@ extension SeeCommand {
}
return CaptureAndDetectionResult(
snapshotId: detectionResult.snapshotId,
snapshotId: snapshotID,
screenshotPath: outputPath,
annotatedPath: observation.files.annotatedScreenshotPath,
elements: detectionResult.elements,

View File

@ -76,7 +76,10 @@ extension SeeCommand {
}
}
func makeObservationRequest(target: DesktopObservationTargetRequest) -> DesktopObservationRequest {
func makeObservationRequest(
target: DesktopObservationTargetRequest,
snapshotID: String? = nil
) -> DesktopObservationRequest {
DesktopObservationRequest(
target: target,
capture: DesktopCaptureOptions(
@ -86,10 +89,11 @@ extension SeeCommand {
),
detection: self.observationDetectionOptions(for: target),
output: DesktopObservationOutputOptions(
path: self.screenshotOutputPath(),
path: self.screenshotOutputPath(snapshotID: snapshotID),
saveRawScreenshot: true,
saveAnnotatedScreenshot: self.annotate && self.allowsAnnotation(for: target),
saveSnapshot: true
saveSnapshot: true,
snapshotID: snapshotID
)
)
}

View File

@ -4,16 +4,17 @@ import PeekabooCore
@available(macOS 14.0, *)
@MainActor
extension SeeCommand {
func renderResults(context: SeeCommandRenderContext) async {
func renderResults(context: SeeCommandRenderContext) throws {
try Task.checkCancellation()
if self.jsonOutput {
await self.outputJSONResults(context: context)
try self.outputJSONResults(context: context)
} else {
await self.outputTextResults(context: context)
try self.outputTextResults(context: context)
}
}
/// Fetches the menu bar summary only when verbose output is requested, with a short timeout.
private func fetchMenuBarSummaryIfEnabled() async -> MenuBarSummary? {
func fetchMenuBarSummaryIfEnabled() async -> MenuBarSummary? {
guard self.verbose else { return nil }
do {
@ -31,27 +32,21 @@ extension SeeCommand {
}
}
/// Timeout helper that is not MainActor-bound, so it can still fire if the main actor is blocked.
/// Drives the deadline independently while the MainActor operation is suspended.
/// Synchronous MainActor calls cannot be preempted.
static func withWallClockTimeout<T: Sendable>(
seconds: TimeInterval,
operation: @escaping @Sendable () async throws -> T
timeoutErrorSeconds: TimeInterval? = nil,
interactionMutationTracker: InteractionMutationTracker? = nil,
operation: @escaping @MainActor @Sendable () async throws -> T
) async throws -> T {
try await withThrowingTaskGroup(of: T.self) { group in
group.addTask {
try await operation()
}
group.addTask {
try await Task.sleep(nanoseconds: UInt64(seconds * 1_000_000_000))
throw CaptureError.detectionTimedOut(seconds)
}
guard let result = try await group.next() else {
throw CaptureError.detectionTimedOut(seconds)
}
group.cancelAll()
return result
}
try await withMainActorCommandTimeout(
seconds: seconds,
operationName: "see",
timeoutError: { CaptureError.detectionTimedOut(timeoutErrorSeconds ?? seconds) },
interactionMutationTracker: interactionMutationTracker,
operation: { try await operation() }
)
}
func performAnalysisDetailed(imagePath: String, prompt: String) async throws -> SeeAnalysisData {
@ -60,12 +55,7 @@ extension SeeCommand {
return SeeAnalysisData(provider: res.provider, model: res.model, text: res.text)
}
private func buildMenuSummaryIfNeeded() async -> MenuBarSummary? {
// Placeholder for future UI summary generation; currently unused.
nil
}
private func outputJSONResults(context: SeeCommandRenderContext) async {
private func outputJSONResults(context: SeeCommandRenderContext) throws {
let uiElements: [UIElementSummary] = context.elements.all.map { element in
UIElementSummary(
id: element.id,
@ -84,10 +74,6 @@ extension SeeCommand {
let snapshotPaths = self.snapshotPaths(for: context)
// Menu bar enumeration can be slow or hang on some setups. Only attempt it in verbose
// mode and bound it with a short timeout so JSON output is responsive by default.
let menuSummary = await self.fetchMenuBarSummaryIfEnabled()
let output = SeeResult(
snapshot_id: context.snapshotId,
screenshot_raw: snapshotPaths.raw,
@ -102,7 +88,7 @@ extension SeeCommand {
analysis: context.analysis,
execution_time: context.executionTime,
ui_elements: uiElements,
menu_bar: menuSummary,
menu_bar: context.menuBar,
truncation: SeeTruncationSummary(metadata: context.metadata),
observation: context.observation
)
@ -137,7 +123,8 @@ extension SeeCommand {
return MenuBarSummary(menus: menus)
}
private func outputTextResults(context: SeeCommandRenderContext) async {
private func outputTextResults(context: SeeCommandRenderContext) throws {
try Task.checkCancellation()
print("🖼️ Screenshot saved to: \(context.screenshotPath)")
if let annotatedPath = context.annotatedPath {
print("📝 Annotated screenshot: \(annotatedPath)")
@ -180,17 +167,6 @@ extension SeeCommand {
print("\n📝 Annotated screenshot created")
}
if let menuSummary = await self.buildMenuSummaryIfNeeded() {
print("\n🧭 Menu Bar Summary")
for menu in menuSummary.menus {
print("- \(menu.title) (\(menu.enabled ? "Enabled" : "Disabled"))")
for item in menu.items.prefix(5) {
let shortcut = item.keyboard_shortcut.map { " [\($0)]" } ?? ""
print("\(item.title)\(shortcut)")
}
}
}
print("\nSnapshot ID: \(context.snapshotId)")
let terminalCapabilities = TerminalDetector.detectCapabilities()
@ -200,9 +176,10 @@ extension SeeCommand {
}
private func snapshotPaths(for context: SeeCommandRenderContext) -> SnapshotPaths {
SnapshotPaths(
raw: context.screenshotPath,
annotated: context.annotatedPath ?? "",
let publishesScreenshotPaths = !self.usesTemporaryScreenshotOutput
return SnapshotPaths(
raw: publishesScreenshotPaths ? context.screenshotPath : "",
annotated: publishesScreenshotPaths ? context.annotatedPath ?? "" : "",
map: self.services.snapshots.getSnapshotStoragePath() + "/\(context.snapshotId)/snapshot.json"
)
}

View File

@ -40,6 +40,7 @@ struct SeeCommandRenderContext {
let analysis: SeeAnalysisData?
let executionTime: TimeInterval
let observation: SeeObservationDiagnostics?
let menuBar: MenuBarSummary?
}
struct UIElementSummary: Codable {

View File

@ -79,7 +79,7 @@ struct SeeCommand: ApplicationResolvable, ErrorHandlingCommand, RuntimeOptionsCo
@RuntimeStorage private var runtime: CommandRuntime?
var runtimeOptions = CommandRuntimeOptions()
private var resolvedRuntime: CommandRuntime {
var resolvedRuntime: CommandRuntime {
guard let runtime else {
preconditionFailure("CommandRuntime must be configured before accessing runtime resources")
}
@ -113,9 +113,11 @@ struct SeeCommand: ApplicationResolvable, ErrorHandlingCommand, RuntimeOptionsCo
@MainActor
mutating func run(using runtime: CommandRuntime) async throws {
self.runtime = runtime
let startTime = Date()
let commandStartedAt = Date()
let logger = self.logger
let overallTimeout = TimeInterval(self.timeoutSeconds ?? ((self.analyze == nil) ? 20 : 60))
let mutationCoordinator = runtime.toolSnapshotMutationCoordinator
let snapshotManager = runtime.services.snapshots
logger.operationStart("see_command", metadata: [
"app": self.app ?? "none",
@ -128,25 +130,92 @@ struct SeeCommand: ApplicationResolvable, ErrorHandlingCommand, RuntimeOptionsCo
let commandCopy = self
do {
runtime.beginInteractionMutation(preservingSnapshotsCreatedAfterBoundary: true)
try await CrossProcessOperationGate.withExclusiveOperation(
named: CrossProcessOperationGate.desktopObservationName
) {
try await withThrowingTaskGroup(of: Void.self) { group in
group.addTask {
try await commandCopy.runImpl(startTime: startTime, logger: logger)
let observationStartedAt = Date()
let observationDeadline = observationStartedAt.addingTimeInterval(overallTimeout)
let scope = MCPToolSnapshotMutationScope(
toolName: "see",
startedAt: observationStartedAt,
effect: .mutationProducingFreshObservation
)
let reservationTimeout = try Self.remainingObservationTimeout(
until: observationDeadline,
overallTimeout: overallTimeout
)
let snapshotID = try await Self.withWallClockTimeout(
seconds: reservationTimeout,
timeoutErrorSeconds: overallTimeout
) {
try await snapshotManager.createSnapshot(pendingAt: observationStartedAt)
}
defer {
if snapshotManager.copiesScreenshotArtifactsIntoStorage {
commandCopy.cleanupTemporaryScreenshotOutput(snapshotID: snapshotID)
}
group.addTask {
try await Task.sleep(nanoseconds: UInt64(overallTimeout * 1_000_000_000))
throw CaptureError.detectionTimedOut(overallTimeout)
}
var observationCompleted = false
do {
let preparationTimeout = try Self.remainingObservationTimeout(
until: observationDeadline,
overallTimeout: overallTimeout
)
let context = try await Self.withWallClockTimeout(
seconds: preparationTimeout,
timeoutErrorSeconds: overallTimeout,
interactionMutationTracker: runtime.observationTimeoutMutationTracker
) {
try await commandCopy.prepareResult(
startTime: commandStartedAt,
logger: logger,
snapshotID: snapshotID
)
}
observationCompleted = true
let publicationTimeout = try Self.remainingObservationTimeout(
until: observationDeadline,
overallTimeout: overallTimeout
)
let published = try await Self.withWallClockTimeout(
seconds: publicationTimeout,
timeoutErrorSeconds: overallTimeout
) {
await mutationCoordinator.completeMutation(
scope.completed(
at: Date(),
preserving: snapshotID,
confirmedMutationCompletedAt: context.metadata.desktopMutationCompletedAt,
observationPreservationAllowed: context.metadata
.desktopMutationPreservationAllowed
),
succeeded: true
)
}
guard published else {
throw PeekabooError.operationError(
message: "Failed to publish the refreshed UI snapshot"
)
}
do {
_ = try await group.next()
group.cancelAll()
} catch {
group.cancelAll()
throw error
try Task.checkCancellation()
try commandCopy.renderResults(context: context)
commandCopy.emitAnnotationStatus(context: context)
logger.operationComplete("see_command", metadata: [
"executionTimeMs": Int(Date().timeIntervalSince(commandStartedAt) * 1000),
"success": true,
])
} catch {
if observationCompleted || !PendingSnapshotCleanupPolicy.shouldPreserveReservation(after: error) {
try? await self.services.snapshots.cleanSnapshot(snapshotId: snapshotID)
}
_ = await mutationCoordinator.completeMutation(
scope.completed(at: Date(), preserving: nil),
succeeded: false
)
throw error
}
}
} catch {
@ -162,13 +231,29 @@ struct SeeCommand: ApplicationResolvable, ErrorHandlingCommand, RuntimeOptionsCo
}
}
private func runImpl(startTime: Date, logger: Logger) async throws {
private static func remainingObservationTimeout(
until deadline: Date,
overallTimeout: TimeInterval
) throws -> TimeInterval {
let remaining = deadline.timeIntervalSinceNow
guard remaining > 0 else {
throw CaptureError.detectionTimedOut(overallTimeout)
}
return remaining
}
private func prepareResult(
startTime: Date,
logger: Logger,
snapshotID: String
) async throws -> SeeCommandRenderContext {
// ScreenCaptureService performs the authoritative permission check inside each capture path.
// Avoid duplicating that TCC probe here; `see` is often called in latency-sensitive loops.
// Perform capture and element detection
logger.verbose("Starting capture and detection phase", category: "Capture")
let captureResult = try await performCaptureWithDetection()
let captureResult = try await performCaptureWithDetection(snapshotID: snapshotID)
try Task.checkCancellation()
logger.verbose("Capture completed successfully", category: "Capture", metadata: [
"snapshotId": captureResult.snapshotId,
"elementCount": captureResult.elements.all.count,
@ -187,25 +272,19 @@ struct SeeCommand: ApplicationResolvable, ErrorHandlingCommand, RuntimeOptionsCo
snapshotId: captureResult.snapshotId,
originalPath: captureResult.screenshotPath
)
try Task.checkCancellation()
if let annotatedPath,
annotatedPath != captureResult.screenshotPath {
try await self.services.snapshots.storeAnnotatedScreenshot(
snapshotId: captureResult.snapshotId,
annotatedScreenshotPath: annotatedPath
)
try Task.checkCancellation()
}
logger.operationComplete("generate_annotations", metadata: [
"annotatedPath": annotatedPath ?? "none",
])
}
if self.annotate, annotationsAllowed, annotatedPath == nil, !self.jsonOutput {
print("\(AgentDisplayTokens.Status.warning) No interactive UI elements found to annotate")
} else if self.annotate, annotationsAllowed, let annotatedPath, !self.jsonOutput {
let interactableElements = captureResult.elements.all.filter(\.isEnabled)
print("📝 Created annotated screenshot with \(interactableElements.count) interactive elements")
self.logger.verbose("Annotated screenshot path: \(annotatedPath)")
}
// Perform AI analysis if requested
var analysisResult: SeeAnalysisData?
if let prompt = analyze {
@ -227,6 +306,7 @@ struct SeeCommand: ApplicationResolvable, ErrorHandlingCommand, RuntimeOptionsCo
imagePath: captureResult.screenshotPath,
prompt: prompt
)
try Task.checkCancellation()
logger.stopTimer("ai_generate")
logger.operationComplete(
"ai_analysis",
@ -238,14 +318,11 @@ struct SeeCommand: ApplicationResolvable, ErrorHandlingCommand, RuntimeOptionsCo
)
}
// Output results
let executionTime = Date().timeIntervalSince(startTime)
logger.operationComplete("see_command", metadata: [
"executionTimeMs": Int(executionTime * 1000),
"success": true,
])
let menuBarSummary = self.jsonOutput ? await self.fetchMenuBarSummaryIfEnabled() : nil
try Task.checkCancellation()
let context = SeeCommandRenderContext(
let executionTime = Date().timeIntervalSince(startTime)
return SeeCommandRenderContext(
snapshotId: captureResult.snapshotId,
screenshotPath: captureResult.screenshotPath,
annotatedPath: annotatedPath,
@ -253,9 +330,20 @@ struct SeeCommand: ApplicationResolvable, ErrorHandlingCommand, RuntimeOptionsCo
elements: captureResult.elements,
analysis: analysisResult,
executionTime: executionTime,
observation: captureResult.observation
observation: captureResult.observation,
menuBar: menuBarSummary
)
await self.renderResults(context: context)
}
private func emitAnnotationStatus(context: SeeCommandRenderContext) {
let annotationsAllowed = self.allowsAnnotationForCurrentCapture()
if self.annotate, annotationsAllowed, context.annotatedPath == nil, !self.jsonOutput {
print("\(AgentDisplayTokens.Status.warning) No interactive UI elements found to annotate")
} else if self.annotate, annotationsAllowed, let annotatedPath = context.annotatedPath, !self.jsonOutput {
let interactableElements = context.elements.all.filter(\.isEnabled)
print("📝 Created annotated screenshot with \(interactableElements.count) interactive elements")
self.logger.verbose("Annotated screenshot path: \(annotatedPath)")
}
}
func getFileSize(_ path: String) -> Int? {
@ -293,8 +381,8 @@ extension SeeCommand: ParsableCommand {
description: "Capture the frontmost window, print structured output, and save annotations."
),
CommandUsageExample(
command: "peekaboo see --app Safari --window-title \"Login\" --json",
description: "Target a specific Safari window to collect stable element IDs."
command: "peekaboo see --app Safari --window-title \"Login\" --json --path /tmp/safari-login.png",
description: "Target a specific Safari window to collect fresh element IDs and keep the capture artifact in /tmp."
),
CommandUsageExample(
command: "peekaboo see --mode screen --screen-index 0 --analyze 'Summarize the dashboard'",

View File

@ -45,7 +45,9 @@ extension PermissionCommand {
mutating func run(using runtime: CommandRuntime) async throws {
self.prepare(using: runtime)
if await self.renderIfAlreadyGranted() { return }
let result = await self.requestScreenRecordingPermission()
let result = await PermissionHelpers.performInteractivePermissionRequest(using: runtime) {
await self.requestScreenRecordingPermission()
}
self.render(result: result)
}
@ -161,7 +163,9 @@ extension PermissionCommand {
mutating func run(using runtime: CommandRuntime) async throws {
self.prepare(using: runtime)
if await self.renderIfAlreadyGranted() { return }
let granted = self.promptAccessibilityDialog()
let granted = await PermissionHelpers.performInteractivePermissionRequest(using: runtime) {
self.promptAccessibilityDialog()
}
self.renderAccessibilityResult(granted: granted)
}
@ -280,7 +284,10 @@ extension PermissionCommand {
}
private func requestEventSynthesizingPermission() async throws -> AgentPermissionActionResult {
let result = try await PermissionHelpers.requestEventSynthesizingPermission(services: self.services)
let result = try await PermissionHelpers.requestEventSynthesizingPermission(
services: self.services,
runtime: self.resolvedRuntime
)
return AgentPermissionActionResult(
action: result.action,
source: result.source,

View File

@ -22,7 +22,12 @@ extension ErrorHandlingCommand {
} else {
Logger.shared
}
outputError(message: error.localizedDescription, code: errorCode, logger: logger)
outputError(
message: errorMessage(for: error),
code: errorCode,
details: errorDetails(for: error),
logger: logger
)
} else {
let errorMessage: String = if let peekabooError = error as? PeekabooError {
peekabooError.errorDescription ?? String(describing: error)
@ -71,22 +76,22 @@ extension ErrorHandlingCommand {
}
private func mapPeekabooErrorToCode(_ error: PeekabooError) -> ErrorCode {
if let lookupCode = self.lookupErrorCode(for: error) {
if let lookupCode = lookupErrorCode(for: error) {
return lookupCode
}
if let permissionCode = self.permissionErrorCode(for: error) {
if let permissionCode = permissionErrorCode(for: error) {
return permissionCode
}
if let timeoutCode = self.timeoutErrorCode(for: error) {
if let timeoutCode = timeoutErrorCode(for: error) {
return timeoutCode
}
if let automationCode = self.automationErrorCode(for: error) {
if let automationCode = automationErrorCode(for: error) {
return automationCode
}
if let inputCode = self.inputErrorCode(for: error) {
if let inputCode = inputErrorCode(for: error) {
return inputCode
}
if let credentialCode = self.credentialErrorCode(for: error) {
if let credentialCode = credentialErrorCode(for: error) {
return credentialCode
}
return .UNKNOWN_ERROR
@ -220,6 +225,36 @@ extension ErrorHandlingCommand {
}
}
func errorMessage(for error: any Error) -> String {
if let bridgeError = error as? PeekabooBridgeErrorEnvelope {
return bridgeError.message
}
return error.localizedDescription
}
func applicationLaunchErrorCode(for error: any Error) -> ErrorCode? {
guard let bridgeError = error as? PeekabooBridgeErrorEnvelope,
bridgeError.code == .notFound
else {
return nil
}
return .APP_NOT_FOUND
}
func errorDetails(for error: any Error) -> String? {
guard let bridgeError = error as? PeekabooBridgeErrorEnvelope else {
return nil
}
var details: [String] = []
if let bridgeDetails = bridgeError.details, !bridgeDetails.isEmpty {
details.append(bridgeDetails)
}
if let permission = bridgeError.permission {
details.append("permission: \(permission.rawValue)")
}
return details.isEmpty ? nil : details.joined(separator: "\n")
}
func errorCode(for focusError: FocusError) -> ErrorCode {
switch focusError {
case .applicationNotRunning:
@ -233,10 +268,29 @@ func errorCode(for focusError: FocusError) -> ErrorCode {
func errorCode(for bridgeError: PeekabooBridgeErrorEnvelope) -> ErrorCode {
switch bridgeError.code {
case .permissionDenied:
switch bridgeError.permission {
case .screenRecording:
.PERMISSION_ERROR_SCREEN_RECORDING
case .accessibility:
.PERMISSION_ERROR_ACCESSIBILITY
case .postEvent:
.PERMISSION_ERROR_EVENT_SYNTHESIZING
case .appleScript:
.PERMISSION_ERROR_APPLESCRIPT
case .none:
.PERMISSION_DENIED
}
case .timeout:
.TIMEOUT
default:
.INTERNAL_SWIFT_ERROR
case .invalidRequest:
.INVALID_ARGUMENT
case .operationNotSupported:
.VALIDATION_ERROR
case .notFound:
.UNKNOWN_ERROR
case .versionMismatch, .unauthorizedClient, .decodingFailed, .internalError, .serverBusy:
.UNKNOWN_ERROR
}
}

View File

@ -20,9 +20,21 @@ struct CommandRuntimeOptions {
var captureEnginePreference: String?
var inputStrategy: UIInputStrategy?
var preferRemote = true
var remoteIsolationRequested = false
var autoStartDaemon = true
var bridgeSocketPath: String?
var requiresElementActions = false
var requiresInspectAccessibilityTree = false
var requiresBrowserMCP = false
var requiresApplicationLaunchOptions = false
var requiresApplicationRelaunch = false
var requiresSurvivingApplicationHost = false
var requiresHostApplicationInventory = false
var requiresImplicitSnapshotInvalidation = false
var requiresCallerDesktopMutationBarrier = false
var usesPerToolSnapshotInvalidation = false
var requiresExactWindowTargetedClicks = false
var requiresPostEventClickPermission = false
func makeConfiguration() -> CommandRuntime.Configuration {
CommandRuntime.Configuration(
@ -39,7 +51,9 @@ struct CommandRuntimeOptions {
if options.captureEnginePreference == nil,
let captureEngine = Self.captureEnginePreference(environment: environment) {
options.captureEnginePreference = captureEngine
options.preferRemote = false
if !options.requiresApplicationLaunchOptions && !options.requiresHostApplicationInventory {
options.preferRemote = false
}
}
return options
}
@ -71,14 +85,32 @@ struct CommandRuntime {
let configuration: Configuration
let hostDescription: String
let selectedRemoteSocketPath: String?
let selectedRemoteHostProcessIdentifier: pid_t?
let snapshotInvalidationRemoteSocketPaths: [String]
let applicationRelaunchAllowed: Bool
let interactionMutationTracker: InteractionMutationTracker
@MainActor let services: any PeekabooServiceProviding
@MainActor let logger: Logger
@MainActor
var observationTimeoutMutationTracker: InteractionMutationTracker? {
if self.selectedRemoteSocketPath == nil || self.interactionMutationTracker.hasPendingDurableMutation {
return self.interactionMutationTracker
}
return nil
}
@MainActor
init(
configuration: Configuration,
services: any PeekabooServiceProviding,
hostDescription: String = "local (in-process)"
hostDescription: String = "local (in-process)",
selectedRemoteSocketPath: String? = nil,
selectedRemoteHostProcessIdentifier: pid_t? = nil,
snapshotInvalidationRemoteSocketPaths: [String] = [],
applicationRelaunchAllowed: Bool = true,
interactionMutationTracker: InteractionMutationTracker = InteractionMutationTracker()
) {
// Keep Tachikoma credential/profile resolution aligned with Peekaboo CLI storage.
PeekabooCore.ConfigurationManager.configureTachikomaProfileDirectory()
@ -86,6 +118,11 @@ struct CommandRuntime {
self.configuration = configuration
self.services = services
self.hostDescription = hostDescription
self.selectedRemoteSocketPath = selectedRemoteSocketPath
self.selectedRemoteHostProcessIdentifier = selectedRemoteHostProcessIdentifier
self.snapshotInvalidationRemoteSocketPaths = snapshotInvalidationRemoteSocketPaths
self.applicationRelaunchAllowed = applicationRelaunchAllowed
self.interactionMutationTracker = interactionMutationTracker
self.logger = Logger.shared
services.installAgentRuntimeDefaults()
@ -148,15 +185,19 @@ extension CommandRuntime {
@MainActor
static func makeDefaultAsync(options: CommandRuntimeOptions) async -> CommandRuntime {
let effectiveOptions = options.applyingEnvironmentOverrides(environment: ProcessInfo.processInfo.environment)
if let override = self.serviceOverride {
if let override = serviceOverride {
return CommandRuntime(options: effectiveOptions, services: override)
}
let resolution = await self.resolveServices(options: effectiveOptions)
let resolution = await resolveServices(options: effectiveOptions)
return CommandRuntime(
configuration: effectiveOptions.makeConfiguration(),
services: resolution.services,
hostDescription: resolution.hostDescription
hostDescription: resolution.hostDescription,
selectedRemoteSocketPath: resolution.selectedRemoteSocketPath,
selectedRemoteHostProcessIdentifier: resolution.selectedRemoteHostProcessIdentifier,
snapshotInvalidationRemoteSocketPaths: resolution.snapshotInvalidationRemoteSocketPaths,
applicationRelaunchAllowed: resolution.applicationRelaunchAllowed
)
}
@ -176,8 +217,7 @@ extension CommandRuntime {
}
@MainActor
private static func resolveServices(options: CommandRuntimeOptions)
async -> (services: any PeekabooServiceProviding, hostDescription: String) {
private static func resolveServices(options: CommandRuntimeOptions) async -> RuntimeHostResolver.Resolution {
await RuntimeHostResolver.resolveServices(options: options)
}
@ -231,10 +271,26 @@ extension CommandRuntime {
BridgeCapabilityPolicy.supportsTargetedHotkeys(for: handshake)
}
static func supportsTargetedTypeActions(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
BridgeCapabilityPolicy.supportsTargetedTypeActions(for: handshake)
}
static func supportsTargetedClicks(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
BridgeCapabilityPolicy.supportsTargetedClicks(for: handshake)
}
static func supportsApplicationLaunchOptions(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
BridgeCapabilityPolicy.supportsApplicationLaunchOptions(for: handshake)
}
static func supportsApplicationRelaunch(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
BridgeCapabilityPolicy.supportsApplicationRelaunch(for: handshake)
}
static func supportsImplicitSnapshotInvalidation(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
BridgeCapabilityPolicy.supportsImplicitSnapshotInvalidation(for: handshake)
}
static func supportsElementActions(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
BridgeCapabilityPolicy.supportsElementActions(for: handshake)
}
@ -247,6 +303,10 @@ extension CommandRuntime {
BridgeCapabilityPolicy.supportsInspectAccessibilityTree(for: handshake)
}
static func supportsBrowserMCP(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
BridgeCapabilityPolicy.supportsBrowserMCP(for: handshake)
}
static func supportsPostEventPermissionRequest(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
BridgeCapabilityPolicy.supportsPostEventPermissionRequest(for: handshake)
}
@ -256,6 +316,11 @@ extension CommandRuntime {
BridgeCapabilityPolicy.targetedHotkeyAvailability(for: handshake)
}
static func targetedTypeAvailability(for handshake: PeekabooBridgeHandshakeResponse)
-> (isEnabled: Bool, unavailableReason: String?, missingPermissions: Set<PeekabooBridgePermissionKind>) {
BridgeCapabilityPolicy.targetedTypeAvailability(for: handshake)
}
static func targetedClickAvailability(for handshake: PeekabooBridgeHandshakeResponse)
-> (isEnabled: Bool, unavailableReason: String?, missingPermissions: Set<PeekabooBridgePermissionKind>) {
BridgeCapabilityPolicy.targetedClickAvailability(for: handshake)
@ -270,7 +335,7 @@ protocol RuntimeOptionsConfigurable {
extension RuntimeOptionsConfigurable {
mutating func setRuntimeOptions(_ options: CommandRuntimeOptions) {
self.runtimeOptions = options
runtimeOptions = options
}
}

View File

@ -42,7 +42,8 @@ enum AutomationServiceBridge {
target: ClickTarget,
clickType: ClickType,
snapshotId: String?,
targetProcessIdentifier: pid_t
targetProcessIdentifier: pid_t,
targetWindowID: Int? = nil
) async throws {
try await Task { @MainActor in
guard let targetedClickService = automation as? any TargetedClickServiceProtocol else {
@ -55,12 +56,28 @@ enum AutomationServiceBridge {
throw self.targetedClickUnavailableError(service: targetedClickService)
}
try await targetedClickService.click(
target: target,
clickType: clickType,
snapshotId: snapshotId,
targetProcessIdentifier: targetProcessIdentifier
)
if let targetWindowID {
guard let exactWindowService = targetedClickService as? any ExactWindowTargetedClickServiceProtocol
else {
throw PeekabooError.serviceUnavailable(
"Background clicks with an exact window require a compatible automation service"
)
}
try await exactWindowService.click(
target: target,
clickType: clickType,
snapshotId: snapshotId,
targetProcessIdentifier: targetProcessIdentifier,
targetWindowID: targetWindowID
)
} else {
try await targetedClickService.click(
target: target,
clickType: clickType,
snapshotId: snapshotId,
targetProcessIdentifier: targetProcessIdentifier
)
}
}.value
}
@ -77,6 +94,31 @@ enum AutomationServiceBridge {
}.value
}
static func typeActions(
automation: any UIAutomationServiceProtocol,
request: TypeActionsRequest,
targetProcessIdentifier: pid_t
) async throws -> TypeResult {
try await Task { @MainActor in
guard let targetedTypeService = automation as? any TargetedTypeServiceProtocol else {
throw PeekabooError.serviceUnavailable(
"Background typing requires an automation service that supports targeted type delivery"
)
}
guard targetedTypeService.supportsTargetedTypeActions else {
throw self.targetedTypeUnavailableError(service: targetedTypeService)
}
return try await targetedTypeService.typeActions(
request.actions,
cadence: request.cadence,
snapshotId: request.snapshotId,
targetProcessIdentifier: targetProcessIdentifier
)
}.value
}
static func scroll(
automation: any UIAutomationServiceProtocol,
request: ScrollRequest
@ -160,6 +202,17 @@ enum AutomationServiceBridge {
)
}
private static func targetedTypeUnavailableError(service: any TargetedTypeServiceProtocol) -> PeekabooError {
if service.targetedTypeRequiresEventSynthesizingPermission {
return .permissionDeniedEventSynthesizing
}
return .serviceUnavailable(
service.targetedTypeUnavailableReason ??
"Remote bridge host does not support background typing; use --no-remote or update the host"
)
}
private static func targetedClickUnavailableError(service: any TargetedClickServiceProtocol) -> PeekabooError {
if service.targetedClickRequiresEventSynthesizingPermission {
return .permissionDeniedEventSynthesizing

View File

@ -1,5 +1,6 @@
import CoreGraphics
import Foundation
import PeekabooAutomationKit
import PeekabooCore
import PeekabooFoundation
@ -126,7 +127,11 @@ func withCommandTimeout<T: Sendable>(
}
let timeoutTask = Task.detached {
try? await Task.sleep(nanoseconds: UInt64(seconds * 1_000_000_000))
do {
try await Task.sleep(nanoseconds: UInt64(seconds * 1_000_000_000))
} catch {
return
}
race.resume(with: Result<T, any Error>.failure(PeekabooError.timeout(
operation: operationName,
duration: seconds
@ -139,6 +144,7 @@ func withCommandTimeout<T: Sendable>(
race.setContinuation(continuation)
}
} onCancel: {
race.resume(with: Result<T, any Error>.failure(CancellationError()))
workTask.cancel()
timeoutTask.cancel()
}
@ -148,6 +154,9 @@ func withCommandTimeout<T: Sendable>(
func withMainActorCommandTimeout<T: Sendable>(
seconds: TimeInterval,
operationName: String,
timeoutError: (@Sendable () -> any Error)? = nil,
desktopMutationWatermarkStore: DesktopMutationWatermarkStore? = nil,
interactionMutationTracker: InteractionMutationTracker? = nil,
operation: @escaping @MainActor () async throws -> T
) async throws -> T {
guard seconds > 0 else {
@ -155,21 +164,37 @@ func withMainActorCommandTimeout<T: Sendable>(
}
let race = TimeoutRace()
let workTask = Task { @MainActor in
do {
let value = try await operation()
race.resume(with: .success(value))
} catch {
race.resume(with: Result<T, any Error>.failure(error))
let pendingMutation = try desktopMutationWatermarkStore?.beginMutation()
do {
try interactionMutationTracker?.retainDurableMutationLease()
} catch {
if let desktopMutationWatermarkStore, let pendingMutation {
try? desktopMutationWatermarkStore.cancelMutation(pendingMutation)
}
throw error
}
let workTask = Task { @MainActor in
let result: Result<T, any Error>
do {
result = try await .success(operation())
} catch {
result = .failure(error)
}
if let desktopMutationWatermarkStore, let pendingMutation {
_ = try? desktopMutationWatermarkStore.completeMutation(pendingMutation)
}
_ = try? interactionMutationTracker?.completeDurableMutation(through: Date())
race.resume(with: result)
}
let timeoutTask = Task.detached {
try? await Task.sleep(nanoseconds: UInt64(seconds * 1_000_000_000))
race.resume(with: Result<T, any Error>.failure(PeekabooError.timeout(
operation: operationName,
duration: seconds
)))
do {
try await Task.sleep(nanoseconds: UInt64(seconds * 1_000_000_000))
} catch {
return
}
let error = timeoutError?() ?? PeekabooError.timeout(operation: operationName, duration: seconds)
race.resume(with: Result<T, any Error>.failure(error))
workTask.cancel()
}
@ -178,6 +203,7 @@ func withMainActorCommandTimeout<T: Sendable>(
race.setContinuation(continuation)
}
} onCancel: {
race.resume(with: Result<T, any Error>.failure(CancellationError()))
workTask.cancel()
timeoutTask.cancel()
}

View File

@ -10,7 +10,7 @@ enum CommanderCLIBinder {
parsedValues: ParsedValues
) throws -> any ParsableCommand {
var command = type.init()
let runtimeOptions = try self.makeRuntimeOptions(from: parsedValues, commandType: type)
let runtimeOptions = try makeRuntimeOptions(from: parsedValues, commandType: type)
if var bindable = command as? any CommanderBindableCommand {
try bindable.applyCommanderValues(.init(parsedValues: parsedValues))
guard let rebound = bindable as? any ParsableCommand else {
@ -43,6 +43,31 @@ enum CommanderCLIBinder {
commandType: (any ParsableCommand.Type)? = nil
) throws -> CommandRuntimeOptions {
var options = CommandRuntimeOptions()
options.requiresApplicationLaunchOptions = Self.requiresApplicationLaunchOptions(commandType)
options.requiresApplicationRelaunch = commandType == AppCommand.RelaunchSubcommand.self
options.requiresSurvivingApplicationHost = commandType == AppCommand.QuitSubcommand.self
options.requiresHostApplicationInventory = Self.requiresHostApplicationInventory(commandType)
options.requiresImplicitSnapshotInvalidation = Self.requiresImplicitSnapshotInvalidation(
commandType,
parsedValues: parsedValues
)
let clipboardMayMutate = commandType == ClipboardCommand.self &&
Self.clipboardMayMutate(parsedValues)
options.requiresCallerDesktopMutationBarrier = commandType == SwitchSubcommand.self ||
commandType == MoveWindowSubcommand.self ||
commandType == CaptureActionCommand.self ||
clipboardMayMutate
options.requiresExactWindowTargetedClicks = Self.requiresExactWindowTargetedClicks(
commandType,
parsedValues: parsedValues
)
options.requiresPostEventClickPermission = Self.requiresPostEventClickPermission(
commandType,
parsedValues: parsedValues
)
options.usesPerToolSnapshotInvalidation = commandType == AgentCommand.self ||
commandType == MCPCommand.Serve.self ||
commandType == InspectUICommand.self
options.verbose = parsedValues.flags.contains("verbose")
options.jsonOutput = parsedValues.flags.contains("jsonOutput")
let values = CommanderBindableValues(parsedValues: parsedValues)
@ -53,7 +78,9 @@ enum CommanderCLIBinder {
.trimmingCharacters(in: .whitespacesAndNewlines),
!captureEngine.isEmpty {
options.captureEnginePreference = captureEngine
options.preferRemote = false
if !options.requiresApplicationLaunchOptions && !options.requiresHostApplicationInventory {
options.preferRemote = false
}
}
if let rawInputStrategy = values.singleOption("inputStrategy")?
.trimmingCharacters(in: .whitespacesAndNewlines),
@ -66,10 +93,10 @@ enum CommanderCLIBinder {
)
}
options.inputStrategy = strategy
options.preferRemote = false
}
if values.flag("no-remote") {
options.preferRemote = false
options.remoteIsolationRequested = true
}
let explicitBridgeSocket = values.singleOption("bridge-socket")?.trimmingCharacters(in: .whitespacesAndNewlines)
if commandType == AgentCommand.self && !values.flag("no-remote") {
@ -80,8 +107,10 @@ enum CommanderCLIBinder {
options.preferRemote = false
options.autoStartDaemon = false
}
if Self.prefersLocalRuntime(commandType), !values.flag("no-remote"),
explicitBridgeSocket?.isEmpty ?? true {
if Self.requiresCallerLocalRuntime(commandType) {
options.preferRemote = false
} else if Self.prefersLocalRuntime(commandType), !values.flag("no-remote"),
explicitBridgeSocket?.isEmpty ?? true {
options.preferRemote = false
}
if let socketPath = explicitBridgeSocket, !socketPath.isEmpty {
@ -90,9 +119,241 @@ enum CommanderCLIBinder {
if commandType == SetValueCommand.self || commandType == PerformActionCommand.self {
options.requiresElementActions = true
}
if commandType == InspectUICommand.self {
options.requiresInspectAccessibilityTree = true
}
if commandType == BrowserCommand.self {
options.requiresBrowserMCP = true
}
return options
}
private static func requiresApplicationLaunchOptions(_ commandType: (any ParsableCommand.Type)?) -> Bool {
commandType == OpenCommand.self ||
commandType == AppCommand.LaunchSubcommand.self ||
commandType == AppCommand.RelaunchSubcommand.self
}
private static func requiresHostApplicationInventory(_ commandType: (any ParsableCommand.Type)?) -> Bool {
commandType == ListCommand.AppsSubcommand.self ||
commandType == AppCommand.ListSubcommand.self
}
private static func requiresImplicitSnapshotInvalidation(
_ commandType: (any ParsableCommand.Type)?,
parsedValues: ParsedValues
) -> Bool {
if commandType == ClipboardCommand.self {
return self.clipboardMayMutate(parsedValues)
}
if commandType == MenuBarCommand.self {
return parsedValues.positional.first?.lowercased() == "click"
}
if commandType == BrowserCommand.self {
return BrowserCommand.actionMayMutate(parsedValues.positional.first ?? "status")
}
if commandType == SeeCommand.self {
return true
}
if self.isInteractivePermissionRequest(commandType) {
return true
}
if commandType == DialogCommand.ListSubcommand.self {
return self.dialogListMayFocus(parsedValues)
}
if commandType == MenuCommand.ListSubcommand.self {
return self.menuListMayFocus(parsedValues)
}
if commandType == ImageCommand.self ||
commandType == CaptureLiveCommand.self ||
commandType == CaptureWatchAlias.self {
return self.captureCommandMayFocus(commandType, parsedValues: parsedValues)
}
return commandType == OpenCommand.self ||
commandType == AppCommand.LaunchSubcommand.self ||
commandType == AppCommand.RelaunchSubcommand.self ||
commandType == AppCommand.QuitSubcommand.self ||
commandType == AppCommand.HideSubcommand.self ||
commandType == AppCommand.UnhideSubcommand.self ||
commandType == AppCommand.SwitchSubcommand.self ||
commandType == ClickCommand.self ||
commandType == MoveCommand.self ||
commandType == TypeCommand.self ||
commandType == PressCommand.self ||
commandType == HotkeyCommand.self ||
commandType == PasteCommand.self ||
commandType == ScrollCommand.self ||
commandType == SwipeCommand.self ||
commandType == DragCommand.self ||
commandType == SetValueCommand.self ||
commandType == PerformActionCommand.self ||
commandType == CaptureActionCommand.self ||
commandType == WindowCommand.FocusSubcommand.self ||
commandType == WindowCommand.CloseSubcommand.self ||
commandType == WindowCommand.MinimizeSubcommand.self ||
commandType == WindowCommand.MaximizeSubcommand.self ||
commandType == WindowCommand.MoveSubcommand.self ||
commandType == WindowCommand.ResizeSubcommand.self ||
commandType == WindowCommand.SetBoundsSubcommand.self ||
commandType == DialogCommand.ClickSubcommand.self ||
commandType == DialogCommand.DismissSubcommand.self ||
commandType == DialogCommand.InputSubcommand.self ||
commandType == DialogCommand.FileSubcommand.self ||
commandType == MenuCommand.ClickSubcommand.self ||
commandType == MenuCommand.ClickExtraSubcommand.self ||
commandType == DockCommand.LaunchSubcommand.self ||
commandType == DockCommand.RightClickSubcommand.self ||
commandType == DockCommand.HideSubcommand.self ||
commandType == DockCommand.ShowSubcommand.self ||
commandType == SwitchSubcommand.self ||
commandType == MoveWindowSubcommand.self ||
commandType == RunCommand.self
}
private static func isInteractivePermissionRequest(
_ commandType: (any ParsableCommand.Type)?
) -> Bool {
commandType == PermissionsCommand.RequestScreenRecordingSubcommand.self ||
commandType == PermissionsCommand.RequestEventSynthesizingSubcommand.self ||
commandType == PermissionCommand.RequestScreenRecordingSubcommand.self ||
commandType == PermissionCommand.RequestAccessibilitySubcommand.self ||
commandType == PermissionCommand.RequestEventSynthesizingSubcommand.self
}
private static func clipboardMayMutate(_ parsedValues: ParsedValues) -> Bool {
let values = CommanderBindableValues(parsedValues: parsedValues)
let positionalAction = values.positionalValue(at: 0)?
.trimmingCharacters(in: .whitespacesAndNewlines)
let action = (positionalAction?.isEmpty == false ? positionalAction : nil) ??
values.singleOption("actionOption") ??
values.singleOption("action")
return ClipboardCommand.actionMayMutate(action)
}
private static func menuListMayFocus(_ parsedValues: ParsedValues) -> Bool {
let values = CommanderBindableValues(parsedValues: parsedValues)
return !values.flag("noAutoFocus")
}
private static func dialogListMayFocus(_ parsedValues: ParsedValues) -> Bool {
let values = CommanderBindableValues(parsedValues: parsedValues)
let hasWindowTarget = values.singleOption("windowId") != nil ||
values.singleOption("windowTitle") != nil ||
values.singleOption("windowIndex") != nil
if hasWindowTarget {
return true
}
guard !values.flag("noAutoFocus") else { return false }
let app = values.singleOption("app")?
.trimmingCharacters(in: .whitespacesAndNewlines)
return app?.isEmpty == false ||
values.singleOption("pid") != nil
}
private static func captureCommandMayFocus(
_ commandType: (any ParsableCommand.Type)?,
parsedValues: ParsedValues
) -> Bool {
let values = CommanderBindableValues(parsedValues: parsedValues)
let focus = values.singleOption("captureFocus")?
.trimmingCharacters(in: .whitespacesAndNewlines)
.lowercased()
guard focus != "background" else { return false }
let app = values.singleOption("app")?
.trimmingCharacters(in: .whitespacesAndNewlines)
let hasApplicationTarget = app?.isEmpty == false || values.singleOption("pid") != nil
if commandType == ImageCommand.self {
let normalizedApp = app?.lowercased()
guard normalizedApp != "menubar", normalizedApp != "frontmost" else { return false }
let mode = values.singleOption("mode")?
.trimmingCharacters(in: .whitespacesAndNewlines)
.lowercased() ?? Self.inferredImageCaptureMode(values)
switch mode {
case "window":
return values.singleOption("windowId") == nil && hasApplicationTarget
case "multi":
return hasApplicationTarget
default:
return false
}
}
let mode = values.singleOption("mode")?
.trimmingCharacters(in: .whitespacesAndNewlines)
.lowercased() ?? Self.inferredLiveCaptureMode(values)
return mode == "window" && hasApplicationTarget
}
private static func inferredImageCaptureMode(_ values: CommanderBindableValues) -> String {
if values.singleOption("region") != nil { return "area" }
if values.singleOption("app") != nil ||
values.singleOption("pid") != nil ||
values.singleOption("windowTitle") != nil ||
values.singleOption("windowIndex") != nil ||
values.singleOption("windowId") != nil {
return "window"
}
return "frontmost"
}
private static func inferredLiveCaptureMode(_ values: CommanderBindableValues) -> String {
if values.singleOption("region") != nil { return "area" }
if values.singleOption("app") != nil ||
values.singleOption("pid") != nil ||
values.singleOption("windowTitle") != nil ||
values.singleOption("windowIndex") != nil {
return "window"
}
return "frontmost"
}
private static func requiresExactWindowTargetedClicks(
_ commandType: (any ParsableCommand.Type)?,
parsedValues: ParsedValues
) -> Bool {
guard commandType == ClickCommand.self else { return false }
let values = CommanderBindableValues(parsedValues: parsedValues)
guard self.usesBackgroundClickDelivery(values) else { return false }
let hasWindowSelector = values.singleOption("windowId") != nil ||
values.singleOption("windowTitle") != nil ||
values.singleOption("windowIndex") != nil
if hasWindowSelector {
return true
}
let hasProcessTarget = values.singleOption("app") != nil || values.singleOption("pid") != nil
return values.singleOption("coords") != nil && hasProcessTarget && !values.flag("globalCoords")
}
private static func requiresPostEventClickPermission(
_ commandType: (any ParsableCommand.Type)?,
parsedValues: ParsedValues
) -> Bool {
guard commandType == ClickCommand.self else { return false }
let values = CommanderBindableValues(parsedValues: parsedValues)
guard self.usesBackgroundClickDelivery(values) else { return false }
if values.singleOption("coords") != nil {
return true
}
// ClickCommand resolves conflicting flags as right-click first, then double-click.
return values.flag("double") && !values.flag("right")
}
private static func usesBackgroundClickDelivery(_ values: CommanderBindableValues) -> Bool {
if values.flag("focusBackground") { return true }
return !values.flag("foreground") &&
!values.flag("noAutoFocus") &&
!values.flag("spaceSwitch") &&
!values.flag("bringToCurrentSpace") &&
values.singleOption("focusTimeoutSeconds") == nil &&
values.singleOption("focusRetryCount") == nil
}
private static func prefersLocalRuntime(_ commandType: (any ParsableCommand.Type)?) -> Bool {
commandType == MCPCommand.Serve.self ||
commandType == ToolsCommand.self ||
@ -111,9 +372,16 @@ enum CommanderCLIBinder {
commandType == ConfigCommand.TestProviderCommand.self ||
commandType == ConfigCommand.RemoveProviderCommand.self ||
commandType == ConfigCommand.ModelsProviderCommand.self ||
commandType == AppCommand.ListSubcommand.self ||
commandType == ListCommand.AppsSubcommand.self ||
commandType == ListCommand.ScreensSubcommand.self
commandType == ListCommand.ScreensSubcommand.self ||
commandType == PermissionsCommand.RequestScreenRecordingSubcommand.self ||
commandType == PermissionCommand.RequestScreenRecordingSubcommand.self ||
commandType == PermissionCommand.RequestAccessibilitySubcommand.self
}
private static func requiresCallerLocalRuntime(_ commandType: (any ParsableCommand.Type)?) -> Bool {
commandType == PermissionsCommand.RequestScreenRecordingSubcommand.self ||
commandType == PermissionCommand.RequestScreenRecordingSubcommand.self ||
commandType == PermissionCommand.RequestAccessibilitySubcommand.self
}
private static func isDaemonCommand(_ commandType: (any ParsableCommand.Type)?) -> Bool {
@ -279,7 +547,9 @@ extension CommanderBindableValues {
options.noAutoFocus = self.flag("noAutoFocus")
options.spaceSwitch = self.flag("spaceSwitch")
options.bringToCurrentSpace = self.flag("bringToCurrentSpace")
options.focusBackground = includeBackgroundDelivery && self.flag("focusBackground")
if includeBackgroundDelivery && self.flag("focusBackground") {
options.focusBackground = true
}
if let timeout: TimeInterval = try decodeOption("focusTimeoutSeconds", as: TimeInterval.self) {
options.focusTimeoutSeconds = timeout
}

View File

@ -12,7 +12,46 @@ enum BridgeCapabilityPolicy {
return false
}
if options.requiresElementActions && !self.supportsElementActions(for: handshake) {
if options.requiresElementActions, !self.supportsElementActions(for: handshake) {
return false
}
if options.requiresInspectAccessibilityTree, !self.supportsInspectAccessibilityTree(for: handshake) {
return false
}
if options.requiresBrowserMCP, !self.supportsBrowserMCP(for: handshake) {
return false
}
if options.requiresApplicationLaunchOptions, !self.supportsApplicationLaunchOptions(for: handshake) {
return false
}
if options.requiresApplicationRelaunch, !self.supportsApplicationRelaunch(for: handshake) {
return false
}
if options.requiresSurvivingApplicationHost, handshake.hostKind != .onDemand {
return false
}
if options.requiresHostApplicationInventory, !self.supportsHostApplicationInventory(for: handshake) {
return false
}
if options.requiresExactWindowTargetedClicks,
!self.supportsExactWindowTargetedClicks(for: handshake) {
return false
}
if options.requiresPostEventClickPermission,
handshake.permissions?.postEvent != true {
return false
}
if options.requiresImplicitSnapshotInvalidation || options.usesPerToolSnapshotInvalidation,
!self.supportsImplicitSnapshotInvalidation(for: handshake) {
return false
}
@ -23,10 +62,53 @@ enum BridgeCapabilityPolicy {
self.targetedHotkeyAvailability(for: handshake).isEnabled
}
static func supportsTargetedTypeActions(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
self.targetedTypeAvailability(for: handshake).isEnabled
}
static func supportsTargetedClicks(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
self.targetedClickAvailability(for: handshake).isEnabled
}
static func supportsApplicationLaunchOptions(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
handshake.negotiatedVersion >= PeekabooBridgeProtocolVersion(major: 1, minor: 9) &&
handshake.supportedOperations.contains(.launchApplicationWithOptions)
}
static func supportsApplicationRelaunch(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
guard handshake.hostKind == .onDemand,
handshake.negotiatedVersion >= PeekabooBridgeProtocolVersion(major: 1, minor: 9),
handshake.supportedOperations.contains(.relaunchApplicationWithOptions)
else {
return false
}
let enabledOperations = handshake.enabledOperations ?? handshake.supportedOperations
return enabledOperations.contains(.relaunchApplicationWithOptions)
}
static func supportsHostApplicationInventory(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
guard handshake.negotiatedVersion >= PeekabooBridgeProtocolVersion(major: 1, minor: 0),
handshake.supportedOperations.contains(.listApplications)
else {
return false
}
let enabledOperations = handshake.enabledOperations ?? handshake.supportedOperations
return enabledOperations.contains(.listApplications)
}
static func supportsImplicitSnapshotInvalidation(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
guard handshake.negotiatedVersion >= PeekabooBridgeProtocolVersion(major: 1, minor: 9),
handshake.supportedOperations.contains(.invalidateImplicitLatestSnapshot)
else {
return false
}
let enabledOperations = handshake.enabledOperations ?? handshake.supportedOperations
return enabledOperations.contains(.invalidateImplicitLatestSnapshot)
}
static func supportsElementActions(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
handshake.negotiatedVersion >= PeekabooBridgeProtocolVersion(major: 1, minor: 3) &&
handshake.supportedOperations.contains(.setValue) &&
@ -43,6 +125,14 @@ enum BridgeCapabilityPolicy {
handshake.supportedOperations.contains(.inspectAccessibilityTree)
}
static func supportsBrowserMCP(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
handshake.negotiatedVersion >= PeekabooBridgeProtocolVersion(major: 1, minor: 4) &&
handshake.supportedOperations.contains(.browserStatus) &&
handshake.supportedOperations.contains(.browserConnect) &&
handshake.supportedOperations.contains(.browserDisconnect) &&
handshake.supportedOperations.contains(.browserExecute)
}
static func supportsPostEventPermissionRequest(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
handshake.negotiatedVersion >= PeekabooBridgeProtocolVersion(major: 1, minor: 2) &&
handshake.supportedOperations.contains(.requestPostEventPermission)
@ -62,7 +152,7 @@ enum BridgeCapabilityPolicy {
return (true, nil, [])
}
let missingPermissions = self.missingPermissions(for: .targetedHotkey, handshake: handshake)
let missingPermissions = missingPermissions(for: .targetedHotkey, handshake: handshake)
guard !missingPermissions.isEmpty else {
return (
false,
@ -90,10 +180,25 @@ enum BridgeCapabilityPolicy {
let enabledOperations = handshake.enabledOperations ?? handshake.supportedOperations
if enabledOperations.contains(.targetedClick) {
return (true, nil, [])
let missingVariantPermissions: Set<PeekabooBridgePermissionKind> =
handshake.permissions?.postEvent == false ? [.postEvent] : []
return (true, nil, missingVariantPermissions)
}
let missingPermissions = self.missingPermissions(for: .targetedClick, handshake: handshake)
let requestAwarePermissions =
handshake.negotiatedVersion >= PeekabooBridgeProtocolVersion(major: 1, minor: 9) &&
handshake.permissionTags[PeekabooBridgeOperation.targetedClick.rawValue]?.isEmpty == true
if requestAwarePermissions,
handshake.permissions?.accessibility == false,
handshake.permissions?.postEvent == false {
return (
false,
"Remote bridge host background clicks require Accessibility or Event Synthesizing permission",
[]
)
}
let missingPermissions = missingPermissions(for: .targetedClick, handshake: handshake)
guard !missingPermissions.isEmpty else {
return (
false,
@ -110,6 +215,47 @@ enum BridgeCapabilityPolicy {
)
}
static func supportsExactWindowTargetedClicks(for handshake: PeekabooBridgeHandshakeResponse) -> Bool {
guard handshake.negotiatedVersion >= PeekabooBridgeProtocolVersion(major: 1, minor: 9),
handshake.supportedOperations.contains(.exactWindowTargetedClick)
else {
return false
}
return (handshake.enabledOperations ?? handshake.supportedOperations)
.contains(.exactWindowTargetedClick)
}
static func targetedTypeAvailability(for handshake: PeekabooBridgeHandshakeResponse)
-> (isEnabled: Bool, unavailableReason: String?, missingPermissions: Set<PeekabooBridgePermissionKind>) {
guard
handshake.negotiatedVersion >= PeekabooBridgeProtocolVersion(major: 1, minor: 8),
handshake.supportedOperations.contains(.targetedTypeActions)
else {
return (false, nil, [])
}
let enabledOperations = handshake.enabledOperations ?? handshake.supportedOperations
if enabledOperations.contains(.targetedTypeActions) {
return (true, nil, [])
}
let missingPermissions = missingPermissions(for: .targetedTypeActions, handshake: handshake)
guard !missingPermissions.isEmpty else {
return (
false,
"Remote bridge host supports background typing, but it is disabled by current permissions",
[]
)
}
return (
false,
"Remote bridge host supports background typing, but current permissions are missing: " +
self.missingPermissionNames(missingPermissions).joined(separator: ", "),
missingPermissions
)
}
private static func missingPermissions(
for operation: PeekabooBridgeOperation,
handshake: PeekabooBridgeHandshakeResponse
@ -117,7 +263,7 @@ enum BridgeCapabilityPolicy {
let requiredPermissions = Set(
handshake.permissionTags[operation.rawValue] ?? Array(operation.requiredPermissions)
)
let grantedPermissions = self.grantedPermissions(from: handshake.permissions)
let grantedPermissions = grantedPermissions(from: handshake.permissions)
return requiredPermissions.subtracting(grantedPermissions)
}

View File

@ -1,8 +1,34 @@
import Darwin
import Foundation
import MachO
import PeekabooBridge
enum DaemonLaunchPolicy {
enum ImplicitRuntimeCandidateRole: Equatable {
case reusableDaemon
case defaultAppFallback
}
struct LaunchResult {
let status: PeekabooDaemonStatus
let processID: pid_t
var ownsObservedDaemon: Bool {
self.status.pid == self.processID
}
}
enum SocketAvailability: Equatable {
case available
case reusableDaemon
case timedOut
}
enum LegacyStopRaceResolution: Equatable {
case keepReplacement
case useLegacy(socketPath: String)
}
static func shouldAutoStartDaemon(
options: CommandRuntimeOptions,
environment: [String: String]
@ -16,7 +42,201 @@ enum DaemonLaunchPolicy {
!socket.isEmpty {
return socket
}
return PeekabooBridgeConstants.peekabooSocketPath
return PeekabooBridgeConstants.daemonSocketPath
}
static func runtimeBuildIdentity(
executableURL: URL? = Bundle.main.executableURL,
executableUUIDProvider: (URL) -> [String] = executableUUIDs
) -> String {
let protocolVersion = PeekabooBridgeConstants.protocolVersion
let identityPrefix = "\(protocolVersion.major).\(protocolVersion.minor)|" +
PeekabooBridgeConstants.buildIdentifier
let resolvedURL = executableURL?.resolvingSymlinksInPath()
if let resolvedURL {
let executableUUIDs = executableUUIDProvider(resolvedURL).sorted()
if !executableUUIDs.isEmpty {
return "\(identityPrefix)|\(executableUUIDs.joined(separator: ","))"
}
}
let executablePath = resolvedURL?.path ?? CommandLine.arguments.first ?? "unknown"
let attributes = try? FileManager.default.attributesOfItem(atPath: executablePath)
let fileSize = (attributes?[.size] as? NSNumber)?.uint64Value ?? 0
let modificationBits = (attributes?[.modificationDate] as? Date)?
.timeIntervalSinceReferenceDate.bitPattern ?? 0
return [
identityPrefix,
executablePath,
"\(fileSize)",
"\(modificationBits)",
].joined(separator: "|")
}
private enum ByteOrder {
case little
case big
}
private nonisolated static func executableUUIDs(_ executableURL: URL) -> [String] {
guard let data = try? Data(contentsOf: executableURL, options: .mappedIfSafe) else {
return []
}
return self.machoUUIDs(in: data)
}
nonisolated static func machoUUIDs(in data: Data) -> [String] {
guard let magic = readUInt32(data, at: 0, order: .little) else { return [] }
switch magic {
case UInt32(FAT_CIGAM), UInt32(FAT_CIGAM_64):
return self.fatMachOUUIDs(
in: data,
order: .big,
uses64BitArchitectureRecords: magic == UInt32(FAT_CIGAM_64)
)
case UInt32(FAT_MAGIC), UInt32(FAT_MAGIC_64):
return self.fatMachOUUIDs(
in: data,
order: .little,
uses64BitArchitectureRecords: magic == UInt32(FAT_MAGIC_64)
)
default:
return self.machOUUID(in: data, sliceOffset: 0).map { [$0] } ?? []
}
}
private nonisolated static func fatMachOUUIDs(
in data: Data,
order: ByteOrder,
uses64BitArchitectureRecords: Bool
) -> [String] {
guard let architectureCount = readUInt32(data, at: 4, order: order) else { return [] }
let recordSize = uses64BitArchitectureRecords ? 32 : 20
guard architectureCount <= 64 else { return [] }
var uuids: [String] = []
for index in 0..<Int(architectureCount) {
let recordOffset = 8 + index * recordSize
let rawSliceOffset: UInt64? = if uses64BitArchitectureRecords {
self.readUInt64(data, at: recordOffset + 8, order: order)
} else {
self.readUInt32(data, at: recordOffset + 8, order: order).map(UInt64.init)
}
guard let rawSliceOffset, rawSliceOffset <= UInt64(Int.max) else { return [] }
if let uuid = machOUUID(in: data, sliceOffset: Int(rawSliceOffset)) {
uuids.append(uuid)
}
}
return uuids
}
private nonisolated static func machOUUID(in data: Data, sliceOffset: Int) -> String? {
guard let magic = readUInt32(data, at: sliceOffset, order: .little) else { return nil }
let order: ByteOrder
let headerSize: Int
switch magic {
case UInt32(MH_MAGIC):
order = .little
headerSize = 28
case UInt32(MH_MAGIC_64):
order = .little
headerSize = 32
case UInt32(MH_CIGAM):
order = .big
headerSize = 28
case UInt32(MH_CIGAM_64):
order = .big
headerSize = 32
default:
return nil
}
guard let commandCount = readUInt32(data, at: sliceOffset + 16, order: order),
let commandBytes = readUInt32(data, at: sliceOffset + 20, order: order),
commandCount <= 16384
else { return nil }
var commandOffset = sliceOffset + headerSize
let commandsEnd = commandOffset + Int(commandBytes)
guard commandsEnd >= commandOffset, commandsEnd <= data.count else { return nil }
for _ in 0..<Int(commandCount) {
guard let command = readUInt32(data, at: commandOffset, order: order),
let rawCommandSize = readUInt32(data, at: commandOffset + 4, order: order)
else { return nil }
let commandSize = Int(rawCommandSize)
guard commandSize >= 8, commandOffset + commandSize <= commandsEnd else { return nil }
if command == UInt32(LC_UUID), commandSize >= 24 {
let uuidRange = (commandOffset + 8)..<(commandOffset + 24)
return data[uuidRange].map { String(format: "%02x", $0) }.joined()
}
commandOffset += commandSize
}
return nil
}
private nonisolated static func readUInt32(_ data: Data, at offset: Int, order: ByteOrder) -> UInt32? {
guard offset >= 0, offset + 4 <= data.count else { return nil }
let bytes = data[offset..<(offset + 4)]
return bytes.enumerated().reduce(UInt32(0)) { partial, pair in
let shift = switch order {
case .little: pair.offset * 8
case .big: (3 - pair.offset) * 8
}
return partial | UInt32(pair.element) << UInt32(shift)
}
}
private nonisolated static func readUInt64(_ data: Data, at offset: Int, order: ByteOrder) -> UInt64? {
guard offset >= 0, offset + 8 <= data.count else { return nil }
let bytes = data[offset..<(offset + 8)]
return bytes.enumerated().reduce(UInt64(0)) { partial, pair in
let shift = switch order {
case .little: pair.offset * 8
case .big: (7 - pair.offset) * 8
}
return partial | UInt64(pair.element) << UInt64(shift)
}
}
static func autoStartSocketPath(
daemonSocketPath: String,
defaultSocketWasOccupiedAndRejected: Bool,
runtimeBuildIdentity: String
) -> String {
guard defaultSocketWasOccupiedAndRejected,
let buildScopedSocketPath = buildScopedDaemonSocketPath(
daemonSocketPath: daemonSocketPath,
runtimeBuildIdentity: runtimeBuildIdentity
)
else {
return daemonSocketPath
}
return buildScopedSocketPath
}
static func buildScopedDaemonSocketPath(
daemonSocketPath: String,
runtimeBuildIdentity: String
) -> String? {
guard self.standardizedSocketPath(daemonSocketPath) ==
self.standardizedSocketPath(PeekabooBridgeConstants.daemonSocketPath)
else { return nil }
return URL(fileURLWithPath: daemonSocketPath)
.deletingLastPathComponent()
.appendingPathComponent("daemon-\(self.stableHash(runtimeBuildIdentity)).sock")
.path
}
private static func stableHash(_ value: String) -> String {
var hash: UInt64 = 14_695_981_039_346_656_037
for byte in value.utf8 {
hash ^= UInt64(byte)
hash &*= 1_099_511_628_211
}
return String(format: "%016llx", hash)
}
static func daemonIdleTimeoutSeconds(environment: [String: String]) -> TimeInterval {
@ -29,20 +249,96 @@ enum DaemonLaunchPolicy {
return value
}
static func shouldMigrateLegacyDaemon(targetSocketPath: String) -> Bool {
self.standardizedSocketPath(targetSocketPath) ==
self.standardizedSocketPath(PeekabooBridgeConstants.daemonSocketPath)
}
static func implicitRuntimeCandidateRole(
socketPath: String,
daemonSocketPath: String,
buildScopedDaemonSocketPath: String? = nil
) -> ImplicitRuntimeCandidateRole? {
let candidate = self.standardizedSocketPath(socketPath)
if candidate == self.standardizedSocketPath(daemonSocketPath) ||
buildScopedDaemonSocketPath.map(self.standardizedSocketPath) == candidate {
return .reusableDaemon
}
if self.shouldMigrateLegacyDaemon(targetSocketPath: daemonSocketPath),
candidate == self.standardizedSocketPath(PeekabooBridgeConstants.peekabooSocketPath) {
return .defaultAppFallback
}
return nil
}
static func isSelectableImplicitRuntimeCandidate(
role: ImplicitRuntimeCandidateRole,
handshake: PeekabooBridgeHandshakeResponse,
daemonStatus: PeekabooDaemonStatus?
) -> Bool {
switch role {
case .reusableDaemon:
daemonStatus.map(DaemonControlClient.isReusableDaemonStatus) == true
case .defaultAppFallback:
handshake.hostKind == .gui ||
daemonStatus.map(DaemonControlClient.isReusableDaemonStatus) == true
}
}
static func onDemandDaemonArguments(socketPath: String, idleTimeoutSeconds: TimeInterval) -> [String] {
[
self.daemonArguments(
socketPath: socketPath,
mode: .auto,
idleTimeoutSeconds: idleTimeoutSeconds
)
}
static func daemonArguments(
socketPath: String,
mode: PeekabooDaemonMode,
pollIntervalMs: Int? = nil,
idleTimeoutSeconds: TimeInterval
) -> [String] {
var arguments = [
"daemon",
"run",
"--mode",
"auto",
mode.rawValue,
"--bridge-socket",
socketPath,
"--idle-timeout-seconds",
String(format: "%.3f", idleTimeoutSeconds),
]
if let pollIntervalMs, pollIntervalMs > 0 {
arguments.append(contentsOf: [
"--poll-interval-ms",
"\(pollIntervalMs)",
])
}
if mode == .auto {
arguments.append(contentsOf: [
"--idle-timeout-seconds",
String(format: "%.3f", idleTimeoutSeconds),
])
}
return arguments
}
static func startOnDemandDaemon(socketPath: String, environment: [String: String]) async -> Bool {
static func migratedDaemonArguments(
socketPath: String,
status: PeekabooDaemonStatus,
fallbackIdleTimeoutSeconds: TimeInterval
) -> [String]? {
guard let mode = DaemonControlClient.migrationMode(for: status) else { return nil }
let idleTimeoutSeconds = status.activity?.idleTimeoutSeconds.flatMap { $0 > 0 ? $0 : nil }
?? fallbackIdleTimeoutSeconds
return self.daemonArguments(
socketPath: socketPath,
mode: mode,
pollIntervalMs: status.windowTracker?.cgPollIntervalMs,
idleTimeoutSeconds: idleTimeoutSeconds
)
}
static func startOnDemandDaemon(socketPath: String, environment: [String: String]) async -> String? {
let client = DaemonControlClient(socketPath: socketPath)
let lockHandle = DaemonPaths.openDaemonStartupLock()
if let fileDescriptor = lockHandle?.fileDescriptor {
@ -55,17 +351,201 @@ enum DaemonLaunchPolicy {
try? lockHandle?.close()
}
if await client.fetchStatus() != nil {
return true
if await client.fetchReusableDaemonStatus() != nil {
return socketPath
}
switch await self.waitForDaemonSocketAvailability(
socketPath: socketPath,
client: client,
timeout: TimeInterval(DaemonControlClient.defaultShutdownWaitSeconds)
) {
case .available:
break
case .reusableDaemon:
return socketPath
case .timedOut:
return nil
}
let fallbackIdleTimeoutSeconds = self.daemonIdleTimeoutSeconds(environment: environment)
var launchArguments = self.daemonArguments(
socketPath: socketPath,
mode: .auto,
idleTimeoutSeconds: fallbackIdleTimeoutSeconds
)
let legacyClient = DaemonControlClient(socketPath: PeekabooBridgeConstants.peekabooSocketPath)
if self.shouldMigrateLegacyDaemon(targetSocketPath: socketPath),
let legacyStatus = await legacyClient.fetchReusableDaemonStatus(),
let migrationArguments = migratedDaemonArguments(
socketPath: socketPath,
status: legacyStatus,
fallbackIdleTimeoutSeconds: fallbackIdleTimeoutSeconds
) {
if DaemonControlClient.supportsSafeMigration(legacyStatus),
DaemonControlClient.isIdleForMigration(legacyStatus) {
launchArguments = migrationArguments
guard let replacement = await launchDaemon(
socketPath: socketPath,
arguments: launchArguments
)
else {
return await self.compatibleLegacyFallbackSocketPath {
await legacyClient.fetchReusableDaemonStatus()
}
}
do {
let stopped = try await legacyClient.stopAndWait(
waitSeconds: DaemonControlClient.defaultShutdownWaitSeconds,
expectedPID: legacyStatus.pid,
requireIdentityMatch: true
)
if !stopped {
if let currentLegacyStatus = await legacyClient.fetchReusableDaemonStatus() {
return await self.resolveLegacyStopRace(
legacyStatus: currentLegacyStatus,
client: client,
replacement: replacement,
replacementSocketPath: socketPath
)
}
}
} catch {
if let currentLegacyStatus = await legacyClient.fetchReusableDaemonStatus() {
return await self.resolveLegacyStopRace(
legacyStatus: currentLegacyStatus,
client: client,
replacement: replacement,
replacementSocketPath: socketPath
)
}
}
return await client.fetchReusableDaemonStatus() != nil ? socketPath : nil
}
if let fallback = self.compatibleLegacyFallbackSocketPath(for: legacyStatus) {
return fallback
}
// An incompatible legacy host cannot satisfy this caller. Leave it running and
// start the current daemon on the free canonical socket instead.
launchArguments = self.daemonArguments(
socketPath: socketPath,
mode: .auto,
idleTimeoutSeconds: fallbackIdleTimeoutSeconds
)
}
return await self.launchDaemon(
socketPath: socketPath,
arguments: launchArguments
) != nil ? socketPath : nil
}
static func compatibleLegacyFallbackSocketPath(for status: PeekabooDaemonStatus) -> String? {
guard DaemonControlPlanner.supportsCurrentDaemon(status) else {
return nil
}
return PeekabooBridgeConstants.peekabooSocketPath
}
static func compatibleLegacyFallbackSocketPath(
refreshingWith fetchStatus: () async -> PeekabooDaemonStatus?
) async -> String? {
guard let currentStatus = await fetchStatus() else { return nil }
return self.compatibleLegacyFallbackSocketPath(for: currentStatus)
}
static func legacyStopRaceResolution(for status: PeekabooDaemonStatus) -> LegacyStopRaceResolution {
if let fallback = self.compatibleLegacyFallbackSocketPath(for: status) {
return .useLegacy(socketPath: fallback)
}
return .keepReplacement
}
static func legacyStopRaceSocketPath(
replacementCleanupSucceeded: Bool,
replacementIsReusable: Bool,
legacySocketPath: String,
replacementSocketPath: String
) -> String? {
if replacementCleanupSucceeded {
return legacySocketPath
}
return replacementIsReusable ? replacementSocketPath : nil
}
private static func resolveLegacyStopRace(
legacyStatus: PeekabooDaemonStatus,
client: DaemonControlClient,
replacement: LaunchResult,
replacementSocketPath: String
) async -> String? {
switch self.legacyStopRaceResolution(for: legacyStatus) {
case .keepReplacement:
return await client.fetchReusableDaemonStatus() != nil ? replacementSocketPath : nil
case let .useLegacy(socketPath):
let cleanedUp = await self.stopReplacement(client: client, replacement: replacement)
var replacementIsReusable = false
if !cleanedUp {
replacementIsReusable = await client.fetchReusableDaemonStatus() != nil
}
return self.legacyStopRaceSocketPath(
replacementCleanupSucceeded: cleanedUp,
replacementIsReusable: replacementIsReusable,
legacySocketPath: socketPath,
replacementSocketPath: replacementSocketPath
)
}
}
static func waitForDaemonSocketAvailability(
socketPath: String,
client: DaemonControlClient,
timeout: TimeInterval
) async -> SocketAvailability {
guard self.bridgeLeaseIsHeld(socketPath: socketPath) else {
return .available
}
let deadline = Date().addingTimeInterval(timeout)
while Date() < deadline {
if await client.fetchReusableDaemonStatus() != nil {
return .reusableDaemon
}
if !self.bridgeLeaseIsHeld(socketPath: socketPath) {
return .available
}
try? await Task.sleep(nanoseconds: 100_000_000)
}
return self.bridgeLeaseIsHeld(socketPath: socketPath) ? .timedOut : .available
}
private static func bridgeLeaseIsHeld(socketPath: String) -> Bool {
let fd = open(
"\(socketPath).lock",
O_RDWR | O_CLOEXEC | O_NOFOLLOW
)
guard fd >= 0 else { return false }
defer { close(fd) }
if flock(fd, LOCK_EX | LOCK_NB) == 0 {
flock(fd, LOCK_UN)
return false
}
return errno == EWOULDBLOCK || errno == EAGAIN
}
static func launchDaemon(
socketPath: String,
arguments: [String],
timeout: TimeInterval = 3
) async -> LaunchResult? {
let executable = CommandLine.arguments.first ?? "/usr/local/bin/peekaboo"
let process = Process()
process.executableURL = URL(fileURLWithPath: executable)
process.arguments = self.onDemandDaemonArguments(
socketPath: socketPath,
idleTimeoutSeconds: self.daemonIdleTimeoutSeconds(environment: environment)
)
process.arguments = arguments
let logHandle = DaemonPaths.openDaemonLogForAppend() ?? FileHandle.nullDevice
process.standardOutput = logHandle
process.standardError = logHandle
@ -74,16 +554,52 @@ enum DaemonLaunchPolicy {
do {
try process.run()
} catch {
return false
return nil
}
let deadline = Date().addingTimeInterval(3)
let deadline = Date().addingTimeInterval(timeout)
let client = DaemonControlClient(socketPath: socketPath)
while Date() < deadline {
if await client.fetchStatus() != nil {
return true
if let status = await client.fetchReusableDaemonStatus() {
let processID = process.processIdentifier
if status.pid != processID, process.isRunning {
process.terminate()
}
return LaunchResult(status: status, processID: processID)
}
try? await Task.sleep(nanoseconds: 100_000_000)
}
return false
if process.isRunning {
process.terminate()
}
return nil
}
static func stopReplacement(
client: DaemonControlClient,
replacement: LaunchResult
) async -> Bool {
guard replacement.ownsObservedDaemon else { return true }
let expectedPID = replacement.processID
let deadline = Date().addingTimeInterval(
TimeInterval(DaemonControlClient.defaultShutdownWaitSeconds)
)
while Date() < deadline {
guard let status = await client.fetchControllableDaemonStatus(),
status.pid == expectedPID
else {
return true
}
_ = try? await client.stopDaemon(expectedPID: expectedPID)
try? await Task.sleep(nanoseconds: 200_000_000)
}
return await client.fetchControllableDaemonStatus()?.pid != expectedPID
}
private static func standardizedSocketPath(_ path: String) -> String {
let expanded = (path as NSString).expandingTildeInPath
return (expanded as NSString).standardizingPath
}
}

View File

@ -6,31 +6,85 @@ import PeekabooCore
@MainActor
enum RuntimeHostResolver {
static func resolveServices(options: CommandRuntimeOptions)
async -> (services: any PeekabooServiceProviding, hostDescription: String) {
struct Resolution {
let services: any PeekabooServiceProviding
let hostDescription: String
let selectedRemoteSocketPath: String?
let selectedRemoteHostProcessIdentifier: pid_t?
let snapshotInvalidationRemoteSocketPaths: [String]
let applicationRelaunchAllowed: Bool
}
struct ImplicitRemoteCandidate: Equatable {
let socketPath: String
let requireReusableDaemon: Bool
let requiredHostKind: PeekabooBridgeHostKind?
let requiresValidatedHistoricalDaemon: Bool
}
struct RemoteCandidatePlan {
let explicitSocket: String?
let daemonSocketPath: String
let runtimeBuildIdentity: String
let buildScopedDaemonSocketPath: String?
let historicalBuildScopedDaemonSocketPaths: [String]
let candidates: [ImplicitRemoteCandidate]
}
struct RemoteCandidateValidation {
let reusableDaemonStatus: PeekabooDaemonStatus?
}
enum InitialRoutingDecision: Equatable {
case local(snapshotInvalidationRemoteSocketPaths: [String])
case remote
}
static func resolveServices(options: CommandRuntimeOptions) async -> Resolution {
let environment = ProcessInfo.processInfo.environment
let envNoRemote = environment["PEEKABOO_NO_REMOTE"]
guard options.preferRemote,
envNoRemote == nil,
options.inputStrategy == nil,
!RuntimeInputPolicyResolver.hasEnvironmentOverride(environment: environment),
!RuntimeInputPolicyResolver.hasConfigOverride(
input: PeekabooAutomation.ConfigurationManager.shared.getConfiguration()?.input
)
else {
return (
let configurationInput = PeekabooAutomation.ConfigurationManager.shared.getConfiguration()?.input
guard self.shouldResolveKnownRemoteEndpoints(
options: options,
environment: environment,
configurationInput: configurationInput
) else {
return Resolution(
services: RuntimeServiceFactory.makeLocalServices(options: options),
hostDescription: "local (in-process)"
hostDescription: "local (in-process)",
selectedRemoteSocketPath: nil,
selectedRemoteHostProcessIdentifier: nil,
snapshotInvalidationRemoteSocketPaths: [],
applicationRelaunchAllowed: true
)
}
let explicitSocket = BridgeSocketResolver.explicitBridgeSocket(options: options, environment: environment)
let candidatePlan = await self.remoteCandidatePlan(options: options, environment: environment)
let explicitSocket = candidatePlan.explicitSocket
let daemonSocketPath = candidatePlan.daemonSocketPath
let runtimeBuildIdentity = candidatePlan.runtimeBuildIdentity
let buildScopedDaemonSocketPath = candidatePlan.buildScopedDaemonSocketPath
let historicalBuildScopedDaemonSocketPaths = candidatePlan.historicalBuildScopedDaemonSocketPaths
let snapshotInvalidationRemoteSocketPaths = snapshotInvalidationRemoteSocketPaths(
explicitSocket: explicitSocket,
daemonSocketPath: daemonSocketPath,
buildScopedDaemonSocketPath: buildScopedDaemonSocketPath,
historicalBuildScopedDaemonSocketPaths: historicalBuildScopedDaemonSocketPaths
)
let daemonSocketPath = DaemonLaunchPolicy.daemonSocketPath(environment: environment)
let candidates: [String] = if let explicitSocket, !explicitSocket.isEmpty {
[explicitSocket]
} else {
[daemonSocketPath]
if case let .local(localSnapshotInvalidationPaths) = initialRoutingDecision(
options: options,
environment: environment,
configurationInput: configurationInput,
knownSnapshotInvalidationRemoteSocketPaths: snapshotInvalidationRemoteSocketPaths
) {
return Resolution(
services: RuntimeServiceFactory.makeLocalServices(options: options),
hostDescription: "local (in-process)",
selectedRemoteSocketPath: nil,
selectedRemoteHostProcessIdentifier: nil,
snapshotInvalidationRemoteSocketPaths: localSnapshotInvalidationPaths,
applicationRelaunchAllowed: true
)
}
let identity = PeekabooBridgeClientIdentity(
@ -40,60 +94,305 @@ enum RuntimeHostResolver {
hostname: Host.current().name
)
if let resolved = await self.resolveRemoteServices(
candidates: candidates,
if let resolved = await resolveRemoteServices(
candidates: candidatePlan.candidates,
identity: identity,
options: options
options: options,
snapshotInvalidationRemoteSocketPaths: snapshotInvalidationRemoteSocketPaths
) {
return resolved
}
if options.autoStartDaemon,
DaemonLaunchPolicy.shouldAutoStartDaemon(options: options, environment: environment),
await DaemonLaunchPolicy.startOnDemandDaemon(socketPath: daemonSocketPath, environment: environment),
let resolved = await self.resolveRemoteServices(
candidates: [daemonSocketPath],
identity: identity,
options: options
) {
return resolved
if DaemonLaunchPolicy.shouldAutoStartDaemon(options: options, environment: environment) {
let rejectedDefaultSocketOccupant =
await DaemonControlClient(socketPath: daemonSocketPath).fetchStatus() != nil
let autoStartSocketPath = DaemonLaunchPolicy.autoStartSocketPath(
daemonSocketPath: daemonSocketPath,
defaultSocketWasOccupiedAndRejected: rejectedDefaultSocketOccupant,
runtimeBuildIdentity: runtimeBuildIdentity
)
if let resolvedDaemonSocket = await DaemonLaunchPolicy.startOnDemandDaemon(
socketPath: autoStartSocketPath,
environment: environment
),
let resolved = await resolveRemoteServices(
candidates: [ImplicitRemoteCandidate(
socketPath: resolvedDaemonSocket,
requireReusableDaemon: true,
requiredHostKind: nil,
requiresValidatedHistoricalDaemon: false
)],
identity: identity,
options: options,
snapshotInvalidationRemoteSocketPaths: snapshotInvalidationRemoteSocketPaths
) {
return resolved
}
}
return (
return Resolution(
services: RuntimeServiceFactory.makeLocalServices(options: options),
hostDescription: "local (in-process)"
hostDescription: "local (in-process fallback)",
selectedRemoteSocketPath: nil,
selectedRemoteHostProcessIdentifier: nil,
snapshotInvalidationRemoteSocketPaths: snapshotInvalidationRemoteSocketPaths,
applicationRelaunchAllowed: !options.requiresApplicationRelaunch
)
}
static func remoteRoutingAllowed(
options: CommandRuntimeOptions,
environment: [String: String],
configurationInput: PeekabooAutomation.Configuration.InputConfig?
) -> Bool {
self.initialRoutingDecision(
options: options,
environment: environment,
configurationInput: configurationInput,
knownSnapshotInvalidationRemoteSocketPaths: []
) == .remote
}
static func remoteCandidatePlan(
options: CommandRuntimeOptions,
environment: [String: String]
) async -> RemoteCandidatePlan {
let explicitSocket = BridgeSocketResolver.explicitBridgeSocket(options: options, environment: environment)
let daemonSocketPath = DaemonLaunchPolicy.daemonSocketPath(environment: environment)
let runtimeBuildIdentity = DaemonLaunchPolicy.runtimeBuildIdentity()
let buildScopedDaemonSocketPath = DaemonLaunchPolicy.buildScopedDaemonSocketPath(
daemonSocketPath: daemonSocketPath,
runtimeBuildIdentity: runtimeBuildIdentity
)
let historicalBuildScopedDaemonSocketPaths: [String] = if self.shouldDiscoverHistoricalDaemons(
explicitSocket: explicitSocket,
daemonSocketPath: daemonSocketPath
) {
await DaemonControlResolver.validatedHistoricalTargets(
daemonSocketPath: daemonSocketPath,
currentBuildScopedSocketPath: buildScopedDaemonSocketPath
)
.filter { DaemonControlPlanner.supportsCurrentDaemon($0.status) }
.map(\.client.socketPath)
} else {
[]
}
let candidates: [ImplicitRemoteCandidate] = if let explicitSocket, !explicitSocket.isEmpty {
[ImplicitRemoteCandidate(
socketPath: explicitSocket,
requireReusableDaemon: false,
requiredHostKind: nil,
requiresValidatedHistoricalDaemon: false
)]
} else {
self.implicitRemoteCandidates(
options: options,
daemonSocketPath: daemonSocketPath,
buildScopedDaemonSocketPath: buildScopedDaemonSocketPath,
historicalBuildScopedDaemonSocketPaths: historicalBuildScopedDaemonSocketPaths
)
}
return RemoteCandidatePlan(
explicitSocket: explicitSocket,
daemonSocketPath: daemonSocketPath,
runtimeBuildIdentity: runtimeBuildIdentity,
buildScopedDaemonSocketPath: buildScopedDaemonSocketPath,
historicalBuildScopedDaemonSocketPaths: historicalBuildScopedDaemonSocketPaths,
candidates: candidates
)
}
static func initialRoutingDecision(
options: CommandRuntimeOptions,
environment: [String: String],
configurationInput: PeekabooAutomation.Configuration.InputConfig?,
knownSnapshotInvalidationRemoteSocketPaths: [String]
) -> InitialRoutingDecision {
guard !self.remoteIsolationRequested(options: options, environment: environment) else {
return .local(snapshotInvalidationRemoteSocketPaths: [])
}
if self.inputPolicyRequiresLocal(
options: options,
environment: environment,
configurationInput: configurationInput
) {
return .local(
snapshotInvalidationRemoteSocketPaths: knownSnapshotInvalidationRemoteSocketPaths
)
}
if !options.preferRemote,
options.requiresImplicitSnapshotInvalidation || options.usesPerToolSnapshotInvalidation {
return .local(
snapshotInvalidationRemoteSocketPaths: knownSnapshotInvalidationRemoteSocketPaths
)
}
guard options.preferRemote else {
return .local(snapshotInvalidationRemoteSocketPaths: [])
}
return .remote
}
static func shouldResolveKnownRemoteEndpoints(
options: CommandRuntimeOptions,
environment: [String: String],
configurationInput: PeekabooAutomation.Configuration.InputConfig?
) -> Bool {
guard !self.remoteIsolationRequested(options: options, environment: environment) else {
return false
}
return options.preferRemote ||
options.requiresImplicitSnapshotInvalidation ||
options.usesPerToolSnapshotInvalidation ||
self.inputPolicyRequiresLocal(
options: options,
environment: environment,
configurationInput: configurationInput
)
}
static func remoteIsolationRequested(
options: CommandRuntimeOptions,
environment: [String: String]
) -> Bool {
options.remoteIsolationRequested || environment["PEEKABOO_NO_REMOTE"] != nil
}
static func snapshotInvalidationRemoteSocketPaths(
explicitSocket: String?,
daemonSocketPath: String,
buildScopedDaemonSocketPath: String? = nil,
historicalBuildScopedDaemonSocketPaths: [String] = []
) -> [String] {
var seen = Set<String>()
var candidatePaths = [
explicitSocket,
PeekabooBridgeConstants.peekabooSocketPath,
daemonSocketPath,
buildScopedDaemonSocketPath,
]
.compactMap(\.self)
candidatePaths.append(contentsOf: historicalBuildScopedDaemonSocketPaths)
return candidatePaths
.map { NSString(string: $0).standardizingPath }
.filter { !$0.isEmpty && seen.insert($0).inserted }
}
static func shouldDiscoverHistoricalDaemons(
explicitSocket: String?,
daemonSocketPath: String
) -> Bool {
explicitSocket == nil && DaemonLaunchPolicy.shouldMigrateLegacyDaemon(targetSocketPath: daemonSocketPath)
}
static func inputPolicyRequiresLocal(
options: CommandRuntimeOptions,
environment: [String: String],
configurationInput: PeekabooAutomation.Configuration.InputConfig?
) -> Bool {
guard !options.requiresApplicationLaunchOptions,
!options.requiresHostApplicationInventory
else {
return false
}
return options.inputStrategy != nil ||
RuntimeInputPolicyResolver.hasEnvironmentOverride(environment: environment) ||
RuntimeInputPolicyResolver.hasConfigOverride(input: configurationInput)
}
static func implicitRemoteCandidates(
options: CommandRuntimeOptions,
daemonSocketPath: String,
buildScopedDaemonSocketPath: String? = nil,
historicalBuildScopedDaemonSocketPaths: [String] = []
) -> [ImplicitRemoteCandidate] {
var seenDaemonPaths = Set<String>()
var daemons: [ImplicitRemoteCandidate] = []
for socketPath in [daemonSocketPath, buildScopedDaemonSocketPath].compactMap(\.self) {
guard seenDaemonPaths.insert(NSString(string: socketPath).standardizingPath).inserted else { continue }
daemons.append(ImplicitRemoteCandidate(
socketPath: socketPath,
requireReusableDaemon: true,
requiredHostKind: nil,
requiresValidatedHistoricalDaemon: false
))
}
for socketPath in historicalBuildScopedDaemonSocketPaths {
guard seenDaemonPaths.insert(NSString(string: socketPath).standardizingPath).inserted else { continue }
daemons.append(ImplicitRemoteCandidate(
socketPath: socketPath,
requireReusableDaemon: true,
requiredHostKind: .onDemand,
requiresValidatedHistoricalDaemon: true
))
}
let gui = ImplicitRemoteCandidate(
socketPath: PeekabooBridgeConstants.peekabooSocketPath,
requireReusableDaemon: false,
requiredHostKind: .gui,
requiresValidatedHistoricalDaemon: false
)
if options.requiresApplicationRelaunch || options.requiresSurvivingApplicationHost {
return daemons
}
if options.requiresApplicationLaunchOptions || options.requiresHostApplicationInventory {
return [gui] + daemons
}
if DaemonLaunchPolicy.shouldMigrateLegacyDaemon(targetSocketPath: daemonSocketPath) {
return daemons + [gui]
}
return daemons
}
private static func resolveRemoteServices(
candidates: [String],
candidates: [ImplicitRemoteCandidate],
identity: PeekabooBridgeClientIdentity,
options: CommandRuntimeOptions
options: CommandRuntimeOptions,
snapshotInvalidationRemoteSocketPaths: [String]
)
async -> (services: any PeekabooServiceProviding, hostDescription: String)? {
for socketPath in candidates {
async -> Resolution? {
for candidate in candidates {
let socketPath = candidate.socketPath
let client = PeekabooBridgeClient(socketPath: socketPath)
do {
let handshake = try await client.handshake(client: identity, requestedHost: nil)
guard BridgeCapabilityPolicy.supportsRemoteRequirements(for: handshake, options: options) else {
continue
}
guard let validation = await self.validateRemoteCandidate(
candidate,
handshake: handshake,
options: options
) else { continue }
let reusableDaemonStatus = validation.reusableDaemonStatus
let targetedHotkeyAvailability = BridgeCapabilityPolicy.targetedHotkeyAvailability(for: handshake)
let targetedTypeAvailability = BridgeCapabilityPolicy.targetedTypeAvailability(for: handshake)
let targetedClickAvailability = BridgeCapabilityPolicy.targetedClickAvailability(for: handshake)
let hostDescription = "remote \(handshake.hostKind.rawValue) via \(socketPath)" +
(handshake.build.map { " (build \($0))" } ?? "")
return (
return Resolution(
services: RemotePeekabooServices(
client: client,
supportsTargetedHotkeys: targetedHotkeyAvailability.isEnabled,
targetedHotkeyUnavailableReason: targetedHotkeyAvailability.unavailableReason,
targetedHotkeyRequiresEventSynthesizingPermission:
targetedHotkeyAvailability.missingPermissions.contains(.postEvent),
supportsTargetedTypeActions: targetedTypeAvailability.isEnabled,
targetedTypeUnavailableReason: targetedTypeAvailability.unavailableReason,
targetedTypeRequiresEventSynthesizingPermission:
targetedTypeAvailability.missingPermissions.contains(.postEvent),
supportsTargetedClicks: targetedClickAvailability.isEnabled,
targetedClickUnavailableReason: targetedClickAvailability.unavailableReason,
targetedClickRequiresEventSynthesizingPermission:
targetedClickAvailability.missingPermissions.contains(.postEvent),
supportsExactWindowTargetedClicks:
BridgeCapabilityPolicy.supportsExactWindowTargetedClicks(for: handshake),
supportsInspectAccessibilityTree: BridgeCapabilityPolicy.supportsInspectAccessibilityTree(
for: handshake
),
@ -102,9 +401,20 @@ enum RuntimeHostResolver {
),
supportsElementActions: BridgeCapabilityPolicy.supportsElementActions(for: handshake),
supportsDesktopObservation: BridgeCapabilityPolicy.supportsDesktopObservation(for: handshake),
allowLocalApplicationFallback: handshake.hostKind == .onDemand
supportsImplicitLatestSnapshotInvalidation:
BridgeCapabilityPolicy.supportsImplicitSnapshotInvalidation(for: handshake),
supportsApplicationLaunchOptions:
BridgeCapabilityPolicy.supportsApplicationLaunchOptions(for: handshake),
supportsApplicationRelaunch:
BridgeCapabilityPolicy.supportsApplicationRelaunch(for: handshake),
allowLocalApplicationFallback: handshake.hostKind == .onDemand,
desktopMutationWatermarkStore: DesktopMutationWatermarkStore()
),
hostDescription: hostDescription
hostDescription: hostDescription,
selectedRemoteSocketPath: NSString(string: socketPath).standardizingPath,
selectedRemoteHostProcessIdentifier: reusableDaemonStatus?.pid,
snapshotInvalidationRemoteSocketPaths: snapshotInvalidationRemoteSocketPaths,
applicationRelaunchAllowed: BridgeCapabilityPolicy.supportsApplicationRelaunch(for: handshake)
)
} catch {
continue
@ -112,4 +422,47 @@ enum RuntimeHostResolver {
}
return nil
}
static func validateRemoteCandidate(
_ candidate: ImplicitRemoteCandidate,
handshake: PeekabooBridgeHandshakeResponse,
options: CommandRuntimeOptions,
fetchReusableDaemonStatus: (String) async -> PeekabooDaemonStatus? = { socketPath in
await DaemonControlClient(socketPath: socketPath).fetchReusableDaemonStatus()
}
) async -> RemoteCandidateValidation? {
guard candidate.requiredHostKind == nil || handshake.hostKind == candidate.requiredHostKind else {
return nil
}
guard BridgeCapabilityPolicy.supportsRemoteRequirements(for: handshake, options: options) else {
return nil
}
let requiresReusableHost = candidate.requireReusableDaemon ||
options.requiresApplicationRelaunch ||
options.requiresSurvivingApplicationHost
let reusableDaemonStatus: PeekabooDaemonStatus? = if requiresReusableHost {
await fetchReusableDaemonStatus(candidate.socketPath)
} else {
nil
}
guard !requiresReusableHost || reusableDaemonStatus != nil else { return nil }
if candidate.requiresValidatedHistoricalDaemon {
guard let reusableDaemonStatus,
DaemonControlResolver.isValidatedHistoricalTarget(
status: reusableDaemonStatus,
socketPath: candidate.socketPath
),
DaemonControlPlanner.supportsCurrentDaemon(reusableDaemonStatus)
else {
return nil
}
}
if options.requiresApplicationRelaunch || options.requiresSurvivingApplicationHost,
reusableDaemonStatus?.pid == nil {
return nil
}
return RemoteCandidateValidation(reusableDaemonStatus: reusableDaemonStatus)
}
}

View File

@ -5,6 +5,9 @@ import PeekabooCore
enum RuntimeServiceFactory {
static func makeLocalServices(options: CommandRuntimeOptions) -> PeekabooServices {
PeekabooServices(
snapshotManager: SnapshotManager(
desktopMutationWatermarkStore: DesktopMutationWatermarkStore()
),
inputPolicy: PeekabooAutomation.ConfigurationManager.shared.getUIInputPolicy(
cliStrategy: options.inputStrategy
)

View File

@ -1,4 +1,5 @@
import Foundation
import PeekabooAutomation
import PeekabooBridge
import Security
@ -11,11 +12,14 @@ struct BridgeDiagnostics {
@MainActor
func run(runtimeOptions: CommandRuntimeOptions) async -> BridgeStatusReport {
let envNoRemote = ProcessInfo.processInfo.environment["PEEKABOO_NO_REMOTE"]
let shouldSkipRemote = !runtimeOptions.preferRemote || envNoRemote != nil
let remoteSkipReason = shouldSkipRemote
? (!runtimeOptions.preferRemote ? "--no-remote" : "PEEKABOO_NO_REMOTE")
: nil
let environment = ProcessInfo.processInfo.environment
let effectiveOptions = runtimeOptions.applyingEnvironmentOverrides(environment: environment)
let configurationInput = PeekabooAutomation.ConfigurationManager.shared.getConfiguration()?.input
let remoteSkipReason = Self.remoteSkipReason(
runtimeOptions: effectiveOptions,
environment: environment,
configurationInput: configurationInput
)
let identity = PeekabooBridgeClientIdentity(
bundleIdentifier: Bundle.main.bundleIdentifier,
@ -24,9 +28,12 @@ struct BridgeDiagnostics {
hostname: Host.current().name
)
let candidates = self.candidateSocketPaths(runtimeOptions: runtimeOptions)
if shouldSkipRemote {
self.logger.debug("Bridge status: remote skipped (\(remoteSkipReason ?? "unknown reason"))")
if let remoteSkipReason {
let candidates = Self.diagnosticSocketPaths(
runtimeOptions: effectiveOptions,
environment: environment
)
self.logger.debug("Bridge status: remote skipped (\(remoteSkipReason))")
return BridgeStatusReport(
remoteSkipped: true,
remoteSkipReason: remoteSkipReason,
@ -36,6 +43,23 @@ struct BridgeDiagnostics {
)
}
let candidatePlan = await RuntimeHostResolver.remoteCandidatePlan(
options: effectiveOptions,
environment: environment
)
let runtimeCandidates = candidatePlan.candidates
let candidates = Self.diagnosticSocketPaths(
runtimeCandidateSocketPaths: runtimeCandidates.map(\.socketPath),
hasExplicitSocket: candidatePlan.explicitSocket != nil
)
var runtimeCandidateByPath: [String: RuntimeHostResolver.ImplicitRemoteCandidate] = [:]
for candidate in runtimeCandidates {
let path = NSString(string: candidate.socketPath).standardizingPath
if runtimeCandidateByPath[path] == nil {
runtimeCandidateByPath[path] = candidate
}
}
var results: [BridgeCandidateReport] = []
var selected: BridgeSelectionReport?
@ -50,9 +74,17 @@ struct BridgeDiagnostics {
)
results.append(.init(socketPath: socketPath, result: .success(report)))
let enabledOps = handshake.enabledOperations ?? handshake.supportedOperations
if selected == nil, enabledOps.contains(.captureScreen) {
selected = .remote(socketPath: socketPath, handshake: report)
let candidatePath = NSString(string: socketPath).standardizingPath
if selected == nil,
let runtimeCandidate = runtimeCandidateByPath[candidatePath] {
let validation = await RuntimeHostResolver.validateRemoteCandidate(
runtimeCandidate,
handshake: handshake,
options: effectiveOptions
)
if validation != nil {
selected = .remote(socketPath: socketPath, handshake: report)
}
}
} catch let envelope as PeekabooBridgeErrorEnvelope {
self.logger.debug(
@ -78,21 +110,90 @@ struct BridgeDiagnostics {
)
}
private func candidateSocketPaths(runtimeOptions: CommandRuntimeOptions) -> [String] {
let envSocket = ProcessInfo.processInfo.environment["PEEKABOO_BRIDGE_SOCKET"]
let explicitSocket = runtimeOptions.bridgeSocketPath ?? envSocket
static func remoteSkipReason(
runtimeOptions: CommandRuntimeOptions,
environment: [String: String],
configurationInput: PeekabooAutomation.Configuration.InputConfig?
) -> String? {
let decision = RuntimeHostResolver.initialRoutingDecision(
options: runtimeOptions,
environment: environment,
configurationInput: configurationInput,
knownSnapshotInvalidationRemoteSocketPaths: []
)
guard case .local = decision else { return nil }
let rawCandidates: [String] = if let explicitSocket, !explicitSocket.isEmpty {
[explicitSocket]
} else {
[
PeekabooBridgeConstants.peekabooSocketPath,
PeekabooBridgeConstants.claudeSocketPath,
PeekabooBridgeConstants.clawdbotSocketPath,
]
if environment["PEEKABOO_NO_REMOTE"] != nil {
return "PEEKABOO_NO_REMOTE"
}
if runtimeOptions.remoteIsolationRequested {
return "--no-remote"
}
if RuntimeHostResolver.inputPolicyRequiresLocal(
options: runtimeOptions,
environment: environment,
configurationInput: configurationInput
) {
return "input strategy policy"
}
return "local runtime policy"
}
static func runtimeCandidateSocketPaths(
runtimeOptions: CommandRuntimeOptions,
environment: [String: String],
historicalBuildScopedDaemonSocketPaths: [String] = []
) -> [String] {
if let explicitPath = BridgeSocketResolver.explicitBridgeSocket(
options: runtimeOptions,
environment: environment
) {
return [explicitPath]
}
return rawCandidates.map { NSString(string: $0).expandingTildeInPath }
let daemonPath = DaemonLaunchPolicy.daemonSocketPath(environment: environment)
let buildScopedPath = DaemonLaunchPolicy.buildScopedDaemonSocketPath(
daemonSocketPath: daemonPath,
runtimeBuildIdentity: DaemonLaunchPolicy.runtimeBuildIdentity()
)
return RuntimeHostResolver.implicitRemoteCandidates(
options: runtimeOptions,
daemonSocketPath: daemonPath,
buildScopedDaemonSocketPath: buildScopedPath,
historicalBuildScopedDaemonSocketPaths: historicalBuildScopedDaemonSocketPaths
).map(\.socketPath)
}
static func diagnosticSocketPaths(
runtimeOptions: CommandRuntimeOptions,
environment: [String: String],
historicalBuildScopedDaemonSocketPaths: [String] = []
) -> [String] {
let runtimePaths = self.runtimeCandidateSocketPaths(
runtimeOptions: runtimeOptions,
environment: environment,
historicalBuildScopedDaemonSocketPaths: historicalBuildScopedDaemonSocketPaths
)
return self.diagnosticSocketPaths(
runtimeCandidateSocketPaths: runtimePaths,
hasExplicitSocket: BridgeSocketResolver.explicitBridgeSocket(
options: runtimeOptions,
environment: environment
) != nil
)
}
private static func diagnosticSocketPaths(
runtimeCandidateSocketPaths runtimePaths: [String],
hasExplicitSocket: Bool
) -> [String] {
if hasExplicitSocket { return runtimePaths }
let additionalPaths = [
PeekabooBridgeConstants.peekabooSocketPath,
PeekabooBridgeConstants.claudeSocketPath,
PeekabooBridgeConstants.clawdbotSocketPath,
]
return runtimePaths + additionalPaths.filter { !runtimePaths.contains($0) }
}
private static func currentTeamIdentifier() -> String? {

View File

@ -9,11 +9,8 @@ struct BridgeCommand: ParsableCommand {
Peekaboo Bridge lets the CLI run permission-bound operations (Screen Recording, Accessibility,
AppleScript) via a host app that already has the needed TCC grants.
By default, Peekaboo prefers a remote host when available:
1) Peekaboo.app
2) Claude.app
3) ClawdBot.app
4) Local in-process fallback (caller needs permissions)
By default, automation commands use the dedicated Peekaboo daemon and fall back to local execution.
Peekaboo.app, Claude.app, and ClawdBot.app sockets are shown for diagnostics and can be selected explicitly.
Examples:
peekaboo bridge status

View File

@ -0,0 +1,365 @@
import Darwin
import Dispatch
import Foundation
private struct CaptureActionProcessLaunchError: LocalizedError {
let message: String
var errorDescription: String? {
self.message
}
}
private final class BoundedPipeOutput: @unchecked Sendable {
private let lock = NSLock()
private nonisolated(unsafe) var data = Data()
private nonisolated(unsafe) var truncated = false
nonisolated func append(_ chunk: Data) {
let maxOutputBytes = 64 * 1024
self.lock.lock()
defer { self.lock.unlock() }
guard self.data.count < maxOutputBytes else {
self.truncated = true
return
}
let remaining = maxOutputBytes - self.data.count
if chunk.count <= remaining {
self.data.append(chunk)
} else {
self.data.append(contentsOf: chunk.prefix(remaining))
self.truncated = true
}
}
nonisolated func finish() -> (String, Bool) {
self.lock.lock()
defer { self.lock.unlock() }
return (String(bytes: self.data, encoding: .utf8) ?? "", self.truncated)
}
}
private final class CaptureActionSignalForwarder: @unchecked Sendable {
private let lock = NSLock()
private let queue = DispatchQueue(label: "boo.peekaboo.capture-action.signals")
private nonisolated(unsafe) var sources: [any DispatchSourceSignal] = []
private nonisolated(unsafe) var previousHandlers: [(Int32, sig_t?)] = []
private nonisolated(unsafe) var cancelled = false
nonisolated init(onSignal: @escaping @Sendable (Int32) -> Void) {
for signalNumber in [SIGINT, SIGTERM] {
self.previousHandlers.append((signalNumber, signal(signalNumber, SIG_IGN)))
let source = DispatchSource.makeSignalSource(signal: signalNumber, queue: self.queue)
source.setEventHandler {
onSignal(signalNumber)
}
source.resume()
self.sources.append(source)
}
}
nonisolated func cancel() {
self.lock.lock()
guard !self.cancelled else {
self.lock.unlock()
return
}
self.cancelled = true
let sources = self.sources
let previousHandlers = self.previousHandlers
self.sources.removeAll()
self.previousHandlers.removeAll()
self.lock.unlock()
for source in sources {
source.cancel()
}
for (signalNumber, previousHandler) in previousHandlers {
signal(signalNumber, previousHandler)
}
}
deinit {
self.cancel()
}
}
private final class CaptureActionProcessBox: @unchecked Sendable {
private let stdoutPipe = Pipe()
private let stderrPipe = Pipe()
private let stdoutOutput = BoundedPipeOutput()
private let stderrOutput = BoundedPipeOutput()
private let lock = NSLock()
private nonisolated(unsafe) var processIdentifier: pid_t?
private nonisolated(unsafe) var timedOut = false
private nonisolated(unsafe) var didExit = false
nonisolated func start(command: [String]) throws {
guard let executable = command.first else {
throw CaptureActionProcessLaunchError(message: "Action command cannot be empty")
}
self.installOutputHandlers()
try self.spawn(executable: executable, arguments: command)
}
nonisolated func waitUntilExit() -> Int32 {
guard let pid = self.currentProcessIdentifier() else { return -1 }
var status: Int32 = 0
while true {
let result = Darwin.waitpid(pid, &status, 0)
if result == pid {
self.markExited()
return Self.exitCode(fromWaitStatus: status)
}
if result == -1, errno == EINTR {
continue
}
self.markExited()
return -1
}
}
nonisolated func terminateAfterTimeout(seconds: TimeInterval) async {
do {
try await Task.sleep(nanoseconds: UInt64(seconds * 1_000_000_000))
} catch {
return
}
guard self.requestTimeoutTermination() else { return }
do {
try await Task.sleep(nanoseconds: 500_000_000)
} catch {
return
}
self.killTimedOutProcessGroup()
}
nonisolated func wasTimedOut() -> Bool {
self.lock.lock()
defer { self.lock.unlock() }
return self.timedOut
}
nonisolated func finishOutput() -> (stdout: (String, Bool), stderr: (String, Bool)) {
let stdoutHandle = self.stdoutPipe.fileHandleForReading
let stderrHandle = self.stderrPipe.fileHandleForReading
stdoutHandle.readabilityHandler = nil
stderrHandle.readabilityHandler = nil
self.drainAvailableNonBlocking(from: stdoutHandle, into: self.stdoutOutput)
self.drainAvailableNonBlocking(from: stderrHandle, into: self.stderrOutput)
stdoutHandle.closeFile()
stderrHandle.closeFile()
return (self.stdoutOutput.finish(), self.stderrOutput.finish())
}
nonisolated func killTimedOutProcessGroup() {
guard self.wasTimedOut(), let pid = self.currentProcessIdentifier() else { return }
self.killProcessGroup(pid: pid, signal: SIGKILL)
}
nonisolated func terminateProcessGroupForCancellation() {
guard let pid = self.currentProcessIdentifier() else { return }
self.killProcessGroup(pid: pid, signal: SIGTERM)
Task.detached {
do {
try await Task.sleep(nanoseconds: 500_000_000)
} catch {
return
}
self.killProcessGroup(pid: pid, signal: SIGKILL)
}
}
nonisolated func forwardSignalToProcessGroup(_ signalNumber: Int32) {
guard let pid = self.currentProcessIdentifier() else { return }
self.killProcessGroup(pid: pid, signal: signalNumber)
}
private nonisolated func spawn(executable: String, arguments: [String]) throws {
let stdoutRead = self.stdoutPipe.fileHandleForReading.fileDescriptor
let stdoutWrite = self.stdoutPipe.fileHandleForWriting.fileDescriptor
let stderrRead = self.stderrPipe.fileHandleForReading.fileDescriptor
let stderrWrite = self.stderrPipe.fileHandleForWriting.fileDescriptor
var fileActions: posix_spawn_file_actions_t?
try Self.check(posix_spawn_file_actions_init(&fileActions), "posix_spawn_file_actions_init")
defer { posix_spawn_file_actions_destroy(&fileActions) }
try Self.check(posix_spawn_file_actions_adddup2(&fileActions, stdoutWrite, STDOUT_FILENO), "dup stdout")
try Self.check(posix_spawn_file_actions_adddup2(&fileActions, stderrWrite, STDERR_FILENO), "dup stderr")
try Self.check(posix_spawn_file_actions_addclose(&fileActions, stdoutRead), "close child stdout read")
try Self.check(posix_spawn_file_actions_addclose(&fileActions, stderrRead), "close child stderr read")
if stdoutWrite != STDOUT_FILENO {
try Self.check(posix_spawn_file_actions_addclose(&fileActions, stdoutWrite), "close child stdout write")
}
if stderrWrite != STDERR_FILENO {
try Self.check(posix_spawn_file_actions_addclose(&fileActions, stderrWrite), "close child stderr write")
}
var attributes: posix_spawnattr_t?
try Self.check(posix_spawnattr_init(&attributes), "posix_spawnattr_init")
defer { posix_spawnattr_destroy(&attributes) }
let flags = Int16(POSIX_SPAWN_SETPGROUP)
try Self.check(posix_spawnattr_setflags(&attributes, flags), "set spawn flags")
try Self.check(posix_spawnattr_setpgroup(&attributes, 0), "set process group")
var argv = Self.makeCStringArray(arguments)
defer { Self.freeCStringArray(argv) }
let environment = ProcessInfo.processInfo.environment.map { key, value in "\(key)=\(value)" }
var envp = Self.makeCStringArray(environment)
defer { Self.freeCStringArray(envp) }
var pid: pid_t = 0
let spawnResult = executable.withCString { executablePath in
posix_spawnp(&pid, executablePath, &fileActions, &attributes, &argv, &envp)
}
self.stdoutPipe.fileHandleForWriting.closeFile()
self.stderrPipe.fileHandleForWriting.closeFile()
try Self.check(spawnResult, "posix_spawnp")
self.lock.lock()
self.processIdentifier = pid
self.lock.unlock()
}
private nonisolated func installOutputHandlers() {
self.stdoutPipe.fileHandleForReading.readabilityHandler = { [stdoutOutput] handle in
let chunk = handle.availableData
if chunk.isEmpty {
handle.readabilityHandler = nil
} else {
stdoutOutput.append(chunk)
}
}
self.stderrPipe.fileHandleForReading.readabilityHandler = { [stderrOutput] handle in
let chunk = handle.availableData
if chunk.isEmpty {
handle.readabilityHandler = nil
} else {
stderrOutput.append(chunk)
}
}
}
private nonisolated func requestTimeoutTermination() -> Bool {
self.lock.lock()
defer { self.lock.unlock() }
guard let pid = self.processIdentifier, !self.didExit else { return false }
self.timedOut = true
self.killProcessGroup(pid: pid, signal: SIGTERM)
return true
}
private nonisolated func currentProcessIdentifier() -> pid_t? {
self.lock.lock()
defer { self.lock.unlock() }
return self.processIdentifier
}
private nonisolated func markExited() {
self.lock.lock()
self.didExit = true
self.lock.unlock()
}
private nonisolated func killProcessGroup(pid: pid_t, signal: Int32) {
_ = Darwin.kill(-pid, signal)
}
private nonisolated func drainAvailableNonBlocking(from handle: FileHandle, into output: BoundedPipeOutput) {
let outputReadChunkBytes = 4096
let fileDescriptor = handle.fileDescriptor
let flags = fcntl(fileDescriptor, F_GETFL)
if flags >= 0 {
_ = fcntl(fileDescriptor, F_SETFL, flags | O_NONBLOCK)
}
var buffer = [UInt8](repeating: 0, count: outputReadChunkBytes)
while true {
let count = Darwin.read(fileDescriptor, &buffer, outputReadChunkBytes)
if count > 0 {
output.append(Data(buffer.prefix(count)))
} else if count == 0 || errno == EAGAIN || errno == EWOULDBLOCK {
break
} else {
break
}
}
}
private nonisolated static func makeCStringArray(_ strings: [String]) -> [UnsafeMutablePointer<CChar>?] {
var pointers = strings.map { strdup($0) }
pointers.append(nil)
return pointers
}
private nonisolated static func freeCStringArray(_ pointers: [UnsafeMutablePointer<CChar>?]) {
for pointer in pointers {
free(pointer)
}
}
private nonisolated static func check(_ code: Int32, _ operation: String) throws {
guard code != 0 else { return }
throw CaptureActionProcessLaunchError(
message: "\(operation) failed: \(String(cString: strerror(code)))"
)
}
private nonisolated static func exitCode(fromWaitStatus status: Int32) -> Int32 {
let signal = status & 0x7F
if signal == 0 {
return (status >> 8) & 0xFF
}
if signal != 0x7F {
return 128 + signal
}
return status
}
}
enum CaptureActionProcessRunner {
nonisolated static func run(
command: [String],
timeoutSeconds: TimeInterval
) async throws -> CaptureActionProcessResult {
let box = CaptureActionProcessBox()
let started = Date()
try box.start(command: command)
let signalForwarder = CaptureActionSignalForwarder { signalNumber in
box.forwardSignalToProcessGroup(signalNumber)
}
defer { signalForwarder.cancel() }
return await withTaskCancellationHandler {
let waitTask = Task.detached { box.waitUntilExit() }
let timeoutTask = Task.detached { await box.terminateAfterTimeout(seconds: timeoutSeconds) }
let exitCode = await waitTask.value
box.killTimedOutProcessGroup()
timeoutTask.cancel()
try? await Task.sleep(nanoseconds: 50_000_000)
let output = box.finishOutput()
let durationMs = Int(Date().timeIntervalSince(started) * 1000)
return CaptureActionProcessResult(
command: command,
exitCode: exitCode,
timedOut: box.wasTimedOut(),
timeoutSeconds: timeoutSeconds,
durationMs: durationMs,
stdout: output.stdout.0,
stderr: output.stderr.0,
stdoutTruncated: output.stdout.1,
stderrTruncated: output.stderr.1
)
} onCancel: {
box.terminateProcessGroupForCancellation()
}
}
}

View File

@ -0,0 +1,649 @@
import Commander
import CoreGraphics
import Foundation
import PeekabooCore
import PeekabooFoundation
@MainActor
struct CaptureActionCommand: ApplicationResolvable, ErrorHandlingCommand, OutputFormattable,
RuntimeOptionsConfigurable {
var app: String?
var pid: Int32?
var mode: String?
var windowTitle: String?
var windowIndex: Int?
var screenIndex: Int?
var region: String?
var captureFocus: LiveCaptureFocus = .auto
var captureEngine: String?
var durationLimit: Double?
var preRollMs: Int?
var postRollMs: Int?
var actionTimeout: Double?
var idleFps: Double?
var activeFps: Double?
var threshold: Double?
var heartbeatSec: Double?
var quietMs: Int?
var highlightChanges = false
var maxFrames: Int?
var maxMb: Int?
var resolutionCap: Double?
var diffStrategy: String?
var diffBudgetMs: Int?
var path: String?
var autocleanMinutes: Int?
var videoOut: String?
var command: [String] = []
@RuntimeStorage private var runtime: CommandRuntime?
var runtimeOptions = CommandRuntimeOptions()
nonisolated(unsafe) static var commandDescription: CommandDescription {
MainActorCommandDescription.describe {
CommandDescription(
commandName: "action",
abstract: "Capture around a child command with pre/post-roll",
discussion: """
Starts adaptive live capture, runs a child command, keeps post-roll, then
stops capture and verifies the resulting artifacts.
Examples:
peekaboo capture action --duration-limit 10 -- echo smoke
peekaboo capture action --mode area --region 0,0,640,360 -- ./test-flow.sh
""",
version: "1.0.0"
)
}
}
private var resolvedRuntime: CommandRuntime {
guard let runtime else {
preconditionFailure("CommandRuntime must be configured before accessing runtime resources")
}
return runtime
}
private var logger: Logger {
self.resolvedRuntime.logger
}
var services: any PeekabooServiceProviding {
self.resolvedRuntime.services
}
func withCaptureFocusMutation(_ operation: () async throws -> Void) async rethrows {
try await self.resolvedRuntime.withCaptureFocusMutation(operation)
}
var jsonOutput: Bool {
self.resolvedRuntime.configuration.jsonOutput
}
var outputLogger: Logger {
self.logger
}
mutating func run(using runtime: CommandRuntime) async throws {
self.runtime = runtime
self.logger.setJsonOutputMode(self.jsonOutput)
self.logger.operationStart("capture_action", metadata: ["mode": self.mode ?? "auto"])
do {
guard !self.command.isEmpty else {
throw ValidationError("Pass the action command after --")
}
let scope = try await resolveScope()
let options = try buildOptions()
let timing = try resolveActionTiming(durationLimit: options.duration)
if scope.kind == .window, let identifier = scope.applicationIdentifier {
try await focusIfNeeded(appIdentifier: identifier)
}
let outputDir = try resolveOutputDirectory()
let deps = WatchCaptureDependencies(
screenCapture: services.screenCapture,
screenService: self.services.screens,
frameSource: nil
)
let config = WatchCaptureConfiguration(
scope: scope,
options: options,
outputRoot: outputDir,
autoclean: WatchAutocleanConfig(minutes: autocleanMinutes ?? 120, managed: path == nil),
sourceKind: .live,
videoIn: nil,
videoOut: CaptureCommandPathResolver.filePath(from: self.videoOut),
keepAllFrames: false
)
let session = WatchCaptureSession(dependencies: deps, configuration: config)
let captureTask = self.startCaptureTask(session: session, scope: scope)
do {
if try await Self.waitForPreRollOrCaptureEnd(
milliseconds: timing.startupGateMs,
captureTask: captureTask
) != nil {
throw ValidationError("Capture ended before action started")
}
self.resolvedRuntime.beginInteractionMutation()
let action = try await CaptureActionProcessRunner.run(
command: self.command,
timeoutSeconds: timing.actionTimeout
)
try await Self.sleep(milliseconds: timing.postRollMs)
session.requestStop()
let capture = try await captureTask.value
let validation = validateArtifacts(capture)
let result = CaptureActionCommandResult(
success: action.succeeded && validation.ok,
action: action,
capture: capture,
validation: validation
)
self.output(result)
self.logger.operationComplete(
"capture_action",
success: result.success,
metadata: ["frames_kept": capture.stats.framesKept]
)
if !result.success {
throw ExitCode(1)
}
} catch {
session.requestStop()
captureTask.cancel()
_ = try? await captureTask.value
throw error
}
} catch let exit as ExitCode {
throw exit
} catch {
handleError(error)
self.logger.operationComplete(
"capture_action",
success: false,
metadata: ["error": error.localizedDescription]
)
throw ExitCode(1)
}
}
private func startCaptureTask(
session: WatchCaptureSession,
scope: CaptureScope
) -> Task<CaptureSessionResult, any Error> {
let runSession: @MainActor @Sendable () async throws -> CaptureSessionResult = {
try await session.run()
}
let enginePreference = liveCaptureEnginePreference(for: scope)
return Task { @MainActor in
if let engineAware = services.screenCapture as? any EngineAwareScreenCaptureServiceProtocol {
try await engineAware.withCaptureEngine(enginePreference, operation: runSession)
} else {
try await runSession()
}
}
}
private func output(_ result: CaptureActionCommandResult) {
if self.jsonOutput {
let error = result.success
? nil
: ErrorInfo(message: result.failureMessage, code: .VALIDATION_ERROR)
let envelope = CaptureActionJSONEnvelope(
success: result.success,
data: result,
messages: nil,
debug_logs: self.outputLogger.getDebugLogs(),
error: error
)
outputJSONCodable(envelope, logger: self.outputLogger)
return
}
print(
"capture(action) kept \(result.capture.stats.framesKept) frames " +
"(dropped \(result.capture.stats.framesDropped))"
)
print("contact sheet: \(result.capture.contactSheet.path)")
print("metadata: \(result.capture.metadataFile)")
if let videoOut = result.capture.videoOut {
print("video: \(videoOut)")
}
print("action exit: \(result.action.exitCode)")
if result.action.timedOut {
print("action timed out after \(String(format: "%.2f", result.action.timeoutSeconds))s")
}
if !result.validation.ok {
print("artifact validation failed: \(result.validation.missing.joined(separator: ", "))")
}
}
private func buildOptions() throws -> CaptureOptions {
let duration = max(1, min(durationLimit ?? 60, 180))
let idle = min(max(idleFps ?? 2, 0.1), 5)
let active = min(max(activeFps ?? 8, 0.5), 15)
let threshold = min(max(threshold ?? 2.5, 0), 100)
let heartbeat = max(heartbeatSec ?? 5, 0)
let quiet = max(quietMs ?? 1000, 0)
let maxFrames = max(maxFrames ?? 800, 1)
let resolutionCap = resolutionCap ?? 1440
let diffStrategy = try CaptureCommandOptionParser.diffStrategy(diffStrategy)
let diffBudgetMs = diffBudgetMs ?? (diffStrategy == .quality ? 30 : nil)
let maxMb = maxMb.flatMap { $0 > 0 ? $0 : nil }
return CaptureOptions(
duration: duration,
idleFps: idle,
activeFps: active,
changeThresholdPercent: threshold,
heartbeatSeconds: heartbeat,
quietMsToIdle: quiet,
maxFrames: maxFrames,
maxMegabytes: maxMb,
highlightChanges: self.highlightChanges,
captureFocus: self.captureFocus,
resolutionCap: resolutionCap,
diffStrategy: diffStrategy,
diffBudgetMs: diffBudgetMs
)
}
private func resolveActionTiming(durationLimit: TimeInterval) throws -> CaptureActionTiming {
let preRoll = max(preRollMs ?? 250, 0)
let postRoll = max(postRollMs ?? 500, 0)
let rollSeconds = Double(preRoll + postRoll) / 1000.0
guard rollSeconds < durationLimit else {
throw ValidationError("--pre-roll-ms + --post-roll-ms must be less than --duration-limit")
}
let defaultActionTimeout = max(0.1, durationLimit - rollSeconds)
let actionTimeout = max(0.1, min(actionTimeout ?? defaultActionTimeout, durationLimit - rollSeconds))
return CaptureActionTiming(
preRollMs: preRoll,
postRollMs: postRoll,
startupGateMs: max(preRoll, 100),
actionTimeout: actionTimeout
)
}
private func resolveOutputDirectory() throws -> URL {
CaptureCommandPathResolver.outputDirectory(from: self.path)
}
private static func sleep(milliseconds: Int) async throws {
guard milliseconds > 0 else { return }
try await Task.sleep(nanoseconds: UInt64(milliseconds) * 1_000_000)
}
private static func waitForPreRollOrCaptureEnd(
milliseconds: Int,
captureTask: Task<CaptureSessionResult, any Error>
) async throws -> CaptureSessionResult? {
try await withThrowingTaskGroup(of: CaptureActionStartupGate.self) { group in
group.addTask {
if milliseconds > 0 {
try await Task.sleep(nanoseconds: UInt64(milliseconds) * 1_000_000)
}
return .preRollElapsed
}
group.addTask {
try await .captureEnded(captureTask.value)
}
guard let first = try await group.next() else {
return nil
}
group.cancelAll()
switch first {
case .preRollElapsed:
return nil
case let .captureEnded(result):
return result
}
}
}
}
private struct CaptureActionTiming {
let preRollMs: Int
let postRollMs: Int
let startupGateMs: Int
let actionTimeout: TimeInterval
}
private enum CaptureActionStartupGate {
case preRollElapsed
case captureEnded(CaptureSessionResult)
}
struct CaptureActionCommandResult: Codable {
let success: Bool
let action: CaptureActionProcessResult
let capture: CaptureSessionResult
let validation: CaptureActionArtifactValidation
var failureMessage: String {
if self.action.timedOut {
return "Action timed out after \(self.action.timeoutSeconds)s"
}
if !self.action.succeeded {
return "Action exited with status \(self.action.exitCode)"
}
return "Capture artifact validation failed"
}
}
struct CaptureActionJSONEnvelope: Codable {
let success: Bool
let data: CaptureActionCommandResult
let messages: [String]?
let debug_logs: [String]
let error: ErrorInfo?
}
struct CaptureActionArtifactValidation: Codable {
let ok: Bool
let checked: [String]
let missing: [String]
}
struct CaptureActionProcessResult: Codable {
let command: [String]
let exitCode: Int32
let timedOut: Bool
let timeoutSeconds: TimeInterval
let durationMs: Int
let stdout: String
let stderr: String
let stdoutTruncated: Bool
let stderrTruncated: Bool
var succeeded: Bool {
!self.timedOut && self.exitCode == 0
}
}
@MainActor
extension CaptureActionCommand {
private func validateArtifacts(_ result: CaptureSessionResult) -> CaptureActionArtifactValidation {
var checked = [result.metadataFile, result.contactSheet.path]
checked.append(contentsOf: result.frames.map(\.path))
if let videoOut = result.videoOut {
checked.append(videoOut)
} else if let expectedVideoOut = CaptureCommandPathResolver.filePath(from: videoOut) {
checked.append(expectedVideoOut)
}
var missing: [String] = []
if result.frames.isEmpty {
missing.append("frame files")
}
for path in checked where !Self.fileExistsAndIsNonEmpty(path) {
missing.append(path)
}
return CaptureActionArtifactValidation(ok: missing.isEmpty, checked: checked, missing: missing)
}
private static func fileExistsAndIsNonEmpty(_ path: String) -> Bool {
let manager = FileManager.default
guard manager.fileExists(atPath: path),
let attributes = try? manager.attributesOfItem(atPath: path),
let size = attributes[.size] as? NSNumber
else {
return false
}
return size.intValue > 0
}
}
@MainActor
extension CaptureActionCommand {
func resolveScope() async throws -> CaptureScope {
let mode = try resolveMode()
switch mode {
case .screen:
let displayInfo = try await displayInfo(for: screenIndex)
return CaptureScope(
kind: .screen,
screenIndex: displayInfo?.index,
displayUUID: displayInfo?.uuid,
windowId: nil,
applicationIdentifier: nil,
windowIndex: nil,
region: nil
)
case .frontmost:
return CaptureScope(kind: .frontmost)
case .window:
let identifier = try resolveApplicationIdentifier()
let windowReference = try await resolveWindowReference(for: identifier)
return CaptureScope(
kind: .window,
screenIndex: nil,
displayUUID: nil,
windowId: windowReference.windowID,
applicationIdentifier: identifier,
windowIndex: windowReference.windowIndex,
region: nil
)
case .area:
let rect = try parseRegion()
return CaptureScope(kind: .region, region: rect)
case .multi:
throw ValidationError("capture action does not support multi-mode captures")
}
}
func resolveMode() throws -> LiveCaptureMode {
if let explicit = mode {
let normalized = explicit.trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
if normalized == "region" { return .area }
guard let mode = LiveCaptureMode(rawValue: normalized) else {
throw ValidationError(
"Unsupported capture action mode '\(explicit)'. Use screen, window, frontmost, or area."
)
}
return mode
}
if self.region != nil { return .area }
if self.app != nil || self.pid != nil || self.windowTitle != nil || self.windowIndex != nil { return .window }
return .frontmost
}
func parseRegion() throws -> CGRect {
guard let region = region?.trimmingCharacters(in: .whitespacesAndNewlines),
!region.isEmpty
else {
throw PeekabooError.invalidInput("Region must be provided when --mode area is set")
}
let parts = region
.split(separator: ",", omittingEmptySubsequences: false)
.map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }
guard parts.count == 4,
let x = Double(parts[0]),
let y = Double(parts[1]),
let width = Double(parts[2]),
let height = Double(parts[3])
else {
throw PeekabooError.invalidInput("Region must be x,y,width,height")
}
guard width > 0, height > 0 else {
throw PeekabooError.invalidInput("Region width and height must be greater than zero")
}
return CGRect(x: x, y: y, width: width, height: height)
}
func focusIfNeeded(appIdentifier: String) async throws {
switch self.captureFocus {
case .background:
return
case .auto:
let options = FocusOptions(
autoFocus: true,
focusTimeout: nil,
focusRetryCount: nil,
spaceSwitch: false,
bringToCurrentSpace: false
)
try await withCaptureFocusMutation {
try await ensureFocused(
applicationName: appIdentifier,
windowTitle: self.windowTitle,
options: options,
services: self.services
)
}
case .foreground:
let options = FocusOptions(
autoFocus: true,
focusTimeout: nil,
focusRetryCount: nil,
spaceSwitch: true,
bringToCurrentSpace: true
)
try await withCaptureFocusMutation {
try await ensureFocused(
applicationName: appIdentifier,
windowTitle: self.windowTitle,
options: options,
services: self.services
)
}
}
}
private func liveCaptureEnginePreference(for scope: CaptureScope) -> CaptureEnginePreference {
let value = (captureEngine ?? self.resolvedRuntime.configuration.captureEnginePreference)?
.trimmingCharacters(in: .whitespacesAndNewlines)
.lowercased()
switch value {
case "modern", "modern-only", "sckit", "sc", "screen-capture-kit", "sck":
return .modern
case "classic", "cg", "legacy", "legacy-only", "false", "0", "no":
return .legacy
default:
return scope.kind == .region ? .legacy : .auto
}
}
private func displayInfo(for index: Int?) async throws -> (index: Int, uuid: String)? {
guard let index else { return nil }
let screens = self.services.screens.listScreens()
guard let match = screens.first(where: { $0.index == index }) else {
throw PeekabooError.invalidInput("Screen index \(index) not found")
}
return (index, "\(match.displayID)")
}
private func resolveWindowReference(for identifier: String) async throws -> (windowID: UInt32?, windowIndex: Int?) {
guard self.windowTitle != nil || self.windowIndex != nil else {
return (nil, nil)
}
let windows = try await WindowServiceBridge.listWindows(
windows: self.services.windows,
target: .application(identifier)
)
let renderable = ObservationTargetResolver.captureCandidates(from: windows)
let selectedWindow: ServiceWindowInfo? = if let title = windowTitle?
.trimmingCharacters(in: .whitespacesAndNewlines),
!title.isEmpty {
renderable.first { $0.title.localizedCaseInsensitiveContains(title) }
} else if let explicitIndex = windowIndex {
renderable.first { $0.index == explicitIndex }
} else {
nil
}
guard let selectedWindow else {
let criteria = self.windowTitle.map { "window title '\($0)' for \(identifier)" }
?? self.windowIndex.map { "window index \($0) for \(identifier)" }
?? "window for \(identifier)"
throw PeekabooError.windowNotFound(criteria: criteria)
}
return (
windowID: UInt32(exactly: selectedWindow.windowID),
windowIndex: selectedWindow.index
)
}
}
extension CaptureActionCommand: ParsableCommand {}
extension CaptureActionCommand: AsyncRuntimeCommand {}
extension CaptureActionCommand: CommanderSignatureProviding {
static func commanderSignature() -> CommandSignature {
let live = CaptureLiveCommand.commanderSignature()
let options = live.options.filter { $0.label != "duration" } + [
.commandOption(
"durationLimit",
help: "Hard capture limit seconds (default 60, max 180)",
long: "duration-limit"
),
.commandOption("preRollMs", help: "Milliseconds to capture before running the action", long: "pre-roll-ms"),
.commandOption("postRollMs", help: "Milliseconds to capture after the action exits", long: "post-roll-ms"),
.commandOption(
"actionTimeout",
help: "Action timeout seconds (defaults to remaining duration)",
long: "action-timeout"
),
.commandOption(
"command",
help: "Command to run; usually pass after --",
long: "command",
parsing: .remaining
),
]
return CommandSignature(
arguments: live.arguments,
options: options,
flags: live.flags,
optionGroups: live.optionGroups
)
}
}
@MainActor
extension CaptureActionCommand: CommanderBindableCommand {
mutating func applyCommanderValues(_ values: CommanderBindableValues) throws {
self.app = values.singleOption("app")
self.pid = try values.decodeOption("pid", as: Int32.self)
self.mode = values.singleOption("mode")
self.windowTitle = values.singleOption("windowTitle")
self.windowIndex = try values.decodeOption("windowIndex", as: Int.self)
self.screenIndex = try values.decodeOption("screenIndex", as: Int.self)
self.region = values.singleOption("region")
if let parsedFocus: LiveCaptureFocus = try values.decodeOptionEnum("captureFocus") {
self.captureFocus = parsedFocus
}
self.captureEngine = values.singleOption("captureEngine")
self.durationLimit = try values.decodeOption("durationLimit", as: Double.self)
self.preRollMs = try values.decodeOption("preRollMs", as: Int.self)
self.postRollMs = try values.decodeOption("postRollMs", as: Int.self)
self.actionTimeout = try values.decodeOption("actionTimeout", as: Double.self)
self.idleFps = try values.decodeOption("idleFps", as: Double.self)
self.activeFps = try values.decodeOption("activeFps", as: Double.self)
self.threshold = try values.decodeOption("threshold", as: Double.self)
self.heartbeatSec = try values.decodeOption("heartbeatSec", as: Double.self)
self.quietMs = try values.decodeOption("quietMs", as: Int.self)
self.maxFrames = try values.decodeOption("maxFrames", as: Int.self)
self.maxMb = try values.decodeOption("maxMb", as: Int.self)
self.resolutionCap = try values.decodeOption("resolutionCap", as: Double.self)
self.diffStrategy = values.singleOption("diffStrategy")
self.diffBudgetMs = try values.decodeOption("diffBudgetMs", as: Int.self)
if values.flag("highlightChanges") { self.highlightChanges = true }
self.path = values.singleOption("path")
self.autocleanMinutes = try values.decodeOption("autocleanMinutes", as: Int.self)
self.videoOut = values.singleOption("videoOut")
self.command = values.optionValues("command")
}
}

View File

@ -69,6 +69,10 @@ struct CaptureLiveCommand: ApplicationResolvable, ErrorHandlingCommand, OutputFo
self.resolvedRuntime.services
}
func withCaptureFocusMutation(_ operation: () async throws -> Void) async rethrows {
try await self.resolvedRuntime.withCaptureFocusMutation(operation)
}
var jsonOutput: Bool {
self.resolvedRuntime.configuration.jsonOutput
}
@ -86,14 +90,14 @@ struct CaptureLiveCommand: ApplicationResolvable, ErrorHandlingCommand, OutputFo
// The capture service performs the authoritative permission check inside
// the serialized capture transaction; an extra CLI-side SCK probe can race
// with concurrent screenshot commands and report transient TCC denial.
let scope = try await self.resolveScope()
let options = try self.buildOptions()
let scope = try await resolveScope()
let options = try buildOptions()
if scope.kind == .window, let identifier = scope.applicationIdentifier {
try await self.focusIfNeeded(appIdentifier: identifier)
try await focusIfNeeded(appIdentifier: identifier)
}
let outputDir = try self.resolveOutputDirectory()
let outputDir = try resolveOutputDirectory()
let deps = WatchCaptureDependencies(
screenCapture: self.services.screenCapture,
screenCapture: services.screenCapture,
screenService: self.services.screens,
frameSource: nil
)
@ -101,7 +105,7 @@ struct CaptureLiveCommand: ApplicationResolvable, ErrorHandlingCommand, OutputFo
scope: scope,
options: options,
outputRoot: outputDir,
autoclean: WatchAutocleanConfig(minutes: self.autocleanMinutes ?? 120, managed: self.path == nil),
autoclean: WatchAutocleanConfig(minutes: autocleanMinutes ?? 120, managed: path == nil),
sourceKind: .live,
videoIn: nil,
videoOut: CaptureCommandPathResolver.filePath(from: self.videoOut),
@ -111,21 +115,21 @@ struct CaptureLiveCommand: ApplicationResolvable, ErrorHandlingCommand, OutputFo
let runSession: @MainActor @Sendable () async throws -> CaptureSessionResult = {
try await session.run()
}
let enginePreference = self.liveCaptureEnginePreference(for: scope)
let result: CaptureSessionResult = if let engineAware = self.services.screenCapture
let enginePreference = liveCaptureEnginePreference(for: scope)
let result: CaptureSessionResult = if let engineAware = services.screenCapture
as? any EngineAwareScreenCaptureServiceProtocol {
try await engineAware.withCaptureEngine(enginePreference, operation: runSession)
} else {
try await runSession()
}
self.output(result)
output(result)
self.logger.operationComplete(
"capture_live",
success: true,
metadata: ["frames_kept": result.stats.framesKept]
)
} catch {
self.handleError(error)
handleError(error)
self.logger.operationComplete(
"capture_live",
success: false,
@ -138,7 +142,7 @@ struct CaptureLiveCommand: ApplicationResolvable, ErrorHandlingCommand, OutputFo
extension CaptureLiveCommand {
private func liveCaptureEnginePreference(for scope: CaptureScope) -> CaptureEnginePreference {
let value = (self.captureEngine ?? self.resolvedRuntime.configuration.captureEnginePreference)?
let value = (captureEngine ?? self.resolvedRuntime.configuration.captureEnginePreference)?
.trimmingCharacters(in: .whitespacesAndNewlines)
.lowercased()

View File

@ -3,7 +3,7 @@ import PeekabooCore
@MainActor
extension CaptureLiveCommand {
func focusIfNeeded(appIdentifier: String) async throws {
switch self.captureFocus {
switch captureFocus {
case .background: return
case .auto:
let options = FocusOptions(
@ -13,12 +13,14 @@ extension CaptureLiveCommand {
spaceSwitch: false,
bringToCurrentSpace: false
)
try await ensureFocused(
applicationName: appIdentifier,
windowTitle: self.windowTitle,
options: options,
services: self.services
)
try await withCaptureFocusMutation {
try await ensureFocused(
applicationName: appIdentifier,
windowTitle: self.windowTitle,
options: options,
services: self.services
)
}
case .foreground:
let options = FocusOptions(
autoFocus: true,
@ -27,12 +29,14 @@ extension CaptureLiveCommand {
spaceSwitch: true,
bringToCurrentSpace: true
)
try await ensureFocused(
applicationName: appIdentifier,
windowTitle: self.windowTitle,
options: options,
services: self.services
)
try await withCaptureFocusMutation {
try await ensureFocused(
applicationName: appIdentifier,
windowTitle: self.windowTitle,
options: options,
services: self.services
)
}
}
}
}

View File

@ -23,7 +23,12 @@ struct CaptureCommand: ParsableCommand {
CommandDescription(
commandName: "capture",
abstract: "Capture live screens/windows or ingest a video and extract frames",
subcommands: [CaptureLiveCommand.self, CaptureVideoCommand.self, CaptureWatchAlias.self],
subcommands: [
CaptureLiveCommand.self,
CaptureActionCommand.self,
CaptureVideoCommand.self,
CaptureWatchAlias.self,
],
showHelpOnEmptyInvocation: true
)
}

View File

@ -17,7 +17,9 @@ extension ImageCommand {
),
observation: ImageObservationDiagnostics(
timings: observation.timings,
diagnostics: observation.diagnostics
diagnostics: observation.diagnostics,
capture: observation.capture,
rawImagePath: observation.files.rawScreenshotPath
)
)
}

View File

@ -4,43 +4,51 @@ import PeekabooCore
@MainActor
extension ImageCommand {
func focusIfNeeded(appIdentifier: String) async throws {
switch self.captureFocus {
switch captureFocus {
case .background:
return
case .auto:
if await self.hasVisibleCaptureWindow(appIdentifier: appIdentifier) {
if try await self.hasVisibleCaptureWindow(appIdentifier: appIdentifier) {
return
}
if self.windowTitle == nil, await self.isAlreadyFrontmost(appIdentifier: appIdentifier) {
if windowTitle == nil, try await self.isAlreadyFrontmost(appIdentifier: appIdentifier) {
return
}
let focusIdentifier = await self.resolveFocusIdentifier(appIdentifier: appIdentifier)
let focusIdentifier = try await resolveFocusIdentifier(appIdentifier: appIdentifier)
let options = FocusOptions(autoFocus: true, spaceSwitch: false, bringToCurrentSpace: false)
try await ensureFocused(
applicationName: focusIdentifier,
windowTitle: self.windowTitle,
options: options,
services: self.services
)
try await withCaptureFocusMutation {
try await ensureFocused(
applicationName: focusIdentifier,
windowTitle: self.windowTitle,
options: options,
services: self.services
)
}
case .foreground:
let focusIdentifier = await self.resolveFocusIdentifier(appIdentifier: appIdentifier)
let focusIdentifier = try await resolveFocusIdentifier(appIdentifier: appIdentifier)
let options = FocusOptions(autoFocus: true, spaceSwitch: true, bringToCurrentSpace: true)
try await ensureFocused(
applicationName: focusIdentifier,
windowTitle: self.windowTitle,
options: options,
services: self.services
)
try await withCaptureFocusMutation {
try await ensureFocused(
applicationName: focusIdentifier,
windowTitle: self.windowTitle,
options: options,
services: self.services
)
}
}
}
private func hasVisibleCaptureWindow(appIdentifier: String) async -> Bool {
guard let app = try? await self.services.applications.findApplication(identifier: appIdentifier) else {
private func hasVisibleCaptureWindow(appIdentifier: String) async throws -> Bool {
guard let app = try await FocusFailurePolicy.optional({
try await services.applications.findApplication(identifier: appIdentifier)
}) else {
return false
}
let lookupIdentifier = app.bundleIdentifier ?? app.name
guard let response = try? await self.services.applications.listWindows(for: lookupIdentifier, timeout: 1) else {
guard let response = try await FocusFailurePolicy.optional({
try await services.applications.listWindows(for: lookupIdentifier, timeout: 1)
}) else {
return false
}
@ -51,7 +59,7 @@ extension ImageCommand {
return false
}
guard let windowTitle = self.windowTitle?.trimmingCharacters(in: .whitespacesAndNewlines),
guard let windowTitle = windowTitle?.trimmingCharacters(in: .whitespacesAndNewlines),
!windowTitle.isEmpty
else {
return true
@ -62,9 +70,13 @@ extension ImageCommand {
}
}
private func isAlreadyFrontmost(appIdentifier: String) async -> Bool {
guard let frontmost = try? await self.services.applications.getFrontmostApplication(),
let target = try? await self.services.applications.findApplication(identifier: appIdentifier)
private func isAlreadyFrontmost(appIdentifier: String) async throws -> Bool {
guard let frontmost = try await FocusFailurePolicy.optional({
try await services.applications.getFrontmostApplication()
}),
let target = try await FocusFailurePolicy.optional({
try await services.applications.findApplication(identifier: appIdentifier)
})
else {
return false
}
@ -72,8 +84,10 @@ extension ImageCommand {
return frontmost.processIdentifier == target.processIdentifier
}
private func resolveFocusIdentifier(appIdentifier: String) async -> String {
guard let app = try? await self.services.applications.findApplication(identifier: appIdentifier) else {
private func resolveFocusIdentifier(appIdentifier: String) async throws -> String {
guard let app = try await FocusFailurePolicy.optional({
try await services.applications.findApplication(identifier: appIdentifier)
}) else {
return appIdentifier
}
return "PID:\(app.processIdentifier)"

View File

@ -1,3 +1,4 @@
import AppKit
import Commander
import Foundation
import PeekabooCore
@ -19,12 +20,127 @@ struct ImageObservationDiagnostics: Codable {
let warnings: [String]
let state_snapshot: SeeDesktopStateSnapshotSummary?
let target: SeeObservationTargetDiagnostics?
let coordinates: ImageCoordinateDiagnostics?
init(timings: ObservationTimings, diagnostics: DesktopObservationDiagnostics) {
init(
timings: ObservationTimings,
diagnostics: DesktopObservationDiagnostics,
capture: CaptureResult? = nil,
rawImagePath: String? = nil
) {
self.spans = timings.spans.map(SeeObservationSpan.init)
self.warnings = diagnostics.warnings
self.warnings = diagnostics.warnings + ImageBlankCaptureDiagnostics.warnings(
rawImagePath: rawImagePath,
capture: capture
)
self.state_snapshot = diagnostics.stateSnapshot.map(SeeDesktopStateSnapshotSummary.init)
self.target = diagnostics.target.map(SeeObservationTargetDiagnostics.init)
self.coordinates = capture.map(ImageCoordinateDiagnostics.init)
}
}
struct ImageCoordinateDiagnostics: Codable {
let coordinate_space: String
let logical_bounds: CGRect?
let image_size_pixels: ImageSizeDiagnostics
let scale_factor: CGFloat?
let screen_index: Int?
let screen_name: String?
init(capture: CaptureResult) {
let metadata = capture.metadata
self.coordinate_space = "global_display_points"
self.logical_bounds = metadata.windowInfo?.bounds ?? metadata.displayInfo?.bounds
self.image_size_pixels = ImageSizeDiagnostics(metadata.size)
self.scale_factor = metadata.diagnostics?.outputScale
?? metadata.displayInfo?.scaleFactor
?? Self.inferredScale(imageSize: metadata.size, bounds: self.logical_bounds)
self.screen_index = metadata.windowInfo?.screenIndex ?? metadata.displayInfo?.index
self.screen_name = metadata.windowInfo?.screenName ?? metadata.displayInfo?.name
}
private static func inferredScale(imageSize: CGSize, bounds: CGRect?) -> CGFloat? {
guard let bounds, bounds.width > .zero else {
return nil
}
return imageSize.width / bounds.width
}
}
struct ImageSizeDiagnostics: Codable {
let width: Double
let height: Double
init(_ size: CGSize) {
self.width = size.width
self.height = size.height
}
}
enum ImageBlankCaptureDiagnostics {
static func warnings(rawImagePath: String?, capture: CaptureResult?) -> [String] {
guard let rawImagePath,
let capture,
capture.metadata.mode == .window,
let data = try? Data(contentsOf: URL(fileURLWithPath: rawImagePath)),
let bitmap = NSBitmapImageRep(data: data)
else {
return []
}
return self.blankWarning(bitmap: bitmap).map { [$0] } ?? []
}
private static func blankWarning(bitmap: NSBitmapImageRep) -> String? {
let width = bitmap.pixelsWide
let height = bitmap.pixelsHigh
guard width > 1, height > 1 else {
return nil
}
let sampleCount = min(width, 20) * min(height, 20)
guard sampleCount > 0 else { return nil }
var alphaSum = 0.0
var luminanceSum = 0.0
var luminanceSquaredSum = 0.0
let xStep = max(1, width / min(width, 20))
let yStep = max(1, height / min(height, 20))
var actualSamples = 0
for y in stride(from: 0, to: height, by: yStep) {
for x in stride(from: 0, to: width, by: xStep) {
guard let color = bitmap.colorAt(x: x, y: y)?.usingColorSpace(.deviceRGB) else {
continue
}
let alpha = Double(color.alphaComponent)
let luminance = Double(0.2126 * color.redComponent + 0.7152 * color.greenComponent + 0.0722 * color
.blueComponent)
alphaSum += alpha
luminanceSum += luminance
luminanceSquaredSum += luminance * luminance
actualSamples += 1
}
}
guard actualSamples > 0 else { return nil }
let alphaMean = alphaSum / Double(actualSamples)
if alphaMean < 0.01 {
return "Captured window image appears transparent; target may be hidden or non-renderable."
}
let luminanceMean = luminanceSum / Double(actualSamples)
let variance = max(0, luminanceSquaredSum / Double(actualSamples) - luminanceMean * luminanceMean)
if variance < 0.0001, luminanceMean < 0.02 {
return "Captured window image appears solid black; target may be occluded, transparent, or non-renderable."
}
if variance < 0.0001, luminanceMean > 0.98 {
return "Captured window image appears blank white; target may be empty or non-renderable."
}
return nil
}
}
@ -82,7 +198,10 @@ extension ImageCommand {
if self.jsonOutput {
outputSuccessCodable(data: output, logger: self.outputLogger)
} else {
captures.map(\.file).forEach { print("📸 \(self.describeSavedFile($0))") }
for capture in captures {
print("📸 \(self.describeSavedFile(capture.file))")
self.printWarnings(capture.observation.warnings)
}
}
}
@ -95,7 +214,10 @@ extension ImageCommand {
if self.jsonOutput {
outputSuccessCodable(data: output, logger: self.outputLogger)
} else {
captures.map(\.file).forEach { print("📸 \(self.describeSavedFile($0))") }
for capture in captures {
print("📸 \(self.describeSavedFile(capture.file))")
self.printWarnings(capture.observation.warnings)
}
print("\n🤖 Analysis (\(analysis.provider)) - \(analysis.model):")
print(analysis.text)
}
@ -117,6 +239,10 @@ extension ImageCommand {
segments.append("\(file.path)")
return segments.joined(separator: " ")
}
private func printWarnings(_ warnings: [String]) {
warnings.forEach { print("⚠️ \($0)") }
}
}
extension ImageFormat {

View File

@ -78,6 +78,10 @@ struct ImageCommand: ApplicationResolvable, ErrorHandlingCommand, OutputFormatta
self.resolvedRuntime.services
}
func withCaptureFocusMutation(_ operation: () async throws -> Void) async rethrows {
try await self.resolvedRuntime.withCaptureFocusMutation(operation)
}
var jsonOutput: Bool {
self.runtime?.configuration.jsonOutput ?? self.runtimeOptions.jsonOutput
}
@ -95,7 +99,7 @@ struct ImageCommand: ApplicationResolvable, ErrorHandlingCommand, OutputFormatta
self.runtime = runtime
self.logger.setJsonOutputMode(self.jsonOutput)
let startMetadata: [String: Any] = [
"mode": self.mode?.rawValue ?? "auto",
"mode": mode?.rawValue ?? "auto",
"app": self.app ?? "none",
"pid": self.pid ?? 0,
"hasAnalyzePrompt": self.analyze != nil
@ -103,28 +107,28 @@ struct ImageCommand: ApplicationResolvable, ErrorHandlingCommand, OutputFormatta
self.logger.operationStart("image_command", metadata: startMetadata)
do {
try self.validateStdoutStreamingOptions()
try validateStdoutStreamingOptions()
// ScreenCaptureService performs the authoritative permission check inside each capture path.
// Avoid preflighting here too; it adds fixed latency to every one-shot screenshot.
let captures = try await CrossProcessOperationGate.withExclusiveOperation(
named: CrossProcessOperationGate.desktopObservationName
) {
try await self.performCapture()
try await performCapture()
}
if self.streamsImageToStdout {
try self.outputImageToStdout(captures)
} else if let prompt = self.analyze, let firstFile = captures.first?.file {
let analysis = try await self.analyzeImage(at: firstFile.path, with: prompt)
self.outputResultsWithAnalysis(captures, analysis: analysis)
if streamsImageToStdout {
try outputImageToStdout(captures)
} else if let prompt = analyze, let firstFile = captures.first?.file {
let analysis = try await analyzeImage(at: firstFile.path, with: prompt)
outputResultsWithAnalysis(captures, analysis: analysis)
} else {
self.outputResults(captures)
outputResults(captures)
}
self.logger.operationComplete("image_command", success: true)
} catch {
self.handleError(error)
handleError(error)
self.logger.operationComplete(
"image_command",
success: false,
@ -155,7 +159,7 @@ extension ImageCommand: CommanderBindableCommand {
mutating func applyCommanderValues(_ values: CommanderBindableValues) throws {
self.app = values.singleOption("app")
self.pid = try values.decodeOption("pid", as: Int32.self)
self.path = values.singleOption("path")
path = values.singleOption("path")
if let parsedMode: CaptureMode = try values.decodeOptionEnum("mode") {
self.mode = parsedMode
}
@ -169,7 +173,7 @@ extension ImageCommand: CommanderBindableCommand {
if let parsedFormat {
self.format = parsedFormat
}
if let path = self.path?.trimmingCharacters(in: .whitespacesAndNewlines),
if let path = path?.trimmingCharacters(in: .whitespacesAndNewlines),
!path.isEmpty {
let expanded = (path as NSString).expandingTildeInPath
let ext = URL(fileURLWithPath: expanded).pathExtension.lowercased()

View File

@ -126,7 +126,7 @@ struct LearnCommand {
5. Recover from errors by trying alternative interactions (menus, hotkeys).
6. Common workflows:
- Screenshot: `image` with `--app` or `--mode screen`.
- Typing: `click` the field, then `type` the text.
- Typing: `click` the field, then `type --app ...` the text; add `--foreground` only if needed.
- Menus: `menu click --path ...`.
- Keyboard shortcuts: `hotkey`.
""", to: &output)

View File

@ -37,8 +37,7 @@ extension ListCommand {
self.logger.setJsonOutputMode(self.jsonOutput)
do {
try await requireScreenRecordingPermission(services: self.services)
let output = try await self.services.applications.listApplications()
let output = try await services.applications.listApplications()
if self.jsonOutput {
outputSuccessCodable(data: output.data, logger: self.outputLogger)
@ -46,7 +45,7 @@ extension ListCommand {
print(CLIFormatter.format(output))
}
} catch {
self.handleError(error)
handleError(error)
throw ExitCode(1)
}
}

View File

@ -32,6 +32,12 @@ extension PermissionsCommand.GrantSubcommand: CommanderSignatureProviding {
}
}
extension PermissionsCommand.RequestScreenRecordingSubcommand: CommanderSignatureProviding {
static func commanderSignature() -> CommandSignature {
CommandSignature()
}
}
extension PermissionsCommand.RequestEventSynthesizingSubcommand: CommanderSignatureProviding {
static func commanderSignature() -> CommandSignature {
CommandSignature()

View File

@ -11,6 +11,7 @@ struct PermissionsCommand: ParsableCommand {
subcommands: [
StatusSubcommand.self,
GrantSubcommand.self,
RequestScreenRecordingSubcommand.self,
RequestEventSynthesizingSubcommand.self,
],
defaultSubcommand: StatusSubcommand.self
@ -134,6 +135,57 @@ extension PermissionsCommand {
}
}
@MainActor
struct RequestScreenRecordingSubcommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfigurable {
struct Result: Codable {
let action: String
let granted: Bool
}
@RuntimeStorage private var runtime: CommandRuntime?
var runtimeOptions = CommandRuntimeOptions()
private var resolvedRuntime: CommandRuntime {
guard let runtime else {
preconditionFailure("CommandRuntime must be configured before accessing runtime resources")
}
return runtime
}
var outputLogger: Logger {
self.resolvedRuntime.logger
}
var jsonOutput: Bool {
self.runtime?.configuration.jsonOutput ?? self.runtimeOptions.jsonOutput
}
@MainActor
mutating func run(using runtime: CommandRuntime) async throws {
self.runtime = runtime
let granted = await PermissionHelpers.performInteractivePermissionRequest(using: runtime) {
runtime.services.permissions.requestScreenRecordingPermission(interactive: true)
}
let result = Result(action: "request-screen-recording", granted: granted)
if self.jsonOutput {
outputSuccessCodable(data: result, logger: self.outputLogger)
return
}
if granted {
print("Screen Recording permission is granted.")
} else {
print("Screen Recording permission was not granted.")
print(
"If no prompt appeared, open System Settings > Privacy & Security > " +
"Screen & System Audio Recording."
)
print("Add or enable the current Peekaboo binary, then restart Peekaboo.")
}
}
}
@MainActor
struct RequestEventSynthesizingSubcommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfigurable {
@RuntimeStorage private var runtime: CommandRuntime?
@ -158,7 +210,10 @@ extension PermissionsCommand {
mutating func run(using runtime: CommandRuntime) async throws {
self.runtime = runtime
do {
let result = try await PermissionHelpers.requestEventSynthesizingPermission(services: runtime.services)
let result = try await PermissionHelpers.requestEventSynthesizingPermission(
services: runtime.services,
runtime: runtime
)
self.render(result)
} catch {
self.handleError(error)
@ -233,6 +288,27 @@ extension PermissionsCommand.GrantSubcommand: CommanderBindableCommand {
}
}
@MainActor
extension PermissionsCommand.RequestScreenRecordingSubcommand: ParsableCommand {
nonisolated(unsafe) static var commandDescription: CommandDescription {
MainActorCommandDescription.describe {
CommandDescription(
commandName: "request-screen-recording",
abstract: "Request Screen Recording permission for the local Peekaboo process"
)
}
}
}
extension PermissionsCommand.RequestScreenRecordingSubcommand: AsyncRuntimeCommand {}
@MainActor
extension PermissionsCommand.RequestScreenRecordingSubcommand: CommanderBindableCommand {
mutating func applyCommanderValues(_ values: CommanderBindableValues) throws {
_ = values
}
}
@MainActor
extension PermissionsCommand.RequestEventSynthesizingSubcommand: ParsableCommand {
nonisolated(unsafe) static var commandDescription: CommandDescription {

View File

@ -6,14 +6,17 @@ import TachikomaMCP
@MainActor
struct ToolsCommand: OutputFormattable, RuntimeOptionsConfigurable {
private static let abstractText = "List available tools with filtering and display options"
private static let descriptionText = "Tools command for listing and filtering available tools"
private static let abstractText = "List the MCP/agent tool catalog"
private static let descriptionText = "Tools command for listing the MCP/agent tool catalog"
static let commandDescription = CommandDescription(
commandName: "tools",
abstract: Self.abstractText,
discussion: """
Display all available Peekaboo tools exposed to agents and the MCP server.
Display the Peekaboo MCP/agent tool catalog. These tools are exposed to agents
and `peekaboo mcp` clients (e.g. Codex, Claude Code, Cursor). Some tools also
have dedicated CLI wrappers, such as `peekaboo browser` and `peekaboo inspect-ui`.
Run `peekaboo --help` for the CLI command list.
Examples:
peekaboo tools # Show all tools

View File

@ -31,6 +31,7 @@ extension ClickCommand: CommanderBindableCommand {
}
self.double = values.flag("double")
self.right = values.flag("right")
self.foreground = values.flag("foreground")
self.focusOptions = try values.makeFocusOptions(includeBackgroundDelivery: true)
}
}
@ -48,12 +49,12 @@ extension ClickCommand: CommanderSignatureProviding {
options: [
.commandOption(
"snapshot",
help: "Snapshot ID (uses latest if not specified)",
help: "Snapshot ID, or 'latest' (uses latest if not specified)",
long: "snapshot"
),
.commandOption(
"on",
help: "Element ID to click (e.g., B1, T2)",
help: "Opaque element ID copied from current see or inspect-ui output",
long: "on"
),
.commandOption(
@ -83,6 +84,11 @@ extension ClickCommand: CommanderSignatureProviding {
help: "Right-click (secondary click)",
long: "right"
),
.commandFlag(
"foreground",
help: "Focus target and send a foreground mouse click",
long: "foreground"
),
.commandFlag(
"globalCoords",
help: "Treat --coords as global screen coordinates even with target options",

View File

@ -14,6 +14,7 @@ struct ClickResult: Codable {
let inputCoordinates: [String: Double]?
let screenCoordinates: [String: Double]?
let targetPoint: InteractionTargetPointDiagnostics?
let deliveryMode: String?
init(
success: Bool,
@ -27,7 +28,8 @@ struct ClickResult: Codable {
coordinateSpace: String? = nil,
inputCoordinates: CGPoint? = nil,
screenCoordinates: CGPoint? = nil,
targetPoint: InteractionTargetPointDiagnostics? = nil
targetPoint: InteractionTargetPointDiagnostics? = nil,
deliveryMode: String? = nil
) {
self.success = success
self.clickedElement = clickedElement
@ -41,5 +43,6 @@ struct ClickResult: Codable {
self.inputCoordinates = inputCoordinates.map { ["x": $0.x, "y": $0.y] }
self.screenCoordinates = screenCoordinates.map { ["x": $0.x, "y": $0.y] }
self.targetPoint = targetPoint
self.deliveryMode = deliveryMode
}
}

View File

@ -25,6 +25,15 @@ extension ClickCommand {
if self.globalCoords && self.coords == nil {
throw ValidationError("--global-coords requires --coords")
}
if self.foreground && self.focusOptions.backgroundDeliveryExplicitlyRequested {
throw ValidationError("--foreground cannot be combined with --focus-background")
}
if self.focusOptions.backgroundDeliveryExplicitlyRequested &&
self.focusOptions.hasForegroundFocusOverrides {
throw ValidationError("--focus-background cannot be combined with focus options")
}
}
func formatElementInfo(_ element: DetectedElement) -> String {
@ -39,7 +48,7 @@ extension ClickCommand {
💡 Hints:
Run 'peekaboo see' first to capture UI elements
Check that the element ID is correct (e.g., B1, T2)
Copy the opaque element ID exactly from current see or inspect-ui output
Element may have disappeared or changed
"""
}

View File

@ -11,10 +11,10 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
@Argument(help: "Element text or query to click")
var query: String?
@Option(help: "Snapshot ID (uses latest if not specified)")
@Option(help: "Snapshot ID, or 'latest' (uses latest if not specified)")
var snapshot: String?
@Option(help: "Element ID to click (e.g., B1, T2)")
@Option(help: "Opaque element ID copied from current see or inspect-ui output")
var on: String?
@Option(name: .customLong("id"), help: "Element ID to click (alias for --on)")
@ -37,6 +37,9 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
@Flag(help: "Right-click (secondary click)")
var right = false
@Flag(help: "Focus target and send a foreground mouse click")
var foreground = false
@OptionGroup var focusOptions: FocusCommandOptions
@RuntimeStorage private var runtime: CommandRuntime?
@ -65,6 +68,20 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
self.runtime?.configuration.jsonOutput ?? self.runtimeOptions.jsonOutput
}
private var deliveryMode: ClickDeliveryMode {
if self.focusOptions.backgroundDeliveryExplicitlyRequested {
return .background
}
if self.foreground || self.focusOptions.hasForegroundFocusOverrides {
return .foreground
}
return .background
}
private var usesBackgroundDelivery: Bool {
self.deliveryMode == .background
}
@MainActor
mutating func run(using runtime: CommandRuntime) async throws {
self.runtime = runtime
@ -72,14 +89,14 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
let startTime = Date()
do {
try self.validate()
try validate()
// Determine click target first to check if we need a snapshot
let clickTarget: ClickTarget
let waitResult: WaitForElementResult
var activeSnapshotId: String
var observationForInvalidation: InteractionObservationContext?
var coordinateResolution: InteractionCoordinateResolution?
var explicitWindowResolution: InteractionWindowResolution?
// Check if we're clicking by coordinates (doesn't need snapshot)
if let coordString = coords {
@ -97,6 +114,9 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
clickTarget = .coordinates(resolvedCoordinates.screenPoint)
waitResult = WaitForElementResult(found: true, element: nil, waitTime: 0)
activeSnapshotId = "" // Not needed for coordinate clicks
if !self.usesBackgroundDelivery {
self.resolvedRuntime.beginInteractionMutation()
}
try await self.focusApplicationIfNeeded(
snapshotId: nil,
coordinateResolution: resolvedCoordinates
@ -106,8 +126,8 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
// InputDriver.click() sends a CGEvent at screen-absolute coordinates,
// so if the target window is not frontmost, the click will land on
// whatever window is at that position (see #90).
if !self.focusOptions.focusBackground {
try await self.verifyFocusForCoordinateClick(coordinateResolution: resolvedCoordinates)
if !self.usesBackgroundDelivery {
try await verifyFocusForCoordinateClick(coordinateResolution: resolvedCoordinates)
}
} else {
@ -120,27 +140,36 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
)
try await observation.validateIfExplicit(using: self.services.snapshots)
explicitWindowResolution = try await self.resolveExplicitWindowSelection(
observation: observation
)
if !self.usesBackgroundDelivery {
self.resolvedRuntime.beginInteractionMutation()
}
try await self.focusApplicationIfNeeded(snapshotId: observation.focusSnapshotId(for: self.target))
// Use whichever element ID parameter was provided
let elementId = self.on ?? self.id
if let elementId {
if !self.focusOptions.focusBackground {
if !self.usesBackgroundDelivery {
let refreshRuntime = self.resolvedRuntime
observation = try await InteractionObservationRefresher.refreshForMissingElementsIfNeeded(
observation,
elementIds: [elementId],
target: self.target,
services: self.services,
logger: self.logger
logger: self.logger,
beforeRefresh: { startedAt in
refreshRuntime.beginInteractionMutation(at: startedAt)
}
)
}
observationForInvalidation = observation
activeSnapshotId = observation.snapshotId ?? ""
clickTarget = .elementId(elementId)
if self.focusOptions.focusBackground {
let element = try await self.cachedElementById(elementId, observation: observation)
if self.usesBackgroundDelivery {
let element = try await cachedElementById(elementId, observation: observation)
waitResult = WaitForElementResult(found: true, element: element, waitTime: 0)
} else {
// Click by element ID with auto-wait
@ -157,14 +186,13 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
}
} else if let searchQuery = query {
if !self.focusOptions.focusBackground {
if !self.usesBackgroundDelivery {
observation = try await self.refreshObservationIfQueryMissing(observation, query: searchQuery)
}
observationForInvalidation = observation
activeSnapshotId = observation.snapshotId ?? ""
if self.focusOptions.focusBackground {
let element = try await self.cachedElementMatching(searchQuery, observation: observation)
if self.usesBackgroundDelivery {
let element = try await cachedElementMatching(searchQuery, observation: observation)
clickTarget = .elementId(element.id)
waitResult = WaitForElementResult(found: true, element: element, waitTime: 0)
} else {
@ -192,19 +220,50 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
}
}
let backgroundProcessIdentifier: pid_t? = if self.usesBackgroundDelivery {
try await self.resolveBackgroundClickProcessIdentifier(
snapshotId: activeSnapshotId.isEmpty ? nil : activeSnapshotId,
coordinateResolution: coordinateResolution,
explicitWindowResolution: explicitWindowResolution
)
} else {
nil
}
// Determine click type
let clickType: ClickType = self.right ? .right : (self.double ? .double : .single)
try await self.performClick(clickTarget, clickType: clickType, snapshotId: activeSnapshotId)
self.resolvedRuntime.beginInteractionMutation()
try await self.performClick(
clickTarget,
clickType: clickType,
snapshotId: activeSnapshotId,
coordinateResolution: coordinateResolution,
explicitWindowResolution: explicitWindowResolution,
backgroundProcessIdentifier: backgroundProcessIdentifier
)
// Brief delay to ensure click is processed
try await Task.sleep(nanoseconds: 20_000_000) // 0.02 seconds
try? await Task.sleep(nanoseconds: 20_000_000) // 0.02 seconds
// Result formatting can await bridge lookups. Freeze the mutation boundary first so
// observations created after the click remain eligible as the next implicit latest.
let snapshotInvalidationCutoff = Date()
let appName = await self.resultApplicationName(
// The click already happened. Advance every host watermark before diagnostics that can
// fail if the action closed, moved, or resized its target window.
await InteractionObservationInvalidator.invalidateAfterClickMutation(
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "click",
through: snapshotInvalidationCutoff
)
try Task.checkCancellation()
let appName = await resultApplicationName(
snapshotId: activeSnapshotId,
coordinateResolution: coordinateResolution
)
let details = try await self.clickOutputDetails(
let details = try await clickOutputDetails(
clickTarget: clickTarget,
waitResult: waitResult,
snapshotId: activeSnapshotId,
@ -219,27 +278,20 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
waitTime: waitResult.waitTime,
executionTime: Date().timeIntervalSince(startTime),
targetApp: appName,
targetWindowId: coordinateResolution?.targetWindowID,
targetWindowTitle: coordinateResolution?.targetWindowTitle,
targetWindowId: explicitWindowResolution?.windowInfo.windowID ?? coordinateResolution?.targetWindowID,
targetWindowTitle: explicitWindowResolution?.windowInfo.title ?? coordinateResolution?
.targetWindowTitle,
coordinateSpace: coordinateResolution?.coordinateSpace.rawValue,
inputCoordinates: coordinateResolution?.inputPoint,
screenCoordinates: coordinateResolution?.screenPoint,
targetPoint: details.targetPointDiagnostics
targetPoint: details.targetPointDiagnostics,
deliveryMode: self.deliveryMode.rawValue
)
if let observationForInvalidation {
await InteractionObservationInvalidator.invalidateAfterMutation(
observationForInvalidation,
snapshots: self.services.snapshots,
logger: self.logger,
reason: "click"
)
}
self.outputSuccess(result)
} catch {
self.handleError(error)
handleError(error)
throw ExitCode.failure
}
}
@ -256,13 +308,11 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
guard let element = waitResult.element else {
return (.zero, "Element ID: \(id)", nil)
}
let resolution = try await InteractionTargetPointResolver.elementCenterResolution(
return try await self.elementOutputDetails(
element: element,
elementId: id,
snapshotId: snapshotId.isEmpty ? nil : snapshotId,
snapshots: self.services.snapshots
snapshotId: snapshotId
)
return (resolution.point, self.formatElementInfo(element), resolution.diagnostics)
case let .coordinates(point):
let diagnostics = if let coordinateResolution {
@ -284,13 +334,44 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
guard let element = waitResult.element else {
return (.zero, "Element matching: \(query)", nil)
}
let resolution = try await InteractionTargetPointResolver.elementCenterResolution(
return try await self.elementOutputDetails(
element: element,
elementId: element.id,
snapshotId: snapshotId.isEmpty ? nil : snapshotId,
snapshotId: snapshotId
)
}
}
private func elementOutputDetails(
element: DetectedElement,
elementId: String,
snapshotId: String
) async throws
-> (location: CGPoint, clickedElement: String?, targetPointDiagnostics: InteractionTargetPointDiagnostics?) {
let resolvedSnapshotId = snapshotId.isEmpty ? nil : snapshotId
do {
let resolution = try await InteractionTargetPointResolver.elementCenterResolution(
element: element,
elementId: elementId,
snapshotId: resolvedSnapshotId,
snapshots: self.services.snapshots
)
return (resolution.point, self.formatElementInfo(element), resolution.diagnostics)
return (resolution.point, formatElementInfo(element), resolution.diagnostics)
} catch let error as CancellationError {
throw error
} catch {
// The click already succeeded; its target may have closed or moved before result formatting.
self.logger.debug("Post-click target diagnostics unavailable: \(error.localizedDescription)")
let point = CGPoint(x: element.bounds.midX, y: element.bounds.midY)
let diagnostics = InteractionTargetPointDiagnostics(
source: InteractionTargetPointSource.element.rawValue,
elementId: elementId,
snapshotId: resolvedSnapshotId,
original: InteractionPoint(point),
resolved: InteractionPoint(point),
windowAdjustment: nil
)
return (point, formatElementInfo(element), diagnostics)
}
}
@ -306,29 +387,37 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
return targetApplicationName
}
if let processIdentifier = coordinateResolution?.targetProcessIdentifier {
return await self.applicationName(processIdentifier: processIdentifier) ?? "PID \(processIdentifier)"
return await applicationName(processIdentifier: processIdentifier) ?? "PID \(processIdentifier)"
}
if let windowID = coordinateResolution?.targetWindowID {
return "window \(windowID)"
}
guard self.focusOptions.focusBackground else {
guard self.usesBackgroundDelivery else {
return await self.frontmostApplicationName()
}
if let pid = self.target.pid {
return await self.applicationName(processIdentifier: pid) ?? "PID \(pid)"
if let pid = target.pid {
return await applicationName(processIdentifier: pid) ?? "PID \(pid)"
}
if let appIdentifier = self.target.app?.trimmingCharacters(in: .whitespacesAndNewlines),
if let appIdentifier = target.app?.trimmingCharacters(in: .whitespacesAndNewlines),
!appIdentifier.isEmpty {
return await (try? self.services.applications.findApplication(identifier: appIdentifier).name) ??
appIdentifier
}
guard !snapshotId.isEmpty,
let snapshot = try? await self.services.snapshots.getUIAutomationSnapshot(snapshotId: snapshotId)
let snapshot = try? await services.snapshots.getUIAutomationSnapshot(snapshotId: snapshotId)
else {
if let detectionResult = try? await services.snapshots.getDetectionResult(snapshotId: snapshotId) {
if let applicationName = detectionResult.metadata.windowContext?.applicationName {
return applicationName
}
if let processId = detectionResult.metadata.windowContext?.applicationProcessId {
return await applicationName(processIdentifier: processId) ?? "PID \(processId)"
}
}
return await self.frontmostApplicationName()
}
@ -337,14 +426,14 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
}
if let processId = snapshot.applicationProcessId {
return await self.applicationName(processIdentifier: processId) ?? "PID \(processId)"
return await applicationName(processIdentifier: processId) ?? "PID \(processId)"
}
return await self.frontmostApplicationName()
}
private func applicationName(processIdentifier: Int32) async -> String? {
guard let output = try? await self.services.applications.listApplications() else {
guard let output = try? await services.applications.listApplications() else {
return nil
}
return output.data.applications.first { $0.processIdentifier == processIdentifier }?.name
@ -354,8 +443,8 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
output(result) {
print("✅ Click successful")
print("🎯 App: \(result.targetApp)")
if self.focusOptions.focusBackground {
print("🎯 Mode: background")
if let deliveryMode = result.deliveryMode {
print("🎯 Mode: \(deliveryMode)")
}
if let coordinateSpace = result.coordinateSpace {
print("🎯 Coordinate space: \(coordinateSpace)")
@ -389,10 +478,37 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
query: query,
target: self.target,
services: self.services,
logger: self.logger
logger: self.logger,
beforeRefresh: { startedAt in
self.resolvedRuntime.beginInteractionMutation(at: startedAt)
}
)
}
private func resolveExplicitWindowSelection(
observation: InteractionObservationContext
) async throws -> InteractionWindowResolution? {
guard self.target.windowId != nil || self.target.windowTitle != nil || self.target.windowIndex != nil else {
return nil
}
let resolution = try await InteractionCoordinateResolver.resolveTargetWindow(
target: self.target,
services: self.services
)
guard self.usesBackgroundDelivery else {
return resolution
}
let snapshotId = try observation.requireSnapshot()
let detectionResult = try await observation.requireDetectionResult(using: self.services.snapshots)
try InteractionWindowSelectionValidator.validate(
resolution: resolution,
snapshotContext: detectionResult.metadata.windowContext,
snapshotId: snapshotId
)
return resolution
}
private func cachedElementById(
_ elementId: String,
observation: InteractionObservationContext
@ -456,21 +572,31 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
return score
}
private func performClick(_ target: ClickTarget, clickType: ClickType, snapshotId: String) async throws {
private func performClick(
_ target: ClickTarget,
clickType: ClickType,
snapshotId: String,
coordinateResolution: InteractionCoordinateResolution?,
explicitWindowResolution: InteractionWindowResolution?,
backgroundProcessIdentifier: pid_t?
) async throws {
let effectiveSnapshotId: String? = if case .coordinates = target {
nil
} else {
snapshotId.isEmpty ? nil : snapshotId
}
if self.focusOptions.focusBackground {
let pid = try await self.resolveBackgroundClickProcessIdentifier(snapshotId: effectiveSnapshotId)
if self.usesBackgroundDelivery {
guard let backgroundProcessIdentifier else {
preconditionFailure("Background process identifier must be resolved before click delivery")
}
try await AutomationServiceBridge.click(
automation: self.services.automation,
target: target,
clickType: clickType,
snapshotId: effectiveSnapshotId,
targetProcessIdentifier: pid
targetProcessIdentifier: backgroundProcessIdentifier,
targetWindowID: explicitWindowResolution?.windowInfo.windowID ?? coordinateResolution?.targetWindowID
)
} else {
try await AutomationServiceBridge.click(
@ -486,7 +612,7 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
snapshotId: String?,
coordinateResolution: InteractionCoordinateResolution? = nil
) async throws {
if self.focusOptions.focusBackground {
if self.usesBackgroundDelivery {
try self.validateBackgroundClickOptions()
return
}
@ -523,40 +649,68 @@ struct ClickCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
}
private func validateBackgroundClickOptions() throws {
if self.focusOptions.focusTimeoutSeconds != nil ||
self.focusOptions.focusRetryCount != nil ||
self.focusOptions.spaceSwitch ||
self.focusOptions.bringToCurrentSpace {
if self.foreground, self.focusOptions.backgroundDeliveryExplicitlyRequested {
throw ValidationError("--foreground cannot be combined with --focus-background")
}
if self.focusOptions.backgroundDeliveryExplicitlyRequested &&
self.focusOptions.hasForegroundFocusOverrides {
throw ValidationError("--focus-background cannot be combined with focus options")
}
}
private func resolveBackgroundClickProcessIdentifier(snapshotId: String?) async throws -> pid_t {
private func resolveBackgroundClickProcessIdentifier(
snapshotId: String?,
coordinateResolution: InteractionCoordinateResolution?,
explicitWindowResolution: InteractionWindowResolution?
) async throws -> pid_t {
if self.target.pid != nil, self.target.app != nil {
throw ValidationError("--focus-background accepts one process target: use --app or --pid")
throw ValidationError("Background click accepts one process target: use --app or --pid")
}
if let pid = self.target.pid {
if let processId = explicitWindowResolution?.targetProcessIdentifier {
return pid_t(processId)
}
if let pid = target.pid {
guard pid > 0 else {
throw ValidationError("--pid must be greater than 0")
}
return pid_t(pid)
}
if let appIdentifier = self.target.app?.trimmingCharacters(in: .whitespacesAndNewlines),
if let appIdentifier = target.app?.trimmingCharacters(in: .whitespacesAndNewlines),
!appIdentifier.isEmpty {
let app = try await self.services.applications.findApplication(identifier: appIdentifier)
let app = try await services.applications.findApplication(identifier: appIdentifier)
return pid_t(app.processIdentifier)
}
if let processId = coordinateResolution?.targetProcessIdentifier {
return pid_t(processId)
}
if let snapshotId,
let snapshot = try? await self.services.snapshots.getUIAutomationSnapshot(snapshotId: snapshotId),
let snapshot = try? await services.snapshots.getUIAutomationSnapshot(snapshotId: snapshotId),
let processId = snapshot.applicationProcessId {
return pid_t(processId)
}
throw ValidationError("--focus-background requires --app, --pid, or a snapshot with process metadata")
if let snapshotId,
let detectionResult = try? await services.snapshots.getDetectionResult(snapshotId: snapshotId),
let processId = detectionResult.metadata.windowContext?.applicationProcessId {
return pid_t(processId)
}
throw ValidationError(
"Background click requires --app, --pid, --window-id, or a snapshot with process metadata; " +
"use --foreground for foreground screen clicks"
)
}
// Error handling is provided by ErrorHandlingCommand protocol
}
private enum ClickDeliveryMode: String {
case background
case foreground
}

View File

@ -31,7 +31,7 @@ extension DragCommand: CommanderSignatureProviding {
),
.commandOption(
"snapshot",
help: "Snapshot ID for element resolution",
help: "Snapshot ID for element resolution, or 'latest'",
long: "snapshot"
),
.commandOption(

View File

@ -25,7 +25,7 @@ struct DragCommand: ErrorHandlingCommand, OutputFormattable {
@Option(help: "Target application (e.g., 'Trash', 'Finder')")
var toApp: String?
@Option(help: "Snapshot ID for element resolution")
@Option(help: "Snapshot ID for element resolution, or 'latest'")
var snapshot: String?
@Option(help: "Duration of drag in milliseconds (default: 500)")
@ -80,12 +80,16 @@ struct DragCommand: ErrorHandlingCommand, OutputFormattable {
fallbackToLatest: needsSnapshot,
snapshots: self.services.snapshots
)
let refreshRuntime = self.resolvedRuntime
observation = try await InteractionObservationRefresher.refreshForMissingElementsIfNeeded(
observation,
elementIds: [self.from, self.to],
target: self.target,
services: self.services,
logger: self.logger
logger: self.logger,
beforeRefresh: { startedAt in
refreshRuntime.beginInteractionMutation(at: startedAt)
}
)
if needsSnapshot {
_ = try await observation.requireDetectionResult(using: self.services.snapshots)
@ -93,6 +97,7 @@ struct DragCommand: ErrorHandlingCommand, OutputFormattable {
try await observation.validateIfExplicit(using: self.services.snapshots)
}
self.resolvedRuntime.beginInteractionMutation()
try await ensureFocused(
snapshotId: observation.focusSnapshotId(for: self.target),
target: self.target,
@ -157,13 +162,13 @@ struct DragCommand: ErrorHandlingCommand, OutputFormattable {
+ "profile=\(movement.profileName)"
)
try await Task.sleep(nanoseconds: 100_000_000)
try? await Task.sleep(nanoseconds: 100_000_000)
await InteractionObservationInvalidator.invalidateAfterMutation(
observation,
snapshots: self.services.snapshots,
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "drag"
)
try Task.checkCancellation()
let result = DragResult(
success: true,
@ -254,11 +259,11 @@ extension DragCommand: ParsableCommand {
Execute click-and-drag operations for moving elements, selecting text, or dragging files.
EXAMPLES:
peekaboo drag --from B1 --to T2
peekaboo drag --from "$SOURCE_ID" --to "$TARGET_ID"
peekaboo drag --from-coords "100,200" --to-coords "400,300"
peekaboo drag --from B1 --to-app Trash
peekaboo drag --from S1 --to-coords "500,250" --duration 2000
peekaboo drag --from T1 --to T5 --modifiers shift
peekaboo drag --from "$SOURCE_ID" --to-app Trash
peekaboo drag --from "$SOURCE_ID" --to-coords "500,250" --duration 2000
peekaboo drag --from "$SOURCE_ID" --to "$TARGET_ID" --modifiers shift
""",
version: "2.0.0",
showHelpOnEmptyInvocation: true

View File

@ -23,10 +23,17 @@ extension HotkeyCommand: CommanderSignatureProviding {
),
.commandOption(
"snapshot",
help: "Snapshot ID (uses latest if not specified)",
help: "Snapshot ID, or 'latest' (uses latest if not specified)",
long: "snapshot"
),
],
flags: [
.commandFlag(
"foreground",
help: "Focus target and send a foreground/global hotkey",
long: "foreground"
),
],
optionGroups: [
InteractionTargetOptions.commanderSignature(),
FocusCommandOptions.commanderSignature(includeBackgroundDelivery: true),

View File

@ -18,12 +18,15 @@ struct HotkeyCommand: ErrorHandlingCommand, OutputFormattable {
@Option(help: "Delay between key press and release in milliseconds")
var holdDuration: Int = 50
@Option(help: "Snapshot ID (uses latest if not specified)")
@Option(help: "Snapshot ID, or 'latest' (uses latest if not specified)")
var snapshot: String?
@Flag(name: .customLong("focus-background"), help: "Send the hotkey to the target process without focusing it")
var focusBackground = false
@Flag(help: "Focus target and send a foreground/global hotkey")
var foreground = false
@OptionGroup var focusOptions: FocusCommandOptions
@RuntimeStorage private var runtime: CommandRuntime?
@ -87,24 +90,25 @@ struct HotkeyCommand: ErrorHandlingCommand, OutputFormattable {
fallbackToLatest: false,
snapshots: self.services.snapshots
)
try await observation.validateIfExplicit(using: self.services.snapshots)
let deliveryMode: String
let targetPID: pid_t?
if self.focusOptions.focusBackground {
let backgroundPID = try await self.backgroundProcessIdentifier(snapshotId: observation.snapshotId)
self.resolvedRuntime.beginInteractionMutation()
if let backgroundPID {
try self.validateBackgroundHotkeyOptions(snapshotId: observation.snapshotId)
let resolvedPID = try await self.resolveBackgroundHotkeyProcessIdentifier()
try await AutomationServiceBridge.hotkey(
automation: self.services.automation,
keys: keysCsv,
holdDuration: self.holdDuration,
targetProcessIdentifier: resolvedPID
targetProcessIdentifier: backgroundPID
)
deliveryMode = "background"
targetPID = resolvedPID
targetPID = backgroundPID
} else {
try await observation.validateIfExplicit(using: self.services.snapshots)
try await ensureFocused(
snapshotId: observation.focusSnapshotId(for: self.target),
target: self.target,
@ -121,9 +125,8 @@ struct HotkeyCommand: ErrorHandlingCommand, OutputFormattable {
targetPID = nil
}
await InteractionObservationInvalidator.invalidateAfterMutationOrLatest(
observation,
snapshots: self.services.snapshots,
await InteractionObservationInvalidator.invalidateAfterMutation(
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "hotkey"
)
@ -139,7 +142,7 @@ struct HotkeyCommand: ErrorHandlingCommand, OutputFormattable {
)
output(result) {
if self.focusOptions.focusBackground {
if targetPID != nil {
print("✅ Hotkey sent")
} else {
print("✅ Hotkey pressed")
@ -158,17 +161,18 @@ struct HotkeyCommand: ErrorHandlingCommand, OutputFormattable {
}
private func validateBackgroundHotkeyOptions(snapshotId: String?) throws {
if snapshotId != nil {
throw ValidationError("--focus-background cannot be combined with --snapshot")
if self.foreground, self.focusOptions.backgroundDeliveryExplicitlyRequested {
throw ValidationError("--foreground cannot be combined with --focus-background")
}
if self.focusOptions.noAutoFocus ||
self.focusOptions.focusTimeoutSeconds != nil ||
self.focusOptions.focusRetryCount != nil ||
self.focusOptions.spaceSwitch ||
self.focusOptions.bringToCurrentSpace {
throw ValidationError("--focus-background cannot be combined with focus options")
if snapshotId != nil {
return
}
try KeyboardDeliverySupport.validateForegroundFlags(
foreground: self.foreground,
focusOptions: self.focusOptions
)
}
private static func parseKeyNames(_ keysString: String) -> [String] {
@ -178,30 +182,26 @@ struct HotkeyCommand: ErrorHandlingCommand, OutputFormattable {
.filter { !$0.isEmpty }
}
private func resolveBackgroundHotkeyProcessIdentifier() async throws -> pid_t {
if self.target.windowId != nil || self.target.windowTitle != nil || self.target.windowIndex != nil {
throw ValidationError("--focus-background supports --app or --pid")
private func backgroundProcessIdentifier(snapshotId: String?) async throws -> pid_t? {
guard self.focusOptions.focusBackground ||
!KeyboardDeliverySupport.shouldUseForeground(foreground: self.foreground, focusOptions: self.focusOptions)
else {
return nil
}
if self.target.app != nil, self.target.pid != nil {
throw ValidationError("--focus-background accepts one target: use --app or --pid")
throw ValidationError("Background hotkey accepts one process target: use --app or --pid")
}
if let pid = self.target.pid {
guard pid > 0 else {
throw ValidationError("--pid must be greater than 0")
}
return pid_t(pid)
}
guard let appIdentifier = self.target.app?.trimmingCharacters(in: .whitespacesAndNewlines),
!appIdentifier.isEmpty
else {
let pid = try await KeyboardDeliverySupport.backgroundProcessIdentifier(
target: self.target,
snapshotId: snapshotId,
services: self.services
)
if self.focusOptions.focusBackground, pid == nil {
throw ValidationError("--focus-background requires --app or --pid")
}
let app = try await self.services.applications.findApplication(identifier: appIdentifier)
return pid_t(app.processIdentifier)
return pid
}
// Error handling is provided by ErrorHandlingCommand protocol
@ -242,7 +242,8 @@ extension HotkeyCommand: ParsableCommand {
peekaboo hotkey --keys "cmd a" # Select all
peekaboo hotkey --keys "cmd,shift,t" # Reopen closed tab
peekaboo hotkey --keys "cmd space" # Spotlight
peekaboo hotkey "cmd,l" --app Safari --focus-background
peekaboo hotkey "cmd,l" --app Safari
peekaboo hotkey "cmd,l" --app Safari --foreground
KEY NAMES:
Modifiers: cmd, shift, alt/option, ctrl, fn
@ -251,8 +252,9 @@ extension HotkeyCommand: ParsableCommand {
Special: space, return, tab, escape, delete, arrow_up, arrow_down, arrow_left, arrow_right
Function: f1-f12
Background hotkeys accept one non-modifier key plus optional modifiers.
Use --focus-background with --app or --pid to target a process without focusing it.
Background hotkeys are used by default when --app, --pid, --window-id,
or a snapshot with process metadata is available. Use --foreground
when the target must receive a foreground/global hotkey.
""",
showHelpOnEmptyInvocation: true
@ -276,6 +278,7 @@ extension HotkeyCommand: CommanderBindableCommand {
}
self.target = try values.makeInteractionTargetOptions()
self.snapshot = values.singleOption("snapshot")
self.foreground = values.flag("foreground")
self.focusOptions = try values.makeFocusOptions(includeBackgroundDelivery: true)
self.focusBackground = self.focusOptions.focusBackground
}

View File

@ -0,0 +1,139 @@
import Commander
import CoreGraphics
import Foundation
import PeekabooCore
import PeekabooFoundation
enum KeyboardDeliveryMode: String {
case background
case foreground
}
enum KeyboardDeliverySupport {
static func backgroundProcessIdentifier(
target: InteractionTargetOptions,
snapshotId: String?,
services: any PeekabooServiceProviding
) async throws -> pid_t? {
try await self.validateWindowSelectionIfNeeded(target: target, services: services)
if let windowId = target.windowId {
return self.processIdentifierForWindow(windowId: CGWindowID(windowId))
}
if let pid = target.pid {
guard pid > 0 else {
throw ValidationError("--pid must be greater than 0")
}
return pid_t(pid)
}
if let appIdentifier = target.app?.trimmingCharacters(in: .whitespacesAndNewlines),
!appIdentifier.isEmpty {
let app = try await services.applications.findApplication(identifier: appIdentifier)
return pid_t(app.processIdentifier)
}
if let snapshotId,
let snapshot = try? await services.snapshots.getUIAutomationSnapshot(snapshotId: snapshotId),
let processId = snapshot.applicationProcessId {
return pid_t(processId)
}
if let snapshotId,
let detectionResult = try? await services.snapshots.getDetectionResult(snapshotId: snapshotId),
let processId = detectionResult.metadata.windowContext?.applicationProcessId {
return pid_t(processId)
}
return nil
}
private static func validateWindowSelectionIfNeeded(
target: InteractionTargetOptions,
services: any PeekabooServiceProviding
) async throws {
guard target.windowTitle != nil || target.windowIndex != nil || target.windowId != nil else {
return
}
guard let windowTarget = try target.toWindowTarget() else {
return
}
let windows = try await services.windows.listWindows(target: windowTarget)
if windows.isEmpty {
throw PeekabooError.windowNotFound(criteria: self.windowCriteriaDescription(target: target))
}
}
static func validateForegroundFlags(
foreground: Bool,
focusOptions: FocusCommandOptions,
backgroundFlagName: String? = nil
) throws {
if foreground, focusOptions.backgroundDeliveryExplicitlyRequested {
throw ValidationError("--foreground cannot be combined with \(backgroundFlagName ?? "--focus-background")")
}
if focusOptions.backgroundDeliveryExplicitlyRequested, focusOptions.hasForegroundFocusOverrides {
throw ValidationError("\(backgroundFlagName ?? "--focus-background") cannot be combined with focus options")
}
}
static func shouldUseForeground(foreground: Bool, focusOptions: FocusCommandOptions) -> Bool {
foreground || focusOptions.hasForegroundFocusOverrides
}
private static func processIdentifierForWindow(windowId: CGWindowID) -> pid_t? {
guard let windows = CGWindowListCopyWindowInfo([.optionAll, .excludeDesktopElements], kCGNullWindowID)
as? [[String: Any]]
else {
return nil
}
return windows.first { window in
self.windowID(from: window[kCGWindowNumber as String]) == windowId
}.flatMap { window in
self.pid(from: window[kCGWindowOwnerPID as String])
}
}
private static func windowID(from value: Any?) -> CGWindowID? {
self.intValue(from: value).map(CGWindowID.init)
}
private static func pid(from value: Any?) -> pid_t? {
self.intValue(from: value).map(pid_t.init)
}
private static func intValue(from value: Any?) -> Int? {
if let number = value as? NSNumber {
return number.intValue
}
if let int = value as? Int {
return int
}
if let int32 = value as? Int32 {
return Int(int32)
}
if let uint32 = value as? UInt32 {
return Int(uint32)
}
return nil
}
private static func windowCriteriaDescription(target: InteractionTargetOptions) -> String {
if let windowTitle = target.windowTitle?.trimmingCharacters(in: .whitespacesAndNewlines),
!windowTitle.isEmpty {
return "window title '\(windowTitle)'"
}
if let windowIndex = target.windowIndex {
return "window index \(windowIndex)"
}
if let windowId = target.windowId {
return "window id \(windowId)"
}
return "target window"
}
}

View File

@ -16,7 +16,7 @@ extension MoveCommand: ParsableCommand {
EXAMPLES:
peekaboo move 100,200 # Move to coordinates
peekaboo move --to "Submit Button" # Move to element by text
peekaboo move --on B3 # Move to element by ID
peekaboo move --on "$ELEMENT_ID" # ID copied from current output
peekaboo move 500,300 --smooth # Smooth movement
peekaboo move --center # Move to screen center
@ -84,7 +84,7 @@ extension MoveCommand: CommanderSignatureProviding {
),
.commandOption(
"on",
help: "Element ID to move to (e.g., B1, T2)",
help: "Opaque element ID copied from current see or inspect-ui output",
long: "on"
),
.commandOption(
@ -109,7 +109,7 @@ extension MoveCommand: CommanderSignatureProviding {
),
.commandOption(
"snapshot",
help: "Snapshot ID for element resolution",
help: "Snapshot ID for element resolution, or 'latest'",
long: "snapshot"
),
],

View File

@ -17,7 +17,7 @@ struct MoveCommand: ErrorHandlingCommand, OutputFormattable {
@Option(help: "Move to element by text/label")
var to: String?
@Option(help: "Element ID to move to (e.g., B1, T2)")
@Option(help: "Opaque element ID copied from current see or inspect-ui output")
var on: String?
@Option(name: .customLong("id"), help: "Element ID to move to (alias for --on)")
@ -41,7 +41,7 @@ struct MoveCommand: ErrorHandlingCommand, OutputFormattable {
@Option(help: "Movement profile: linear (default) or human.")
var profile: String?
@Option(help: "Snapshot ID for element resolution")
@Option(help: "Snapshot ID for element resolution, or 'latest'")
var snapshot: String?
@RuntimeStorage private var runtime: CommandRuntime?
@ -130,6 +130,7 @@ struct MoveCommand: ErrorHandlingCommand, OutputFormattable {
)
// Perform the movement
self.resolvedRuntime.beginInteractionMutation()
try await AutomationServiceBridge.moveMouse(
automation: self.services.automation,
to: targetLocation,
@ -220,6 +221,7 @@ struct MoveCommand: ErrorHandlingCommand, OutputFormattable {
}
private func focusForCoordinateTarget() async throws {
self.resolvedRuntime.beginInteractionMutation()
try await ensureFocused(
snapshotId: nil,
target: self.target,
@ -239,8 +241,12 @@ struct MoveCommand: ErrorHandlingCommand, OutputFormattable {
elementIds: [elementId],
target: self.target,
services: self.services,
logger: self.logger
logger: self.logger,
beforeRefresh: { startedAt in
self.resolvedRuntime.beginInteractionMutation(at: startedAt)
}
)
self.resolvedRuntime.beginInteractionMutation()
try await ensureFocused(
snapshotId: observation.focusSnapshotId(for: self.target),
target: self.target,
@ -277,9 +283,13 @@ struct MoveCommand: ErrorHandlingCommand, OutputFormattable {
query: query,
target: self.target,
services: self.services,
logger: self.logger
logger: self.logger,
beforeRefresh: { startedAt in
self.resolvedRuntime.beginInteractionMutation(at: startedAt)
}
)
let activeSnapshotId = try observation.requireSnapshot()
self.resolvedRuntime.beginInteractionMutation()
try await ensureFocused(
snapshotId: observation.focusSnapshotId(for: self.target),
target: self.target,

View File

@ -4,6 +4,13 @@ import Commander
extension PasteCommand: CommanderSignatureProviding {
static func commanderSignature() -> CommandSignature {
CommandSignature(
arguments: [
.make(
label: "text",
help: "Text to paste",
isOptional: true
),
],
options: [
.commandOption("textOption", help: "Text to paste (alternative to positional argument)", long: "text"),
.commandOption("filePath", help: "Path to file to paste", long: "file-path"),
@ -23,6 +30,11 @@ extension PasteCommand: CommanderSignatureProviding {
],
flags: [
.commandFlag("allowLarge", help: "Allow payloads larger than 10 MB", long: "allow-large"),
.commandFlag(
"foreground",
help: "Focus target and send foreground/global Cmd+V",
long: "foreground"
),
],
optionGroups: [
InteractionTargetOptions.commanderSignature(),
@ -47,6 +59,7 @@ extension PasteCommand: CommanderBindableCommand {
self.restoreDelayMs = delay
}
self.allowLarge = values.flag("allowLarge")
self.foreground = values.flag("foreground")
self.target = try values.makeInteractionTargetOptions()
self.focusOptions = try values.makeFocusOptions()

View File

@ -4,7 +4,7 @@ import PeekabooCore
import PeekabooFoundation
import UniformTypeIdentifiers
/// Sets clipboard content, pastes (Cmd+V), then restores the prior clipboard.
/// Pastes text through background typing when targeted, otherwise uses clipboard + Cmd+V.
@available(macOS 14.0, *)
@MainActor
struct PasteCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfigurable {
@ -37,6 +37,8 @@ struct PasteCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
@OptionGroup var target: InteractionTargetOptions
@OptionGroup var focusOptions: FocusCommandOptions
@Flag(help: "Focus target and send foreground/global Cmd+V")
var foreground = false
@RuntimeStorage private var runtime: CommandRuntime?
var runtimeOptions = CommandRuntimeOptions()
@ -78,14 +80,62 @@ struct PasteCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
do {
try self.target.validate()
try KeyboardDeliverySupport.validateForegroundFlags(
foreground: self.foreground,
focusOptions: self.focusOptions
)
let request = try self.makeWriteRequest()
try await ensureFocused(
snapshotId: nil,
target: self.target,
options: self.focusOptions,
services: self.services
)
let targetPID = try await self.backgroundProcessIdentifier()
if let targetPID,
let text = self.resolvedText {
let setResult = try Self.readResult(for: request)
self.resolvedRuntime.beginInteractionMutation()
_ = try await AutomationServiceBridge.typeActions(
automation: self.services.automation,
request: TypeActionsRequest(
actions: [.text(text)],
cadence: .fixed(milliseconds: 0),
snapshotId: nil
),
targetProcessIdentifier: targetPID
)
await InteractionObservationInvalidator.invalidateAfterMutation(
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "paste"
)
let result = PasteResult(
success: true,
pastedUti: setResult.utiIdentifier,
pastedSize: setResult.data.count,
pastedTextPreview: setResult.textPreview,
previousClipboardPresent: false,
restoredUti: nil,
restoredSize: nil,
restoreDelayMs: 0,
deliveryMode: KeyboardDeliveryMode.background.rawValue,
targetPID: Int(targetPID)
)
self.output(result) {
print("✅ Pasted text")
print("📋 Pasted: \(setResult.utiIdentifier) (\(setResult.data.count) bytes)")
print("🎯 Mode: background to PID \(targetPID)")
}
return
}
self.resolvedRuntime.beginInteractionMutation()
if targetPID == nil {
try await ensureFocused(
snapshotId: nil,
target: self.target,
options: self.focusOptions,
services: self.services
)
}
let priorClipboard = try? self.services.clipboard.get(prefer: nil)
let restoreSlot = "paste-\(UUID().uuidString)"
@ -108,13 +158,22 @@ struct PasteCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
let setResult = try self.services.clipboard.set(request)
try await AutomationServiceBridge.hotkey(
automation: self.services.automation,
keys: "cmd,v",
holdDuration: 50
)
await InteractionObservationInvalidator.invalidateLatestSnapshot(
using: self.services.snapshots,
if let targetPID {
try await AutomationServiceBridge.hotkey(
automation: self.services.automation,
keys: "cmd,v",
holdDuration: 50,
targetProcessIdentifier: targetPID
)
} else {
try await AutomationServiceBridge.hotkey(
automation: self.services.automation,
keys: "cmd,v",
holdDuration: 50
)
}
await InteractionObservationInvalidator.invalidateAfterMutation(
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "paste"
)
@ -127,17 +186,23 @@ struct PasteCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
previousClipboardPresent: priorClipboard != nil,
restoredUti: restoreResult?.utiIdentifier,
restoredSize: restoreResult?.data.count,
restoreDelayMs: self.restoreDelayMs
restoreDelayMs: self.restoreDelayMs,
deliveryMode: targetPID == nil ? KeyboardDeliveryMode.foreground.rawValue :
KeyboardDeliveryMode.background.rawValue,
targetPID: targetPID.map(Int.init)
)
self.output(result) {
print("✅ Pasted (Cmd+V) and restored clipboard")
print("✅ Pasted and restored clipboard")
print("📋 Pasted: \(setResult.utiIdentifier) (\(setResult.data.count) bytes)")
if priorClipboard != nil {
print("♻️ Restored: \(restoreResult?.utiIdentifier ?? "unknown")")
} else {
print("🧹 Restored: cleared (prior clipboard empty)")
}
if let targetPID {
print("🎯 Mode: background to PID \(targetPID)")
}
}
} catch {
self.handleError(error)
@ -181,6 +246,51 @@ struct PasteCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
throw ValidationError("Provide text, --file-path/--image-path, or --data-base64 with --uti")
}
private static func readResult(for request: ClipboardWriteRequest) throws -> ClipboardReadResult {
guard let primary = request.representations.first else {
throw ClipboardServiceError.writeFailed("No representations provided.")
}
let textPreview: String? = if let text = request.alsoText {
Self.makePreview(text)
} else if primary.utiIdentifier == UTType.plainText.identifier ||
primary.utiIdentifier == UTType.utf8PlainText.identifier,
let string = String(data: primary.data, encoding: .utf8) {
Self.makePreview(string)
} else {
nil
}
return ClipboardReadResult(
utiIdentifier: primary.utiIdentifier,
data: primary.data,
textPreview: textPreview
)
}
private static func makePreview(_ text: String) -> String {
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
let max = 80
guard trimmed.count > max else { return trimmed }
let head = trimmed.prefix(max)
return "\(head)..."
}
private func backgroundProcessIdentifier() async throws -> pid_t? {
guard !KeyboardDeliverySupport.shouldUseForeground(
foreground: self.foreground,
focusOptions: self.focusOptions
) else {
return nil
}
return try await KeyboardDeliverySupport.backgroundProcessIdentifier(
target: self.target,
snapshotId: nil,
services: self.services
)
}
}
struct PasteResult: Codable {
@ -192,6 +302,8 @@ struct PasteResult: Codable {
let restoredUti: String?
let restoredSize: Int?
let restoreDelayMs: Int
let deliveryMode: String
let targetPID: Int?
}
@MainActor
@ -204,12 +316,15 @@ extension PasteCommand: ParsableCommand {
discussion: """
This command reduces drift in automation flows by collapsing:
1) clipboard set
2) Cmd+V paste
2) paste delivery
3) clipboard restore
into one operation.
Background text delivery is used by default when a target process is known;
binary payloads use background Cmd+V. Add --foreground for focused/global paste.
EXAMPLES:
peekaboo paste \"Hello\" --app TextEdit
peekaboo paste \"Hello\" --app TextEdit --foreground
peekaboo paste --text \"Hello\" --app TextEdit --window-title \"Untitled\"
peekaboo paste --data-base64 \"$BASE64\" --uti public.rtf --also-text \"fallback\" --app TextEdit
peekaboo paste --file-path /tmp/snippet.png --app Notes

View File

@ -12,7 +12,7 @@ struct PerformActionCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOpt
@Option(help: "Accessibility action name, e.g. AXPress, AXShowMenu, AXIncrement")
var action: String?
@Option(help: "Snapshot ID (uses latest if not specified)")
@Option(help: "Snapshot ID, or 'latest' (uses latest if not specified)")
var snapshot: String?
@RuntimeStorage private var runtime: CommandRuntime?
@ -52,6 +52,7 @@ struct PerformActionCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOpt
let observation = await self.resolveObservationContext()
try await observation.validateIfExplicit(using: self.services.snapshots)
let startTime = Date()
self.resolvedRuntime.beginInteractionMutation()
let result = try await AutomationServiceBridge.performAction(
automation: self.services.automation,
target: target,
@ -59,8 +60,7 @@ struct PerformActionCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOpt
snapshotId: observation.snapshotId
)
await InteractionObservationInvalidator.invalidateAfterMutation(
observation,
snapshots: self.services.snapshots,
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "perform-action"
)
@ -116,7 +116,7 @@ extension PerformActionCommand: ParsableCommand {
Invokes an accessibility action without synthesizing a mouse or keyboard event.
EXAMPLES:
peekaboo perform-action --on B1 --action AXPress
peekaboo perform-action --on "$ELEMENT_ID" --action AXPress
peekaboo perform-action --on Stepper --action AXIncrement
""",
showHelpOnEmptyInvocation: true
@ -145,7 +145,11 @@ extension PerformActionCommand: CommanderSignatureProviding {
help: "Accessibility action name, e.g. AXPress, AXShowMenu, AXIncrement",
long: "action"
),
.commandOption("snapshot", help: "Snapshot ID (uses latest if not specified)", long: "snapshot"),
.commandOption(
"snapshot",
help: "Snapshot ID, or 'latest' (uses latest if not specified)",
long: "snapshot"
),
]
)
}

View File

@ -33,10 +33,17 @@ extension PressCommand: CommanderSignatureProviding {
),
.commandOption(
"snapshot",
help: "Snapshot ID (uses latest if not specified)",
help: "Snapshot ID, or 'latest' (uses latest if not specified)",
long: "snapshot"
),
],
flags: [
.commandFlag(
"foreground",
help: "Focus target and send foreground/global key presses",
long: "foreground"
),
],
optionGroups: [
InteractionTargetOptions.commanderSignature(),
FocusCommandOptions.commanderSignature(),

View File

@ -21,10 +21,13 @@ struct PressCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
@Option(help: "Hold duration for each key in milliseconds")
var hold: Int = 50
@Option(help: "Snapshot ID (uses latest if not specified)")
@Option(help: "Snapshot ID, or 'latest' (uses latest if not specified)")
var snapshot: String?
@OptionGroup var focusOptions: FocusCommandOptions
@Flag(help: "Focus target and send foreground/global key presses")
var foreground = false
@RuntimeStorage private var runtime: CommandRuntime?
var runtimeOptions = CommandRuntimeOptions()
@ -76,23 +79,44 @@ struct PressCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
)
try await observation.validateIfExplicit(using: self.services.snapshots)
try await ensureFocused(
snapshotId: observation.focusSnapshotId(for: self.target),
target: self.target,
options: self.focusOptions,
services: self.services
)
let targetPID = try await self.backgroundProcessIdentifier(snapshotId: observation.snapshotId)
self.resolvedRuntime.beginInteractionMutation()
if targetPID == nil {
try await ensureFocused(
snapshotId: observation.focusSnapshotId(for: self.target),
target: self.target,
options: self.focusOptions,
services: self.services
)
}
let normalizedKeys = self.keys.map { $0.lowercased() }
var completedPresses = 0
for repetition in 0..<self.count {
for (index, key) in normalizedKeys.indexed() {
try await AutomationServiceBridge.hotkey(
automation: self.services.automation,
keys: key,
holdDuration: self.hold
)
if let targetPID {
guard let specialKey = SpecialKey(rawValue: key) else {
throw ValidationError(
"Unknown key: '\(key)'. Run 'peekaboo press --help' for available keys."
)
}
_ = try await AutomationServiceBridge.typeActions(
automation: self.services.automation,
request: TypeActionsRequest(
actions: [.key(specialKey)],
cadence: .fixed(milliseconds: 0),
snapshotId: observation.snapshotId
),
targetProcessIdentifier: targetPID
)
} else {
try await AutomationServiceBridge.hotkey(
automation: self.services.automation,
keys: key,
holdDuration: self.hold
)
}
completedPresses += 1
let isLastKey = index == normalizedKeys.count - 1
@ -103,9 +127,8 @@ struct PressCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
}
}
await InteractionObservationInvalidator.invalidateAfterMutationOrLatest(
observation,
snapshots: self.services.snapshots,
await InteractionObservationInvalidator.invalidateAfterMutation(
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "press"
)
@ -116,6 +139,9 @@ struct PressCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
keys: keys,
totalPresses: completedPresses,
count: self.count,
deliveryMode: targetPID == nil ? KeyboardDeliveryMode.foreground.rawValue :
KeyboardDeliveryMode.background.rawValue,
targetPID: targetPID.map(Int.init),
executionTime: Date().timeIntervalSince(startTime)
)
@ -125,6 +151,9 @@ struct PressCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
if self.count > 1 {
print("🔢 Repeated: \(self.count) times")
}
if let targetPID {
print("🎯 Mode: background to PID \(targetPID)")
}
print("📊 Total presses: \(completedPresses)")
print("⏱️ Completed in \(String(format: "%.2f", Date().timeIntervalSince(startTime)))s")
}
@ -139,6 +168,10 @@ struct PressCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
mutating func validate() throws {
try self.target.validate()
try KeyboardDeliverySupport.validateForegroundFlags(
foreground: self.foreground,
focusOptions: self.focusOptions
)
guard self.count >= 1 else {
throw ValidationError("--count must be greater than 0")
}
@ -154,6 +187,21 @@ struct PressCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
}
}
}
private func backgroundProcessIdentifier(snapshotId: String?) async throws -> pid_t? {
guard !KeyboardDeliverySupport.shouldUseForeground(
foreground: self.foreground,
focusOptions: self.focusOptions
) else {
return nil
}
return try await KeyboardDeliverySupport.backgroundProcessIdentifier(
target: self.target,
snapshotId: snapshotId,
services: self.services
)
}
}
// MARK: - JSON Output Structure
@ -163,6 +211,8 @@ struct PressResult: Codable {
let keys: [String]
let totalPresses: Int
let count: Int
let deliveryMode: String
let targetPID: Int?
let executionTime: TimeInterval
}
@ -181,6 +231,7 @@ extension PressCommand: ParsableCommand {
EXAMPLES:
peekaboo press return # Press Enter/Return
peekaboo press return --app TextEdit # Background-target TextEdit
peekaboo press tab --count 3 # Press Tab 3 times
peekaboo press escape # Press Escape
peekaboo press delete # Press Backspace/Delete
@ -238,6 +289,7 @@ extension PressCommand: CommanderBindableCommand {
self.hold = hold
}
self.snapshot = values.singleOption("snapshot")
self.foreground = values.flag("foreground")
self.focusOptions = try values.makeFocusOptions()
}
}

View File

@ -21,7 +21,7 @@ extension ScrollCommand: CommanderSignatureProviding {
),
.commandOption(
"snapshot",
help: "Snapshot ID (uses latest if not specified)",
help: "Snapshot ID, or 'latest' (uses latest if not specified)",
long: "snapshot"
),
.commandOption(

View File

@ -18,7 +18,7 @@ struct ScrollCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsCon
@Option(help: "Element ID to scroll on (from 'see' command)")
var on: String?
@Option(help: "Snapshot ID (uses latest if not specified)")
@Option(help: "Snapshot ID, or 'latest' (uses latest if not specified)")
var snapshot: String?
@Option(help: "Delay between scroll ticks in milliseconds")
@ -76,12 +76,16 @@ struct ScrollCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsCon
)
if let elementId = self.on {
let refreshRuntime = self.resolvedRuntime
observation = try await InteractionObservationRefresher.refreshForMissingElementsIfNeeded(
observation,
elementIds: [elementId],
target: self.target,
services: self.services,
logger: self.logger
logger: self.logger,
beforeRefresh: { startedAt in
refreshRuntime.beginInteractionMutation(at: startedAt)
}
)
_ = try await observation.requireDetectionResult(using: self.services.snapshots)
} else {
@ -89,6 +93,7 @@ struct ScrollCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsCon
}
// Ensure window is focused before scrolling
self.resolvedRuntime.beginInteractionMutation()
try await ensureFocused(
snapshotId: observation.focusSnapshotId(for: self.target),
target: self.target,
@ -109,6 +114,11 @@ struct ScrollCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsCon
automation: self.services.automation,
request: scrollRequest
)
await InteractionObservationInvalidator.invalidateAfterMutation(
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "scroll"
)
AutomationEventLogger.log(
.scroll,
"direction=\(self.direction) amount=\(self.amount) smooth=\(self.smooth) "
@ -140,13 +150,6 @@ struct ScrollCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsCon
}
let scrollLocation = scrollResolution.point
await InteractionObservationInvalidator.invalidateAfterMutation(
observation,
snapshots: self.services.snapshots,
logger: self.logger,
reason: "scroll"
)
// Output results
let outputPayload = ScrollResult(
success: true,

View File

@ -12,7 +12,7 @@ struct SetValueCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsC
@Option(help: "Element ID or query to set")
var on: String?
@Option(help: "Snapshot ID (uses latest if not specified)")
@Option(help: "Snapshot ID, or 'latest' (uses latest if not specified)")
var snapshot: String?
@RuntimeStorage private var runtime: CommandRuntime?
@ -52,6 +52,7 @@ struct SetValueCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsC
let observation = await self.resolveObservationContext()
try await observation.validateIfExplicit(using: self.services.snapshots)
let startTime = Date()
self.resolvedRuntime.beginInteractionMutation()
let result = try await AutomationServiceBridge.setValue(
automation: self.services.automation,
target: target,
@ -59,8 +60,7 @@ struct SetValueCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsC
snapshotId: observation.snapshotId
)
await InteractionObservationInvalidator.invalidateAfterMutation(
observation,
snapshots: self.services.snapshots,
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "set-value"
)
@ -119,7 +119,7 @@ extension SetValueCommand: ParsableCommand {
Sets a settable accessibility value without synthesizing keystrokes.
EXAMPLES:
peekaboo set-value "hello" --on T1
peekaboo set-value "hello" --on "$ELEMENT_ID"
peekaboo set-value "42" --on "Search"
""",
showHelpOnEmptyInvocation: true
@ -147,7 +147,11 @@ extension SetValueCommand: CommanderSignatureProviding {
options: [
.commandOption("value", help: "Value to set (alternative to positional argument)", long: "value"),
.commandOption("on", help: "Element ID or query to set", long: "on"),
.commandOption("snapshot", help: "Snapshot ID (uses latest if not specified)", long: "snapshot"),
.commandOption(
"snapshot",
help: "Snapshot ID, or 'latest' (uses latest if not specified)",
long: "snapshot"
),
]
)
}

View File

@ -26,7 +26,7 @@ extension SwipeCommand: CommanderSignatureProviding {
),
.commandOption(
"snapshot",
help: "Snapshot ID (uses latest if not specified)",
help: "Snapshot ID, or 'latest' (uses latest if not specified)",
long: "snapshot"
),
.commandOption(

View File

@ -20,7 +20,7 @@ struct SwipeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
@Option(help: "Destination coordinates (x,y)")
var toCoords: String?
@Option(help: "Snapshot ID (uses latest if not specified)")
@Option(help: "Snapshot ID, or 'latest' (uses latest if not specified)")
var snapshot: String?
@Option(help: "Duration of the swipe in milliseconds")
@ -98,12 +98,16 @@ struct SwipeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
fallbackToLatest: needsSnapshotForElements,
snapshots: self.services.snapshots
)
let refreshRuntime = self.resolvedRuntime
observation = try await InteractionObservationRefresher.refreshForMissingElementsIfNeeded(
observation,
elementIds: [self.from, self.to],
target: self.target,
services: self.services,
logger: self.logger
logger: self.logger,
beforeRefresh: { startedAt in
refreshRuntime.beginInteractionMutation(at: startedAt)
}
)
if needsSnapshotForElements {
@ -112,6 +116,7 @@ struct SwipeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
try await observation.validateIfExplicit(using: self.services.snapshots)
}
self.resolvedRuntime.beginInteractionMutation()
try await ensureFocused(
snapshotId: observation.focusSnapshotId(for: self.target),
target: self.target,
@ -173,13 +178,13 @@ struct SwipeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConf
)
// Small delay to ensure swipe is processed
try await Task.sleep(nanoseconds: 100_000_000) // 0.1 seconds
try? await Task.sleep(nanoseconds: 100_000_000) // 0.1 seconds
await InteractionObservationInvalidator.invalidateAfterMutation(
observation,
snapshots: self.services.snapshots,
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "swipe"
)
try Task.checkCancellation()
let outputPayload = SwipeResult(
success: true,
@ -244,20 +249,20 @@ extension SwipeCommand: ParsableCommand {
EXAMPLES:
# Swipe between UI elements
peekaboo swipe --from B1 --to B5 --snapshot 12345
peekaboo swipe --from "$SOURCE_ID" --to "$TARGET_ID" --snapshot "$SNAPSHOT_ID"
# Swipe with coordinates
peekaboo swipe --from-coords 100,200 --to-coords 300,400
# Mixed mode: element to coordinates
peekaboo swipe --from T1 --to-coords 500,300 --duration 1000
peekaboo swipe --from "$SOURCE_ID" --to-coords 500,300 --duration 1000
# Slow swipe for precise gesture
peekaboo swipe --from-coords 50,50 --to-coords 400,400 --duration 2000
USAGE:
You can specify source and destination using either:
- Element IDs from a previous 'see' command
- Opaque element IDs copied from current 'see' or 'inspect-ui' output
- Direct coordinates
- A mix of both

View File

@ -18,7 +18,7 @@ extension TypeCommand: CommanderSignatureProviding {
),
.commandOption(
"snapshot",
help: "Snapshot ID (uses latest if not specified)",
help: "Snapshot ID, or 'latest' (uses latest if not specified)",
long: "snapshot"
),
.commandOption(
@ -28,7 +28,7 @@ extension TypeCommand: CommanderSignatureProviding {
),
.commandOption(
"profile",
help: "Typing profile: human (default) or linear",
help: "Typing profile: linear (default) or human",
long: "profile"
),
.commandOption(
@ -63,6 +63,11 @@ extension TypeCommand: CommanderSignatureProviding {
help: "Clear the field before typing (Cmd+A, Delete)",
long: "clear"
),
.commandFlag(
"foreground",
help: "Focus target and send foreground keyboard input",
long: "foreground"
),
],
optionGroups: [
InteractionTargetOptions.commanderSignature(),

View File

@ -12,6 +12,8 @@ struct TypeCommandResult: Codable {
let executionTime: TimeInterval
let wordsPerMinute: Int?
let profile: String
let deliveryMode: String
let targetPID: Int?
}
struct TypeCommandActionSummary: Codable {

View File

@ -13,17 +13,17 @@ struct TypeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfi
@Option(name: .customLong("text"), help: "Text to type (alternative to positional argument)")
var textOption: String?
@Option(help: "Snapshot ID (uses latest if not specified)")
@Option(help: "Snapshot ID, or 'latest' (uses latest if not specified)")
var snapshot: String?
@Option(help: "Delay between keystrokes in milliseconds")
var delay: Int = 2
var delay: Int = 0
@Option(name: .customLong("wpm"), help: "Approximate human typing speed (words per minute)")
var wordsPerMinute: Int?
@Option(name: .customLong("profile"), help: "Typing profile: human (default) or linear")
var profileOption: String? = TypingProfile.human.rawValue
@Option(name: .customLong("profile"), help: "Typing profile: linear (default) or human")
var profileOption: String?
@Flag(names: [.customLong("return"), .long], help: "Press return/enter after typing")
var pressReturn = false
@ -40,6 +40,9 @@ struct TypeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfi
@Flag(help: "Clear the field before typing (Cmd+A, Delete)")
var clear = false
@Flag(help: "Focus target and send foreground keyboard input")
var foreground = false
@OptionGroup var target: InteractionTargetOptions
@OptionGroup var focusOptions: FocusCommandOptions
@ -83,7 +86,7 @@ struct TypeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfi
let selection = TypingProfile(rawValue: profileOption.lowercased()) {
return selection
}
return .human
return self.wordsPerMinute == nil ? .linear : .human
}
private var resolvedWordsPerMinute: Int {
@ -108,16 +111,23 @@ struct TypeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfi
let actions = try self.buildActions()
let observation = await self.resolveObservationContext()
try await observation.validateIfExplicit(using: self.services.snapshots)
self.warnIfFocusUnknown(snapshotId: observation.snapshotId)
try await self.focusIfNeeded(snapshotId: observation.focusSnapshotId(for: self.target))
let typeResult = try await self.executeTypeActions(actions: actions, snapshotId: observation.snapshotId)
let targetPID = try await self.backgroundProcessIdentifier(snapshotId: observation.snapshotId)
self.resolvedRuntime.beginInteractionMutation()
if targetPID == nil {
self.warnIfFocusUnknown(snapshotId: observation.snapshotId)
try await self.focusIfNeeded(snapshotId: observation.focusSnapshotId(for: self.target))
}
let typeResult = try await self.executeTypeActions(
actions: actions,
snapshotId: observation.snapshotId,
targetProcessIdentifier: targetPID
)
await InteractionObservationInvalidator.invalidateAfterMutation(
observation,
snapshots: self.services.snapshots,
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "type"
)
self.renderResult(typeResult, actions: actions, startTime: startTime)
self.renderResult(typeResult, actions: actions, startTime: startTime, targetProcessIdentifier: targetPID)
} catch {
self.handleError(error)
throw ExitCode.failure
@ -175,6 +185,10 @@ struct TypeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfi
mutating func validate() throws {
try self.target.validate()
try KeyboardDeliverySupport.validateForegroundFlags(
foreground: self.foreground,
focusOptions: self.focusOptions
)
if let option = self.profileOption,
TypingProfile(rawValue: option.lowercased()) == nil {
throw ValidationError("--profile must be either 'human' or 'linear'")
@ -209,12 +223,43 @@ struct TypeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfi
)
}
private func executeTypeActions(actions: [TypeAction], snapshotId: String?) async throws -> TypeResult {
private func executeTypeActions(
actions: [TypeAction],
snapshotId: String?,
targetProcessIdentifier: pid_t?
) async throws -> TypeResult {
let request = TypeActionsRequest(actions: actions, cadence: self.typingCadence, snapshotId: snapshotId)
if let targetProcessIdentifier {
return try await AutomationServiceBridge.typeActions(
automation: self.services.automation,
request: request,
targetProcessIdentifier: targetProcessIdentifier
)
}
return try await AutomationServiceBridge.typeActions(automation: self.services.automation, request: request)
}
private func renderResult(_ typeResult: TypeResult, actions: [TypeAction], startTime: Date) {
private func backgroundProcessIdentifier(snapshotId: String?) async throws -> pid_t? {
guard !KeyboardDeliverySupport.shouldUseForeground(
foreground: self.foreground,
focusOptions: self.focusOptions
) else {
return nil
}
return try await KeyboardDeliverySupport.backgroundProcessIdentifier(
target: self.target,
snapshotId: snapshotId,
services: self.services
)
}
private func renderResult(
_ typeResult: TypeResult,
actions: [TypeAction],
startTime: Date,
targetProcessIdentifier: pid_t?
) {
let specialKeys = max(typeResult.keyPresses - typeResult.totalCharacters, 0)
let result = TypeCommandResult(
success: true,
@ -227,7 +272,10 @@ struct TypeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfi
actions: actions.map(Self.actionSummary),
executionTime: Date().timeIntervalSince(startTime),
wordsPerMinute: self.resolvedProfile == .human ? self.resolvedWordsPerMinute : nil,
profile: self.resolvedProfile.rawValue
profile: self.resolvedProfile.rawValue,
deliveryMode: targetProcessIdentifier == nil ? KeyboardDeliveryMode.foreground.rawValue :
KeyboardDeliveryMode.background.rawValue,
targetPID: targetProcessIdentifier.map(Int.init)
)
output(result) {
@ -238,6 +286,9 @@ struct TypeCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfi
if specialKeys > 0 {
print("🔑 Special keys: \(specialKeys)")
}
if let targetProcessIdentifier {
print("🎯 Mode: background to PID \(targetProcessIdentifier)")
}
print("📊 Total characters: \(typeResult.totalCharacters)")
switch self.resolvedProfile {
case .human:
@ -286,6 +337,7 @@ extension TypeCommand: CommanderBindableCommand {
self.escape = values.flag("escape")
self.delete = values.flag("delete")
self.clear = values.flag("clear")
self.foreground = values.flag("foreground")
self.target = try values.makeInteractionTargetOptions()
self.focusOptions = try values.makeFocusOptions()
}
@ -301,11 +353,12 @@ extension TypeCommand: ParsableCommand {
commandName: "type",
abstract: "Type text or send keyboard input",
discussion: """
The 'type' command sends keyboard input to the focused element.
It can type regular text or send special key combinations.
The 'type' command sends keyboard input to a targeted app/window,
snapshot process, or the current focused element. Background delivery
is used by default when a target process is known.
EXAMPLES:
peekaboo type "Hello World" # Type text with human cadence (default: 140 WPM)
peekaboo type "Hello World" --app TextEdit # Background-target TextEdit
peekaboo type "user@example.com" # Type email
peekaboo type "text" --delay 0 # Type at maximum speed
peekaboo type "text" --delay 50 # Type slower (50ms between keys)
@ -335,12 +388,13 @@ extension TypeCommand: ParsableCommand {
\\\\ - Literal backslash
FOCUS MANAGEMENT:
Provide --app/--pid/window targeting or a snapshot for focus guarantees.
Provide --app/--pid/window targeting or a snapshot for background delivery.
Use --foreground only when the target requires focused keyboard input.
Without a target, keys are injected into the current focused element.
HUMAN TYPING:
Use --profile human (default) for realistic cadence; override speed with --wpm (80-220).
Use --profile linear for deterministic timing via --delay.
TYPING CADENCE:
Linear typing is the default and uses --delay (0ms by default).
Use --profile human or --wpm (80-220) for realistic cadence.
""",
showHelpOnEmptyInvocation: true

View File

@ -0,0 +1,352 @@
import Commander
import Foundation
import PeekabooCore
import TachikomaMCP
@MainActor
struct BrowserCommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfigurable {
var action = "status"
var channel: String?
var pageId: Int?
var url: String?
var navigationType: String?
var uid: String?
var toUid: String?
var text: String?
var value: String?
var key: String?
var submitKey: String?
var dialogAction: String?
var includeSnapshot = false
var double = false
var noBringToFront = false
var background = false
var timeout: Int?
var pageSize: Int?
var pageIndex: Int?
var types: [String] = []
var resourceTypes: [String] = []
var includePreserved = false
var messageId: Int?
var requestId: Int?
var requestFilePath: String?
var responseFilePath: String?
var path: String?
var format: String?
var quality: Int?
var fullPage = false
var traceAction: String?
var noReload = false
var noAutoStop = false
var insightSetId: String?
var insightName: String?
var mcpTool: String?
var mcpArgsJson: String?
var runtimeOptions: CommandRuntimeOptions = {
var options = CommandRuntimeOptions()
options.requiresBrowserMCP = true
return options
}()
@RuntimeStorage private var runtime: CommandRuntime?
static let commandDescription = CommandDescription(
commandName: "browser",
abstract: "Control Chrome page content through the browser MCP tool",
discussion: """
Dedicated CLI wrapper around Peekaboo's browser MCP tool. Use it for DOM/page
operations such as status, connect, navigate, snapshot, click, fill, type,
screenshots, console/network inspection, and performance traces.
Examples:
peekaboo browser status --json
peekaboo browser connect --channel chrome
peekaboo browser navigate --url https://example.com
peekaboo browser snapshot --path /tmp/page.txt
"""
)
private var resolvedRuntime: CommandRuntime {
guard let runtime else {
preconditionFailure("CommandRuntime must be configured before accessing runtime resources")
}
return runtime
}
private var services: any PeekabooServiceProviding {
self.resolvedRuntime.services
}
private var logger: Logger {
self.resolvedRuntime.logger
}
var jsonOutput: Bool {
self.resolvedRuntime.configuration.jsonOutput
}
var outputLogger: Logger {
self.logger
}
mutating func setRuntimeOptions(_ options: CommandRuntimeOptions) {
var options = options
options.requiresBrowserMCP = true
self.runtimeOptions = options
}
mutating func run(using runtime: CommandRuntime) async throws {
self.runtime = runtime
self.logger.setJsonOutputMode(self.jsonOutput)
do {
let arguments = try self.arguments()
if Self.actionMayMutate(self.action) {
self.resolvedRuntime.beginInteractionMutation()
}
let context = MCPToolContext(services: self.services)
let tool = BrowserTool(context: context)
let response = try await tool.execute(arguments: ToolArguments(raw: arguments))
try MCPToolCommandOutput.output(
tool: tool.name,
response: response,
jsonOutput: self.jsonOutput,
logger: self.outputLogger
)
} catch let exit as ExitCode {
throw exit
} catch {
self.handleError(error)
throw ExitCode(1)
}
}
static func actionMayMutate(_ rawAction: String) -> Bool {
let normalized = rawAction
.trimmingCharacters(in: .whitespacesAndNewlines)
.replacingOccurrences(of: "-", with: "_")
guard let action = BrowserAction(rawValue: normalized) else { return false }
switch action {
case .status, .connect, .disconnect, .listPages, .waitFor, .snapshot, .console, .network, .screenshot:
return false
case .selectPage, .closePage, .newPage, .navigate, .click, .fill, .fillForm, .drag, .hover, .type,
.pressKey, .uploadFile, .handleDialog, .performanceTrace, .call:
return true
}
}
private func arguments() throws -> [String: Any] {
let normalizedAction = self.action
.trimmingCharacters(in: .whitespacesAndNewlines)
.replacingOccurrences(of: "-", with: "_")
guard BrowserAction(rawValue: normalizedAction) != nil else {
throw ValidationError("Unsupported browser action '\(self.action)'")
}
var arguments: [String: Any] = ["action": normalizedAction]
self.add(self.channel, as: "channel", to: &arguments)
self.add(self.pageId, as: "page_id", to: &arguments)
self.add(self.url, as: "url", to: &arguments)
self.add(self.navigationType, as: "navigation_type", to: &arguments)
self.add(self.uid, as: "uid", to: &arguments)
self.add(self.toUid, as: "to_uid", to: &arguments)
self.add(self.text, as: "text", to: &arguments)
self.add(self.value, as: "value", to: &arguments)
self.add(self.key, as: "key", to: &arguments)
self.add(self.submitKey, as: "submit_key", to: &arguments)
self.add(self.dialogAction, as: "dialog_action", to: &arguments)
self.addFlag(self.includeSnapshot, as: "include_snapshot", to: &arguments)
self.addFlag(self.double, as: "double", to: &arguments)
if self.noBringToFront {
arguments["bring_to_front"] = false
}
self.addFlag(self.background, as: "background", to: &arguments)
self.add(self.timeout, as: "timeout", to: &arguments)
self.add(self.pageSize, as: "page_size", to: &arguments)
self.add(self.pageIndex, as: "page_index", to: &arguments)
if !self.types.isEmpty {
arguments["types"] = self.types
}
if !self.resourceTypes.isEmpty {
arguments["resource_types"] = self.resourceTypes
}
self.addFlag(self.includePreserved, as: "include_preserved", to: &arguments)
self.add(self.messageId, as: "message_id", to: &arguments)
self.add(self.requestId, as: "request_id", to: &arguments)
self.add(self.requestFilePath, as: "request_file_path", to: &arguments)
self.add(self.responseFilePath, as: "response_file_path", to: &arguments)
self.add(self.path, as: "path", to: &arguments)
self.add(self.format, as: "format", to: &arguments)
self.add(self.quality, as: "quality", to: &arguments)
self.addFlag(self.fullPage, as: "full_page", to: &arguments)
self.add(self.traceAction, as: "trace_action", to: &arguments)
if self.noReload {
arguments["reload"] = false
}
if self.noAutoStop {
arguments["auto_stop"] = false
}
self.add(self.insightSetId, as: "insight_set_id", to: &arguments)
self.add(self.insightName, as: "insight_name", to: &arguments)
self.add(self.mcpTool, as: "mcp_tool", to: &arguments)
if let mcpArgsJson {
do {
_ = try MCPArgumentParsing.parseJSONObject(mcpArgsJson)
} catch {
throw ValidationError("--mcp-args-json must be a JSON object")
}
arguments["mcp_args_json"] = mcpArgsJson
}
return arguments
}
private func add(_ value: String?, as key: String, to arguments: inout [String: Any]) {
guard let value, !value.isEmpty else { return }
arguments[key] = value
}
private func add(_ value: Int?, as key: String, to arguments: inout [String: Any]) {
guard let value else { return }
arguments[key] = value
}
private func addFlag(_ value: Bool, as key: String, to arguments: inout [String: Any]) {
if value {
arguments[key] = true
}
}
}
extension BrowserCommand: ParsableCommand {}
extension BrowserCommand: AsyncRuntimeCommand {}
extension BrowserCommand: CommanderSignatureProviding {
static func commanderSignature() -> CommandSignature {
CommandSignature(
arguments: [
.make(
label: "action",
help: "Browser action (default: status)",
isOptional: true
),
],
options: [
.commandOption("channel", help: "Chrome channel", long: "channel"),
.commandOption("pageId", help: "Chrome DevTools page ID", long: "page-id"),
.commandOption("url", help: "URL for navigate/new-page", long: "url"),
.commandOption(
"navigationType",
help: "Navigation type: url|back|forward|reload",
long: "navigation-type"
),
.commandOption("uid", help: "Element uid from browser snapshot", long: "uid"),
.commandOption("toUid", help: "Drop target uid for drag", long: "to-uid"),
.commandOption("text", help: "Text for type/wait/dialog", long: "text"),
.commandOption("value", help: "Value for fill", long: "value"),
.commandOption("key", help: "Key or key combination for press-key", long: "key"),
.commandOption("submitKey", help: "Optional key after type", long: "submit-key"),
.commandOption("dialogAction", help: "Dialog action: accept|dismiss", long: "dialog-action"),
.commandOption("timeout", help: "Timeout in milliseconds", long: "timeout"),
.commandOption("pageSize", help: "Console/network page size", long: "page-size"),
.commandOption("pageIndex", help: "Console/network page index", long: "page-index"),
OptionDefinition.make(
label: "types",
names: [.long("type"), .aliasLong("types")],
help: "Console message type; repeat or comma-separate",
parsing: .singleValue
),
OptionDefinition.make(
label: "resourceTypes",
names: [.long("resource-type"), .aliasLong("resource-types")],
help: "Network resource type; repeat or comma-separate",
parsing: .singleValue
),
.commandOption("messageId", help: "Console message ID", long: "message-id"),
.commandOption("requestId", help: "Network request ID", long: "request-id"),
.commandOption("requestFilePath", help: "Path for saving a request body", long: "request-file-path"),
.commandOption("responseFilePath", help: "Path for saving a response body", long: "response-file-path"),
.commandOption("path", help: "Output path for snapshot/screenshot/trace", long: "path"),
.commandOption("format", help: "Screenshot format: png|jpeg|webp", long: "format"),
.commandOption("quality", help: "Screenshot quality for jpeg/webp", long: "quality"),
.commandOption("traceAction", help: "Trace action: start|stop|analyze", long: "trace-action"),
.commandOption("insightSetId", help: "Trace insight set ID", long: "insight-set-id"),
.commandOption("insightName", help: "Trace insight name", long: "insight-name"),
.commandOption("mcpTool", help: "Advanced browser MCP tool for call action", long: "mcp-tool"),
.commandOption(
"mcpArgsJson",
help: "Advanced JSON object args for call/fill-form",
long: "mcp-args-json"
),
],
flags: [
.commandFlag(
"includeSnapshot",
help: "Include fresh snapshot when supported",
long: "include-snapshot"
),
.commandFlag("double", help: "Double-click for click", long: "double"),
.commandFlag("noBringToFront", help: "Do not bring selected page to front", long: "no-bring-to-front"),
.commandFlag("background", help: "Open new page in background", long: "background"),
.commandFlag(
"includePreserved",
help: "Include preserved console/network data",
long: "include-preserved"
),
.commandFlag("fullPage", help: "Capture full-page screenshot", long: "full-page"),
.commandFlag("noReload", help: "Do not reload when starting a trace", long: "no-reload"),
.commandFlag("noAutoStop", help: "Do not auto-stop performance trace", long: "no-auto-stop"),
]
)
}
}
extension BrowserCommand: CommanderBindableCommand {
mutating func applyCommanderValues(_ values: CommanderBindableValues) throws {
self.action = values.positionalValue(at: 0) ?? "status"
self.channel = values.singleOption("channel")
self.pageId = try values.decodeOption("pageId", as: Int.self)
self.url = values.singleOption("url")
self.navigationType = values.singleOption("navigationType")
self.uid = values.singleOption("uid")
self.toUid = values.singleOption("toUid")
self.text = values.singleOption("text")
self.value = values.singleOption("value")
self.key = values.singleOption("key")
self.submitKey = values.singleOption("submitKey")
self.dialogAction = values.singleOption("dialogAction")
self.includeSnapshot = values.flag("includeSnapshot")
self.double = values.flag("double")
self.noBringToFront = values.flag("noBringToFront")
self.background = values.flag("background")
self.timeout = try values.decodeOption("timeout", as: Int.self)
self.pageSize = try values.decodeOption("pageSize", as: Int.self)
self.pageIndex = try values.decodeOption("pageIndex", as: Int.self)
self.types = Self.splitCSV(values.optionValues("types"))
self.resourceTypes = Self.splitCSV(values.optionValues("resourceTypes"))
self.includePreserved = values.flag("includePreserved")
self.messageId = try values.decodeOption("messageId", as: Int.self)
self.requestId = try values.decodeOption("requestId", as: Int.self)
self.requestFilePath = values.singleOption("requestFilePath")
self.responseFilePath = values.singleOption("responseFilePath")
self.path = values.singleOption("path")
self.format = values.singleOption("format")
self.quality = try values.decodeOption("quality", as: Int.self)
self.fullPage = values.flag("fullPage")
self.traceAction = values.singleOption("traceAction")
self.noReload = values.flag("noReload")
self.noAutoStop = values.flag("noAutoStop")
self.insightSetId = values.singleOption("insightSetId")
self.insightName = values.singleOption("insightName")
self.mcpTool = values.singleOption("mcpTool")
self.mcpArgsJson = values.singleOption("mcpArgsJson")
}
private static func splitCSV(_ values: [String]) -> [String] {
values.flatMap { value in
value.split(separator: ",")
.map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }
.filter { !$0.isEmpty }
}
}
}

View File

@ -0,0 +1,138 @@
import Commander
import Foundation
import PeekabooCore
import TachikomaMCP
@MainActor
struct InspectUICommand: ErrorHandlingCommand, OutputFormattable, RuntimeOptionsConfigurable {
var appTarget: String?
var snapshot: String?
var maxDepth: Int?
var maxElements: Int?
var maxChildren: Int?
var runtimeOptions: CommandRuntimeOptions = {
var options = CommandRuntimeOptions()
options.requiresInspectAccessibilityTree = true
return options
}()
@RuntimeStorage private var runtime: CommandRuntime?
static let commandDescription = CommandDescription(
commandName: "inspect-ui",
abstract: "Inspect accessible UI text through the inspect_ui MCP tool",
discussion: """
Dedicated CLI wrapper around Peekaboo's inspect_ui MCP tool. Use this for
accessibility-tree text inspection when `see` screenshots are too broad.
Examples:
peekaboo inspect-ui --app-target TextEdit
peekaboo inspect-ui --snapshot 1234 --max-elements 200 --json
"""
)
private var resolvedRuntime: CommandRuntime {
guard let runtime else {
preconditionFailure("CommandRuntime must be configured before accessing runtime resources")
}
return runtime
}
private var services: any PeekabooServiceProviding {
self.resolvedRuntime.services
}
private var logger: Logger {
self.resolvedRuntime.logger
}
var jsonOutput: Bool {
self.resolvedRuntime.configuration.jsonOutput
}
var outputLogger: Logger {
self.logger
}
mutating func setRuntimeOptions(_ options: CommandRuntimeOptions) {
var options = options
options.requiresInspectAccessibilityTree = true
self.runtimeOptions = options
}
mutating func run(using runtime: CommandRuntime) async throws {
self.runtime = runtime
self.logger.setJsonOutputMode(self.jsonOutput)
do {
let context = MCPToolContext(
services: self.services,
snapshotMutationCoordinator: runtime.toolSnapshotMutationCoordinator
)
let tool = InspectUITool(context: context)
let response = try await context.execute(
tool: tool,
arguments: ToolArguments(raw: self.arguments())
)
try MCPToolCommandOutput.output(
tool: tool.name,
response: response,
jsonOutput: self.jsonOutput,
logger: self.outputLogger
)
} catch let exit as ExitCode {
throw exit
} catch {
self.handleError(error)
throw ExitCode(1)
}
}
private func arguments() -> [String: Any] {
var arguments: [String: Any] = [:]
self.add(self.appTarget, as: "app_target", to: &arguments)
self.add(self.snapshot, as: "snapshot", to: &arguments)
self.add(self.maxDepth, as: "max_depth", to: &arguments)
self.add(self.maxElements, as: "max_elements", to: &arguments)
self.add(self.maxChildren, as: "max_children", to: &arguments)
return arguments
}
private func add(_ value: String?, as key: String, to arguments: inout [String: Any]) {
guard let value, !value.isEmpty else { return }
arguments[key] = value
}
private func add(_ value: Int?, as key: String, to arguments: inout [String: Any]) {
guard let value else { return }
arguments[key] = value
}
}
extension InspectUICommand: ParsableCommand {}
extension InspectUICommand: AsyncRuntimeCommand {}
extension InspectUICommand: CommanderSignatureProviding {
static func commanderSignature() -> CommandSignature {
CommandSignature(
options: [
.commandOption("appTarget", help: "App name, bundle ID, PID, or frontmost", long: "app-target"),
.commandOption("snapshot", help: "Existing UI snapshot ID", long: "snapshot"),
.commandOption("maxDepth", help: "Maximum accessibility-tree depth", long: "max-depth"),
.commandOption("maxElements", help: "Maximum elements to inspect", long: "max-elements"),
.commandOption("maxChildren", help: "Maximum children per node", long: "max-children"),
]
)
}
}
extension InspectUICommand: CommanderBindableCommand {
mutating func applyCommanderValues(_ values: CommanderBindableValues) throws {
self.appTarget = values.singleOption("appTarget")
self.snapshot = values.singleOption("snapshot")
self.maxDepth = try values.decodeOption("maxDepth", as: Int.self)
self.maxElements = try values.decodeOption("maxElements", as: Int.self)
self.maxChildren = try values.decodeOption("maxChildren", as: Int.self)
}
}

View File

@ -36,6 +36,7 @@ extension MCPCommand {
@MainActor
mutating func run(using runtime: CommandRuntime) async throws {
var localDaemon: PeekabooDaemon?
do {
guard let transportType = Self.transportType(named: self.transport) else {
runtime.logger.setJsonOutputMode(runtime.configuration.jsonOutput)
@ -51,20 +52,53 @@ extension MCPCommand {
if runtime.services is RemotePeekabooServices {
runtime.logger.debug("MCP: using remote Bridge host; skipping local daemon startup")
} else {
let daemon = PeekabooDaemon(configuration: .mcp())
await daemon.start()
let daemon = PeekabooDaemon(configuration: .embeddedMCP())
localDaemon = daemon
try await daemon.startChecked()
}
let server = try await PeekabooMCPServer()
let mutationCoordinator = runtime.toolSnapshotMutationCoordinator
let toolContext = Self.makeToolContext(
services: runtime.services,
snapshotMutationCoordinator: mutationCoordinator
)
let server = try await PeekabooMCPServer(toolContext: toolContext)
try await server.serve(transport: transportType, port: self.port)
await Self.stopLocalDaemon(localDaemon)
} catch let exitCode as ExitCode {
await Self.stopLocalDaemon(localDaemon)
throw exitCode
} catch {
await Self.stopLocalDaemon(localDaemon)
runtime.logger.error("Failed to start MCP server: \(error)")
throw ExitCode.failure
}
}
private static func stopLocalDaemon(_ daemon: PeekabooDaemon?) async {
guard let daemon, await daemon.requestStop() else { return }
await daemon.waitUntilStopped()
}
static func makeToolContext(
services: any PeekabooServiceProviding,
snapshotMutationCoordinator: (any MCPToolSnapshotMutationCoordinating)?
) -> MCPToolContext {
let snapshotExecutionGate: MCPToolSnapshotExecutionGate
if let agent = services.agent as? PeekabooAgentService {
agent.configureSnapshotMutationCoordinator(snapshotMutationCoordinator)
snapshotExecutionGate = agent.snapshotExecutionGate
} else {
snapshotExecutionGate = MCPToolSnapshotExecutionGate()
}
return MCPToolContext(
services: services,
snapshotMutationCoordinator: snapshotMutationCoordinator,
snapshotExecutionGate: snapshotExecutionGate
)
}
static func transportType(named name: String) -> PeekabooCore.TransportType? {
switch name.lowercased() {
case "stdio": .stdio

View File

@ -0,0 +1,88 @@
import Commander
import Foundation
import MCP
import PeekabooCore
import TachikomaMCP
struct MCPToolCommandPayload: Codable {
let tool: String
let isError: Bool
let content: [MCP.Tool.Content]
let text: String
let meta: Value?
}
struct MCPToolCommandJSONEnvelope: Codable {
let success: Bool
let data: MCPToolCommandPayload
let messages: [String]?
let debug_logs: [String]
let error: ErrorInfo?
}
@MainActor
enum MCPToolCommandOutput {
static func payload(tool: String, response: ToolResponse) -> MCPToolCommandPayload {
MCPToolCommandPayload(
tool: tool,
isError: response.isError,
content: response.content,
text: response.content.map(self.summary).joined(separator: "\n"),
meta: response.meta
)
}
static func output(
tool: String,
response: ToolResponse,
jsonOutput: Bool,
logger: Logger
) throws {
let payload = self.payload(tool: tool, response: response)
if jsonOutput {
let error = response.isError
? ErrorInfo(message: payload.text, code: .VALIDATION_ERROR)
: nil
let envelope = MCPToolCommandJSONEnvelope(
success: !response.isError,
data: payload,
messages: nil,
debug_logs: logger.getDebugLogs(),
error: error
)
outputJSONCodable(envelope, logger: logger)
} else if !payload.text.isEmpty {
print(payload.text)
}
if response.isError {
throw ExitCode(1)
}
}
private static func summary(for content: MCP.Tool.Content) -> String {
switch content {
case let .text(text, _, _):
return text
case let .image(data, mimeType, _, _):
return "[Image: \(mimeType), base64 bytes: \(data.count)]"
case let .audio(data, mimeType, _, _):
return "[Audio: \(mimeType), base64 bytes: \(data.count)]"
case let .resource(resource, _, _):
if let text = resource.text {
return text
} else if let blob = resource.blob {
return "[Resource: \(resource.uri), blob bytes: \(blob.count)]"
} else {
return "[Resource: \(resource.uri)]"
}
case let .resourceLink(uri, name, title, _, mimeType, _):
let label = title ?? name
if let mimeType {
return "[Resource Link: \(label) \(uri), type: \(mimeType)]"
} else {
return "[Resource Link: \(label) \(uri)]"
}
}
}
}

View File

@ -26,6 +26,18 @@ struct FocusCommandOptions: CommanderParsable, FocusOptionsProtocol {
set { self.focusBackgroundStorage = newValue }
}
var backgroundDeliveryExplicitlyRequested: Bool {
self.focusBackgroundStorage == true
}
var hasForegroundFocusOverrides: Bool {
self.noAutoFocus ||
self.focusTimeoutSeconds != nil ||
self.focusRetryCount != nil ||
self.spaceSwitch ||
self.bringToCurrentSpace
}
init() {}
// MARK: FocusOptionsProtocol

View File

@ -34,7 +34,41 @@ enum FocusTargetResolver {
}
}
enum FocusFailurePolicy {
static func optional<T>(_ operation: () async throws -> T) async throws -> T? {
do {
try Task.checkCancellation()
let result = try await operation()
try Task.checkCancellation()
return result
} catch {
try self.rethrowCancellation(error)
return nil
}
}
static func flatteningOptional<T>(_ operation: () async throws -> T?) async throws -> T? {
do {
try Task.checkCancellation()
let result = try await operation()
try Task.checkCancellation()
return result
} catch {
try self.rethrowCancellation(error)
return nil
}
}
static func rethrowCancellation(_ error: any Error) throws {
if error is CancellationError {
throw error
}
try Task.checkCancellation()
}
}
/// Ensure the target window is focused before executing a command.
@MainActor
func ensureFocused(
snapshotId: String? = nil,
windowID: CGWindowID? = nil,
@ -43,6 +77,7 @@ func ensureFocused(
options: any FocusOptionsProtocol,
services: any PeekabooServiceProviding
) async throws {
try Task.checkCancellation()
guard options.autoFocus else {
return
}
@ -54,6 +89,7 @@ func ensureFocused(
} else {
nil as UIAutomationSnapshot?
}
try Task.checkCancellation()
let targetRequest = FocusTargetResolver.resolve(
windowID: windowID,
@ -66,7 +102,9 @@ func ensureFocused(
case let .windowId(windowID):
windowID
case let .bestWindow(applicationName, windowTitle):
try? await focusService.findBestWindow(applicationName: applicationName, windowTitle: windowTitle)
try await FocusFailurePolicy.flatteningOptional {
try await focusService.findBestWindow(applicationName: applicationName, windowTitle: windowTitle)
}
case nil:
nil
}
@ -74,7 +112,9 @@ func ensureFocused(
guard let windowID = targetWindow else {
if case let .bestWindow(applicationName, _) = targetRequest {
_ = try await services.applications.findApplication(identifier: applicationName)
try Task.checkCancellation()
try await services.applications.activateApplication(identifier: applicationName)
try Task.checkCancellation()
}
return
}
@ -86,8 +126,10 @@ func ensureFocused(
bringToCurrentSpace: options.bringToCurrentSpace
)
try Task.checkCancellation()
do {
try await focusService.focusWindow(windowID: windowID, options: focusOptions)
try Task.checkCancellation()
} catch let error as FocusError {
switch error {
case .windowNotFound, .axElementNotFound:
@ -99,19 +141,25 @@ func ensureFocused(
fallbackTargets.append(.frontmost)
for target in fallbackTargets {
try Task.checkCancellation()
do {
try await WindowServiceBridge.focusWindow(windows: services.windows, target: target)
try Task.checkCancellation()
return
} catch {
try FocusFailurePolicy.rethrowCancellation(error)
fallbackErrors.append(error)
}
}
if let appName = applicationName {
try Task.checkCancellation()
do {
try await services.applications.activateApplication(identifier: appName)
try Task.checkCancellation()
return
} catch {
try FocusFailurePolicy.rethrowCancellation(error)
fallbackErrors.append(error)
}
}
@ -124,6 +172,7 @@ func ensureFocused(
}
/// Ensure focus using shared interaction target flags (`--app/--pid/--window-title/--window-index`).
@MainActor
func ensureFocused(
snapshotId: String? = nil,
target: InteractionTargetOptions,

View File

@ -12,7 +12,7 @@ enum InteractionCoordinateResolver {
services: any PeekabooServiceProviding,
forceGlobal: Bool = false
) async throws -> InteractionCoordinateResolution {
guard target.hasAnyTarget, !forceGlobal else {
guard target.hasAnyTarget else {
return InteractionCoordinateResolution(
inputPoint: inputPoint,
screenPoint: inputPoint,
@ -22,8 +22,33 @@ enum InteractionCoordinateResolver {
)
}
let hasWindowSelector = target.windowId != nil || target.windowTitle != nil || target.windowIndex != nil
if forceGlobal, !hasWindowSelector {
return InteractionCoordinateResolution(
inputPoint: inputPoint,
screenPoint: inputPoint,
coordinateSpace: .global,
windowInfo: nil,
targetApplication: nil
)
}
let windowResolution = try await self.resolveTargetWindow(target: target, services: services)
return try self.resolveTargetWindowCoordinates(
inputPoint,
windowInfo: windowResolution.windowInfo,
targetApplication: windowResolution.targetApplication,
forceGlobal: forceGlobal
)
}
static func resolveTargetWindow(
target: InteractionTargetOptions,
services: any PeekabooServiceProviding
) async throws -> InteractionWindowResolution {
guard let windowTarget = try target.toWindowTarget() else {
throw ValidationError("Coordinate target could not be resolved from the supplied target options.")
throw ValidationError("Window target could not be resolved from the supplied target options.")
}
let windowInfo = try await self.resolveWindowInfo(
@ -31,18 +56,12 @@ enum InteractionCoordinateResolver {
target: target,
services: services
)
let targetApplication = await self.resolveTargetApplication(
let targetApplication = try await self.resolveTargetApplication(
windowInfo: windowInfo,
target: target,
services: services
)
return try self.resolveTargetWindowCoordinates(
inputPoint,
windowInfo: windowInfo,
targetApplication: targetApplication
)
return InteractionWindowResolution(windowInfo: windowInfo, targetApplication: targetApplication)
}
static func resolveTargetWindowCoordinates(
@ -51,7 +70,7 @@ enum InteractionCoordinateResolver {
targetApplication: ServiceApplicationInfo?,
forceGlobal: Bool = false
) throws -> InteractionCoordinateResolution {
guard let windowInfo, !forceGlobal else {
guard let windowInfo else {
return InteractionCoordinateResolution(
inputPoint: inputPoint,
screenPoint: inputPoint,
@ -61,6 +80,16 @@ enum InteractionCoordinateResolver {
)
}
if forceGlobal {
return InteractionCoordinateResolution(
inputPoint: inputPoint,
screenPoint: inputPoint,
coordinateSpace: .global,
windowInfo: windowInfo,
targetApplication: targetApplication
)
}
try self.validate(inputPoint: inputPoint, within: windowInfo)
let screenPoint = CGPoint(
@ -109,9 +138,19 @@ enum InteractionCoordinateResolver {
windowInfo: ServiceWindowInfo,
target: InteractionTargetOptions,
services: any PeekabooServiceProviding
) async -> ServiceApplicationInfo? {
if let identifier = try? target.resolveApplicationIdentifierOptional(),
let application = try? await services.applications.findApplication(identifier: identifier) {
) async throws -> ServiceApplicationInfo? {
if let identifier = try target.resolveApplicationIdentifierOptional() {
let application = try await services.applications.findApplication(identifier: identifier)
if target.windowId != nil {
let applicationWindows = try await services.windows.listWindows(
target: .application("PID:\(application.processIdentifier)")
)
try self.validateWindowOwnership(
windowInfo: windowInfo,
application: application,
applicationWindows: applicationWindows
)
}
return application
}
@ -137,6 +176,19 @@ enum InteractionCoordinateResolver {
return nil
}
static func validateWindowOwnership(
windowInfo: ServiceWindowInfo,
application: ServiceApplicationInfo,
applicationWindows: [ServiceWindowInfo]
) throws {
guard applicationWindows.contains(where: { $0.windowID == windowInfo.windowID }) else {
throw ValidationError(
"Window \(windowInfo.windowID) does not belong to \(application.name) " +
"(PID \(application.processIdentifier))"
)
}
}
private static func targetDescription(_ target: InteractionTargetOptions) -> String {
if let windowId = target.windowId {
return "window id \(windowId)"
@ -165,6 +217,46 @@ enum InteractionCoordinateResolver {
}
}
struct InteractionWindowResolution {
let windowInfo: ServiceWindowInfo
let targetApplication: ServiceApplicationInfo?
var targetProcessIdentifier: Int32? {
self.targetApplication?.processIdentifier
}
}
enum InteractionWindowSelectionValidator {
static func validate(
resolution: InteractionWindowResolution,
snapshotContext: WindowContext?,
snapshotId: String
) throws {
guard let snapshotWindowID = snapshotContext?.windowID else {
throw ValidationError(
"Snapshot '\(snapshotId)' does not identify an exact window; " +
"capture a fresh snapshot for the selected window"
)
}
guard snapshotWindowID == resolution.windowInfo.windowID else {
throw ValidationError(
"Snapshot '\(snapshotId)' belongs to window \(snapshotWindowID), but the explicit selector " +
"resolved window \(resolution.windowInfo.windowID)"
)
}
if let snapshotPID = snapshotContext?.applicationProcessId,
let selectedPID = resolution.targetProcessIdentifier,
snapshotPID != selectedPID {
throw ValidationError(
"Snapshot '\(snapshotId)' belongs to PID \(snapshotPID), but the selected window " +
"belongs to PID \(selectedPID)"
)
}
}
}
struct InteractionCoordinateResolution {
let inputPoint: CGPoint
let screenPoint: CGPoint

View File

@ -53,17 +53,29 @@ struct InteractionObservationContext {
snapshots: any SnapshotManagerProtocol
) async -> InteractionObservationContext {
if let explicitSnapshotId = normalizedSnapshotId(rawSnapshot) {
return InteractionObservationContext(
explicitSnapshotId: explicitSnapshotId,
snapshotId: explicitSnapshotId,
source: .explicit
)
guard self.isLatestAlias(explicitSnapshotId) else {
return InteractionObservationContext(
explicitSnapshotId: explicitSnapshotId,
snapshotId: explicitSnapshotId,
source: .explicit
)
}
return await self.latestSnapshotContext(from: snapshots)
}
guard fallbackToLatest else {
return InteractionObservationContext(explicitSnapshotId: nil, snapshotId: nil, source: .none)
return InteractionObservationContext(
explicitSnapshotId: nil,
snapshotId: nil,
source: .none
)
}
return await self.latestSnapshotContext(from: snapshots)
}
private static func latestSnapshotContext(from snapshots: any SnapshotManagerProtocol) async
-> InteractionObservationContext {
if let latestSnapshotId = await snapshots.getMostRecentSnapshot() {
return InteractionObservationContext(
explicitSnapshotId: nil,
@ -79,12 +91,32 @@ struct InteractionObservationContext {
let trimmed = snapshotId?.trimmingCharacters(in: .whitespacesAndNewlines)
return trimmed?.isEmpty == false ? trimmed : nil
}
private static func isLatestAlias(_ snapshotId: String) -> Bool {
switch snapshotId.lowercased() {
case "latest", "most-recent", "most_recent":
true
default:
false
}
}
}
@MainActor
struct InteractionObservationRefreshDependencies {
let desktopObservation: any DesktopObservationServiceProtocol
let snapshots: any SnapshotManagerProtocol
let beginMutation: ((Date) -> Void)?
init(
desktopObservation: any DesktopObservationServiceProtocol,
snapshots: any SnapshotManagerProtocol,
beginMutation: ((Date) -> Void)? = nil
) {
self.desktopObservation = desktopObservation
self.snapshots = snapshots
self.beginMutation = beginMutation
}
}
@MainActor
@ -94,7 +126,8 @@ enum InteractionObservationRefresher {
elementIds: [String?],
target: InteractionTargetOptions,
services: any PeekabooServiceProviding,
logger: Logger
logger: Logger,
beforeRefresh: ((Date) -> Void)? = nil
) async throws -> InteractionObservationContext {
var refreshed = observation
for elementId in elementIds.compactMap(\.self) {
@ -103,7 +136,8 @@ enum InteractionObservationRefresher {
elementId: elementId,
target: target,
services: services,
logger: logger
logger: logger,
beforeRefresh: beforeRefresh
)
}
return refreshed
@ -114,7 +148,8 @@ enum InteractionObservationRefresher {
query: String,
target: InteractionTargetOptions,
services: any PeekabooServiceProviding,
logger: Logger
logger: Logger,
beforeRefresh: ((Date) -> Void)? = nil
) async throws -> InteractionObservationContext {
try await self.refreshForMissingQueryIfNeeded(
observation,
@ -122,7 +157,8 @@ enum InteractionObservationRefresher {
target: target,
dependencies: InteractionObservationRefreshDependencies(
desktopObservation: services.desktopObservation,
snapshots: services.snapshots
snapshots: services.snapshots,
beginMutation: beforeRefresh
),
logger: logger
)
@ -159,7 +195,8 @@ enum InteractionObservationRefresher {
elementId: String,
target: InteractionTargetOptions,
services: any PeekabooServiceProviding,
logger: Logger
logger: Logger,
beforeRefresh: ((Date) -> Void)? = nil
) async throws -> InteractionObservationContext {
try await self.refreshForMissingElementIfNeeded(
observation,
@ -167,7 +204,8 @@ enum InteractionObservationRefresher {
target: target,
dependencies: InteractionObservationRefreshDependencies(
desktopObservation: services.desktopObservation,
snapshots: services.snapshots
snapshots: services.snapshots,
beginMutation: beforeRefresh
),
logger: logger
)
@ -207,31 +245,87 @@ enum InteractionObservationRefresher {
logger: Logger
) async throws -> InteractionObservationContext {
let requestTarget = try target.observationTargetRequest()
let result = try await dependencies.desktopObservation.observe(DesktopObservationRequest(
target: requestTarget,
capture: DesktopCaptureOptions(
engine: .auto,
scale: .logical1x,
visualizerMode: .screenshotFlash
),
detection: DesktopDetectionOptions(mode: .accessibility, allowWebFocusFallback: true),
output: DesktopObservationOutputOptions(saveSnapshot: true)
))
guard let refreshedSnapshotId = result.elements?.snapshotId else {
return observation
let observationStartedAt = Date()
dependencies.beginMutation?(observationStartedAt)
let snapshotID = try await dependencies.snapshots.createSnapshot(pendingAt: observationStartedAt)
let result: DesktopObservationResult
do {
result = try await dependencies.desktopObservation.observe(DesktopObservationRequest(
target: requestTarget,
capture: DesktopCaptureOptions(
engine: .auto,
scale: .logical1x,
visualizerMode: .screenshotFlash
),
detection: DesktopDetectionOptions(mode: .accessibility, allowWebFocusFallback: true),
output: DesktopObservationOutputOptions(
saveSnapshot: true,
snapshotID: snapshotID
)
))
guard result.elements != nil else {
try? await dependencies.snapshots.cleanSnapshot(snapshotId: snapshotID)
_ = try? await dependencies.snapshots.invalidateImplicitLatestSnapshot(through: Date())
return observation
}
let publication = try self.certifiedPublicationBoundary(
for: result,
observationStartedAt: observationStartedAt
)
_ = try await dependencies.snapshots.invalidateImplicitLatestSnapshot(
through: publication.cutoff,
preserving: snapshotID,
preservedAt: publication.preservedAt
)
guard await dependencies.snapshots.getMostRecentSnapshot() == snapshotID else {
throw PeekabooError.snapshotStale(
"The refreshed observation was superseded before it could be published"
)
}
} catch {
if !PendingSnapshotCleanupPolicy.shouldPreserveReservation(after: error) {
try? await dependencies.snapshots.cleanSnapshot(snapshotId: snapshotID)
}
_ = try? await dependencies.snapshots.invalidateImplicitLatestSnapshot(through: Date())
throw error
}
logger.debug(
"Refreshed implicit observation snapshot '\(refreshedSnapshotId)' for \(reason)"
"Refreshed implicit observation snapshot '\(snapshotID)' for \(reason)"
)
return InteractionObservationContext(
explicitSnapshotId: nil,
snapshotId: refreshedSnapshotId,
snapshotId: snapshotID,
source: .latest
)
}
private static func certifiedPublicationBoundary(
for result: DesktopObservationResult,
observationStartedAt: Date
) throws -> (cutoff: Date, preservedAt: Date) {
let completedAtValues = [
result.diagnostics.desktopMutationCompletedAt,
result.elements?.metadata.desktopMutationCompletedAt,
].compactMap(\.self)
let preservationValues = [
result.diagnostics.desktopMutationPreservationAllowed,
result.elements?.metadata.desktopMutationPreservationAllowed,
].compactMap(\.self)
let hasCertificate = !completedAtValues.isEmpty || !preservationValues.isEmpty
guard hasCertificate else { return (observationStartedAt, Date()) }
guard let completedAt = completedAtValues.max(),
!preservationValues.isEmpty,
preservationValues.allSatisfy(\.self)
else {
throw PeekabooError.snapshotStale(
"The refreshed observation overlapped another desktop mutation"
)
}
let cutoff = max(observationStartedAt, completedAt)
return (cutoff, cutoff)
}
private static func containsElement(
matching query: String,
in detectionResult: ElementDetectionResult

View File

@ -1,38 +1,440 @@
import Foundation
import PeekabooBridge
import PeekabooCore
import PeekabooFoundation
@MainActor
final class InteractionMutationTracker {
private let desktopMutationWatermarkStore: DesktopMutationWatermarkStore
private var pendingDesktopMutation: DesktopMutationWatermarkStore.PendingMutation?
private var durableMutationLeaseCount = 0
private(set) var mutationStartedAt: Date?
private(set) var mutationSequence: UInt64 = 0
private var successfulCompletionCutoff: Date?
private var failedInvalidationCutoff: Date?
private(set) var preservedSnapshotID: String?
private(set) var preservedAt: Date?
init(desktopMutationWatermarkStore: DesktopMutationWatermarkStore = DesktopMutationWatermarkStore()) {
self.desktopMutationWatermarkStore = desktopMutationWatermarkStore
}
var hasFailedInvalidationAttempt: Bool {
self.failedInvalidationCutoff != nil
}
var hasPendingDurableMutation: Bool {
self.pendingDesktopMutation != nil
}
@discardableResult
func begin(
at cutoff: Date = Date(),
preservingSnapshotsCreatedAfterBoundary: Bool = false
) -> Date {
if self.mutationSequence < UInt64.max {
self.mutationSequence += 1
}
if let failedInvalidationCutoff, cutoff > failedInvalidationCutoff {
self.failedInvalidationCutoff = nil
}
if self.mutationStartedAt == nil {
self.mutationStartedAt = cutoff
}
if preservingSnapshotsCreatedAfterBoundary {
self.successfulCompletionCutoff = max(self.successfulCompletionCutoff ?? cutoff, cutoff)
} else {
self.successfulCompletionCutoff = nil
}
self.preservedSnapshotID = nil
self.preservedAt = nil
return self.mutationStartedAt ?? cutoff
}
func preserveFreshObservation(
snapshotId: String,
startedAt: Date,
preservedAt: Date,
preservationAllowed: Bool = true
) {
guard self.mutationStartedAt != nil else { return }
guard preservationAllowed else {
self.successfulCompletionCutoff = nil
self.preservedSnapshotID = nil
self.preservedAt = nil
return
}
self.successfulCompletionCutoff = max(self.successfulCompletionCutoff ?? startedAt, startedAt)
self.preservedSnapshotID = snapshotId
self.preservedAt = preservedAt
}
@discardableResult
func beginDurableMutation(at startedAt: Date = Date()) throws -> Bool {
guard self.pendingDesktopMutation == nil else { return false }
self.pendingDesktopMutation = try self.desktopMutationWatermarkStore.beginMutation(at: startedAt)
self.durableMutationLeaseCount = 1
return true
}
func retainDurableMutationLease(at startedAt: Date = Date()) throws {
if self.pendingDesktopMutation == nil {
self.pendingDesktopMutation = try self.desktopMutationWatermarkStore.beginMutation(at: startedAt)
self.durableMutationLeaseCount = 1
} else {
self.durableMutationLeaseCount += 1
}
}
func completeDurableMutation(
through cutoff: Date
) throws -> DesktopMutationWatermarkStore.MutationCompletion? {
guard let pendingDesktopMutation else { return nil }
if self.durableMutationLeaseCount > 1 {
self.durableMutationLeaseCount -= 1
return nil
}
let completion = try self.desktopMutationWatermarkStore.completeMutation(
pendingDesktopMutation,
through: cutoff
)
self.pendingDesktopMutation = nil
self.durableMutationLeaseCount = 0
return completion
}
func cancelDurableMutation() throws {
guard let pendingDesktopMutation else { return }
if self.durableMutationLeaseCount > 1 {
self.durableMutationLeaseCount -= 1
return
}
try self.desktopMutationWatermarkStore.cancelMutation(pendingDesktopMutation)
self.pendingDesktopMutation = nil
self.durableMutationLeaseCount = 0
}
func withPendingDurableMutationVisible<T>(
createdByCurrentCommand: Bool,
operation: () async throws -> T
) async rethrows -> T {
guard createdByCurrentCommand, let pendingDesktopMutation else {
return try await operation()
}
return try await DesktopMutationWatermarkStore.withPendingMutationVisible(
pendingDesktopMutation,
operation: operation
)
}
func invalidationCutoff(commandCompletedAt completion: Date, succeeded: Bool) -> Date? {
guard self.mutationStartedAt != nil else { return nil }
if let failedInvalidationCutoff {
return failedInvalidationCutoff
}
if succeeded, let successfulCompletionCutoff {
return successfulCompletionCutoff
}
return completion
}
func markInvalidationFailed(through cutoff: Date) {
guard let mutationStartedAt, mutationStartedAt <= cutoff else { return }
self.failedInvalidationCutoff = min(self.failedInvalidationCutoff ?? cutoff, cutoff)
}
func markInvalidated(through cutoff: Date) {
guard let mutationStartedAt, mutationStartedAt <= cutoff else { return }
self.mutationStartedAt = nil
self.successfulCompletionCutoff = nil
self.failedInvalidationCutoff = nil
self.preservedSnapshotID = nil
self.preservedAt = nil
}
}
@MainActor
extension InteractionObservationContext {
@discardableResult
func invalidateAfterMutation(using snapshots: any SnapshotManagerProtocol) async throws -> String? {
guard self.source == .latest, let snapshotId else {
func invalidateAfterMutation(
using snapshots: any SnapshotManagerProtocol,
through cutoff: Date = Date()
) async throws -> String? {
guard source == .latest, let snapshotId else {
return nil
}
try await snapshots.cleanSnapshot(snapshotId: snapshotId)
guard try await snapshots.invalidateImplicitLatestSnapshot(through: cutoff) != nil else {
return nil
}
return snapshotId
}
static func invalidateLatestSnapshot(using snapshots: any SnapshotManagerProtocol) async throws -> String? {
guard let latestSnapshotId = await snapshots.getMostRecentSnapshot() else {
return nil
}
try await snapshots.cleanSnapshot(snapshotId: latestSnapshotId)
return latestSnapshotId
static func invalidateLatestSnapshot(
using snapshots: any SnapshotManagerProtocol,
through cutoff: Date = Date(),
preserving snapshotId: String? = nil,
preservedAt: Date? = nil
) async throws -> String? {
try await snapshots.invalidateImplicitLatestSnapshot(
through: cutoff,
preserving: snapshotId,
preservedAt: preservedAt
)
}
}
@MainActor
enum InteractionObservationInvalidator {
struct MutationTargets {
let snapshots: any SnapshotManagerProtocol
let selectedRemoteSocketPath: String?
let remoteSocketPaths: [String]
let socketExists: (String) -> Bool
let makeLocalSnapshotManager: () -> any SnapshotManagerProtocol
let makeRemoteSnapshotManager: (String) async throws -> (any SnapshotManagerProtocol)?
let mutationTracker: InteractionMutationTracker?
init(
snapshots: any SnapshotManagerProtocol,
selectedRemoteSocketPath: String?,
remoteSocketPaths: [String],
socketExists: @escaping (String) -> Bool = { FileManager.default.fileExists(atPath: $0) },
makeLocalSnapshotManager: @escaping () -> any SnapshotManagerProtocol = {
SnapshotManager(desktopMutationWatermarkStore: DesktopMutationWatermarkStore())
},
makeRemoteSnapshotManager: @escaping (String) async throws -> (any SnapshotManagerProtocol)? = {
try await InteractionObservationInvalidator.makeRemoteSnapshotManager(socketPath: $0)
},
mutationTracker: InteractionMutationTracker? = nil
) {
self.snapshots = snapshots
self.selectedRemoteSocketPath = selectedRemoteSocketPath
self.remoteSocketPaths = remoteSocketPaths
self.socketExists = socketExists
self.makeLocalSnapshotManager = makeLocalSnapshotManager
self.makeRemoteSnapshotManager = makeRemoteSnapshotManager
self.mutationTracker = mutationTracker
}
}
@discardableResult
static func invalidateAfterClickMutation(
targets: MutationTargets,
logger: Logger,
reason: String,
through cutoff: Date = Date()
) async -> Bool {
await self.invalidateAfterMutation(
targets: targets,
logger: logger,
reason: reason,
through: cutoff
)
}
@discardableResult
static func invalidateAfterMutation(
targets: MutationTargets,
logger: Logger,
reason: String,
through cutoff: Date = Date(),
preserving snapshotId: String? = nil,
preservedAt: Date? = nil
) async -> Bool {
let succeeded = await invalidateLatestSnapshotsAcrossKnownHosts(
using: targets.snapshots,
selectedRemoteSocketPath: targets.selectedRemoteSocketPath,
remoteSocketPaths: targets.remoteSocketPaths,
logger: logger,
reason: reason,
through: cutoff,
preserving: snapshotId,
preservedAt: preservedAt,
logFailures: targets.mutationTracker?.mutationStartedAt == nil,
socketExists: targets.socketExists,
makeLocalSnapshotManager: targets.makeLocalSnapshotManager,
makeRemoteSnapshotManager: targets.makeRemoteSnapshotManager
)
if succeeded {
targets.mutationTracker?.markInvalidated(through: cutoff)
} else {
targets.mutationTracker?.markInvalidationFailed(through: cutoff)
}
return succeeded
}
@discardableResult
static func invalidateLatestSnapshotsAcrossKnownHosts(
using snapshots: any SnapshotManagerProtocol,
selectedRemoteSocketPath: String?,
remoteSocketPaths: [String],
logger: Logger,
reason: String,
through cutoff: Date = Date(),
preserving snapshotId: String? = nil,
preservedAt: Date? = nil,
logFailures: Bool = true,
socketExists: (String) -> Bool = { FileManager.default.fileExists(atPath: $0) },
makeLocalSnapshotManager: () -> any SnapshotManagerProtocol = {
SnapshotManager(desktopMutationWatermarkStore: DesktopMutationWatermarkStore())
},
makeRemoteSnapshotManager: (String) async throws -> (any SnapshotManagerProtocol)? = {
try await InteractionObservationInvalidator.makeRemoteSnapshotManager(socketPath: $0)
}
) async -> Bool {
var requiredManagers: [any SnapshotManagerProtocol] = [snapshots]
let selectedPath = selectedRemoteSocketPath.map { NSString(string: $0).standardizingPath }
if selectedPath != nil {
requiredManagers.append(makeLocalSnapshotManager())
}
let requiredSucceeded = await self.invalidateLatestSnapshots(
using: requiredManagers,
logger: logger,
reason: reason,
through: cutoff,
preserving: snapshotId,
preservedAt: preservedAt,
logFailures: logFailures
)
var alternateManagers: [(path: String, manager: any SnapshotManagerProtocol)] = []
var seenPaths = Set<String>()
for rawPath in remoteSocketPaths {
let path = NSString(string: rawPath).standardizingPath
guard !path.isEmpty,
path != selectedPath,
seenPaths.insert(path).inserted,
socketExists(path)
else { continue }
do {
if let manager = try await makeRemoteSnapshotManager(path) {
alternateManagers.append((path: path, manager: manager))
}
} catch {
if self.isStaleSocketProbeFailure(
error,
socketPath: path,
socketExists: socketExists
) {
logger.debug(
"Skipping stale snapshot invalidation endpoint at \(path) after \(reason)"
)
continue
}
if logFailures {
logger.warn(
"Skipping unavailable alternate snapshot endpoint at \(path) after \(reason): " +
error.localizedDescription
)
} else {
logger.debug(
"Skipping unavailable alternate snapshot endpoint at \(path) after \(reason)"
)
}
}
}
for alternate in alternateManagers {
let succeeded = await self.invalidateLatestSnapshot(
using: alternate.manager,
logger: logger,
reason: reason,
through: cutoff,
preserving: snapshotId,
preservedAt: preservedAt,
logFailures: false
)
if !succeeded {
logger.debug(
"Skipping unavailable alternate snapshot endpoint at \(alternate.path) after \(reason)"
)
}
}
return requiredSucceeded
}
private static func isStaleSocketProbeFailure(
_ error: any Error,
socketPath: String,
socketExists: (String) -> Bool
) -> Bool {
guard socketExists(socketPath) else {
return true
}
guard let posixError = error as? POSIXError else {
return false
}
return switch posixError.code {
case .ECONNREFUSED, .ENOENT, .ENOTSOCK:
true
default:
false
}
}
private static func makeRemoteSnapshotManager(
socketPath: String
) async throws -> (any SnapshotManagerProtocol)? {
let client = PeekabooBridgeClient(socketPath: socketPath, requestTimeoutSec: 1)
let identity = PeekabooBridgeClientIdentity(
bundleIdentifier: Bundle.main.bundleIdentifier,
teamIdentifier: nil,
processIdentifier: getpid(),
hostname: Host.current().name
)
let handshake = try await client.handshake(client: identity, requestedHost: nil)
guard BridgeCapabilityPolicy.supportsImplicitSnapshotInvalidation(for: handshake) else {
return nil
}
return RemoteSnapshotManager(
client: client,
supportsImplicitLatestSnapshotInvalidation: true,
desktopMutationWatermarkStore: DesktopMutationWatermarkStore()
)
}
@discardableResult
static func invalidateLatestSnapshots(
using snapshotManagers: [any SnapshotManagerProtocol],
logger: Logger,
reason: String,
through cutoff: Date = Date(),
preserving snapshotId: String? = nil,
preservedAt: Date? = nil,
logFailures: Bool = true
) async -> Bool {
var succeeded = true
for snapshots in snapshotManagers {
guard await self.invalidateLatestSnapshot(
using: snapshots,
logger: logger,
reason: reason,
through: cutoff,
preserving: snapshotId,
preservedAt: preservedAt,
logFailures: logFailures
) else {
succeeded = false
continue
}
}
return succeeded
}
static func invalidateAfterMutation(
_ observation: InteractionObservationContext,
snapshots: any SnapshotManagerProtocol,
logger: Logger,
reason: String
reason: String,
through cutoff: Date = Date()
) async {
do {
if let invalidatedSnapshotId = try await observation.invalidateAfterMutation(using: snapshots) {
if let invalidatedSnapshotId = try await observation.invalidateAfterMutation(
using: snapshots,
through: cutoff
) {
logger.debug(
"Invalidated implicit latest snapshot '\(invalidatedSnapshotId)' after \(reason)"
)
@ -48,7 +450,8 @@ enum InteractionObservationInvalidator {
_ observation: InteractionObservationContext,
snapshots: any SnapshotManagerProtocol,
logger: Logger,
reason: String
reason: String,
through cutoff: Date = Date()
) async {
switch observation.source {
case .explicit:
@ -58,34 +461,221 @@ enum InteractionObservationInvalidator {
observation,
snapshots: snapshots,
logger: logger,
reason: reason
reason: reason,
through: cutoff
)
case .none:
await self.invalidateLatestSnapshot(
using: snapshots,
logger: logger,
reason: reason
reason: reason,
through: cutoff
)
}
}
@discardableResult
static func invalidateLatestSnapshot(
using snapshots: any SnapshotManagerProtocol,
logger: Logger,
reason: String
) async {
reason: String,
through cutoff: Date = Date(),
preserving snapshotId: String? = nil,
preservedAt: Date? = nil,
logFailures: Bool = true
) async -> Bool {
do {
if let invalidatedSnapshotId = try await InteractionObservationContext.invalidateLatestSnapshot(
using: snapshots
using: snapshots,
through: cutoff,
preserving: snapshotId,
preservedAt: preservedAt
) {
logger.debug(
"Invalidated latest snapshot '\(invalidatedSnapshotId)' after \(reason)"
"Invalidated implicit latest snapshot '\(invalidatedSnapshotId)' after \(reason)"
)
}
return true
} catch {
logger.warn(
"Failed to invalidate latest snapshot after \(reason): \(error.localizedDescription)"
)
if logFailures {
logger.warn(
"Failed to invalidate latest snapshot after \(reason): \(error.localizedDescription)"
)
}
return false
}
}
}
@MainActor
extension CommandRuntime {
func withCaptureFocusMutation(
_ operation: () async throws -> Void
) async rethrows {
self.beginInteractionMutation()
try await operation()
self.beginInteractionMutation(preservingSnapshotsCreatedAfterBoundary: true)
}
@discardableResult
func beginInteractionMutation(
at cutoff: Date = Date(),
preservingSnapshotsCreatedAfterBoundary: Bool = false
) -> Date {
interactionMutationTracker.begin(
at: cutoff,
preservingSnapshotsCreatedAfterBoundary: preservingSnapshotsCreatedAfterBoundary
)
}
func preserveFreshObservation(
snapshotId: String,
startedAt: Date,
preservedAt: Date,
preservationAllowed: Bool = true
) {
interactionMutationTracker.preserveFreshObservation(
snapshotId: snapshotId,
startedAt: startedAt,
preservedAt: preservedAt,
preservationAllowed: preservationAllowed
)
}
var interactionMutationTargets: InteractionObservationInvalidator.MutationTargets {
.init(
snapshots: services.snapshots,
selectedRemoteSocketPath: selectedRemoteSocketPath,
remoteSocketPaths: snapshotInvalidationRemoteSocketPaths,
mutationTracker: interactionMutationTracker
)
}
var toolSnapshotMutationCoordinator: any MCPToolSnapshotMutationCoordinating {
RuntimeMCPToolSnapshotMutationCoordinator(
targets: .init(
snapshots: services.snapshots,
selectedRemoteSocketPath: selectedRemoteSocketPath,
remoteSocketPaths: snapshotInvalidationRemoteSocketPaths
),
logger: logger,
mutationTracker: interactionMutationTracker
)
}
}
@MainActor
private final class RuntimeMCPToolSnapshotMutationCoordinator: MCPToolSnapshotMutationCoordinating {
private let targets: InteractionObservationInvalidator.MutationTargets
private let logger: Logger
private let mutationTracker: InteractionMutationTracker
private let hasRemoteSelection: Bool
private var preparedLocalMutationIDs: Set<UUID> = []
private var completedPreparedMutationIDs: Set<UUID> = []
init(
targets: InteractionObservationInvalidator.MutationTargets,
logger: Logger,
mutationTracker: InteractionMutationTracker
) {
self.targets = targets
self.logger = logger
self.mutationTracker = mutationTracker
self.hasRemoteSelection = targets.selectedRemoteSocketPath != nil
}
func prepareMutation(_ scope: MCPToolSnapshotMutationScope) throws {
guard scope.effect != .freshObservation else { return }
let needsCallerBarrier = !self.hasRemoteSelection || scope.effect != .mutationProducingFreshObservation
if needsCallerBarrier {
guard try self.mutationTracker.beginDurableMutation(at: scope.startedAt) else {
throw PeekabooError.operationError(
message: "A previous local desktop mutation barrier is still pending"
)
}
self.preparedLocalMutationIDs.insert(scope.id)
}
self.mutationTracker.begin(
at: scope.startedAt,
preservingSnapshotsCreatedAfterBoundary: scope.effect == .mutationProducingFreshObservation
)
}
func completeMutationBarrier(
_ scope: MCPToolSnapshotMutationScope
) throws -> MCPToolMutationBarrierCompletion? {
guard self.preparedLocalMutationIDs.contains(scope.id) else { return nil }
let completion = try self.mutationTracker.completeDurableMutation(
through: scope.completedAt ?? Date()
)
self.preparedLocalMutationIDs.remove(scope.id)
self.completedPreparedMutationIDs.insert(scope.id)
return completion.map {
MCPToolMutationBarrierCompletion(
cutoff: $0.cutoff,
allowsObservationPreservation: $0.allowsObservationPreservation
)
}
}
@discardableResult
func completeMutation(_ scope: MCPToolSnapshotMutationScope, succeeded: Bool) async -> Bool {
let completedPreparedMutation = self.completedPreparedMutationIDs.remove(scope.id) != nil
let defersToOuterCommandBarrier = !self.hasRemoteSelection &&
scope.effect == .mutationProducingFreshObservation &&
!completedPreparedMutation &&
self.mutationTracker.hasPendingDurableMutation
if defersToOuterCommandBarrier {
if succeeded,
let preservedSnapshotID = scope.preservedSnapshotID,
let completedAt = scope.completedAt {
self.mutationTracker.preserveFreshObservation(
snapshotId: preservedSnapshotID,
startedAt: scope.confirmedMutationCompletedAt ?? scope.startedAt,
preservedAt: completedAt
)
}
return true
}
let sharedWatermark = self.targets.snapshots.effectiveImplicitLatestInvalidationWatermark
let wantsPreservation = succeeded &&
scope.effect == .mutationProducingFreshObservation &&
scope.preservedSnapshotID != nil
let preservationBoundary = scope.confirmedMutationCompletedAt ?? scope.startedAt
let publicationAllowed = !wantsPreservation ||
((scope.observationPreservationAllowed ?? true) &&
(sharedWatermark.map { $0 <= preservationBoundary } ?? true))
let effectiveSucceeded = succeeded && publicationAllowed
let requestedCutoff = scope.invalidationCutoff(succeeded: effectiveSucceeded)
let cutoff = max(requestedCutoff, sharedWatermark ?? requestedCutoff)
let preservedSnapshotID = effectiveSucceeded ? scope.preservedSnapshotID : nil
if let preservedSnapshotID, let completedAt = scope.completedAt {
self.mutationTracker.preserveFreshObservation(
snapshotId: preservedSnapshotID,
startedAt: cutoff,
preservedAt: completedAt
)
}
let invalidated = await InteractionObservationInvalidator.invalidateAfterMutation(
targets: self.targets,
logger: self.logger,
reason: "\(scope.toolName) tool execution",
through: cutoff,
preserving: preservedSnapshotID,
preservedAt: preservedSnapshotID == nil ? nil : scope.completedAt
)
if !invalidated {
let retried = await InteractionObservationInvalidator.invalidateAfterMutation(
targets: self.targets,
logger: self.logger,
reason: "\(scope.toolName) tool execution retry",
through: cutoff,
preserving: preservedSnapshotID,
preservedAt: preservedSnapshotID == nil ? nil : scope.completedAt
)
return retried && (!succeeded || effectiveSucceeded)
}
return !succeeded || effectiveSucceeded
}
}

View File

@ -12,7 +12,7 @@ enum SnapshotValidation {
throw PeekabooError.snapshotNotFound(
"""
Snapshot '\(snapshotId)' was not found (or has no UI map). \
Run 'peekaboo see' again or omit --snapshot to use the most recent snapshot.
Run 'peekaboo see' again, omit --snapshot, or pass --snapshot latest.
"""
)
}

View File

@ -9,11 +9,6 @@ extension AppCommand {
@MainActor
struct LaunchSubcommand {
@MainActor
static var launcher: any ApplicationLaunching = ApplicationLaunchEnvironment.launcher
@MainActor
static var resolver: any ApplicationURLResolving = ApplicationURLResolverEnvironment.resolver
static let commandDescription = CommandDescription(
commandName: "launch",
abstract: "Launch an application",
@ -88,14 +83,12 @@ extension AppCommand {
self.prepare(using: runtime)
do {
try self.validateInputs()
let url = try self.resolveApplicationURL()
let launchedApp = try await self.launchApplication(at: url, name: self.displayName(for: url))
try await self.waitIfNeeded(for: launchedApp)
self.activateIfNeeded(launchedApp)
await self.invalidateFocusSnapshotIfNeeded()
self.resolvedRuntime.beginInteractionMutation()
let launchedApp = try await launchApplication()
await invalidateSnapshotsAfterLaunch()
self.renderLaunchSuccess(app: launchedApp)
} catch {
self.handleError(error)
handleError(error, customCode: applicationLaunchErrorCode(for: error))
throw ExitCode(1)
}
}
@ -112,41 +105,19 @@ extension AppCommand {
}
}
private func resolveApplicationURL() throws -> URL {
try Self.resolver.resolveApplication(appIdentifier: self.requestedAppIdentifier, bundleId: self.bundleId)
}
private func displayName(for url: URL) -> String {
(try? url.resourceValues(forKeys: [.localizedNameKey]).localizedName) ?? self.requestedAppIdentifier
}
private var requestedAppIdentifier: String {
self.app ?? self.bundleId ?? "unknown"
self.bundleId ?? self.app ?? "unknown"
}
private func waitIfNeeded(for app: any RunningApplicationHandle) async throws {
guard self.waitUntilReady else { return }
try await self.waitForApplicationReady(app)
}
private func activateIfNeeded(_ app: any RunningApplicationHandle) {
guard self.shouldFocusAfterLaunch else { return }
if !app.activate(options: []) {
self.logger
.error("Launch succeeded but failed to focus \(app.localizedName ?? self.requestedAppIdentifier)")
}
}
private func invalidateFocusSnapshotIfNeeded() async {
guard self.shouldFocusAfterLaunch else { return }
await InteractionObservationInvalidator.invalidateLatestSnapshot(
using: self.services.snapshots,
private func invalidateSnapshotsAfterLaunch() async {
await InteractionObservationInvalidator.invalidateAfterMutation(
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "app launch focus"
reason: "app launch"
)
}
private func renderLaunchSuccess(app: any RunningApplicationHandle) {
private func renderLaunchSuccess(app: ServiceApplicationInfo) {
struct LaunchResult: Codable {
let action: String
let app_name: String
@ -157,10 +128,10 @@ extension AppCommand {
let data = LaunchResult(
action: "launch",
app_name: app.localizedName ?? self.requestedAppIdentifier,
app_name: app.name,
bundle_id: app.bundleIdentifier ?? "unknown",
pid: app.processIdentifier,
is_ready: app.isFinishedLaunching
is_ready: app.isFinishedLaunching ?? !self.waitUntilReady
)
AutomationEventLogger.log(
.app,
@ -168,34 +139,22 @@ extension AppCommand {
)
output(data) {
print("✓ Launched \(app.localizedName ?? self.requestedAppIdentifier) (PID: \(app.processIdentifier))")
print("✓ Launched \(app.name) (PID: \(app.processIdentifier))")
}
}
private func launchApplication(at url: URL, name: String) async throws -> any RunningApplicationHandle {
if self.openTargets.isEmpty {
return try await Self.launcher.launchApplication(at: url, activates: self.shouldFocusAfterLaunch)
} else {
let urls = try self.openTargets.map { try Self.resolveOpenTarget($0) }
return try await Self.launcher.launchApplication(
url,
opening: urls,
activates: self.shouldFocusAfterLaunch
)
}
}
private func waitForApplicationReady(
_ app: any RunningApplicationHandle,
timeout: TimeInterval = 10
) async throws {
let startTime = Date()
while !app.isFinishedLaunching {
if Date().timeIntervalSince(startTime) > timeout {
throw PeekabooError.timeout("Application did not become ready within \(Int(timeout)) seconds")
}
try await Task.sleep(nanoseconds: 100_000_000) // 0.1 second
}
private func launchApplication() async throws -> ServiceApplicationInfo {
let urls = try openTargets.map { try Self.resolveOpenTarget($0) }
let applicationIdentifier = self.bundleId == nil
? self.app.map { ApplicationIdentifierResolver.resolve($0) }
: nil
return try await self.services.applications.launchApplication(request: ApplicationLaunchRequest(
applicationIdentifier: applicationIdentifier,
applicationBundleIdentifier: self.bundleId,
openURLs: urls,
activates: self.shouldFocusAfterLaunch,
waitUntilReady: self.waitUntilReady
))
}
static func resolveOpenTarget(
@ -229,10 +188,10 @@ extension AppCommand.LaunchSubcommand: AsyncRuntimeCommand, ErrorHandlingCommand
@MainActor
extension AppCommand.LaunchSubcommand: CommanderBindableCommand {
mutating func applyCommanderValues(_ values: CommanderBindableValues) throws {
self.app = try values.decodeOptionalPositional(0, label: "app")
self.bundleId = values.singleOption("bundleId")
self.waitUntilReady = values.flag("waitUntilReady")
self.noFocus = values.flag("noFocus")
self.openTargets = values.optionValues("open")
app = try values.decodeOptionalPositional(0, label: "app")
bundleId = values.singleOption("bundleId")
waitUntilReady = values.flag("waitUntilReady")
noFocus = values.flag("noFocus")
openTargets = values.optionValues("open")
}
}

View File

@ -45,6 +45,23 @@ extension AppCommand {
self.resolvedRuntime.configuration.jsonOutput
}
static func filteredApplications(
_ applications: [ServiceApplicationInfo],
includeHidden: Bool,
includeBackground: Bool
) -> [ServiceApplicationInfo] {
applications.filter { app in
if !includeHidden, app.isHidden {
return false
}
if !includeBackground,
app.activationPolicy == .accessory || app.activationPolicy == .prohibited {
return false
}
return true
}
}
/// Enumerate running applications, apply filtering flags, and emit the chosen output representation.
@MainActor
mutating func run(using runtime: CommandRuntime) async throws {
@ -53,12 +70,11 @@ extension AppCommand {
do {
let appsOutput = try await self.services.applications.listApplications()
// Filter based on flags
let filtered = appsOutput.data.applications.filter { app in
if !self.includeHidden && app.isHidden { return false }
if !self.includeBackground && app.name.isEmpty { return false }
return true
}
let filtered = Self.filteredApplications(
appsOutput.data.applications,
includeHidden: self.includeHidden,
includeBackground: self.includeBackground
)
struct AppInfo: Codable {
let name: String

View File

@ -114,6 +114,12 @@ extension AppCommand {
var results: [AppQuitInfo] = []
for target in quitApps {
if target.pid == self.resolvedRuntime.selectedRemoteHostProcessIdentifier {
throw PeekabooError.invalidInput(
"Cannot quit the daemon host executing this command; use a different runtime host"
)
}
self.resolvedRuntime.beginInteractionMutation()
let success = await (try? self.services.applications.quitApplication(
identifier: target.identifier,
force: self.force

View File

@ -9,11 +9,6 @@ extension AppCommand {
@MainActor
struct RelaunchSubcommand {
@MainActor
static var launcher: any ApplicationLaunching = ApplicationLaunchEnvironment.launcher
@MainActor
static var resolver: any ApplicationURLResolving = ApplicationURLResolverEnvironment.resolver
static let commandDescription = CommandDescription(
commandName: "relaunch",
abstract: "Quit and relaunch an application"
@ -68,41 +63,45 @@ extension AppCommand {
self.runtime = runtime
do {
guard self.resolvedRuntime.applicationRelaunchAllowed else {
throw PeekabooError.serviceUnavailable(
"Relaunch requires a surviving daemon host; the selected bridge is unavailable or GUI-hosted"
)
}
// Find the application first
let appIdentifier = try self.resolveApplicationIdentifier()
let appInfo = try await resolveApplication(appIdentifier, services: self.services)
let appIdentifier = try resolveApplicationIdentifier()
let appInfo = try await resolveApplication(appIdentifier, services: services)
let originalPID = appInfo.processIdentifier
guard originalPID != self.resolvedRuntime.selectedRemoteHostProcessIdentifier else {
throw PeekabooError.serviceUnavailable(
"Cannot relaunch the selected daemon through itself; use another bridge host"
)
}
let processIdentifier = "PID:\(originalPID)"
// Step 1: Quit the app
let quitSuccess = try await self.services.applications.quitApplication(
identifier: processIdentifier,
force: self.force
guard self.wait.isFinite, self.wait >= 0 else {
throw PeekabooError.invalidInput("Relaunch wait must be a finite, non-negative number of seconds")
}
let launchIdentifier = appInfo.bundleIdentifier == nil ? (appInfo.bundlePath ?? appInfo.name) : nil
self.resolvedRuntime.beginInteractionMutation()
let launchedApp = try await services.applications.relaunchApplication(
request: ApplicationRelaunchRequest(
targetIdentifier: processIdentifier,
launchRequest: ApplicationLaunchRequest(
applicationIdentifier: launchIdentifier,
applicationBundleIdentifier: appInfo.bundleIdentifier,
activates: true,
waitUntilReady: self.waitUntilReady
),
force: self.force,
waitSeconds: self.wait
)
)
await InteractionObservationInvalidator.invalidateAfterMutation(
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "app relaunch focus"
)
if !quitSuccess {
throw PeekabooError
.commandFailed(
"Failed to quit \(appInfo.name) (PID: \(originalPID)). The app may have unsaved changes."
)
}
// Wait for the app to actually terminate
try await self.waitUntilTerminated(identifier: processIdentifier, appName: appInfo.name)
// Step 2: Wait the specified duration
if self.wait > 0 {
try await Task.sleep(nanoseconds: UInt64(self.wait * 1_000_000_000))
}
// Step 3: Launch the app
let appURL = try self.resolveLaunchURL(for: appInfo)
let launchedApp = try await Self.launcher.launchApplication(at: appURL, activates: true)
// Wait until ready if requested
if self.waitUntilReady {
try await self.waitUntilReady(launchedApp)
}
struct RelaunchResult: Codable {
let action: String
@ -123,53 +122,22 @@ extension AppCommand {
bundle_id: appInfo.bundleIdentifier,
quit_forced: self.force,
wait_time: self.wait,
launch_success: launchedApp.isFinishedLaunching || !self.waitUntilReady
launch_success: !self.waitUntilReady || launchedApp.isFinishedLaunching == true
)
output(data) {
print("✓ Relaunched \(appInfo.name)")
print(" Old PID: \(originalPID) → New PID: \(launchedApp.processIdentifier)")
if self.waitUntilReady {
print(" Status: \(launchedApp.isFinishedLaunching ? "Ready" : "Launching...")")
print(" Status: \(launchedApp.isFinishedLaunching == true ? "Ready" : "Launching...")")
}
}
} catch {
handleError(error)
handleError(error, customCode: applicationLaunchErrorCode(for: error))
throw ExitCode(1)
}
}
private func waitUntilTerminated(identifier: String, appName: String) async throws {
var terminateWaitTime = 0.0
while await self.services.applications.isApplicationRunning(identifier: identifier),
terminateWaitTime < 5.0 {
try await Task.sleep(nanoseconds: 100_000_000) // 0.1 seconds
terminateWaitTime += 0.1
}
if await self.services.applications.isApplicationRunning(identifier: identifier) {
throw PeekabooError.timeout("App \(appName) did not terminate within 5 seconds")
}
}
private func resolveLaunchURL(for appInfo: ServiceApplicationInfo) throws -> URL {
if let bundleID = appInfo.bundleIdentifier {
return try Self.resolver.resolveBundleIdentifier(bundleID)
}
if let bundlePath = appInfo.bundlePath {
return URL(fileURLWithPath: bundlePath)
}
throw PeekabooError.commandFailed("No bundle ID or path available to relaunch \(appInfo.name)")
}
private func waitUntilReady(_ app: any RunningApplicationHandle) async throws {
var readyWaitTime = 0.0
while !app.isFinishedLaunching && readyWaitTime < 10.0 {
try await Task.sleep(nanoseconds: 100_000_000) // 0.1 seconds
readyWaitTime += 0.1
}
}
}
}
@ -180,12 +148,12 @@ extension AppCommand.RelaunchSubcommand: AsyncRuntimeCommand, ErrorHandlingComma
@MainActor
extension AppCommand.RelaunchSubcommand: CommanderBindableCommand {
mutating func applyCommanderValues(_ values: CommanderBindableValues) throws {
self.app = try values.decodePositional(0, label: "app")
self.pid = try values.decodeOption("pid", as: Int32.self)
app = try values.decodePositional(0, label: "app")
pid = try values.decodeOption("pid", as: Int32.self)
if let wait: TimeInterval = try values.decodeOption("wait", as: TimeInterval.self) {
self.wait = wait
}
self.force = values.flag("force")
self.waitUntilReady = values.flag("waitUntilReady")
force = values.flag("force")
waitUntilReady = values.flag("waitUntilReady")
}
}

View File

@ -98,6 +98,7 @@ struct AppCommand: ParsableCommand {
let appIdentifier = try self.resolveApplicationIdentifier()
let appInfo = try await resolveApplication(appIdentifier, services: self.services)
self.resolvedRuntime.beginInteractionMutation()
try await self.services.applications.hideApplication(identifier: appIdentifier)
let data = [
@ -178,6 +179,7 @@ struct AppCommand: ParsableCommand {
let appIdentifier = try self.resolveApplicationIdentifier()
let appInfo = try await resolveApplication(appIdentifier, services: self.services)
self.resolvedRuntime.beginInteractionMutation()
try await self.services.applications.unhideApplication(identifier: appIdentifier)
// Activate if requested
@ -271,6 +273,7 @@ struct AppCommand: ParsableCommand {
if self.verify {
throw ValidationError("Verify is only supported with --to (not --cycle)")
}
self.resolvedRuntime.beginInteractionMutation()
try await self.services.automation.hotkey(keys: "cmd,tab", holdDuration: 0)
struct CycleResult: Codable {
@ -280,8 +283,8 @@ struct AppCommand: ParsableCommand {
let data = CycleResult(action: "cycle", success: true)
await InteractionObservationInvalidator.invalidateLatestSnapshot(
using: self.services.snapshots,
await InteractionObservationInvalidator.invalidateAfterMutation(
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "app switch cycle"
)
@ -291,7 +294,13 @@ struct AppCommand: ParsableCommand {
AutomationEventLogger.log(.app, "switch action=cycle success=true")
} else if let targetApp = to {
let appInfo = try await resolveApplication(targetApp, services: self.services)
self.resolvedRuntime.beginInteractionMutation()
try await self.services.applications.activateApplication(identifier: appInfo.name)
await InteractionObservationInvalidator.invalidateAfterMutation(
targets: self.resolvedRuntime.interactionMutationTargets,
logger: self.logger,
reason: "app switch"
)
if self.verify {
try await self.verifyFrontmostApp(expected: appInfo)
}
@ -310,11 +319,6 @@ struct AppCommand: ParsableCommand {
success: true
)
await InteractionObservationInvalidator.invalidateLatestSnapshot(
using: self.services.snapshots,
logger: self.logger,
reason: "app switch"
)
output(data) {
print("✓ Switched to \(appInfo.name)")
}

View File

@ -0,0 +1,17 @@
import Foundation
enum ApplicationIdentifierResolver {
static func resolve(
_ value: String,
cwd: String = FileManager.default.currentDirectoryPath
) -> String {
let trimmed = value.trimmingCharacters(in: .whitespacesAndNewlines)
guard trimmed.contains("/") else { return trimmed }
let expanded = NSString(string: trimmed).expandingTildeInPath
let absolutePath = expanded.hasPrefix("/")
? expanded
: NSString(string: cwd).appendingPathComponent(expanded)
return URL(fileURLWithPath: absolutePath).standardizedFileURL.path
}
}

View File

@ -1,124 +0,0 @@
import AppKit
import Foundation
import PeekabooCore
// MARK: - Running Application Handle
@MainActor
protocol RunningApplicationHandle {
var localizedName: String? { get }
var bundleIdentifier: String? { get }
var processIdentifier: Int32 { get }
var isFinishedLaunching: Bool { get }
var isActive: Bool { get }
@discardableResult
func activate(options: NSApplication.ActivationOptions) -> Bool
}
@MainActor
extension NSRunningApplication: RunningApplicationHandle {}
// MARK: - Launcher abstraction
@MainActor
protocol ApplicationLaunching {
func launchApplication(at url: URL, activates: Bool) async throws -> any RunningApplicationHandle
func launchApplication(_ url: URL, opening documents: [URL], activates: Bool) async throws
-> any RunningApplicationHandle
func openTarget(_ targetURL: URL, handlerURL: URL?, activates: Bool) async throws -> any RunningApplicationHandle
}
@MainActor
enum ApplicationLaunchEnvironment {
static var launcher: any ApplicationLaunching = NSWorkspaceApplicationLauncher()
}
@MainActor
final class NSWorkspaceApplicationLauncher: ApplicationLaunching {
func launchApplication(at url: URL, activates: Bool) async throws -> any RunningApplicationHandle {
let configuration = NSWorkspace.OpenConfiguration()
configuration.activates = activates
return try await NSWorkspace.shared.openApplication(at: url, configuration: configuration)
}
func launchApplication(
_ url: URL,
opening documents: [URL],
activates: Bool
) async throws -> any RunningApplicationHandle {
let configuration = NSWorkspace.OpenConfiguration()
configuration.activates = activates
return try await NSWorkspace.shared.open(documents, withApplicationAt: url, configuration: configuration)
}
func openTarget(_ targetURL: URL, handlerURL: URL?, activates: Bool) async throws -> any RunningApplicationHandle {
if let handlerURL {
return try await self.launchApplication(handlerURL, opening: [targetURL], activates: activates)
} else {
let configuration = NSWorkspace.OpenConfiguration()
configuration.activates = activates
return try await NSWorkspace.shared.open(targetURL, configuration: configuration)
}
}
}
// MARK: - Application URL resolver
@MainActor
protocol ApplicationURLResolving {
func resolveApplication(appIdentifier: String, bundleId: String?) throws -> URL
func resolveBundleIdentifier(_ bundleId: String) throws -> URL
}
@MainActor
enum ApplicationURLResolverEnvironment {
static var resolver: any ApplicationURLResolving = DefaultApplicationURLResolver()
}
@MainActor
final class DefaultApplicationURLResolver: ApplicationURLResolving {
func resolveApplication(appIdentifier: String, bundleId: String?) throws -> URL {
if let bundleId {
return try self.resolveBundleIdentifier(bundleId)
}
if let bundleURL = NSWorkspace.shared.urlForApplication(withBundleIdentifier: appIdentifier) {
return bundleURL
}
if let namedURL = self.findApplicationByName(appIdentifier) {
return namedURL
}
if appIdentifier.contains("/") {
return URL(fileURLWithPath: appIdentifier)
}
throw NotFoundError.application(appIdentifier)
}
func resolveBundleIdentifier(_ bundleId: String) throws -> URL {
guard let url = NSWorkspace.shared.urlForApplication(withBundleIdentifier: bundleId) else {
throw NotFoundError.application("Bundle ID: \(bundleId)")
}
return url
}
private func findApplicationByName(_ name: String) -> URL? {
let searchPaths = [
"/Applications",
"/System/Applications",
"~/Applications",
"/Applications/Utilities"
].map { NSString(string: $0).expandingTildeInPath }
for path in searchPaths {
let appPath = "\(path)/\(name).app"
if FileManager.default.fileExists(atPath: appPath) {
return URL(fileURLWithPath: appPath)
}
}
return nil
}
}

View File

@ -101,6 +101,9 @@ struct ClipboardCommand: OutputFormattable, RuntimeOptionsConfigurable {
self.logger.setJsonOutputMode(self.jsonOutput)
let action = try self.resolvedAction()
if Self.actionMayMutate(action) {
self.resolvedRuntime.beginInteractionMutation()
}
switch action.lowercased() {
case "get":
try self.handleGet()
@ -121,6 +124,13 @@ struct ClipboardCommand: OutputFormattable, RuntimeOptionsConfigurable {
// MARK: - Actions
static func actionMayMutate(_ action: String?) -> Bool {
guard let action = action?.trimmingCharacters(in: .whitespacesAndNewlines).lowercased() else {
return false
}
return ["set", "load", "clear", "restore"].contains(action)
}
private func resolvedAction() throws -> String {
let positionalAction = self.action?.trimmingCharacters(in: .whitespacesAndNewlines)
let optionAction = self.actionOption?.trimmingCharacters(in: .whitespacesAndNewlines)

View File

@ -15,7 +15,7 @@ extension DaemonCommand {
}
}
@Option(name: .long, help: "Daemon mode (manual, mcp)")
@Option(name: .long, help: "Daemon mode (manual, auto)")
var mode: String = "manual"
@Option(name: .long, help: "Override bridge socket path")
@ -33,23 +33,41 @@ extension DaemonCommand {
mutating func run(using runtime: CommandRuntime) async throws {
self.runtime = runtime
let pollInterval = TimeInterval(Double(self.pollIntervalMs ?? 1000) / 1000.0)
let socketPath = self.bridgeSocket ?? PeekabooBridgeConstants.peekabooSocketPath
let normalizedMode = self.mode.lowercased()
let config: PeekabooDaemon.Configuration = if normalizedMode == "auto" {
.auto(
bridgeSocketPath: socketPath,
windowPollInterval: pollInterval,
idleTimeout: self.idleTimeoutSeconds ?? CommandRuntime.defaultDaemonIdleTimeoutSeconds
)
} else if normalizedMode == "mcp" {
.mcp(bridgeSocketPath: socketPath, windowPollInterval: pollInterval)
} else {
.manual(bridgeSocketPath: socketPath, windowPollInterval: pollInterval)
}
let config = try Self.configuration(
mode: self.mode,
bridgeSocket: self.bridgeSocket,
pollInterval: pollInterval,
idleTimeoutSeconds: self.idleTimeoutSeconds
)
let daemon = PeekabooDaemon(configuration: config)
await daemon.runUntilStop()
try await daemon.runUntilStopChecked()
}
static func configuration(
mode: String,
bridgeSocket: String?,
pollInterval: TimeInterval,
idleTimeoutSeconds: Double?
) throws -> PeekabooDaemon.Configuration {
let normalizedMode = mode.lowercased()
if normalizedMode == "mcp" {
throw ValidationError(
"Standalone MCP daemon mode is unavailable; use `peekaboo mcp` so the MCP transport owns its lifecycle."
)
}
return if normalizedMode == "auto" {
.auto(
bridgeSocketPath: bridgeSocket ?? PeekabooBridgeConstants.daemonSocketPath,
windowPollInterval: pollInterval,
idleTimeout: idleTimeoutSeconds ?? CommandRuntime.defaultDaemonIdleTimeoutSeconds
)
} else {
.manual(
bridgeSocketPath: bridgeSocket ?? PeekabooBridgeConstants.daemonSocketPath,
windowPollInterval: pollInterval
)
}
}
mutating func applyCommanderValues(_ values: CommanderBindableValues) throws {

View File

@ -1,4 +1,5 @@
import Commander
import Darwin
import Foundation
import PeekabooBridge
import PeekabooFoundation
@ -48,60 +49,198 @@ extension DaemonCommand {
mutating func run(using runtime: CommandRuntime) async throws {
self.runtime = runtime
let socketPath = self.bridgeSocket ?? PeekabooBridgeConstants.peekabooSocketPath
let defaultSocketPath = PeekabooBridgeConstants.daemonSocketPath
let buildScopedSocketPath = DaemonLaunchPolicy.buildScopedDaemonSocketPath(
daemonSocketPath: defaultSocketPath,
runtimeBuildIdentity: DaemonLaunchPolicy.runtimeBuildIdentity()
)
let lockHandle = DaemonPaths.openDaemonStartupLock()
if let fileDescriptor = lockHandle?.fileDescriptor {
flock(fileDescriptor, LOCK_EX)
}
defer {
if let fileDescriptor = lockHandle?.fileDescriptor {
flock(fileDescriptor, LOCK_UN)
}
try? lockHandle?.close()
}
let targets = await DaemonControlResolver.targets(explicitSocket: self.bridgeSocket)
let action = DaemonControlPlanner.startAction(
targets: targets,
explicitSocket: self.bridgeSocket,
defaultSocketPath: defaultSocketPath,
buildScopedSocketPath: buildScopedSocketPath
)
guard let destination = try await self.resolveDestination(action: action, targets: targets) else { return }
let socketPath = destination.socketPath
let promotionTarget = destination.promotionTarget
let client = DaemonControlClient(socketPath: socketPath)
if let status = await client.fetchStatus() {
self.output(status) {
DaemonStatusPrinter.render(status: status)
let migratesLegacyTarget = DaemonControlPlanner.shouldMigrateLegacyTarget(
explicitSocket: self.bridgeSocket,
destinationSocketPath: socketPath,
defaultSocketPath: defaultSocketPath,
targets: targets
)
let legacyTarget = migratesLegacyTarget
? targets.first {
$0.isLegacyDefault && DaemonControlClient.isReusableDaemonStatus($0.status)
}
: nil
if let legacyTarget {
guard DaemonControlClient.supportsSafeMigration(legacyTarget.status) else {
throw PeekabooError.operationError(
message: "Legacy daemon predates safe migration; run `peekaboo daemon stop`, then retry start"
)
}
guard DaemonControlClient.isIdleForMigration(legacyTarget.status) else {
throw PeekabooError.operationError(
message: "Legacy daemon has active requests; retry start after they finish"
)
}
return
}
let executable = Self.resolveExecutablePath()
let process = Process()
process.executableURL = URL(fileURLWithPath: executable)
var args = ["daemon", "run", "--mode", "manual"]
if let bridgeSocket {
args.append(contentsOf: ["--bridge-socket", bridgeSocket])
}
if let pollIntervalMs {
args.append(contentsOf: ["--poll-interval-ms", "\(pollIntervalMs)"])
}
process.arguments = args
let logHandle = DaemonPaths.openDaemonLogForAppend() ?? FileHandle.nullDevice
process.standardOutput = logHandle
process.standardError = logHandle
process.standardInput = FileHandle.nullDevice
try process.run()
let deadline = Date().addingTimeInterval(TimeInterval(self.waitSeconds))
while Date() < deadline {
if let status = await client.fetchStatus() {
switch await DaemonLaunchPolicy.waitForDaemonSocketAvailability(
socketPath: socketPath,
client: client,
timeout: TimeInterval(max(self.waitSeconds, DaemonControlClient.defaultShutdownWaitSeconds))
) {
case .available:
break
case .reusableDaemon:
if let status = await client.fetchReusableDaemonStatus() {
guard status.mode == .manual else {
throw PeekabooError.operationError(
message: "Daemon at \(socketPath) remained in auto mode; retry start when it is idle"
)
}
self.output(status) {
DaemonStatusPrinter.render(status: status)
}
return
}
try await Task.sleep(nanoseconds: 200_000_000)
case .timedOut:
throw PeekabooError.operationError(message: "Daemon socket is still shutting down")
}
throw PeekabooError.operationError(message: "Daemon did not start within \(self.waitSeconds)s")
}
private static func resolveExecutablePath() -> String {
if let path = CommandLine.arguments.first {
return path
let arguments = DaemonLaunchPolicy.daemonArguments(
socketPath: socketPath,
mode: .manual,
pollIntervalMs: self.pollIntervalMs ?? promotionTarget?.status.windowTracker?.cgPollIntervalMs
?? legacyTarget?.status.windowTracker?.cgPollIntervalMs,
idleTimeoutSeconds: CommandRuntime.defaultDaemonIdleTimeoutSeconds
)
guard let replacement = await DaemonLaunchPolicy.launchDaemon(
socketPath: socketPath,
arguments: arguments,
timeout: TimeInterval(self.waitSeconds)
)
else {
throw PeekabooError.operationError(message: "Daemon did not start within \(self.waitSeconds)s")
}
if let legacyTarget {
do {
let stopped = try await legacyTarget.client.stopAndWait(
waitSeconds: max(self.waitSeconds, DaemonControlClient.defaultShutdownWaitSeconds),
expectedPID: legacyTarget.status.pid,
requireIdentityMatch: true
)
if !stopped,
await legacyTarget.client.fetchReusableDaemonStatus() != nil {
throw PeekabooError.operationError(message: "Legacy daemon refused migration stop request")
}
} catch {
if await legacyTarget.client.fetchReusableDaemonStatus() != nil {
let cleanedUp = await DaemonLaunchPolicy.stopReplacement(
client: client,
replacement: replacement
)
if !cleanedUp {
throw PeekabooError.operationError(
message: "Legacy migration failed and replacement cleanup timed out"
)
}
throw error
}
}
}
let status = replacement.status
self.output(status) {
DaemonStatusPrinter.render(status: status)
}
return "/usr/local/bin/peekaboo"
}
}
}
extension DaemonCommand.Start: AsyncRuntimeCommand {}
private struct DaemonStartDestination {
let socketPath: String
let promotionTarget: DaemonControlTarget?
}
@MainActor
extension DaemonCommand.Start {
fileprivate func resolveDestination(
action: DaemonStartAction,
targets: [DaemonControlTarget]
) async throws -> DaemonStartDestination? {
switch action {
case let .useExisting(socketPath):
guard let target = targets.first(where: { $0.client.socketPath == socketPath }) else {
throw PeekabooError.operationError(message: "Selected daemon disappeared; retry start")
}
self.output(target.status) {
DaemonStatusPrinter.render(status: target.status)
}
return nil
case let .launchManual(socketPath):
return DaemonStartDestination(socketPath: socketPath, promotionTarget: nil)
case let .promoteAutoToManual(socketPath, pid):
guard let target = targets.first(where: { $0.client.socketPath == socketPath }) else {
throw PeekabooError.operationError(message: "Selected daemon disappeared; retry start")
}
do {
guard try await target.client.stopAndWait(
waitSeconds: max(self.waitSeconds, DaemonControlClient.defaultShutdownWaitSeconds),
expectedPID: pid,
requireIdentityMatch: true
)
else {
throw PeekabooError.operationError(
message: "Daemon at \(socketPath) refused a safe stop; retry when it is idle"
)
}
} catch let error as PeekabooError {
throw error
} catch {
throw PeekabooError.operationError(
message: "Could not safely promote daemon at \(socketPath): \(error.localizedDescription)"
)
}
return DaemonStartDestination(socketPath: socketPath, promotionTarget: target)
case let .rejectBusy(socketPath):
throw PeekabooError.operationError(
message: "Daemon at \(socketPath) has active requests; retry start after they finish"
)
case let .rejectUnsafe(socketPath):
throw PeekabooError.operationError(
message: "Daemon at \(socketPath) cannot be safely promoted; " +
"stop it explicitly with `peekaboo daemon stop --bridge-socket \(socketPath)`, then retry"
)
case let .rejectIncompatible(socketPath):
throw PeekabooError.operationError(
message: "Daemon at \(socketPath) is incompatible with this build; " +
"stop it with `peekaboo daemon stop --bridge-socket \(socketPath)`, then retry"
)
}
}
}
@MainActor
extension DaemonCommand.Start: CommanderBindableCommand {
mutating func applyCommanderValues(_ values: CommanderBindableValues) throws {

View File

@ -40,12 +40,24 @@ extension DaemonCommand {
mutating func run(using runtime: CommandRuntime) async throws {
self.runtime = runtime
let socketPath = self.bridgeSocket ?? PeekabooBridgeConstants.peekabooSocketPath
let client = DaemonControlClient(socketPath: socketPath)
let targets = await DaemonControlResolver.targets(explicitSocket: self.bridgeSocket)
if let status = await client.fetchStatus() {
self.output(status) {
DaemonStatusPrinter.render(status: status)
if let target = DaemonControlPlanner.preferredStatusTarget(
targets,
explicitSocket: self.bridgeSocket
) {
let additionalSocketPaths = DaemonControlPlanner.additionalSocketPaths(
in: targets,
excluding: target
)
if !additionalSocketPaths.isEmpty {
self.logger.warn(
"Additional Peekaboo daemon detected at \(additionalSocketPaths.joined(separator: ", ")); " +
"reporting \(target.client.socketPath)"
)
}
self.output(target.status) {
DaemonStatusPrinter.render(status: target.status)
}
} else {
let stopped = PeekabooDaemonStatus(running: false)

View File

@ -18,8 +18,8 @@ extension DaemonCommand {
@Option(name: .long, help: "Override bridge socket path")
var bridgeSocket: String?
@Option(name: .long, help: "Seconds to wait for daemon shutdown (default 3)")
var waitSeconds: Int = 3
@Option(name: .long, help: "Seconds to wait for daemon shutdown (default 12)")
var waitSeconds: Int = DaemonControlClient.defaultShutdownWaitSeconds
@RuntimeStorage private var runtime: CommandRuntime?
var runtimeOptions = CommandRuntimeOptions()
@ -45,10 +45,9 @@ extension DaemonCommand {
mutating func run(using runtime: CommandRuntime) async throws {
self.runtime = runtime
let socketPath = self.bridgeSocket ?? PeekabooBridgeConstants.peekabooSocketPath
let client = DaemonControlClient(socketPath: socketPath)
let targets = await DaemonControlResolver.targets(explicitSocket: self.bridgeSocket)
guard let status = await client.fetchStatus() else {
guard !targets.isEmpty else {
let stopped = PeekabooDaemonStatus(running: false)
self.output(stopped) {
DaemonStatusPrinter.render(status: stopped)
@ -56,28 +55,25 @@ extension DaemonCommand {
return
}
if status.mode == nil {
if targets.contains(where: { $0.status.mode == nil }) {
throw PeekabooError.operationError(message: "Connected host does not support daemon stop")
}
let stopped = try await client.stopDaemon()
guard stopped else {
throw PeekabooError.operationError(message: "Daemon refused stop request")
}
let deadline = Date().addingTimeInterval(TimeInterval(self.waitSeconds))
while Date() < deadline {
if await client.fetchStatus() == nil {
let stopped = PeekabooDaemonStatus(running: false)
self.output(stopped) {
DaemonStatusPrinter.render(status: stopped)
}
return
for target in targets {
guard try await target.client.stopAndWait(
waitSeconds: self.waitSeconds,
expectedPID: target.status.pid,
requireIdentityMatch: DaemonControlClient.supportsSafeMigration(target.status)
)
else {
throw PeekabooError.operationError(message: "Daemon refused stop request")
}
try await Task.sleep(nanoseconds: 200_000_000)
}
throw PeekabooError.operationError(message: "Daemon did not stop within \(self.waitSeconds)s")
let stopped = PeekabooDaemonStatus(running: false)
self.output(stopped) {
DaemonStatusPrinter.render(status: stopped)
}
}
}
}

View File

@ -1,4 +1,5 @@
import Commander
import Darwin
import Foundation
import PeekabooBridge
@ -23,10 +24,22 @@ struct DaemonCommand: ParsableCommand {
}
struct DaemonControlClient {
static let defaultShutdownWaitSeconds =
Int(ceil(PeekabooBridgeConstants.defaultRequestTimeoutSeconds)) + 2
let socketPath: String
let requestTimeoutSec: TimeInterval
init(
socketPath: String,
requestTimeoutSec: TimeInterval = PeekabooBridgeConstants.defaultRequestTimeoutSeconds
) {
self.socketPath = socketPath
self.requestTimeoutSec = requestTimeoutSec
}
func fetchStatus() async -> PeekabooDaemonStatus? {
let client = PeekabooBridgeClient(socketPath: self.socketPath)
let client = PeekabooBridgeClient(socketPath: socketPath, requestTimeoutSec: self.requestTimeoutSec)
do {
return try await client.daemonStatus()
} catch let envelope as PeekabooBridgeErrorEnvelope {
@ -39,11 +52,96 @@ struct DaemonControlClient {
}
}
func stopDaemon() async throws -> Bool {
let client = PeekabooBridgeClient(socketPath: self.socketPath)
func stopDaemon(expectedPID: pid_t? = nil) async throws -> Bool {
let client = PeekabooBridgeClient(socketPath: socketPath, requestTimeoutSec: self.requestTimeoutSec)
if let expectedPID {
return try await client.daemonStop(expectedPID: expectedPID)
}
return try await client.daemonStop()
}
func fetchControllableDaemonStatus() async -> PeekabooDaemonStatus? {
guard let status = await fetchStatus(),
Self.isControllableDaemonStatus(status)
else {
return nil
}
return status
}
func fetchReusableDaemonStatus() async -> PeekabooDaemonStatus? {
guard let status = await fetchStatus(),
Self.isReusableDaemonStatus(status)
else {
return nil
}
return status
}
static func isControllableDaemonStatus(_ status: PeekabooDaemonStatus) -> Bool {
status.mode != nil
}
static func isReusableDaemonStatus(_ status: PeekabooDaemonStatus) -> Bool {
status.mode == .auto || status.mode == .manual
}
static func migrationMode(for status: PeekabooDaemonStatus) -> PeekabooDaemonMode? {
self.isReusableDaemonStatus(status) ? status.mode : nil
}
static func isIdleForMigration(_ status: PeekabooDaemonStatus) -> Bool {
status.activity?.activeRequests ?? 0 == 0
}
static func supportsSafeMigration(_ status: PeekabooDaemonStatus) -> Bool {
status.supportsConditionalStop == true
}
func stopAndWait(
waitSeconds: Int,
expectedPID: pid_t?,
requireIdentityMatch: Bool = false
) async throws -> Bool {
var requestError: (any Error)?
var accepted = false
do {
accepted = try await self.stopDaemon(
expectedPID: requireIdentityMatch ? expectedPID : nil
)
} catch {
requestError = error
}
if !accepted, requestError == nil {
return false
}
let deadline = Date().addingTimeInterval(TimeInterval(waitSeconds))
while Date() < deadline {
if await self.fetchControllableDaemonStatus() == nil {
if let expectedPID {
if !Self.isProcessAlive(expectedPID) {
return true
}
} else if requestError == nil {
return true
}
}
try await Task.sleep(nanoseconds: 200_000_000)
}
if let requestError {
throw requestError
}
return false
}
private static func isProcessAlive(_ pid: pid_t) -> Bool {
if kill(pid, 0) == 0 { return true }
return errno != ESRCH
}
private func fallbackHandshake(client: PeekabooBridgeClient) async -> PeekabooDaemonStatus? {
let identity = PeekabooBridgeClientIdentity(
bundleIdentifier: Bundle.main.bundleIdentifier,
@ -54,9 +152,10 @@ struct DaemonControlClient {
do {
let handshake = try await client.handshake(client: identity)
let bridge = PeekabooDaemonBridgeStatus(
socketPath: self.socketPath,
socketPath: socketPath,
hostKind: handshake.hostKind,
allowedOperations: handshake.supportedOperations
allowedOperations: handshake.supportedOperations,
availableOperationNames: handshake.supportedOperations.map(\.rawValue).sorted()
)
return PeekabooDaemonStatus(
running: true,
@ -74,6 +173,308 @@ struct DaemonControlClient {
}
}
struct DaemonControlTarget {
let client: DaemonControlClient
let status: PeekabooDaemonStatus
let role: DaemonControlTargetRole
var isLegacyDefault: Bool {
self.role == .legacyDefault
}
}
enum DaemonControlTargetRole: Equatable {
case explicit
case defaultDaemon
case buildScopedDaemon
case legacyDefault
}
struct DaemonSocketFileCandidate: Equatable {
let path: String
let isSocket: Bool
let ownerUID: uid_t
}
enum DaemonStartAction: Equatable {
case useExisting(socketPath: String)
case launchManual(socketPath: String)
case promoteAutoToManual(socketPath: String, pid: pid_t)
case rejectBusy(socketPath: String)
case rejectUnsafe(socketPath: String)
case rejectIncompatible(socketPath: String)
}
enum DaemonControlPlanner {
private static let currentOperationNames: Set<String> = [
PeekabooBridgeOperation.launchApplicationWithOptions.rawValue,
PeekabooBridgeOperation.relaunchApplicationWithOptions.rawValue,
PeekabooBridgeOperation.invalidateImplicitLatestSnapshot.rawValue,
]
static func supportsCurrentDaemon(_ status: PeekabooDaemonStatus) -> Bool {
guard let bridge = status.bridge else { return false }
let availableNames = Set(
bridge.availableOperationNames ?? bridge.allowedOperations.map(\.rawValue)
)
return self.currentOperationNames.isSubset(of: availableNames)
}
static func preferredStatusTarget(
_ targets: [DaemonControlTarget],
explicitSocket: String?
) -> DaemonControlTarget? {
if explicitSocket != nil {
return targets.first
}
let defaultTarget = targets.first { $0.role == .defaultDaemon }
if let defaultTarget, self.isCurrentReusableTarget(defaultTarget) {
return defaultTarget
}
let scopedTargets = targets.filter { $0.role == .buildScopedDaemon }
if let scopedTarget = scopedTargets.first(where: self.isCurrentReusableTarget) {
return scopedTarget
}
return defaultTarget ?? scopedTargets.first ?? targets.first
}
static func additionalSocketPaths(
in targets: [DaemonControlTarget],
excluding selected: DaemonControlTarget
) -> [String] {
targets
.filter { $0.client.socketPath != selected.client.socketPath }
.map(\.client.socketPath)
}
static func startAction(
targets: [DaemonControlTarget],
explicitSocket: String?,
defaultSocketPath: String,
buildScopedSocketPath: String?
) -> DaemonStartAction {
if let explicitSocket {
guard let target = targets.first else {
return .launchManual(socketPath: explicitSocket)
}
return self.action(forExisting: target)
}
let defaultTarget = targets.first { $0.role == .defaultDaemon }
let scopedTargets = targets.filter { $0.role == .buildScopedDaemon }
if let defaultTarget, self.isCurrentReusableTarget(defaultTarget) {
return self.action(forExisting: defaultTarget)
}
if let scopedTarget = scopedTargets.first(where: self.isCurrentReusableTarget) {
return self.action(forExisting: scopedTarget)
}
if defaultTarget != nil, let buildScopedSocketPath {
if scopedTargets.contains(where: { $0.client.socketPath == buildScopedSocketPath }) {
return .rejectIncompatible(socketPath: buildScopedSocketPath)
}
return .launchManual(socketPath: buildScopedSocketPath)
}
if let defaultTarget {
return .rejectIncompatible(socketPath: defaultTarget.client.socketPath)
}
return .launchManual(socketPath: defaultSocketPath)
}
static func shouldMigrateLegacyTarget(
explicitSocket: String?,
destinationSocketPath: String,
defaultSocketPath: String,
targets: [DaemonControlTarget]
) -> Bool {
explicitSocket == nil &&
NSString(string: destinationSocketPath).standardizingPath ==
NSString(string: defaultSocketPath).standardizingPath &&
!targets.contains { $0.role == .defaultDaemon }
}
private static func action(forExisting target: DaemonControlTarget) -> DaemonStartAction {
guard DaemonControlClient.isReusableDaemonStatus(target.status) else {
return .rejectIncompatible(socketPath: target.client.socketPath)
}
guard target.status.mode == .auto else {
return .useExisting(socketPath: target.client.socketPath)
}
guard DaemonControlClient.isIdleForMigration(target.status) else {
return .rejectBusy(socketPath: target.client.socketPath)
}
guard DaemonControlClient.supportsSafeMigration(target.status),
let pid = target.status.pid
else {
return .rejectUnsafe(socketPath: target.client.socketPath)
}
return .promoteAutoToManual(socketPath: target.client.socketPath, pid: pid)
}
private static func isCurrentReusableTarget(_ target: DaemonControlTarget) -> Bool {
DaemonControlClient.isReusableDaemonStatus(target.status) && self.supportsCurrentDaemon(target.status)
}
}
enum DaemonControlResolver {
private static let historicalProbeTimeoutSeconds: TimeInterval = 1
static func defaultSocketPaths() -> [String] {
let buildScopedPath = DaemonLaunchPolicy.buildScopedDaemonSocketPath(
daemonSocketPath: PeekabooBridgeConstants.daemonSocketPath,
runtimeBuildIdentity: DaemonLaunchPolicy.runtimeBuildIdentity()
)
return [PeekabooBridgeConstants.daemonSocketPath, buildScopedPath].compactMap(\.self)
}
static func historicalBuildScopedSocketPaths(
daemonSocketPath: String,
currentBuildScopedSocketPath: String?,
candidates: [DaemonSocketFileCandidate],
currentUID: uid_t = getuid()
) -> [String] {
let daemonDirectory = Self.standardizedSocketPath(
URL(fileURLWithPath: daemonSocketPath).deletingLastPathComponent().path
)
let excludedPaths = Set([daemonSocketPath, currentBuildScopedSocketPath].compactMap { path in
path.map(Self.standardizedSocketPath)
})
return candidates
.filter { candidate in
candidate.isSocket &&
candidate.ownerUID == currentUID &&
Self.standardizedSocketPath(
URL(fileURLWithPath: candidate.path).deletingLastPathComponent().path
) == daemonDirectory &&
Self.isBuildScopedSocketName(URL(fileURLWithPath: candidate.path).lastPathComponent) &&
!excludedPaths.contains(Self.standardizedSocketPath(candidate.path))
}
.map(\.path)
.sorted()
}
static func isValidatedHistoricalTarget(
status: PeekabooDaemonStatus,
socketPath: String
) -> Bool {
guard status.running,
DaemonControlClient.isReusableDaemonStatus(status),
DaemonControlClient.supportsSafeMigration(status),
status.pid.map({ $0 > 0 }) == true,
let bridge = status.bridge,
bridge.hostKind == .onDemand,
standardizedSocketPath(bridge.socketPath) == standardizedSocketPath(socketPath)
else {
return false
}
let operationNames = Set(bridge.availableOperationNames ?? bridge.allowedOperations.map(\.rawValue))
return operationNames.contains(PeekabooBridgeOperation.daemonStatus.rawValue) &&
operationNames.contains(PeekabooBridgeOperation.daemonStop.rawValue)
}
static func targets(explicitSocket: String?) async -> [DaemonControlTarget] {
if let explicitSocket {
let client = DaemonControlClient(socketPath: explicitSocket)
guard let status = await client.fetchStatus() else { return [] }
return [DaemonControlTarget(client: client, status: status, role: .explicit)]
}
var targets: [DaemonControlTarget] = []
let defaultSocketPaths = self.defaultSocketPaths()
for (index, socketPath) in defaultSocketPaths.enumerated() {
let client = DaemonControlClient(socketPath: socketPath)
if let status = await client.fetchControllableDaemonStatus() {
targets.append(DaemonControlTarget(
client: client,
status: status,
role: index == 0 ? .defaultDaemon : .buildScopedDaemon
))
}
}
await targets.append(contentsOf: self.validatedHistoricalTargets(
daemonSocketPath: PeekabooBridgeConstants.daemonSocketPath,
currentBuildScopedSocketPath: defaultSocketPaths.dropFirst().first
))
let legacyClient = DaemonControlClient(socketPath: PeekabooBridgeConstants.peekabooSocketPath)
if let status = await legacyClient.fetchControllableDaemonStatus() {
targets.append(DaemonControlTarget(
client: legacyClient,
status: status,
role: .legacyDefault
))
}
return targets
}
static func validatedHistoricalTargets(
daemonSocketPath: String,
currentBuildScopedSocketPath: String?
) async -> [DaemonControlTarget] {
var targets: [DaemonControlTarget] = []
for socketPath in self.discoveredHistoricalBuildScopedSocketPaths(
daemonSocketPath: daemonSocketPath,
currentBuildScopedSocketPath: currentBuildScopedSocketPath
) {
let client = DaemonControlClient(
socketPath: socketPath,
requestTimeoutSec: self.historicalProbeTimeoutSeconds
)
guard let status = await client.fetchControllableDaemonStatus(),
self.isValidatedHistoricalTarget(status: status, socketPath: socketPath)
else {
continue
}
targets.append(DaemonControlTarget(
client: client,
status: status,
role: .buildScopedDaemon
))
}
return targets
}
private static func discoveredHistoricalBuildScopedSocketPaths(
daemonSocketPath: String,
currentBuildScopedSocketPath: String?
) -> [String] {
let directoryURL = URL(fileURLWithPath: daemonSocketPath).deletingLastPathComponent()
guard let urls = try? FileManager.default.contentsOfDirectory(
at: directoryURL,
includingPropertiesForKeys: nil,
options: [.skipsHiddenFiles]
)
else {
return []
}
let candidates = urls.compactMap { url -> DaemonSocketFileCandidate? in
var info = stat()
guard lstat(url.path, &info) == 0 else { return nil }
return DaemonSocketFileCandidate(
path: url.path,
isSocket: info.st_mode & mode_t(S_IFMT) == mode_t(S_IFSOCK),
ownerUID: info.st_uid
)
}
return self.historicalBuildScopedSocketPaths(
daemonSocketPath: daemonSocketPath,
currentBuildScopedSocketPath: currentBuildScopedSocketPath,
candidates: candidates
)
}
private static func isBuildScopedSocketName(_ name: String) -> Bool {
guard name.hasPrefix("daemon-"), name.hasSuffix(".sock") else { return false }
let hash = name.dropFirst("daemon-".count).dropLast(".sock".count)
return hash.count == 16 && hash.allSatisfy { ("0"..."9").contains($0) || ("a"..."f").contains($0) }
}
private static func standardizedSocketPath(_ path: String) -> String {
NSString(string: path).standardizingPath
}
}
enum DaemonPaths {
static func daemonLogURL() -> URL {
let root = FileManager.default.homeDirectoryForCurrentUser
@ -131,7 +532,7 @@ enum DaemonStatusPrinter {
print("------")
print("Socket: \(bridge.socketPath)")
print("Host: \(bridge.hostKind.rawValue)")
print("Ops: \(bridge.allowedOperations.count)")
print("Ops: \(bridge.availableOperationNames?.count ?? bridge.allowedOperations.count)")
}
if let permissions = status.permissions {

View File

@ -44,6 +44,9 @@ extension DialogCommand {
do {
try self.target.validate()
if self.focusOptions.autoFocus {
self.resolvedRuntime.beginInteractionMutation()
}
try await ensureFocused(
snapshotId: nil,
target: self.target,
@ -54,6 +57,7 @@ extension DialogCommand {
let resolvedWindowTitle = try await self.target.resolveWindowTitleOptional(services: self.services)
let appHint = try await DialogCommand.resolveDialogAppHint(target: self.target, services: self.services)
self.resolvedRuntime.beginInteractionMutation()
let result = try await self.services.dialogs.clickButton(
buttonText: self.button,
windowTitle: resolvedWindowTitle,

Some files were not shown because too many files have changed in this diff Show More