crabbox/docs/features/interactive-desktop-vnc.md

Interactive Desktop And VNC

Read when:

• choosing a desktop target for browser/UI QA;
• opening a lease with VNC or WebVNC;
• diagnosing stale WebVNC viewers, bridge disconnects, or broken desktop sessions;
• driving desktop input from agents without hand-written xdotool;
• deciding which layer owns desktop setup, browser state, screenshots, or credentials.

Crabbox treats desktop access as a lease capability, not a separate remote access product. A desktop lease still uses the normal Crabbox boundaries: provider lifecycle, per-lease SSH keys, SSH tunnels, idle expiry, cleanup, and run history. VNC is a way to inspect or drive the visible session inside that boundary.

Quick Start

crabbox warmup --desktop --browser
crabbox webvnc --id blue-lobster --open
crabbox webvnc status --id blue-lobster
crabbox desktop doctor --id blue-lobster
crabbox vnc --id blue-lobster --open
crabbox screenshot --id blue-lobster --output desktop.png

AWS Windows and EC2 Mac use the same VNC command once the desktop lease exists:

crabbox warmup --provider aws --target windows --desktop
crabbox vnc --id crimson-crab --open

CRABBOX_AWS_MAC_HOST_ID=h-... \
  crabbox warmup --provider aws --target macos --desktop --market on-demand
crabbox vnc --id silver-squid --open

Static hosts are explicit and host-managed:

crabbox vnc --provider ssh --target macos --static-host mac-studio.local --host-managed --open
crabbox vnc --provider ssh --target windows --static-host win-dev.local --host-managed --open

What Crabbox Owns

Crabbox owns:

• the lease lifecycle and cleanup;
• per-lease SSH keys and known_hosts scoping;
• SSH local forwarding to the target's loopback VNC service;
• generated per-lease VNC or OS passwords for managed desktop leases;
• desktop=true and browser=true lease metadata;
• screenshots and desktop launch commands that operate inside the lease.

Scenario systems such as Mantis own:

• product-specific login and app credentials;
• browser profile import/export;
• screenshots that prove a bug before and after a fix;
• PR comments, issue triage, and artifact summaries.

Support Matrix

Target	Managed by Crabbox	Desktop access	Primary page
Linux on Hetzner	Yes	Xvfb/XFCE/x11vnc over SSH tunnel	Linux VNC
Linux on AWS	Yes	Xvfb/XFCE/x11vnc over SSH tunnel	Linux VNC
Linux on Azure	Yes	Xvfb/XFCE/x11vnc over SSH tunnel	Linux VNC
AWS Windows	Yes	TightVNC over SSH tunnel	Windows VNC
AWS EC2 Mac	Yes	Screen Sharing/VNC over SSH tunnel	macOS VNC
Azure Windows	No	SSH/sync/run only	Azure
Static Linux	Host-managed	Existing loopback VNC service	Linux VNC
Static macOS	Host-managed	Existing Screen Sharing/VNC	macOS VNC
Static Windows	Host-managed	Existing VNC service	Windows VNC
Blacksmith Testbox	No	Not exposed through Crabbox VNC today	Blacksmith Testbox

Commands

Use crabbox webvnc for the authenticated coordinator portal. This is the preferred path for human demos because --open preloads the VNC password in the local browser fragment:

crabbox webvnc --id blue-lobster --open
crabbox webvnc status --id blue-lobster
crabbox webvnc reset --id blue-lobster --open

Use crabbox vnc for a native VNC client when WebVNC status/reset says the portal/browser path is unhealthy or when you need a native client feature:

crabbox vnc --id blue-lobster
crabbox vnc --id blue-lobster --network tailscale
crabbox vnc --id blue-lobster --open

WebVNC uses the same runner-side VNC service as crabbox vnc. The difference is the viewer path: a local crabbox webvnc process keeps an SSH tunnel open, connects to the coordinator with a one-use bridge ticket, and the browser uses bundled noVNC from the authenticated portal. The portal does not connect to the runner by itself; the local bridge must keep running.

WebVNC supports collaborative viewing. The local bridge keeps a warm pool of backend VNC sessions (default 4 slots), the first browser viewer controls the lease, and additional viewers join as read-only observers. Any viewer — a new observer or the prior controller — can press take over to become the controller; whoever loses control stays connected as an observer and sees who took over. Observer mode is intended for trusted shared leases; it is not a hostile-client security boundary.

The portal toolbar supports explicit clipboard exchange. Paste reads the local browser clipboard, forwards it to the remote VNC server, and sends the target paste shortcut. Copy-remote is enabled after the remote server publishes clipboard text and then writes that text to the local browser clipboard on click; browsers generally block fully automatic clipboard writes without a user gesture.

Use crabbox screenshot when you need a PNG without taking over the session:

crabbox screenshot --id blue-lobster --output desktop.png

Use crabbox artifacts when QA needs a durable proof bundle instead of a single screenshot:

crabbox artifacts collect --id blue-lobster --all --output artifacts/blue-lobster
crabbox artifacts publish --dir artifacts/blue-lobster --pr 123 --storage s3 --bucket qa-artifacts

Use crabbox desktop launch to start a browser or app inside the visible session without keeping the SSH command attached:

crabbox desktop launch --id blue-lobster --browser --url https://example.com --webvnc --open

For human demos, Crabbox keeps launched browsers windowed so the remote desktop panel, title bar, and surrounding session remain visible. Use desktop launch --fullscreen only when you intentionally want browser-only video or capture output.

Use crabbox desktop doctor --id <lease> before blaming WebVNC. It checks the lease's desktop session, VNC service, input tooling, browser binary, ffmpeg, screen geometry, and screenshot capture, then separately reports WebVNC bridge/viewer status with one-line repair suggestions.

Failure output is designed for rescue-first debugging. When a desktop command cannot prove the expected state, Crabbox prints the failed layer as problem: browser not launched, problem: input stack dead, problem: VNC bridge disconnected, problem: WebVNC daemon not running, or similar, followed by an exact rescue: command. WebVNC status/reset also prints the exact native crabbox vnc ... --open fallback when the native viewer is the better next step.

Use first-class input helpers instead of hand-rolled xdotool:

crabbox desktop click --id blue-lobster --x 640 --y 420
crabbox desktop paste --id blue-lobster --text "peter@example.com"
printf 'peter@example.com' | crabbox desktop paste --id blue-lobster
crabbox desktop type --id blue-lobster --text "hello"
crabbox desktop key --id blue-lobster ctrl+l
crabbox desktop key blue-lobster ctrl+l

Prefer desktop paste or symbol-aware desktop type for emails, passwords, URLs, and text containing characters such as @ or +; raw key-symbol typing can vary with the target X keyboard layout. desktop key is for shortcuts and special keys, and supports both --id <lease> <keys> and positional <lease> <keys> forms.

Network Model

Managed VNC is tunnel-first:

• VNC binds to 127.0.0.1:5900 on the target.
• The cloud firewall/security group opens SSH only, not VNC.
• crabbox vnc forwards a local port such as localhost:5901 to remote 127.0.0.1:5900.
• --network tailscale changes only the SSH endpoint used by that tunnel.
• WebVNC keeps the same local SSH tunnel and adds an authenticated browser websocket through the coordinator.
• WebVNC browser websockets are paired with local bridge backend sessions inside the coordinator Durable Object. One viewer is the controller; other viewers are observers until they press take over. If a browser view disconnects, only its paired backend session is reset and the local command reconnects a fresh bridge slot for the next portal retry.
• crabbox webvnc status reports the local daemon pid/log, SSH tunnel command, target VNC reachability, coordinator bridge/viewer state, recent bridge events, portal URL/password, and the exact native crabbox vnc ... --open fallback. The fallback preserves explicit --network public or --network tailscale selections.
• crabbox webvnc reset closes only the selected lease's WebVNC sockets, stops only that lease's verified local WebVNC daemon, restarts the target desktop/VNC services, then prints the fresh portal URL.
• WebVNC and desktop commands print rescue commands inline when the bridge, viewer, browser launch, VNC target, or input stack fails, so operators do not need to dig through troubleshooting docs during a demo.

Crabbox does not bind managed VNC directly to a public IP or Tailscale 100.x address. Static hosts can expose direct host:5900 only when the operator has already made that endpoint reachable on a trusted network.

Browser State

--browser guarantees a browser binary and env such as BROWSER and CHROME_BIN; it does not create, unlock, sync, or migrate a logged-in profile. On managed Linux leases, these env vars point to a Crabbox wrapper that disables Chrome/Chromium first-run and default-browser prompts for repeatable VNC use. On managed targets, manual browser login through VNC lasts only for that lease unless the caller intentionally exports an artifact. On static hosts, any existing browser profile belongs to that host.

For repeatable logged-in tests, use scenario-owned state such as a Playwright storage-state file or an app-specific short-lived token. Avoid syncing full browser profile directories between operating systems; browser credentials are often machine- and user-encrypted.

Security Rules

• Never expose managed VNC directly to the public internet.
• Do not expose managed VNC directly on a Tailscale interface.
• Prefer SSH local forwarding such as localhost:5901 -> 127.0.0.1:5900.
• Generate per-lease passwords for managed desktop leases.
• Redact passwords from logs, provider metadata, and run records.
• Keep TTL and idle-timeout cleanup in force.
• Require --host-managed before opening static-host VNC prompts.

Where To Go Next

• Linux VNC: Hetzner/AWS Linux desktop services and static Linux.
• Windows VNC: AWS Windows, native Windows static hosts, and WSL2 boundaries.
• macOS VNC: AWS EC2 Mac and static Mac Screen Sharing.
• AWS: AWS target matrix, capacity, AMIs, and EC2 Mac host requirements.
• Hetzner: Linux-only managed Hetzner behavior.
• Blacksmith Testbox: delegated Testbox behavior and why VNC is not a Crabbox feature there yet.
• vnc command, webvnc command, screenshot command, desktop command, artifacts command, egress command.
• Mediated egress: per-app browser/app egress through the operator machine for Discord, Slack, and similar source-IP-sensitive QA.