openclaw/openclaw-windows-node
openclaw/openclaw-windows-node
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity
392 Commits 134 Branches 17 Tags 7.1 MiB
master
Commit Graph

50 Commits

Author SHA1 Message Date
Régis Brid
b0ba9affa2
audio: Whisper STT + Piper TTS as MCP-callable node capabilities (#288) 
* Add Windows STT transcribe capability

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* stt: privacy hardening, localization, and test coverage

Review-driven cleanup on top of the initial stt.transcribe capability.
No behavior change for successful invocations.

Privacy:
* SttCapability no longer echoes the caller-supplied language tag in the
  "Invalid language tag" error, and no longer interpolates the underlying
  exception's Message into "Transcribe failed". Both could end up in the
  recent-activity stream and BuildSupportBundle output, which can be
  shared off-device. Full detail still goes to the local logger.
* App.OnNodeInvokeCompleted now sanitizes failed-invoke details for
  privacy-sensitive commands (stt.transcribe, camera.snap/clip,
  screen.snapshot/record). Recent activity and support bundles record
  only "privacy-sensitive | <ms> | error" instead of the raw error
  string. Non-privacy-sensitive commands keep the error text since it is
  useful for diagnostics and does not carry mic/camera args.
* Models.cs PermissionDiagnostics microphone detail now mentions
  stt.transcribe instead of "future voice features", so users hitting
  0x800455A0 see microphone in their permissions checklist as relevant.

Refactors for testability (no behavior change):
* New Services/NodeInvokeActivityFormatter.cs owns GetPrivacyClass and
  BuildDetails. App.OnNodeInvokeCompleted delegates to it.
* New Services/NodeCapabilityGating.cs owns the optional-capability
  predicates. NodeService.RegisterCapabilities calls into it instead of
  inlining "_settings?.NodeXxxEnabled" checks. Privacy-sensitive
  defaults stay off; everything else stays default-on.
* Both helpers are linked into OpenClaw.Tray.Tests.

Localization:
* SettingsWindow.xaml gains x:Uid for every TTS and STT control. The
  literal Text/Header/PlaceholderText values are kept as dev-time
  fallbacks, matching the SettingsTokenTextBox and SettingsMcpDescription
  pattern already in the file.
* en-us, fr-fr, nl-nl, zh-cn, and zh-tw .resw files gain matching
  entries for the 14 new TTS/STT keys. Brand names (ElevenLabs),
  command names (tts.speak, stt.transcribe, gateway.nodes.allowCommands,
  MSIX), BCP-47 tags, and the eleven_multilingual_v2 model identifier
  are kept verbatim across all locales.
* SettingsMcpDescription.Text in all five locales now lists "microphone"
  and "speakers" alongside camera/screen/canvas so the local MCP-server
  description reflects the full Phase 1 + Phase 2 voice surface.

Tests:
* Two new privacy regression tests in CapabilityTests verify that an
  invalid language and a thrown handler exception never leak their text
  into the response error.
* New NodeInvokeActivityFormatterTests pin the privacy-class table, the
  sanitized details for privacy-sensitive failures, and the full
  ActivityStreamService.BuildSupportBundle path.
* New NodeCapabilityGatingTests pin that tts.speak and stt.transcribe
  default off (including for null settings) and that the two capabilities
  are independent consent surfaces.
* New SettingsWindowLocalizationCoverageTests parses SettingsWindow.xaml
  and asserts every new TTS/STT x:Uid resolves to the expected
  .Header/.Text/.Content/.PlaceholderText keys in en-us.
* ActivityStreamServiceTests and NodeInvokeActivityFormatterTests now
  share a non-parallel xUnit collection because ActivityStreamService is
  a static singleton; running both classes in parallel could otherwise
  cause flaky support-bundle assertions.
* NodeCapabilityGatingTests cleans up its temp settings directories.

Cleanup:
* Drop "Phase 2" wording from SpeechToTextService.cs; the resw section
  comments referring to "Phase 1 TTS / Phase 2 STT" are likewise
  reworded to plain "TTS / STT settings". Phase numbering is a planning
  artifact and should not appear in the codebase.

Validation:
* .\build.ps1
* dotnet test tests/OpenClaw.Shared.Tests/OpenClaw.Shared.Tests.csproj
  --no-restore  (1173 passed, 20 skipped)
* dotnet test tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj
  --no-restore  (465 passed)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove fake/sample data from 6 UI pages

Replace constructor-injected sample data with empty/loading states:
- UsagePage: remove fabricated provider costs and daily data
- SessionsPage: remove 3 fake AI conversation sessions
- NodesPage: remove fake Desktop-PC/MacBook-Pro nodes
- ChannelsPage: remove fake Telegram/WhatsApp channels
- SkillsPage: remove fake skills and stale 'API not yet wired' warning
- CronPage: remove fake cron jobs, stale warning, fix hardcoded defaults

All pages now show proper empty states until real gateway data arrives.
The Skills and Cron APIs were already fully wired; the warnings were
simply outdated and misleading.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: add voice/audio support with local Whisper STT

Add full voice interaction capabilities to the Windows node:

Core audio pipeline:
- NAudio WASAPI microphone capture with MTA thread initialization
- Energy-based voice activity detection with hysteresis
- Whisper.net speech-to-text with multi-threaded inference
- Pre-buffer to capture speech onset before VAD triggers
- Auto-download of Whisper models from HuggingFace

Voice overlay window:
- Modern WinUI 3 floating window with Mica backdrop and custom title bar
- Chat-style transcript bubbles with segment consolidation
- Real-time audio level visualization
- Start/Stop, Mute, and Settings controls

STT node capability:
- stt.listen and stt.status MCP commands for agent-initiated listening
- Follows existing capability pattern (like TTS)

Voice settings page:
- Model size selection (tiny/base/small) with download management
- Language selection (auto-detect + 9 languages)
- Silence timeout slider
- TTS voice picker with Windows neural voice enumeration
- ElevenLabs provider configuration
- Voice preview button

Integration:
- Tray menu Voice item
- Ctrl+Alt+Shift+V global hotkey for push-to-talk
- Deep links: openclaw://voice, openclaw://voice-stop
- Gateway chat responses shown in voice overlay
- TTS response playback with mic muting to prevent echo
- Capabilities page STT toggle
- Hub navigation Voice & Audio page

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Merge master into user/rbrid/stt-capability

Master refactored 8 separate windows into a unified Hub app (#272), which
removed src/OpenClaw.Tray.WinUI/Windows/SettingsWindow.xaml(.cs) and
WebChatWindow.xaml.cs. Node-capability toggles now live in
Pages/CapabilitiesPage as a code-built list (one icon + label per
capability) instead of an XAML page with x:Uid-localized headers.

Conflict resolution and re-integration:

* Accepted master's deletion of SettingsWindow.xaml, SettingsWindow.xaml.cs,
  and WebChatWindow.xaml.cs. The TTS/STT controls and code-behind that this
  branch added to those files are obsolete with the new Hub UI.

* Pages/CapabilitiesPage.xaml.cs gains a Speech-to-Text toggle alongside the
  existing Camera/Canvas/Screen/Location/TTS toggles, plus 'stt' in the
  active-capabilities summary string. This is the natural minimal alignment
  with the new pattern: one capability = one entry in the toggle list.

* The TTS provider / ElevenLabs key/voice/model UI that this branch had
  added is dropped because master removed the corresponding settings
  surface entirely. The backend services (TextToSpeechService,
  ElevenLabsTextToSpeechClient) and the SettingsManager keys are intact;
  the values can be set via direct settings.json edit until a new UI
  surface lands.

* Resolved 5 .resw conflicts (en-us, fr-fr, nl-nl, zh-cn, zh-tw) by taking
  master's content. All TTS/STT resource keys this branch had added are
  removed because the controls referencing them are gone. The earlier
  SettingsMcpDescription update (adding 'microphone' and 'speakers' to
  the capability list) is outside the conflict region and is preserved.

* Deleted tests/OpenClaw.Tray.Tests/SettingsWindowLocalizationCoverageTests.cs.
  It pinned that 14 specific x:Uids on SettingsWindow.xaml had matching
  resw entries; the controls and the file no longer exist.

Refactors from this branch survived the auto-merge cleanly:
* App.xaml.cs OnNodeInvokeCompleted still delegates to
  NodeInvokeActivityFormatter for privacy-class scrubbing.
* NodeService.RegisterCapabilities still calls NodeCapabilityGating
  predicates for every optional capability, including TTS and STT.

Validation:
* .\build.ps1
* dotnet test tests/OpenClaw.Shared.Tests/OpenClaw.Shared.Tests.csproj
  --no-restore  (1183 passed, 20 skipped)
* dotnet test tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj
  --no-restore  (418 passed; restore required first because master's
  Tray.Tests now links GatewayDiscoveryService.cs which needs Zeroconf)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* stt/tts: refill settings UI gaps after the unified Hub merge

Master's Hub refactor (#272) removed the per-capability detail UI that
previously lived on SettingsWindow. The capability backends are intact but
have no in-app surface anymore: STT had no way to set the BCP-47 language
tag, and TTS had no way to pick the provider, ElevenLabs API key, voice
ID, or model without hand-editing settings.json.

CapabilitiesPage.xaml gains two new detail cards beneath the capability
toggle grid, mirroring the existing McpCard pattern (visible only when
the capability is enabled):

* SttCard:
    - Language TextBox bound to SttLanguage.
    - Commits on LostFocus or Enter.
    - Empty input restores the "en-US" default rather than persisting "".
    - Validates with SttCapability.NormalizeLanguageTag before saving so a
      typo in Settings cannot ship a broken default to the WinRT recognizer.
    - Status text never echoes the user-supplied tag back on the failure
      path; only the local UI affordance shows it (the activity stream
      / support bundle path was already privacy-scrubbed by an earlier
      commit on this branch).

* TtsCard:
    - Provider ComboBox (Windows built-in / ElevenLabs).
    - ElevenLabs sub-panel becomes visible only when that provider is
      selected. Holds API key (PasswordBox), voice ID, and model.
    - API key handling: when a key is already saved we render a fixed
      mask sentinel ("••••••••") instead of any plaintext. Saving the
      form treats the sentinel as "keep current key" so the user can
      change voice ID / model without retyping the key, and rotation
      requires explicitly typing a new key. The on-disk DPAPI encryption
      done by SettingsManager is unchanged.
    - All ElevenLabs fields commit on LostFocus.

SttCapability.NormalizeLanguageTag is promoted from private to public so
the UI validates against exactly the rule the wire protocol applies. No
behavior change for the capability itself.

Validation:
* .\build.ps1
* dotnet test tests/OpenClaw.Shared.Tests/OpenClaw.Shared.Tests.csproj
  --no-restore  (1183 passed, 20 skipped)
* dotnet test tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj
  --no-restore  (418 passed)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: extend privacy class + tests for stt.listen and stt.status

NodeInvokeActivityFormatter.GetPrivacyClass now classifies any stt.*
command as privacy-sensitive, not just stt.transcribe. This catches
stt.listen (microphone capture) and stt.status (engine internals)
under the same scrubbing rules in the activity stream / support
bundle, and keeps the rule simple ("anything in the stt namespace").

Tests added:
* GetPrivacyClass: stt.listen, stt.status, stt.future-command rows.
* PrivacySensitive_FailedInvoke_OmitsErrorTextFromDetails: theory
  rows for stt.listen and stt.status alongside the existing
  stt.transcribe / camera.* / screen.* coverage.
* SttCapabilityTests: full coverage of the unified surface
  - Listen: timeoutMs clamps (below min, above max), default language
    "auto", invalid language rejected without echo, handler not
    wired, handler exception sanitized to "Listen failed", segments +
    engine metadata round-trip, cancellation.
  - Status: handler not wired, handler exception sanitized to
    "Status failed", per-engine readiness round-trip with download
    progress.
  - NormalizeLanguageTag: BCP-47 tags + "auto" sentinel
    (case-insensitive, normalized to lowercase) accepted; underscore
    / spaces / "automatic" rejected.
* SettingsRoundTripTests: round-trips SttEngine, SttModelName,
  SttSilenceTimeout, VoiceTtsEnabled, VoiceAudioFeedback through
  SettingsData.ToJson / FromJson.

Validation:
* .\build.ps1
* dotnet test tests/OpenClaw.Shared.Tests --no-restore  (1266 passed,
  20 skipped)
* dotnet test tests/OpenClaw.Tray.Tests --no-restore  (425 passed)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: STM, locale audit, and coverage tests for STT/TTS card

* Added E:\OpenClawWindowsNode\Audio_STM.md — full STRIDE analysis
  of the merged audio surface (assets, trust boundaries, per-component
  threats, cross-references to code + tests, follow-up backlog).
* Promoted every new STT/TTS card string in CapabilitiesPage.xaml to
  x:Uid + resw entries across all five locales (en-us, fr-fr, nl-nl,
  zh-cn, zh-tw): engine picker labels, language input + help, "More
  voice settings…" link, TTS provider picker, ElevenLabs sub-panel
  fields. Brand names (ElevenLabs), the "auto" BCP-47 sentinel, and
  the eleven_multilingual_v2 model identifier are kept verbatim and
  registered as InvariantOrDeferred in LocalizationValidationTests.
* Added CapabilitiesPageLocalizationCoverageTests — pins every new
  STT/TTS x:Uid against expected resw key suffixes (.Text, .Header,
  .Content, .PlaceholderText) so a future hardcoded-string regression
  fails fast.

Validation:
* .\build.ps1
* dotnet test tests/OpenClaw.Shared.Tests --no-restore  (1266 passed,
  20 skipped)
* dotnet test tests/OpenClaw.Tray.Tests --no-restore  (461 passed)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: extract SttEngineSelector + tests for engine selection rules

The engine-selection logic that NodeService.OnSttTranscribeAsync /
OnSttListenAsync / OnSttStatusAsync inline-implemented is now a pure
helper in Services/SttEngineSelector.cs and is consumed identically
from all three handlers. No behavior change.

Selector rules (pinned by SttEngineSelectorTests, 21 cases):
* Whisper preference + Whisper ready → Whisper, no fallback.
* Whisper preference + Whisper NOT ready + WinRT ready → WinRT,
  fallbackReason="whisper-model-not-ready". Happy degradation while
  the model downloads on first launch.
* Whisper preference + neither ready → keep Whisper preference,
  fallbackReason="whisper-and-winrt-unavailable". Dispatch fails;
  the user's preference is reported unchanged so stt.status is
  honest about what they asked for.
* WinRT preference + WinRT ready → WinRT, no fallback.
* WinRT preference + WinRT ready + Whisper ALSO ready → still WinRT.
  Critical invariant: explicit user choice is never silently upgraded
  to Whisper when the model finishes downloading.
* WinRT preference + WinRT NOT ready → keep WinRT,
  fallbackReason="winrt-unavailable". Same invariant: do not fall
  back to Whisper without explicit user opt-in.
* null/empty/whitespace/unknown engine string → treat as Whisper
  preference. A typo in settings.json must not hard-fail STT.
* Case- and whitespace-insensitive parsing of "whisper" / "winrt".

Engine identifier constants are mirrored locally on
SttEngineSelector.SharedConstants (free of cross-assembly deps);
MirroredConstantsMatchSttCapability pins they stay in sync.

Validation:
* .\build.ps1
* dotnet test tests/OpenClaw.Shared.Tests --no-restore  (1266 passed,
  20 skipped)
* dotnet test tests/OpenClaw.Tray.Tests --no-restore  (482 passed)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: security review fixes from STM walkthrough

Findings from the post-merge security review (full review recorded in
the session at files/security-review.md and reflected in the STM
follow-up backlog):

CRITICAL (1 fixed, 1 deferred):
* I-1 — UI now warns that selecting WinRT honors the Windows Online
  speech recognition toggle and may upload audio to Microsoft when
  that toggle is on. CapabilitiesPage SttEngineHint text updated to
  steer users to Whisper for fully local processing.
* S-4 / T-1 — DEFERRED: SHA-256 verification of the Whisper model
  (download AND load time) requires embedding canonical hashes for
  tiny / base / small from HuggingFace. Tracked as a Critical
  pre-GA follow-up in Audio_STM.md section 6, not blocking this
  merge. (Existing TLS + system trust chain remains the only check.)

HIGH (3 fixed):
* S-3 / D-1 — NodeService.OnSttListenAsync now enforces a 1-second
  cooldown between successive stt.listen invocations. Imperceptible
  to a real user but throttles a hostile loop from a compromised
  gateway. Throws InvalidOperationException("Listen rate limit")
  which the SttCapability sanitization wraps as "Listen failed".
* D-7 — AudioPipeline.CleanupCapture now wraps event-detach,
  capture.Dispose, and CTS dispose in independent try/catch blocks
  so a failure in one step doesn't leak the NAudio WasapiCapture
  COM object (which would hold the mic LED lit until process exit).
  Also added CleanupCapture() calls in StartAsync's two catch
  branches so the mic is released after a failed start.
* I-2 — VoiceOverlayWindow audit confirmed no transcript text reaches
  ActivityStreamService. Status: PIN, no code change needed.

MEDIUM (1 fixed):
* NEW-1 — TtsCapability previously returned \$"Speak failed: {ex.Message}",
  which can leak ElevenLabs key prefixes from 401 responses or
  device names from OS audio errors into the support bundle.
  Now returns a fixed "Speak failed" matching the SttCapability
  pattern. NodeInvokeActivityFormatter.GetPrivacyClass also now
  classifies tts.* as privacy-sensitive (was metadata) so failed-
  invoke details are uniformly scrubbed.

PIN (no change needed, confirmed by review):
* T-3 — SttModelName path-traversal: WhisperModelManager validates
  against the {tiny, base, small} allow-list before any Path.Combine.
* I-4 — ElevenLabs key DPAPI-encrypted at rest.
* I-5 — ElevenLabs key UI shows masked sentinel; plaintext never
  re-rendered after save.
* I-8 / PI-5 — stt.status response carries no PII (only readiness
  strings, engine name, capability flags, numeric download progress).
* PI-3 — Validation/handler errors don't echo caller input or
  exception text across stt.* and now tts.* as well.

Test additions:
* Speak_HandlerException_DoesNotLeakExceptionMessageIntoError —
  pins the new TTS privacy invariant with an "ElevenLabs 401:
  invalid key sk-secret-prefix" payload.
* Speak_ReturnsError_WhenHandlerThrows updated to assert the exact
  sanitized "Speak failed" message instead of leaking ex.Message.
* GetPrivacyClass theory rows now cover tts.speak and
  tts.future-command as privacy-sensitive (was metadata).

Validation:
* .\build.ps1
* dotnet test tests/OpenClaw.Shared.Tests --no-restore  (1271
  passed, 20 skipped)
* dotnet test tests/OpenClaw.Tray.Tests --no-restore  (483 passed)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: drop WinRT SpeechRecognizer + SAPI fallback; Whisper-only

Both legacy stacks are removed; SttCapability now dispatches every
stt.* call to a single Whisper engine via VoiceService. When the
Whisper model is not yet downloaded, handlers return a clear error
pointing the caller at the Voice Settings page download button —
there is no automatic fallback engine.

Rationale (from the discussion with Ranjesh):
* WinRT SpeechRecognizer is an old API that fails to activate in
  unpackaged tray builds (the long-standing 0x800455A0 issue) and,
  when the OS Online speech recognition toggle is on, may upload
  audio to Microsoft cloud — at odds with our local-first posture.
* System.Speech (desktop SAPI) is even older and has no value over
  Whisper for any modern scenario.
* Carrying two engines complicated the merge with no real upside
  now that Whisper.net runs reliably on every supported PC.

Removed:
* src/OpenClaw.Tray.WinUI/Services/SpeechToText/SpeechToTextService.cs
  (the WinRT + SAPI engine).
* src/OpenClaw.Tray.WinUI/Services/SttEngineSelector.cs (no engines
  to select between).
* tests/OpenClaw.Tray.Tests/SttEngineSelectorTests.cs.
* System.Speech NuGet package reference (was duplicated; both
  copies removed).
* SttEngine setting (SettingsData + SettingsManager round-trip).
* SttCapability.EngineWinRt and DefaultEngine constants.
* SttTranscribeResult.EngineFallbackReason and
  SttListenResult.EngineFallbackReason — no fallback to report.
* CapabilitiesPage Engine ComboBox + the engine-related UI strings
  in all five locales.
* The "Windows built-in may upload audio" caveat (no longer relevant).

Simplified:
* SttStatusResult: replaced PreferredEngine/EffectiveEngine plus
  per-engine readiness blocks with a single Engine + Readiness pair
  (engine is always "whisper" today; the field stays so a future
  engine doesn't break the wire).
* NodeService.OnSttTranscribeAsync / OnSttListenAsync /
  OnSttStatusAsync: dropped selector logic + WinRT marshalling.
  When VoiceService.IsWhisperReady is false, throw clear
  "Whisper model not downloaded" — wrapped to "Transcribe failed"
  / "Listen failed" by SttCapability's privacy sanitizer.
* CapabilitiesPage STT card hint surfaces model download state
  ("Whisper model is ready" / "downloading" / "not downloaded —
  open More voice settings…").
* McpToolBridge curated descriptions: drop engineFallbackReason
  field and the per-engine blocks from stt.status.

Tests:
* CapabilityTests.Status_ReturnsEngineReadiness rewritten for the
  flat shape; now also asserts no language/path strings appear in
  the JSON (tightens PI-5 enforcement).
* SettingsRoundTripTests: dropped SttEngine field assertions.
* CapabilitiesPageLocalizationCoverageTests: dropped engine ComboBox
  Uids from the contract list.
* LocalizationValidationTests: removed the engine ComboBox keys
  from the InvariantOrDeferred allow-list (no longer needed; the
  invariants list now only protects "auto", "ElevenLabs", and
  "eleven_multilingual_v2").

Validation:
* .\build.ps1
* dotnet test tests/OpenClaw.Shared.Tests --no-restore  (1271 passed,
  20 skipped)
* dotnet test tests/OpenClaw.Tray.Tests --no-restore  (456 passed)

Audio_STM.md and Audio_FollowUps.md updated to reflect the engine
removal (smaller test-seam refactor surface; I-1 "WinRT online
speech caveat" follow-up is retired).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: add Piper TTS provider via Sherpa-ONNX

Adds a third TTS provider, "piper", that runs Piper voices fully
locally on this PC through the official Sherpa-ONNX .NET binding
(org.k2fsa.sherpa.onnx 1.13.0). No cloud egress; the voice model
downloads once to %LOCALAPPDATA%\OpenClawTray\models\piper\<voice-id>\
and is reused across calls.

Backend (OpenClaw.Shared/Audio/PiperVoiceManager.cs):
* Curated catalog of 6 starter voices (en-US ×2, en-GB, fr-FR, de-DE,
  zh-CN) sourced from the sherpa-onnx tts-models GitHub release
  tarballs — these are repackaged Piper voices that include the
  language-specific espeak-ng-data, so the user only downloads one
  archive per voice instead of model + tokens + espeak separately.
* Download with progress callback; extraction via OS-bundled tar.exe
  (Win10 1803+); atomic per-voice directory layout; cleanup of
  partial files on failure or cancellation.
* IsVoiceDownloaded / GetVoiceSize / DeleteVoice for the (forthcoming)
  Voice Settings page UI.
* TODO marker for SHA-256 verification (Audio_FollowUps.md §2).

Tray service (OpenClawTray/Services/TextToSpeech/PiperTextToSpeechClient.cs):
* Wraps SherpaOnnx.OfflineTts; loads one voice at a time and reuses
  the loaded model across calls (load is the expensive ~200-500 ms
  step). Single-flight gate prevents concurrent generates from
  racing the same TTS instance.
* Inference runs on a background Task so cancellation can race the
  synthesis.
* Converts Sherpa's 32-bit float PCM samples to a standard 16-bit
  PCM mono WAV blob the WinUI MediaPlayer can play with no further
  transcoding.

Wiring (OpenClaw.Tray.WinUI/Services/TextToSpeech/TextToSpeechService.cs):
* Third branch in SpeakAsync's provider dispatch. SpeakWithPiperAsync
  resolves the voice from args.VoiceId or settings.TtsPiperVoiceId,
  fails with a "voice not downloaded" error pointing the user at
  Voice Settings if the file isn't present, and otherwise reuses the
  cached PiperTextToSpeechClient (rebuilds it only when the voice id
  changes).
* TextToSpeechService.PiperVoices exposed so the Voice Settings page
  can drive download / delete from the same instance.

UI (OpenClaw.Tray.WinUI/Pages/CapabilitiesPage.xaml + .xaml.cs):
* Added Piper as the first ComboBoxItem on the TTS provider picker
  ("Piper (local ML, recommended)"). Resw entries across all 5
  locales (en-us, fr-fr, nl-nl, zh-cn, zh-tw).
* UpdateTtsCard reads TtsProvider with a 3-way switch (piper /
  windows / elevenlabs); unknown / null defaults to Piper.

Capability + settings:
* TtsCapability.PiperProvider = "piper" wire constant.
* SettingsData.TtsPiperVoiceId / SettingsManager.TtsPiperVoiceId,
  default "en_US-amy-low" (~50 MB, smallest English voice).
  Round-trip preserved through Save/Load.

Tests:
* SettingsRoundTripTests asserts TtsPiperVoiceId persists.
* CapabilitiesPageLocalizationCoverageTests pins the new
  CapabilitiesPage_TtsProviderPiper x:Uid against en-us.
* PiperVoiceManager + PiperTextToSpeechClient have no unit tests
  yet — same blocker as the rest of the audio engine layer
  (Audio_FollowUps.md §1: needs interface extraction first).

Audio_FollowUps.md §3 updated with a "Status update — basic Piper
plumbing landed" subsection enumerating exactly what shipped and
what remains (Voice download UI, manager tests, SHA-256 verification,
spike validation).

Validation:
* .\build.ps1
* dotnet test tests/OpenClaw.Shared.Tests --no-restore  (1271 passed,
  20 skipped)
* dotnet test tests/OpenClaw.Tray.Tests --no-restore  (462 passed)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: voice download UI, Piper-as-default, first-listen polish

User-visible

* New Piper voice download panel on the Voice & Audio page
  (catalog of 6 voices, download with progress, delete, preview).
* Piper is now the default TTS provider for fresh installs.
* Read responses aloud toggle now drives every chat reply,
  not only voice-overlay sessions.
* Voice Overlay's Settings button opens the Voice & Audio page
  (was a no-op stub).
* First Whisper auto-download surfaces a status line in the
  Voice Overlay so the user knows the silent ~140 MB fetch is
  why nothing is being transcribed yet.
* Speech Model card refreshes its 'Model ready / Download
  required' status whenever the page becomes visible, even if
  NodeService hasn't wired its VoiceService yet.
* Stale 'Windows built-in' fallback text removed from the
  Speech-to-Text card description (5 locales). Whisper has been
  the only engine since ff11467.
* Width bumps so labels no longer truncate (the Speech Model
  size combo, the Provider combo).
* Dropped 'STT' jargon from the Language ComboBox header.
* Fixed misleading '~50-80 MB each' Piper size copy (real range
  is ~25-150 MB depending on quality).

Plumbing

* New SettingsRequested event on VoiceOverlayWindow; App hooks
  it to ShowHub('voice').
* TtsCapability.ResolveProvider falls back to Piper.
* App.OnNotificationReceived no longer gates TTS on
  VoiceMode != Inactive.
* VoiceSettingsPage.UpdateModelStatus queries the file system
  via WhisperModelManager directly so it works before
  NodeService finishes lazy-init of VoiceService.
* VoiceService.InitializeAsync fires DiagnosticMessage events
  around silent VAD/Whisper auto-downloads.

Tests: Shared 1271 / Tray 462 (default-provider asserts updated).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: address rubber-duck review (Highs #2-#5, Mediums #6-#8, Low #9)

High #2: Reuse a singleton TextToSpeechService for chat replies
* App.SpeakResponseAsync now goes through NodeService.TextToSpeech
  (a new public accessor on the existing _textToSpeechService field)
  instead of constructing a fresh service per call. Cached Piper
  client is reused across replies; the service-internal _playbackGate
  + _activePlayer now actually serialize back-to-back replies, and
  Interrupt=true takes effect.

High #3: Per-provider VoiceId routing
* New TtsWindowsVoiceId setting (round-tripped via SettingsManager
  + SettingsData; SettingsRoundTripTests assert it).
* SpeakResponseAsync no longer passes _settings.TtsElevenLabsVoiceId
  as a generic VoiceId; the per-provider Speak* paths each look up
  their own setting (TtsPiperVoiceId / TtsWindowsVoiceId /
  TtsElevenLabsVoiceId).
* SpeakWithWindowsAsync falls back to TtsWindowsVoiceId when
  args.VoiceId is blank.
* VoiceSettingsPage.OnWindowsVoiceChanged writes TtsWindowsVoiceId
  (was overwriting TtsElevenLabsVoiceId, a real cross-provider bug).

High #4: stt.listen returns a complete utterance, not the first segment
* New AudioPipeline.UtteranceTranscribed event fires once per silence-
  bounded utterance with all Whisper segments aggregated and an
  immutable Segments snapshot.
* VoiceService bubbles it as UtteranceCompleted.
* ListenOnceAsync subscribes to UtteranceCompleted (drops the
  per-fragment accumulator) so multi-segment utterances no longer
  return truncated text.

High #5: Voice Overlay submits one chat message per utterance
* OnTranscriptionReceived keeps the per-fragment streaming bubble
  update; chat submission moved to a new OnUtteranceCompleted
  handler so the gateway sees one message per spoken utterance.

Medium #6: Per-asset cancellation tokens in VoiceSettingsPage
* Split _downloadCts into _whisperDownloadCts and _piperDownloadCts
  so starting a Piper download no longer cancels an in-flight
  Whisper download (and vice versa).

Medium #7: Preflight tar.exe before Piper download
* PiperVoiceManager.EnsureExtractorAvailable runs a fast
  `tar --version` check before any network I/O. Downlevel Windows
  users now get a clear actionable error instead of a wasted ~50-150
  MB download that would later fail at extraction.

Medium #8: Refresh stale MCP tool descriptions
* stt.transcribe / stt.listen / stt.status now describe the single
  Whisper engine surface (no preferredEngine / effectiveEngine /
  engineFallbackReason); stt.listen description explicitly notes
  the result is the full silence-bounded utterance.
* tts.speak description includes `piper` in the provider list and
  notes the fresh-install default.
* Updated McpToolBridgeTests assertion for the new shape.

Low #9: Per-asset single-flight in download managers
* Both WhisperModelManager and PiperVoiceManager wrap their
  Download*Async in a static ConcurrentDictionary<string,Task> keyed
  on the canonical asset ID. Concurrent calls for the same asset
  await the same in-flight Task instead of racing on the same .tmp
  file. Failed downloads remove themselves from the table so a fresh
  retry isn't blocked.

Tests: Shared 1271 / Tray 462. Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: SHA-256 verification of Whisper models and Piper voices

Critical (rubber-duck #1) — fail-closed integrity check before install.

* New `Sha256` field on `WhisperModelInfo` and `PiperVoiceInfo`.
* All 9 catalog entries (3 Whisper models + 6 Piper voices) carry a
  pinned lowercase-hex SHA-256, captured against the live HuggingFace
  and sherpa-onnx GitHub releases on 2026-05-05.
* Download core methods now:
    1. Refuse outright if the catalog entry has no pinned hash
       (`InvalidOperationException`).
    2. Compute SHA-256 of the temp file BEFORE the atomic rename
       (Whisper) or BEFORE the tar extraction (Piper).
    3. On mismatch, throw `System.Security.SecurityException`,
       delete the temp file, and let the catch block tear down any
       half-installed directory. Sanitized message — does NOT echo
       the actual hash (no confirmation oracle).
* New `AssetHashPinningTests` enforces that every catalog entry has
  a 64-hex-char SHA-256 and an https URL — future additions that
  forget the hash now break the build.

Audio_FollowUps.md §2 updated:
* Status block at the top documents what landed today.
* Pre-public-release TODO list trimmed to: independent re-verification
  of the pinned hashes, on-load verification (not just on download),
  and a future signed-manifest format so updates don't require a tray
  rebuild. The original detailed design notes are preserved as the
  spec for that next iteration.

Tests: Shared 1275 / Tray 462. Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: Download Model button works without VoiceService

OnDownloadClick previously routed through VoiceService.DownloadModelAsync,
which silently no-op'd whenever _voiceService was null — and _voiceService
is only constructed inside NodeService.RegisterCapabilities (which runs on
Connect / StartLocalOnly, and only when NodeSttEnabled is true). A user
who toggled STT on without reconnecting, or who hadn't enabled MCP-only
mode, would tap Download and see nothing happen.

Construct a WhisperModelManager directly from
SettingsManager.SettingsDirectoryPath and download via that. Same
on-disk result as the VoiceService auto-download path, but available
regardless of NodeService lifecycle state. Same SHA-256 verification
applies (the manager owns it).

Tests: Tray 462 (no change in surface).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ux: Companion rename, expanded NavView memory, right-click opens Hub

Three coordinated tweaks based on the morning UX review.

1. Right-click on the tray icon now opens BOTH the popup quick-menu
   AND the companion app window. ShowHub gained an `activate` flag;
   for this code path we call ShowHub(activate:false) so the Hub
   surfaces via AppWindow.Show(activateWindow:false) and the popup
   (which is light-dismiss) stays the foreground window. Without this
   the Hub's Activate() would steal focus and dismiss the popup.

2. NavigationView pane mode is now expanded by default and remembered
   across sessions. PaneDisplayMode flipped from Auto to Left, and a
   new HubNavPaneOpen setting (default true) is round-tripped via
   SettingsManager / SettingsData. PaneOpening / PaneClosing handlers
   on HubWindow persist the user's last toggle. SettingsRoundTripTests
   covers the new field.

3. Renamed the mascot from 'Molty' to 'Companion' across the surface:

   User-facing strings:
   * VoiceOverlayWindow Title and header text → `Companion Voice`.
   * VoiceSettingsPage section header → `🔊 Companion Voice`.
   * Both Preview-button sample texts (Windows + Piper) now say
     `Hello! This is your Companion speaking.`.

   Code identifiers (HomePage):
   * MoltyRing → CompanionRing
   * MoltyProgressRing → CompanionProgressRing
   * UpdateMoltyRing → UpdateCompanionRing
   * Comment `<!-- Molty mascot -->` → `<!-- Companion mascot -->`

   `grep -i molty src/` returns zero hits.

Tests: Shared 1275 / Tray 462. Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ux: rubber-duck #2 — restore minimized Hub on right-click; pin pane default

Two findings from the second rubber-duck pass.

Medium: ShowHub(activate:false) was a no-op when the Hub was previously
minimized. AppWindow.Show(activateWindow:false) does not restore minimized
windows. Detect OverlappedPresenter.State == Minimized first and
Restore(activateWindow:false) so the window actually surfaces behind the
popup, then call Show.

Low: regression test for HubNavPaneOpen migration. Settings files written
before this field existed must deserialize to true (NavView expanded).
Added an explicit FromJson(\"{}\") assertion plus pinned the field's
default in MissingFields_UseDefaults and BackwardCompatibility_OldSettings*
so a future refactor can't silently flip new installs to a collapsed pane.

Tests: Tray 463 (one new). Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: deep-link parser strips trailing slash before query (#-)

The Windows shell canonicalizes openclaw://send?args=... to
openclaw://send/?args=... before handing it to us. The previous
implementation called TrimEnd('/') on the WHOLE remainder before
splitting off the query, so the trailing slash before the '?' was
never trimmed and Path came out as 'send/' instead of 'send'.

Trim the slash from the path SEGMENT after splitting off the query.

Three new theory cases pin the regression for send / agent / activity
deep links — categories that all carry query parameters in the
launcher canonicalized form. Existing TrailingSlash test (no query)
still passes with the new placement.

Credit to the parallel Copilot session for catching this.

Tests: Shared 1275 / Tray 466 (3 new). Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: don't drop final utterance on stop or timeout; bound transcription queue; normalize BCP-47

Three coordinated STT pipeline fixes from the latest review.

#1 (High) — Buffered speech was discarded on Stop/Timeout
* AudioPipeline.StopAsync used to call _cts.Cancel() BEFORE flushing,
  and the flush passed the canceled token straight into Whisper.net
  (which honored cancel and dropped the final utterance). Reordered
  to: stop capture -> flush with a fresh CancellationToken.None ->
  cancel _cts -> cleanup. Adds an overrideToken parameter on
  TranscribeSamplesAsync so the flush can opt out of the pipeline cancel.
* VoiceService.ListenOnceAsync used to throw TimeoutException as soon
  as the linkedCts fired, even when speech was actively buffered. It
  now waits on Task.WhenAny(utteranceTcs, timeoutSentinel), and on
  timeout it gives pipeline.StopAsync up to 2 s to flush — only then
  reports timeout. stt.transcribe inherits this fix.

#3 (Medium) — Whisper.net language mismatch
* SpeechToTextService.NormalizeForWhisper trims BCP-47 input down to
  the 2-letter ISO 639-1 primary subtag that Whisper.net's WithLanguage
  call expects. `en-US` -> `en`, `zh-Hans-CN` -> `zh`, garbage
  -> `auto`. Capability validator + MCP docs continue to advertise
  the wider BCP-47 shape (no breaking change for callers); this fixes
  the gap to Whisper.
* Result.Language now echoes the normalized form so the caller sees
  what Whisper actually used.

#4 (Medium) — Unbounded transcription queue
* Each VAD-bounded segment fired `_ = Task.Run(TranscribeSamplesAsync)`
  with no in-flight cap. SpeechToTextService gates Whisper work but
  callbacks accumulate behind the gate, each holding a sample buffer.
  Now bounded with Interlocked counter + MaxConcurrentTranscriptions
  cap (2). Excess segments are dropped with a clear DiagnosticMessage
  rather than silently queued — better UX than getting stale utterances
  arriving minutes after the user stopped speaking.

Tests: Shared 1291 / Tray 466 (16 new normalizer tests). Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: SHA-256 verification of Silero VAD model

Closes the inconsistency the buddy review flagged: Whisper and Piper
download paths are hash-pinned and fail closed on mismatch, but the
Silero VAD download path (VoiceService.DownloadVadModelAsync) was just
HTTPS + system trust chain — no integrity verification before File.Move
into the models directory.

* New SileroVadModelManifest holds the URL, SHA-256, and approximate
  size as public constants in OpenClaw.Shared.Audio. Hash captured from
  the upstream raw URL on 2026-05-05; same pre-public-release re-verify
  TODO as the other manifests (Audio_FollowUps.md §2).
* DownloadVadModelAsync now hashes the temp file with SHA-256 BEFORE
  the atomic rename. On mismatch it throws SecurityException and the
  catch block tears down the .tmp file. Sanitized error — does not echo
  the actual hash (no confirmation oracle).
* AssetHashPinningTests gains a SileroVadModel_HasPinnedSha256 case so
  a future renaming/forgetting of the constant trips the build.

Tests: Shared 1292 (1 new). Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: bring skill.md back in sync with capability registry

The SkillMdDriftTests pinning test was failing — 14 commands present in
McpToolBridge.KnownCommands had no matching ### heading in skill.md:

* The 4 new entries this branch added: stt.transcribe, stt.listen,
  stt.status, tts.speak.
* 10 pre-existing app.* entries (app.navigate, app.status, app.sessions,
  app.agents, app.nodes, app.config.get, app.settings.get, app.settings.set,
  app.menu, app.search) that already drifted before the audio work.

Fixing them all in one pass so the test goes green and stays green.
Each new section follows the existing format: H3 heading, brief
description, JSON-shaped param block, return shape.

Privacy + provider notes added for stt.* and tts.* so agent readers
understand: stt.* is local Whisper only and requires NodeSttEnabled,
tts.* defaults to Piper (local neural).

Tests: SkillMdDriftTests now passes. Shared 1292 / Tray 466. Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ux: throttle Whisper/Piper download progress UI; wire Re-download button

Two manual-test follow-ups on the Voice Settings page.

* Throttle progress UI updates to >=150 ms intervals on both the
  Whisper and Piper download paths. The streaming downloads emit a
  progress callback every ~80 KB chunk, so a 466 MB model produces
  ~5,800 dispatcher hops (Progress<T> + DispatcherQueue.TryEnqueue
  doubled the load). The dispatcher queue saturated and the app
  appeared frozen mid-download. Coalescing limits the rate to a few
  updates per second, with a forced final 100% report so the user
  never sees a stuck "99%" right before "Model ready". Also dropped
  the redundant inner DispatcherQueue.TryEnqueue (Progress<T> already
  marshals to the captured UI SyncContext).

* Re-download button now actually re-downloads. WhisperModelManager
  short-circuits DownloadModelAsync when the file is already present,
  so OnDownloadClick now calls the existing DeleteModel(modelName)
  first when the file is on disk. Net effect: delete -> fresh fetch
  -> SHA-256 re-verify -> atomic rename. Same on-disk result.

Tests: Shared 1292 / Tray 466 (no test surface change). Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* i18n: localize VoiceSettingsPage and VoiceOverlayWindow surfaces

Closes the buddy review's last finding. The new voice UI was English-only
hard-coded in both XAML and code-behind, while the rest of the tray (and
the freshly redone CapabilitiesPage STT/TTS card) reads from .resw via
x:Uid + LocalizationHelper.GetString.

Coverage:

* VoiceSettingsPage.xaml — every user-facing TextBlock / Header /
  ComboBoxItem / Button content / placeholder gets x:Uid (page title,
  card headers, STT toggle, model + language combos, voice chat
  controls, all 3 TTS provider items, Piper download/delete/preview,
  ElevenLabs slot, privacy note).
* VoiceOverlayWindow.xaml — header text, status badge, empty state,
  status text, start/stop label, mute + settings tooltips.
* VoiceSettingsPage.xaml.cs and VoiceOverlayWindow.xaml.cs — runtime
  status messages (download progress, model-ready, preview failures,
  pipeline state transitions, mute/listen state) now read from
  LocalizationHelper.GetString. Format strings use Lf(...) so {0}/{1}
  placeholders are honored under CurrentCulture.

Translations pinned for en-us / fr-fr / nl-nl / zh-cn / zh-tw —
~95 new keys per locale (475 total resw entries). Translations are
best-effort; native speakers should review pre-public-release.

LocalizationValidationTests:
* AllLocales_HaveExactlySameKeysAsEnUs ✅
* Resources_AreTranslatedAllOrNoneAcrossNonEnglishLocales ✅
  (added VoiceSettingsPage_StatusError + ElevenLabs sample-ID
  placeholder keys to the InvariantOrDeferred list — they're
  intentionally identical across locales)

Build green. Shared 1292 / Tray 466.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: stt.transcribe is now a true fixed-duration capture

Closes the buddy review's stt.transcribe finding. The handler used to
adapt SttTranscribeArgs into SttListenArgs and call ListenOnceAsync,
which inherited VAD-based silence shutdown — so a 5 000 ms request
would return after 1 s if the user stopped speaking. The advertised
contract (skill.md, McpToolBridge) promises bounded fixed-duration
capture, not silence-bounded.

Implementation:

* AudioPipeline.CaptureFixedDurationAsync — new top-level method that
  starts WASAPI capture, accumulates every resampled+gain-applied
  16 kHz mono sample into _fixedCaptureBuffer for exactly durationMs
  (or until cancellation), then returns the buffer. OnDataAvailable
  branches on a new _fixedCaptureMode flag and bypasses the VAD path
  entirely in this mode.

* VoiceService.TranscribeFixedDurationAsync — wraps
  CaptureFixedDurationAsync + SpeechToTextService.TranscribeAsync and
  returns SttTranscribeResult directly. Empty buffer (cancelled
  immediately or no audio) returns transcribed=false rather than
  throwing.

* NodeService.OnSttTranscribeAsync now calls TranscribeFixedDurationAsync
  instead of bouncing through ListenOnceAsync.

stt.listen behavior is unchanged.

Tests: Shared 1292 / Tray 466. Build green. (No new tests — exercising
this path requires a real WASAPI device. The capture/transcribe
boundary is tightly coupled to NAudio + Whisper.net, which were the
test seams already deferred to Audio_FollowUps.md §1.)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* i18n: localize VoiceOverlayWindow root window title

Adds x:Uid="VoiceOverlayWindow" on the WindowEx root, plus the
VoiceOverlayWindow_winexWindowEx_2.Title key in all 5 locale resw
files. Listed in InvariantOrDeferredResourceKeys so the parity test
allows the title to read identical "Companion Voice" in every locale —
matches the existing convention for ChatWindow / HubWindow /
CanvasWindow / TrayMenuWindow.

The visible header text and runtime status messages were already
localized; this just closes the gap on the actual OS-level window
title (alt-tab, taskbar).

Build green. Shared 1292 / Tray 466.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: gate stt.* on file presence, not in-memory load state

The MCP / wire-side stt.transcribe and stt.listen entry points
short-circuited with "Whisper model not downloaded" whenever
_voiceService.IsWhisperReady was false. That property reads
SpeechToTextService.IsModelLoaded — which is true only after the
model has been LOADED INTO MEMORY by EnsureInitializedAsync.

On a freshly-launched tray (or any state where the user hasn't
opened the Voice Overlay yet), the .bin file is on disk but the
model isn't loaded. The pre-flight check rejected the call before
the inner TranscribeFixedDurationAsync / ListenOnceAsync could run
EnsureInitializedAsync to load it lazily.

Net result: every first MCP STT call after launch failed with a
misleading "model not downloaded" error, even though the file was
right there.

Switch the pre-flight check to IsModelDownloaded (file on disk).
The lazy load happens inside the inner call as it always did.

Verified end-to-end via the local MCP HTTP server: tools/call
stt.transcribe with maxDurationMs:5000 returned a real transcript
("Hello, how is everybody doing?") on first invocation after a
fresh tray launch.

Tests: Shared 1292 / Tray 466. Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ux: voice UI testing round — localization, shutdown, and Capabilities cleanup

Three buckets of fixes from this afternoon's manual testing pass.

i18n: dot-suffix lookup bug in code-behind
* LocalizationHelper.GetString(X.Text) returns the raw key when the
  resource name has a dot — XAML x:Uid resolution interprets the
  trailing .Text as a property suffix, but direct programmatic
  lookup doesn't, so the resource map can't find it. Six call sites
  were displaying literal keys like "VoiceOverlayWindow_StatusBadge.Text"
  in the running UI.
* Added six dot-free code-only keys (BadgeReady, StatusReadyMessage,
  ButtonStartListening, ButtonDownloadModel, PiperButtonDownloadVoice,
  PreviewVoiceButtonContent) translated across all 5 locales, and
  swapped the call sites in VoiceOverlayWindow.xaml.cs and
  VoiceSettingsPage.xaml.cs to use them.

audio: Voice Overlay "Failed to encode audio features" on Stop
* Mid-encode interruptions from Whisper.net don't surface as a clean
  OperationCanceledException — they bubble up as misleading errors
  like "Failed to encode audio features." Pressing Stop while a
  transcription Task.Run was in-flight produced exactly that toast.
* AudioPipeline.StopAsync now drains in-flight transcriptions for up
  to 3 s before cancelling \_cts, so the user's last utterance has a
  chance to actually complete.
* TranscribeSamplesAsync's catch block suppresses errors when
  \_isStopping or the cancel token is set — those are expected
  shutdown-induced interruptions, not user-visible failures. Also
  sanitized the diagnostic toast (no raw ex.Message).

Capabilities page rework
* Removed the redundant Language TextBox + label + help + status
  block. The Voice & Audio page already owns the language picker via
  a curated ComboBox (the textbox accepted any string and silently
  failed validation on garbage like "foobar", which was a paper cut).
* "More voice settings…" hyperlink stays as the deep-link.
* Speech-to-Text card hint now reads file presence directly via a
  fresh WhisperModelManager rooted at SettingsManager.SettingsDirectoryPath
  (instead of hub.VoiceServiceInstance?.IsWhisperReady, which is null
  on a freshly-launched tray and reads "loaded into memory" rather
  than "file on disk"). Same trick used by VoiceSettingsPage's
  UpdateModelStatus.
* Updated the Capabilities help text in all 5 locales to say "Two-letter
  ISO 639-1 code (e.g. en, fr, ja)" instead of "BCP-47 tag (e.g. en-US,
  fr-FR, ja-JP)" — matches what NormalizeForWhisper actually accepts
  (region is stripped). (Help text is now only consumed by the language
  picker on Voice & Audio, but the resw key was renamed/repurposed to
  match.)
* Dropped the now-orphan SttLanguageLabel/TextBox/Help resw entries
  from all 5 locales, the CapabilitiesPageLocalizationCoverageTests
  catalog, and the LocalizationValidationTests invariant list.

Tests: Shared 1292 / Tray 460 (6 fewer cases — the
CapabilitiesPageLocalizationCoverageTests theory shrank by 3 keys ×
2 non-en locales). Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* i18n: align VoiceOverlayWindow root x:Uid with WindowEx convention

The Title key in resw was VoiceOverlayWindow_winexWindowEx_2.Title, but
the root x:Uid was just "VoiceOverlayWindow" — so WinUI's auto-derived
property-suffix lookup (Window-typed elements get the _winexWindowEx_2
suffix) couldn't find a match and the title fell back to the XAML
default. Aligned the x:Uid to "VoiceOverlayWindow_winexWindowEx_2",
matching the existing pattern used by ChatWindow / HubWindow /
CanvasWindow / TrayMenuWindow.

(Also: the buddy's parallel "trailing whitespace in resw" finding is
already addressed by subsequent commits — XmlDocument.Save normalized
the formatting; `Get-Content | -match '\s+\$'` returns 0 on every
locale today.)

Build green. Tray 460.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* privacy: stop leaking ex.Message into voice UI status text

The voice settings handlers and the Voice Overlay's start/stop catch
were formatting raw exception messages straight into user-facing UI
status text (and from there potentially into screenshots, error toasts,
support bundles, the activity stream). ex.Message can carry URLs,
local paths, hash digests, HTTP body fragments, or other implementation
detail that the user shouldn't see.

Seven call sites updated:
* VoiceSettingsPage.xaml.cs — Whisper download error, Piper download
  failure, Piper delete failure, Piper preview failure, Windows voice
  enumeration failure, Windows preview failure (6 sites).
* VoiceOverlayWindow.xaml.cs — overlay start/stop catch (1 site).

For each: full ex (message + type + stack) is logged via Logger.Error
or _logger.Error; the UI shows a generic localized message that ends in
"(see Debug log)" so users know where the detail lives.

Resw side:
* Six error-string keys in all 5 locales had their {0} format
  placeholders replaced with self-contained generic messages
  (translated, not just placeholder-stripped).
* VoiceSettingsPage_StatusError dropped from
  LocalizationValidationTests.InvariantOrDeferredResourceKeys — it
  used to be flagged invariant because the placeholder made every
  locale identical; with real translations it now varies and shouldn't
  be exempt.

Tests: Tray 460. Build green.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* audio: include stt.listen + stt.status in DangerousCommands

These two commands were already wired up in NodeService and advertised by
SttCapability, but the gateway's Windows platform-default policy hides any
command that isn't either platform-default (system.*, browser.proxy) or in
the node's DangerousCommands opt-in list. Only stt.transcribe was in that
list, so chat agents only saw stt.transcribe even when NodeSttEnabled was
on.

Adding stt.listen and stt.status lets them get the same explicit gateway
opt-in treatment as stt.transcribe, so once the operator allows them in
gateway.nodes.allowCommands they flow through to the agent's tools list.

Verified end-to-end: after re-pair, chat reports the full 24-command list
including stt.listen, stt.status, and tts.speak.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(audio): isolate shared download cancellation

Keep Whisper model and Piper voice single-flight downloads alive when one caller cancels its wait, and cover retry/cancellation behavior with focused tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(tray): keep right-click to context menu only

Restore tray right-click behavior so it opens only the menu instead of also showing the companion hub.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(voice): allow local overlay without node pairing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Ranjesh Jaganathan <ranjeshj@microsoft.com>
Co-authored-by: Scott Hanselman <scott@hanselman.com>
 2026-05-07 16:22:18 -04:00
Régis Brid
f2fa038bd0
Companion application refactoring (#272) 
* feat: unified Hub window with NavigationView, slim tray menu, and inline toggles

Consolidate 8 separate windows into a single Hub app:
- New HubWindow with NavigationView (Chat, Home, Activity, Settings pages)
- Chat page embeds gateway Control UI via WebView2 (default landing page)
- Home page with live status cards, quick actions, and activity feed
- Settings page with Expander sections, Test Connection, SSH tunnel fields
- Activity page with category filters and live ActivityStreamService binding
- Slim tray menu: status + 3 inline toggles + Hub/QuickSend + Settings/Exit
- Acrylic backdrop on tray flyout, auto-collapsing nav, page transitions
- Deep links (openclaw://) redirected to Hub pages
- Deleted 5 old windows: StatusDetail, ActivityStream, NotificationHistory, WebChat, Settings
- WebView2 event handler cleanup on page Unloaded (code review fix)
- Deferred page init to avoid null Settings during Frame.Navigate

25 files changed, 1350 insertions(+), 2033 deletions(-) — net code reduction

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: settings validation, CSS sidebar hiding, toggle labels, and SSH tunnel support

- Settings save validates gateway URL and SSH tunnel fields before saving
- Test Connection supports SSH tunnel mode (starts temp tunnel for test)
- SSH toggle auto-updates gateway URL field (shows loopback in tunnel mode)
- Chat page injects CSS to hide web Control UI sidebar (no dual navigation)
- Tray toggle switches hide On/Off labels to prevent clipping

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: full native UI with 12 pages, config editor, dual connection, and live gateway data

New pages: Sessions, Channels, Usage, Nodes, Cron, Skills, Config, About
- Sessions: live session list with reset/delete/compact actions
- Channels: channel health cards with start/stop controls
- Usage: cost breakdown, provider stats, daily costs
- Nodes: node inventory with capabilities and device ID copy
- Cron: scheduled jobs with run/remove actions (gateway wired)
- Skills: installed skills status (gateway wired)
- Config: TreeView + detail panel with editable values via config.set protocol
- About: version info, debug tools, documentation links

Infrastructure:
- Dual WebSocket connection: operator client (UI data) + node client (commands)
- 14 new gateway methods: cron.*, skills.*, config.* with JsonElement.Clone()
- Data caching in HubWindow for instant page navigation
- Session startedAt parsing fix (handles number + string timestamps)
- Hub window synced after settings reconnect
- Tray test updated for Auto scrollbar

Build: 0 errors | Tests: 774 passing (652 shared + 122 tray)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* UX Round 2: Agent events, discovery, pairing, models, presence, menu redesign, timer removal

Features:
- Agent Events page with stream filters and 400-event ring buffer
- Gateway discovery via mDNS (Zeroconf) with scan UI in General page
- Node/Device pairing UI with approve/reject in Nodes page
- Models list in Sessions page via models.list gateway method
- Instances page rewired to presence data from handshake snapshot
- Context cards in tray menu (session summary + token usage bars)
- Pairing pending count in tray menu

Menu & UX:
- Merged status + toggle into rich header card with status dot
- Permission toggles section (browser, camera, exec, canvas, screen)
- Renamed hub to Windows Companion with smart disconnect navigation
- Custom title bar with live connection status + gateway version
- Single-click tray -> chat, double-click -> hub
- Chat window pre-warmed on startup, hides instead of closing

Architecture:
- Removed all background timers (10s poll + 30s health check)
- On-demand data loading only (pages fetch on navigation)
- Fixed ParseSessions to merge instead of clear-then-rebuild (no flicker)
- Fixed HandleAgentEvent sessionKey parsing (was reading from root, not payload)
- Symmetric subscribe/unsubscribe for all new gateway events
- Caches cleared on disconnect, seeded into HubWindow on open

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Restructure navigation: agents as hierarchical nav items under Gateway

- Replace flat Agent section with hierarchical Agents > {agentId} > {sub-pages} structure
- Move Conversations shortcut to top level, pointing to SessionsPage
- Remove AgentSelectorNav ComboBox, replaced with dynamic nav tree (RebuildAgentNavItems)
- Add FindAndSelectNavItem for recursive nested nav item selection
- Add agent: tag parsing (ResolveAgentPageType, ParseAgentIdFromTag)
- Update NavigateTo with legacy flat tag mapping to new agent: prefix format
- Strip Zone B (Agent Roster) and Zone C (Capability Toggles) from HomePage
- Remove Node Mode toggle from SettingsPage (lives in CapabilitiesPage)
- Update command palette to use agent-scoped tags
- NavigateToDefault now goes to Home instead of Conversations

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* UX Round 3: Hearth redesign, hierarchical nav, command palette, agent APIs

Navigation restructure:
- 16 flat pages → domain-grouped hierarchical nav (Gateway, This Computer)
- Each agent gets expandable nav item with Sessions, Events, Skills, Workspace
- Dynamic agent nav built from agents.list gateway response
- Nodes nested under Instances (superset relationship)
- Cron moved to gateway section (gateway-wide, not per-agent)
- Connection page extracted from Settings into Gateway section
- Settings simplified to local-only (startup, notifications)

New pages:
- ConnectionPage — gateway URL, token, SSH tunnel, discovery, status
- CapabilitiesPage — device toggles + node status indicator
- WorkspacePage — TabView per-file viewer via agents.files.list/get
- BindingsPage — routing rules viewer (channel→agent)
- ConversationsPage — cross-agent session browser (legacy)

Command palette (Ctrl+K / Ctrl+F):
- Inline overlay with light dismiss (click outside, Escape, Enter)
- 20+ navigation + 5 toggle + dynamic session commands
- Fuzzy substring filtering

Home page (The Hearth):
- Molty status indicator with colored ring
- Natural language status text
- Quick action buttons

Agent-scoped data:
- Sessions, Skills pass agentId to gateway calls
- Agent Events filtered by sessionKey prefix
- Workspace scoped to current agent in hierarchy

Real gateway APIs:
- agents.list → dynamic nav + agent roster
- agents.files.list/get → workspace file viewer
- Cached agents list for hub seeding on open

Architecture:
- Canvas window styled with Mica + custom title bar
- Open Canvas menu item uses NodeService.ShowCanvasWindow()

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(tray): add rich session tooltips and connected devices section

- Add rich ToolTip to each session label showing model, provider,
  channel, thinking level, token breakdown, context window, status,
  and age
- Add Connected Devices section between context summary and Permissions
  with online indicator dots, platform badges, and rich tooltips
- Show connected client count from presence data in status header subtitle

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Redesign session and device tray menu with rich compact cards

Replace plain text session/device rows with structured Grid cards featuring:
- Status dot (green/amber/gray) + name + model/platform badge + chevron
- Token usage with percentage and color-coded progress bar
- Channel badges and capability icons for devices
- Section headers with right-aligned summary stats

Add AddFlyoutCustomItem() to TrayMenuWindow for custom UIElement
flyout items with hover-to-show and click-to-navigate behavior.

Build detailed side flyout panels with headers, token breakdowns,
capability listings, and session metadata.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* UX Round 4: Chat popup, rich tray menu, schema config editor

Chat panel:
- Tray-anchored borderless popup (bottom-right, DPI-aware)
- WS_EX_TOOLWINDOW + no caption/frame (hidden from taskbar)
- Auto-hide on deactivate, instant show (no animation — WebView2 incompatible)
- Single-click toggle, double-click opens Hub (400ms detection)
- Chat window recreated on settings change (stale URL fix)

Tray menu:
- Rich 2-line session cards: status dot, model badge, token progress bar
- Rich device cards: capability emoji strip, platform badge
- Flyout detail panels: non-interactive TextBlocks (not menu items)
- Session flyout: model/provider, channel, ASCII token bar, thinking/verbose
- Device flyout: capabilities merged with commands (cap as header, cmds indented)
- Dynamic capability toggles under local device (from node.Capabilities)
- Flyout dismisses on any menu item hover (including separators/headers/toggles)
- Section headers with aggregates (sessions/tokens, online/caps)
- AddFlyoutCustomItem + AddToggleItem indent support

Schema config editor:
- SchemaConfigEditor UserControl renders from JSON Schema
- Supports string/number/boolean/enum/array/nested objects
- Sensitive field detection (PasswordBox)
- Fallback RenderConfigDirectly when schema unavailable
- Config detail panel uses schema for selected tree node
- Pending changes preserved on save failure

Command palette:
- Rebuilt as inline overlay Grid (light dismiss: Escape + click-outside)
- Ctrl+F added as alternate shortcut
- TextBox replaces AutoSuggestBox (proper Escape handling)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* UX Round 5: Connection dashboard, tray polish, review fixes

Connection Management:
- Redesigned ConnectionPage with 6-section card layout: status card with
  live gateway info, gateway discovery picker with mDNS scan, setup code
  paste (from openclaw qr), manual connection expander, device identity
  card with pairing status and copyable approval command, connection log
- Auto-discovery when disconnected or unconfigured
- Setup code applies both bootstrapToken and Token for immediate connect
- PreferredGatewayId persisted in settings

Tray Menu Polish:
- Compact 3-column ToggleButton grid for capability toggles (all 7 shown)
- Split header subtitle into 2 lines (connection details + node status)
- Auth failure warning shown inline
- Reconnect and Connection quick actions
- Dropped 'Open' prefix from action items, added bottom padding

Hub Window:
- Removed XAML KeyboardAccelerators (caused tooltip flicker on hover)
- Replaced with PreviewKeyDown handler for Ctrl+K/F
- Added ReconnectAction, LastGatewaySelf, node state properties

Connection Lifecycle Fixes (from 3-model rubber-duck review):
- Capability toggles now use ReconnectNodeServiceOnly() instead of full
  teardown — no longer kills gateway client or chat window
- Reconnect action uses lightweight ReconnectGateway() (preserves chat)
- SyncHubNodeState() pushes live pairing/identity to hub on every
  node status and pairing change
- Gateway matching uses host:port comparison (not full URL with scheme)
- Discovery service disposed on page Unloaded
- Connection log refreshes on every status change
- SanitizeUrl guards against port -1
- Null-conditional restored on _hub?.RaiseSettingsSaved()
- Synthesized current gateway entry doesn't mutate cached list

Other:
- Instant single-click chat toggle (removed double-click debounce)
- Catch-all ShowHub(action) for menu nav tags
- SSH tunnel section flattened (removed redundant nested expander)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* UX Round 6: Token bar ProgressBar, title bar search, MCP toggle

Session flyout:
- Replaced ASCII token bar (█░) with WinUI ProgressBar (green/orange/red)
- Built flyout as native UIElement (StackPanel) instead of text items
- Added AddFlyoutCustomItem(UIElement, UIElement, action) overload

Title bar search:
- Replaced hidden command palette overlay with AutoSuggestBox in title bar
- Standard Windows pattern, always visible, Ctrl+K/F focuses it
- Lobster icon 14px → 20px, title shortened to 'OpenClaw'
- Removed overlay XAML, smoke layer, and palette methods

MCP server toggle:
- Added Local MCP Server card on Capabilities page
- Toggle, endpoint URL display, Copy Token/URL buttons
- Shows token readiness status

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix session flyout to match device flyout style

Replaced custom UIElement session flyout panel with simple
TrayMenuFlyoutItem list matching the device flyout pattern.
ProgressBar stays in the main menu session card only.
Removed unused AddFlyoutCustomItem(UIElement, UIElement) overload
and ShowCascadingFlyoutElement helper.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* UX Round 7: Nav restructure, App MCP, config editor, review fixes

Navigation:
- Sessions, Agent Events, Skills promoted to top-level with agent filter
- Agents folded to one level (direct nav → Workspace)
- Instances merged into Nodes with Connected Clients section
- Title restored to 'OpenClaw Windows Companion'
- Title bar: 48px height, responsive search (Ctrl+E), lobster 20px

Config page:
- Schema-driven tree (objects only, no leaf nodes)
- Editor + Raw JSON tabs
- config.patch sends { raw, baseHash } (hash from config.get response)
- Subtitle shows actual config file path from gateway
- All expanders open by default

App MCP capability (10 tools):
- app.navigate/status/sessions/agents/nodes/config.get
- app.settings.get/set with security allowlist (no secrets)
- app.menu/search for tray and command palette testing
- All handlers return structured data (not stringified JSON)
- Sessions filter by session key prefix (not channel)

Bug fixes:
- AgentEventsPage: init NRE guard, filter applies to display
- CapabilitiesPage: MCP toggle suppress during init
- SessionsPage: removed unused agent filter
- Config save: proper baseHash from gateway hash field

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* 5-model adversarial review fixes + regression tests

Fixes from Opus 4.7, Sonnet 4.5, GPT-5.4, GPT-5.2, Opus 4.6 reviews:
- Guard IndexOutOfRangeException on empty session Status
- Fix TCS hang when DispatcherQueue unavailable in app.navigate
- Static s_emptyObject replacing leaked JsonDocument in config tree
- Always prune stale sessions (removed incomingKeys.Count > 0 guard)
- try/catch/finally on 6 async void handlers (Channels, Sessions, Config)
- Seed ALL cached data before NavigateToDefault in ShowHub
- Move CurrentStatus/_cachedCommands inside DispatcherQueue in UpdateStatus
- Raw JSON tab uses 'parsed' not wrapper object
- Null-safe Subtitle in search handler
- Invalidate command cache on agent switch
- Dispose SemaphoreSlim in GatewayDiscoveryService

Regression tests (9 new):
- AppCapabilityTests: category, commands, CanHandle, handlers, errors
- SessionInfo empty Status guard
- ParseSessions empty array clears sessions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Tray menu: header redesign, dismiss fix, session reliability

Header:
- Split into brand header (🦞 OpenClaw) + Gateway section
- Gateway section: status dot, version/host:port, node status, labeled
  ToggleButton ('Connected'/'Disconnected') with tooltip
- Gateway info clickable → opens Connection page
- Menu dismisses after connect/disconnect toggle (avoids stale header)

Dismiss:
- Unified 150ms delayed foreground check for all deactivation cases
- Checks this window, flyout child, and owner parent before dismissing
- Fixes: click-away dismisses everything, hover between items doesn't
- Set _isShown=true in ShowAtCursor (was missing, broke dismiss guard)

Sessions:
- Removed connection status gate — show cached sessions always
- Zero gateway requests on menu open (health check was clearing sessions
  via ParseSessions in the response)
- Session cards click → 'sessions' top-level route

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Connection UX: localhost probe, auth errors, token prompt, gateway switching

Discovery:
- Localhost probe enumerates listening TCP ports via GetActiveTcpListeners
- Probes for gateway HTML signature (<title>OpenClaw Control</title>)
- Excludes MCP port (8765) to avoid false positives
- Runs in parallel with mDNS, results merged

Connection page:
- Auth error InfoBar with contextual guidance (token/pairing/password/signature)
- HubWindow.LastAuthError forwarded from OnAuthenticationFailed
- Cleared on successful connect and new connection attempts
- Token prompt always shows when switching gateways (pre-fills current token)
- Cancel button on token prompt
- Discovery list refreshes after connecting to show ✓

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Prepare UX experiments branch for PR

Fix gateway discovery host resolution, harden branch-introduced security paths, complete localization coverage, remove newly introduced dead code, refresh documentation, wire functional UX flows, and stabilize the Config page rendering path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix MCP-only tray startup

Initialize the local node service when MCP mode is enabled even if gateway node mode is disabled, so MCP-only tray launches start the HTTP server used by integration tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Ranjesh Jaganathan <ranjeshj@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-05-04 20:48:24 -07:00
Régis Brid
e0c40985a7
feat: add Windows node text-to-speech (#253) 
Adds a focused Windows node text-to-speech capability as the first stable voice-support primitive.

- adds the shared `tts.speak` capability and MCP/gateway documentation
- wires Windows and ElevenLabs TTS behind opt-in tray settings
- protects the ElevenLabs API key with DPAPI
- adds shared and tray tests for capability behavior, settings, and ElevenLabs requests

This lands the focused TTS foundation from the broader Voice Mode discussion in #120 so remaining voice UX/STT/repeater work can build on top in smaller follow-up PRs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-05-01 11:31:58 -07:00
Mike Harsh
1433349d10
feat(tray): add onboarding wizard updates (#241) 
* Snap Reactor framework as OpenClawTray.Infrastructure

- Copy microsoft/microsoft-ui-reactor src/Reactor/ (249 C# files, 12 modules)
- Rename namespace Microsoft.UI.Reactor -> OpenClawTray.Infrastructure
- Create OpenClawTray.Infrastructure.csproj (net10.0, WinAppSDK 1.8)
- Add ProjectReference from OpenClaw.Tray.WinUI
- Add project to moltbot-windows-hub.slnx
- Fix C# 14 field keyword conflict in ValidationContext.cs
- Exclude ReactorApplication.xaml (library mode, host app owns Application)
- Update global.json rollForward to latestMajor
- Full solution builds clean (0 errors)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Implement OnboardingWindow host with Reactor pages

- OnboardingWindow.cs: WindowEx host with ReactorHostControl, Mica backdrop, 720x752
- OnboardingApp.cs: Root Reactor component with UseNavigation, step indicator, back/next
- OnboardingState.cs: Shared state with mode-dependent page order (matches macOS flow)
- WelcomePage.cs: Page 0 - welcome title + security notice card
- ConnectionPage.cs: Page 1 - local/remote/later gateway selection
- ReadyPage.cs: Page 9 - feature summary with emoji rows
- Placeholder stubs for Wizard/Permissions/Chat pages (Phase 3)
- Full solution builds clean via build.ps1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Wire first-run detection and tray menu to OnboardingWindow

- First-run: ShowOnboardingAsync() replaces ShowSetupWizardAsync() in OnLaunched
- Tray menu: 'setup' action now opens OnboardingWindow instead of SetupWizardWindow
- OnboardingCompleted event mirrors existing SetupCompleted reconnection logic
- Old ShowSetupWizardAsync() preserved for backward compatibility
- Full build passes clean

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove SetupWizardWindow, redirect all call sites to OnboardingWindow

- Remove ShowSetupWizardAsync() and _setupWizard field
- Redirect deep link OpenSetup handler to ShowOnboardingAsync()
- SetupWizardWindow.cs retained but no longer wired from App.xaml.cs
- All build.ps1 targets pass clean

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Sprint 1: Enhanced pages + shared widgets (4 parallel tasks)

Welcome Page (op-dlw):
- Lobster icon, security warning card with ⚠️, trust model bullet points
- Two-card layout (orange warning + gray trust explanation)

Connection Page (op-24b):
- Local/Remote/Later radio choices with ●/○ indicators and emoji icons
- Conditional gateway URL + token fields for Local/Remote modes
- Local pre-fills ws://localhost:18789, Test Connection button
- Two-way binding to OnboardingState and SettingsManager

Ready Page (op-qrh):
- 🎉 celebration icon, mode-specific info card
- Feature action rows with icon + title + subtitle
- Launch at Login toggle
- Configure Later / Remote info cards

Shared Widgets (op-5xl):
- OnboardingCard: Rounded card with white background
- FeatureRow: Icon + title + subtitle row component
- StepIndicator: Dot-based navigation indicator
- GlowingIcon: 🦞 lobster icon (animation-ready)

All 4 tasks implemented in parallel. Full build passes clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* OnboardingApp nav + localization (27 keys × 5 locales)

OnboardingApp (op-fix):
- Integrated GlowingIcon header and StepIndicator widget
- Layout matches macOS: icon → page content → nav bar
- Phase 3 placeholder pages with clear labels

Localization (op-4jl):
- 27 onboarding keys added to all 5 locale .resw files
- en-us, fr-fr, nl-nl, zh-cn, zh-tw
- Covers: title, nav buttons, welcome, connection, ready pages

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Sprint 2+3: All pages + polish (6 parallel tasks)

Wizard Page (op-y0w):
- Native offline fallback: gateway URL, token, node mode toggle
- Test Connection button with status feedback
- TODO comments for future WebSocket RPC integration

Permissions Page (op-9mr):
- 5 Windows permissions: Notifications, Camera, Mic, Screen Capture, Location
- Status indicators (✅/⚪) with Open Settings buttons
- Status message area for feedback

Chat Page (op-e38):
- 'Meet your Agent' MVP chat UI
- Agent welcome bubble (blue) + user message bubbles (gray)
- Text input + Send button, footer note about full WebView2 integration

Mica + Theming (op-dl8):
- Non-resizable window via OverlappedPresenter
- Mica backdrop confirmed, window size matches spec

Page Transitions (op-xh9):
- Spring slide transition on NavigationHost (dampingRatio: 0.86)
- Matches macOS interactiveSpring(response: 0.5, dampingFraction: 0.86)

Accessibility (op-61d):
- To be enhanced in Sprint 4 integration pass

All pages wired into OnboardingApp. Full build passes clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* WizardStepView renderer + integration validation

WizardStepView (op-2oj):
- Dynamic renderer for all 7 gateway RPC step types
- Note, Text (with Sensitive/password), Confirm, Select, MultiSelect, Progress, Action
- WizardStepProps record + WizardStepType enum
- Switch expression renders type-appropriate UI with OnSubmit callback

Integration (op-28l):
- Solution file already includes OpenClawTray.Infrastructure (done in Sprint 0)
- build.ps1 builds WinUI with ProjectReference chain — no changes needed
- All 774 tests pass (652 Shared + 122 Tray, 0 failures)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add onboarding unit tests (13 new tests, 135 total Tray tests)

OnboardingStateTests:
- GetPageOrder: Local includes Wizard, Remote excludes it, Later is minimal
- GetPageOrder: NoChat mode excludes Chat for all modes
- GetPageOrder: Always starts with Welcome, ends with Ready
- Defaults: Mode=Local, ShowChat=true
- Complete: fires Finished event, calls Settings.Save()

All 774+ tests pass (652 Shared + 135 Tray, 0 failures).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add WizardStepProps and WizardStepType unit tests

Tests WizardStepType enum (7 values) and WizardStepProps record defaults.
All 145 Tray tests pass (0 failures).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add inner-loop dev scripts for testing onboarding UX

- dev-loop.ps1: Build + kill + launch cycle with -Clean (first-run) and -Tail (logs)
- test-sandbox.wsb: Windows Sandbox config with mapped build output for clean-state testing
- setup-sandbox-network.ps1: Port proxy setup for sandbox-to-WSL gateway connectivity

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix NullRef on first render, duplicate lobster, Border(null!) crash

- OnboardingWindow: use ctx.UseState(state) in mount function for props persistence
- WelcomePage: remove duplicate lobster icon (OnboardingApp header has the persistent one)
- StepIndicator: Border(TextBlock('')) instead of Border(null!) to avoid runtime NullRef

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix nav bar positioning + visual test framework + bug fixes

Nav bar fix:
- Fixed NavigationHost height to 520px so nav bar stays at consistent position
- All pages render within the same content area, nav bar never jumps
- Replaced Spring transition with 200ms Slide (prevents overlap on fast navigation)
- Compacted WelcomePage: merged security+trust cards, reduced font sizes
- Reduced GlowingIcon from 64px to 48px, tightened margins

Bug fixes:
- Fixed NullRef on first render (ctx.UseState for mount props persistence)
- Fixed duplicate lobster icon (removed from WelcomePage, kept in OnboardingApp header)
- Fixed Border(null!) crash in StepIndicator

Visual test framework:
- visual-test.ps1: P/Invoke window finding + UIAutomation button clicking
- Screenshot capture via PrintWindow/CopyFromScreen (note: GDI capture fails on Dev Box/Cloud PC)
- Baseline + after screenshots in visual-test-output/

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* SlideInOnly transition + RenderTargetBitmap visual capture

SlideInOnlyTransition (NavigationTransition.cs + TransitionEngine.cs):
- New transition type: instantly hides old page (opacity=0), slides+fades new in
- Direction auto-reverses on back nav (Push=right, Pop=left)
- 200ms duration with cubic-bezier easing
- Zero flicker — old page is invisible before new one starts animating

RenderTargetBitmap visual capture (OnboardingWindow.cs):
- In-app capture via WinUI RenderTargetBitmap API
- Works on Dev Box/Cloud PC (no physical display needed)
- Triggered by OPENCLAW_VISUAL_TEST=1 env var
- Auto-captures on initial load and every page navigation (PageChanged event)
- Saves PNGs to OPENCLAW_VISUAL_TEST_DIR
- All 6 pages validated via LLM visual analysis

OnboardingState.cs:
- Added PageChanged event for capture integration

All 145 tests pass. Full build clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Permissions page button alignment: use Grid layout for right-aligned buttons

Changed PermissionRow from HStack to Grid with ['1*', 'Auto'] columns so
'Open Settings' buttons are consistently right-aligned and stacked vertically,
matching the pattern used in ConnectionPage.cs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Permissions page: left-align status emojis in own column

Move status emojis (✅, ❌, ⚠️) from inline with permission name into
a dedicated Grid column 0 with Auto width. Changes the row Grid from
2 columns [1*, Auto] to 3 columns [Auto, 1*, Auto]:
- Column 0: Status emoji, fixed width, left-aligned
- Column 1: Permission icon + name + description, fills remaining
- Column 2: Open Settings button, right-aligned

This ensures all status emojis form a clean vertical line.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* wip: latest onboarding fixes pre-upstream-merge

Checkpoint of in-progress work before merging origin/master to pick up
GatewayTopologyClassifier, SshTunnelCommandLine, SshTunnelService, and
updated SettingsWindow connection logic.

Includes:
- Permissions page alignment fixes
- ConnectionPage gateway auth + pairing flow
- New onboarding services (GatewayHealthCheck, InputValidator,
  LocalGatewayApprover, PermissionChecker, SetupCodeDecoder,
  WizardStepParser)
- Tests for those services
- Localization keys across 5 locales
- Inner-loop dev scripts and e2e helpers
- Onboarding + auth-fix proposal docs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(onboarding): redesign Connection page to match new UX mockup

Implements the redesigned Connection page from connection-page-mockup.html:

- Five gateway modes (was three): Local / WSL / Remote / SSH Tunnel /
  Configure Later. WSL and SSH are added to ConnectionMode and reuse
  the Local page-order in OnboardingState.GetPageOrder().
- Setup Code row gains explicit Paste and QR-import buttons in addition
  to the existing focus-paste behavior. QR decoding is extracted from
  SetupWizardWindow into a reusable Helpers/QrSetupCodeReader so it can
  be invoked from Reactor pages without depending on the wizard window.
- Animated SSH panel renders inline when SSH mode is selected: 2x2 grid
  of SSH User / Host / Remote Port / Local Port plus a live preview line
  generated via SshTunnelCommandLine.BuildArguments(...). Settings are
  written through to SettingsManager.SshTunnel*. App gains a
  EnsureSshTunnelStarted() shim so TestConnection can spin up the
  managed tunnel before health-checking ws://127.0.0.1:<localPort>.
- Topology detection line renders the GatewayTopologyClassifier output
  (DisplayName/Transport/Detail) live as the user changes modes / SSH
  fields, matching the mockup's '● Detected: ...' line.
- Page content is wrapped in a ScrollView and the onboarding window is
  resized to 720x900 to fit the additional rows in the SSH layout.
- App exposes GetOnboardingWindowHandle() so the QR FileOpenPicker can
  initialize against the onboarding HWND.
- Two new optional environment variables aid visual testing without
  requiring UI automation:
    * OPENCLAW_ONBOARDING_START_ROUTE = <OnboardingRoute name>
    * OPENCLAW_ONBOARDING_START_MODE  = <ConnectionMode name>

Adds new locale keys for the SSH/WSL/QR/Topology surface in all five
locales (en-us authoritative; fr-fr, nl-nl, zh-cn, zh-tw machine-
translated and flagged for human review in the PR description).

Adds tests/OpenClaw.Tray.Tests/ConnectionPageTopologyTests.cs covering:
- 5-mode page-order parity (Wsl/Ssh behave like Local).
- GatewayTopologyClassifier outputs for the canonical mode→URL mapping.
- SshTunnelCommandLine preview includes both forwards (gateway +
  browser-proxy +2) and validates user/host.

Validation (per AGENTS.md):
- ./build.ps1: all projects succeed.
- dotnet test Shared:  967 passed / 20 skipped / 0 failed.
- dotnet test Tray:    350 passed / 0 failed (8 new).
- Visual capture in OPENCLAW_VISUAL_TEST mode for both Local and SSH
  modes; matches mockup layout.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* security: remove hardcoded WSL gateway dev token from e2e test

The fallback token was a dev-gateway secret that got flagged by GitHub secret scanning. Token now must come from WSL openclaw.json (preferred) or OPENCLAW_GATEWAY_TOKEN env var; the script fails fast if neither is available.

Note: the leaked token should be rotated by regenerating the dev gateway config in WSL.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore(infra): prune unused Reactor modules (Charting/Data/Yoga/FlexPanel/DataGrid/PropertyGrid)

Per PR feedback: the tray only uses Core/Hosting/Navigation/Elements/Hooks/
Animation/Markdown/Accessibility/Input from the Reactor snap. Removed:
- Charting/ (D3 charts not used by onboarding)
- Data/ (datasource/grid binding not used)
- Yoga/ (FlexPanel not used; tray uses StackElement-based HStack/VStack)
- Controls/DataGrid (cascading: depends on Data+Charting)
- Controls/PropertyGrid (cascading: depends on Data)
- Pruned Yoga/FlexPanel hooks from Core/Element.cs, ElementPool.cs,
  Reconciler.Mount.cs, Reconciler.Update.cs, Elements/Dsl.cs, ElementExtensions.cs
- Pruned Charting hooks from Core/AccessibilityScanner.cs and Hosting/ReactorHost.cs
- Removed UseDataSource from Core/Component.cs
- Removed FieldDescriptor overload from Controls/Validation/FormField.cs
- Removed ResizeGripRegistration call sites (lived in DataGrid)

Build clean. Tray tests 350/350 pass. Shared tests 967/967 pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor(onboarding): replace Reactor snap with FunctionalUI helper

Replace the vendored Reactor-derived infrastructure project with a tiny OpenClaw-owned FunctionalUI helper layer used by onboarding.

Remove unused charting, data, markdown, devtools, validation, localization, input, animation, and broad control infrastructure from the PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: remove local workflow files from onboarding PR

Remove Beads and Gastown hook files so the tray onboarding PR only contains product UI changes and required app support.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: remove extraneous artifacts from onboarding PR

Remove local visual outputs, sandbox/provisioning scripts, e2e scratch automation, and upstream planning docs from the tray onboarding PR.

Keep the remaining changes focused on the product onboarding flow and supporting app code.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix tray onboarding runtime issues

Remove inconsistent gray onboarding panels, stabilize connection mode selection, fix FunctionalUI reparenting during conditional renders, and add runtime hooks needed for tray window capture and WebChat error rendering.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address onboarding pairing feedback

Remove the local gateway auto-approval shortcut and use the existing pairing command copy/notification flow instead. Also scope bootstrap operator handshakes to the gateway handoff profile, skip Chat for Configure Later, and dispose onboarding state safely.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Constrain bootstrap auth to onboarding setup codes

Keep the default gateway client auth payload and chat URL construction aligned with the existing tray app, while allowing onboarding setup-code handoff to opt into bootstrap auth scopes explicitly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Tighten gateway security follow-ups

Preserve MCP-only onboarding completion routing, remove the unused public connect auth token getter, and add regression coverage for default operator scopes and paired bootstrap handoff auth.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Mike Harsh <mharsh@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-05-01 09:12:35 -07:00
Scott Hanselman
5ef3707509 chore: clean root artifacts and rename solution 
Remove tracked root diff and merge-analysis artifacts that were left from prior PR review work.

Rename the solution file from the historical moltbot-windows-hub.slnx to openclaw-windows-node.slnx and update docs/tests that use it for developer guidance or repo-root discovery.

Validation: ./build.ps1; dotnet test ./tests/OpenClaw.Shared.Tests/OpenClaw.Shared.Tests.csproj --no-restore; dotnet test ./tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj --no-restore; dotnet build ./openclaw-windows-node.slnx --no-restore.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-30 01:00:20 -07:00
Scott Hanselman
65542764f2 docs: sync mission control parity status 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 07:01:37 -07:00
Scott Hanselman
6b880ccf96 feat: expose tray diagnostics copy actions 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 06:49:09 -07:00
Scott Hanselman
16d3003356 feat: add tray ssh restart action 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 06:38:48 -07:00
Scott Hanselman
62d7e0c28b feat: add debug bundle entrypoint 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 06:32:21 -07:00
Scott Hanselman
4184e12751 feat: add tray support debug actions 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 06:24:13 -07:00
Scott Hanselman
ec4ce4aafc feat: add summary copy entrypoints 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 06:17:46 -07:00
Scott Hanselman
582cec5557 feat: add diagnostics copy entrypoints 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 06:13:14 -07:00
Scott Hanselman
51c14f38d1 feat: add port diagnostics entrypoint 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 06:08:12 -07:00
Scott Hanselman
65339ab658 feat: add manual update check entrypoint 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 05:57:54 -07:00
Scott Hanselman
8bba19ca9d feat: add browser setup guidance entrypoint 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 05:44:45 -07:00
Scott Hanselman
0ca2014244 fix: redact support context paths 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 05:13:58 -07:00
Scott Hanselman
18b43a50e8 feat: add support debug entrypoints 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 05:04:09 -07:00
Scott Hanselman
44262559b9 feat: add command center ssh restart 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 04:50:48 -07:00
Scott Hanselman
9bce622daa docs: sync deep link list 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-27 04:41:10 -07:00
Scott Hanselman
99f803e5ba feat: add tray command center diagnostics 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-26 23:11:00 -07:00
Scott Hanselman
f121395b06 feat: add safe device parity commands 
Add canvas.a2ui.pushJSONL as a Mac-compatible alias for the existing A2UI JSONL push handler.

Add device.info and device.status using the shared OpenClawKit payload shape, with Windows metadata/status sources plus tests and docs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-26 22:43:16 -07:00
Scott Hanselman
ea6cf23add docs: update Windows node parity audit 2026-04-26 22:26:56 -07:00
Scott Hanselman
dba534250c feat: add screen recording parity 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-26 22:18:59 -07:00
Scott Hanselman
c1296be7fd
feat: add camera.clip, location.get, and canvas local file serving
Some checks are pending
Build and Test / test (push) Waiting to run
Details
Build and Test / build (win-arm64) (push) Blocked by required conditions
Details
Build and Test / build (win-x64) (push) Blocked by required conditions
Details
Build and Test / build-msix (ARM64, win-arm64) (push) Blocked by required conditions
Details
Build and Test / build-msix (x64, win-x64) (push) Blocked by required conditions
Details
Build and Test / build-extension (arm64) (push) Blocked by required conditions
Details
Build and Test / build-extension (x64) (push) Blocked by required conditions
Details
Build and Test / release (push) Blocked by required conditions
Details 
Adds Windows node camera.clip, location.get, canvas local file serving/reload support, gateway command parity docs, and aligns screen capture with gateway-canonical screen.snapshot. Also makes camera.clip choose a supported Windows MediaCapture record stream and skips startup update prompts in debug builds for local node debugging.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-25 23:19:15 -07:00
Scott Hanselman
e772b5e6e9 security: harden system.run approval checks 
Closes #184 by blocking dangerous environment overrides and by re-evaluating nested shell-wrapper payloads and chained commands against the exec approval policy.

This extends the partial env-only approach discussed in PR #186 so the Windows node closes both vectors called out in the issue.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-19 21:33:02 -07:00
Sytone
1d836390e9
feat: SSH tunnel gateway, device identity, reconnect hardening 
Adds SSH local port-forward support for secure remote gateway access, Ed25519 device identity for operator auth, enhanced Quick Send with error remediation, reconnect resilience, and OpenClaw.Cli validator tool.

Includes security fix: SSH user/host input validation to prevent command injection.

615 tests pass (516 shared + 99 tray).

Contributed by @sytone
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-01 00:03:43 -07:00
github-actions[bot]
37a5f9453d
docs: add SETUP.md and POWERTOYS.md guides 
Comprehensive Windows Node setup guide and PowerToys Command Palette extension documentation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-31 23:44:18 -07:00
Scott Hanselman
882937299a docs: remove stale WinForms references, update test counts and capabilities 
- README.md: Fix project table (OpenClaw.Tray → OpenClaw.Tray.WinUI),
  remove WinForms run command, add system.run.prepare and system.which
  to capability table and allowCommands JSON, remove '(investigating)'
  from canvas.a2ui commands
- DEVELOPMENT.md: Remove OpenClaw.Tray/ from structure, add
  OpenClaw.Tray.Tests/, update test counts (88 → 571), fix CI section
- build.ps1: Fix broken 'Tray' target to point at WinUI .csproj,
  remove WinForms from default build and run instructions
- docs/VERSIONING.md: Remove reference to deleted OpenClaw.Tray.csproj
- docs/TEST_COVERAGE.md: Full rewrite (88 → 571 tests, .NET 9 → 10)
- docs/CODE_REVIEW.md: Update project names, test counts, .NET version
- docs/WINDOWS_NODE_TESTING.md: Mark system.run as implemented, update
  capability descriptions
- docs/WINDOWS_NODE_ARCHITECTURE.md: Add historical planning note,
  update current state table (Node mode now implemented)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-17 20:38:13 -07:00
Scott Hanselman
f0ccbc8403
chore: clean up stale files and fix repo URLs (#51) 
- Remove input.json.bak (stray backup file)
- Update README project structure (remove old OpenClaw.Tray, add
  WinUI and test projects)
- Fix installer URLs to point at openclaw/openclaw-windows-node
  instead of old shanselman/moltbot-windows-hub
 2026-03-16 22:25:32 -07:00
Scott Hanselman
3025daaa71 feat(winui): add activity stream and richer node observability 
- add shared models and gateway client handling for session, usage, and node.list telemetry

- add ActivityStreamService plus ActivityStreamWindow flyout with filtering, clear, and dashboard jump actions

- wire WinUI tray actions/status surfaces for activity, node/session/usage updates, and one-time discoverability tip

- document exec-policy troubleshooting in README, including Windows policy path and powershell.exe argv[0] matching

- expand shared tests for model parsing and gateway client behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-02-16 23:02:57 -08:00
Scott Hanselman
4d8d43bcfe Improve web chat security UX and bump to 0.4.2
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (win-arm64) (push) Has been cancelled
Details
Build and Test / build (win-x64) (push) Has been cancelled
Details
Build and Test / build-msix (ARM64, win-arm64) (push) Has been cancelled
Details
Build and Test / build-msix (x64, win-x64) (push) Has been cancelled
Details
Build and Test / build-extension (arm64) (push) Has been cancelled
Details
Build and Test / build-extension (x64) (push) Has been cancelled
Details
Build and Test / release (push) Has been cancelled
Details 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-02-16 21:49:13 -08:00
Scott Hanselman
c5e5a291e9 Fix issue #18 README allowCommands docs
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (win-arm64) (push) Has been cancelled
Details
Build and Test / build (win-x64) (push) Has been cancelled
Details
Build and Test / build-msix (ARM64, win-arm64) (push) Has been cancelled
Details
Build and Test / build-msix (x64, win-x64) (push) Has been cancelled
Details
Build and Test / build-extension (arm64) (push) Has been cancelled
Details
Build and Test / build-extension (x64) (push) Has been cancelled
Details
Build and Test / release (push) Has been cancelled
Details 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-02-16 21:22:08 -08:00
Scott Hanselman
199f855729
feat: enhanced notification categorization pipeline (#12) (#15) 
Replace keyword-only classification with a layered pipeline:
structured metadata (intent/channel) > user rules > keyword fallback.

- Add Channel, Agent, Intent, Tags fields to OpenClawNotification
- Extract NotificationCategorizer class with layered pipeline
- Add NotifyChatResponses toggle to suppress chat toasts
- Add UserNotificationRule model for custom regex/keyword rules
- Map error notifications to urgent setting
- Add 30+ unit tests for categorizer pipeline
- Document categorization system in docs/
 2026-02-07 17:24:11 -08:00
Scott Hanselman
3fc96748d5 feat: MSIX dual-distribution packaging 
Add MSIX packaging support alongside existing Inno Setup/ZIP distribution.
The build now supports both packaged (MSIX) and unpackaged (EXE) modes
via the PackageMsix MSBuild property.

MSIX packaging enables:
- Native Windows camera/microphone consent prompts
- Package identity for OS integration
- Protocol activation for openclaw:// deep links via manifest

Changes:
- csproj: Conditional WindowsPackageType (None vs MSIX) toggled by
  -p:PackageMsix=true. Self-contained in both modes. ApplicationVersion
  synced from Version property for MSIX.
- Package.appxmanifest: Declares webcam, microphone, internetClient
  capabilities. Protocol handler for openclaw://. Visual assets.
- PackageHelper.cs: Runtime detection of packaged vs unpackaged mode
  via Windows.ApplicationModel.Package.Current
- App.xaml.cs: Protocol activation handling via AppInstance lifecycle API
  for MSIX deep links. Works alongside existing command-line deep links.
- DeepLinkHandler.cs: Skip registry URI registration when packaged
  (manifest handles it)
- ci.yml: New build-msix job (x64 + ARM64) parallel to existing build.
  MSIX signing via Azure Trusted Signing. MSIX artifacts in releases.
  Null-check on MSIX find step.
- Visual assets: Generated PNG logos for MSIX package tiles
- README: Added MSIX build instructions

Build modes:
  dotnet build -r win-arm64                     # Unpackaged (default)
  dotnet build -r win-arm64 -p:PackageMsix=true # MSIX package

Note: MSIX Publisher in Package.appxmanifest must match the Azure
Trusted Signing certificate subject exactly. Currently set to
'CN=Scott Hanselman' - update if cert subject differs.
 2026-02-07 15:34:49 -08:00
Scott Hanselman
4003e58d8f feat: system.run + exec approval policy 
Implement system.run command execution with swappable ICommandRunner
interface and glob-pattern-based exec approval policy for security.

ICommandRunner abstraction:
- ICommandRunner interface with CommandRequest/CommandResult models
- LocalCommandRunner: Process.Start with pwsh/cmd/powershell support
- Supports shell selection, cwd, timeout, env vars, args
- Designed for future DockerCommandRunner/WslCommandRunner swap-in

Exec approval policy:
- ExecApprovalPolicy engine with glob-pattern matching (*, ?)
- First-match-wins rule evaluation with Allow/Deny/Ask actions
- JSON persistence to exec-policy.json in data directory
- Default policy: allows read-only commands (echo, Get-*, hostname),
  denies destructive ops (rm, shutdown, Format-*, registry)
- system.execApprovals.get: retrieve current policy rules
- system.execApprovals.set: update policy remotely

Node integration:
- SystemCapability checks approval policy before executing commands
- Denied commands return error with policy rule explanation
- NodeService wires LocalCommandRunner + ExecApprovalPolicy

Tests (191 total, all passing):
- 30 exec approval tests: policy evaluation, persistence, patterns,
  SystemCapability integration, default rules validation
- system.run unit + integration tests
- Fixed: NodeCapabilityBase arg helpers guard default(JsonElement)

Docs updated:
- README: exec approvals commands, usage examples, gateway config
- Architecture doc: capability matrix, roadmap checkboxes updated
 2026-02-07 14:16:15 -08:00
Scott Hanselman
e3eeef3367 feat: add Windows node mode capabilities 
Implement node protocol, pairing, and invoke result handling.

Add canvas/screen/camera capabilities with WebView2, GDI capture, and MediaCapture frame reader fallback.

Expose node mode toggle/status in tray settings and update Windows node docs/testing guide.
 2026-02-03 21:25:46 -08:00
Scott Hanselman
bb06fb12c4 Fix WebView2 in unpackaged WinUI on ARM64 
- Enable WindowsAppSDKSelfContained for registration-free WinRT activation
- Build script auto-detects architecture (win-arm64 / win-x64)
- WebChatWindow shows error details instead of silently falling back
- Remove conditional self-contained (always needed for WebView2)

The issue: WebView2 COM activation fails in unpackaged apps because
WinRT registration isn't available. Self-contained mode bundles the
'Undocked RegFree WinRT' which enables COM activation without MSIX.
 2026-02-01 15:14:55 -08:00
Scott Hanselman
7b63816368 Add Windows SDK and WebView2 to prerequisites 
- build.ps1 now checks for WebView2 Runtime
- README lists all prerequisites for WinUI build
- WebView2 check is informational (app falls back to browser)
 2026-02-01 14:35:35 -08:00
Scott Hanselman
758ce08cf0 Upgrade all projects to .NET 10 
- OpenClaw.Tray.WinUI: net9.0 -> net10.0
- OpenClaw.Shared: multi-target -> net10.0 single target
- Updated build.ps1 to only require .NET 10
- Updated README with .NET 10 only prerequisites
- Updated issue #10 comment
 2026-02-01 14:34:50 -08:00
Scott Hanselman
5f18e7b530 Add build.ps1 script with prerequisite detection 
- build.ps1 checks for .NET 9, .NET 10, Windows SDK
- Provides clear guidance on what's missing
- Supports building individual projects or all
- Fix README: correct dotnet run command (needs full .csproj path)
- Expanded Quick Start section with multiple build options

Fixes #10
 2026-02-01 14:29:51 -08:00
Scott Hanselman
6353e05929 Complete OpenClaw rename: fix remaining references 
- CI artifacts: moltbot-commandpalette -> openclaw-commandpalette
- CODE_REVIEW.md: Update class names to OpenClaw versions
- README files: moltbot.com -> openclaw.ai
- Keep repo URLs as-is (can't change without renaming repo)
 2026-01-30 10:28:16 -08:00
copilot-swe-agent[bot]
c3c7f4a7aa Rename all Moltbot references to OpenClaw 
- Renamed all project directories and files
- Updated all namespaces and class names
- Updated deep link scheme from moltbot:// to openclaw://
- Updated AppData directories to OpenClawTray
- Updated README with OpenClaw references and added history note
- Updated all configuration and build files
- Kept mascot name "Molty" unchanged

Co-authored-by: shanselman <2892+shanselman@users.noreply.github.com>
 2026-01-30 10:16:18 -08:00
Scott Hanselman
83d61eb641 Fix Command Palette extension and bundle in installer
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (win-arm64) (push) Has been cancelled
Details
Build and Test / build (win-x64) (push) Has been cancelled
Details
Build and Test / build-extension (arm64) (push) Has been cancelled
Details
Build and Test / build-extension (x64) (push) Has been cancelled
Details
Build and Test / release (push) Has been cancelled
Details 
- Fix exe name mismatch: Moltbot.exe -> Moltbot.CommandPalette.exe in manifest
- Add AppListEntry=none to hide from Start Menu (only show in Command Palette)
- Use OpenUrlCommand from toolkit instead of custom DeepLinkCommand
- Add cmdpal-dev.ps1 script for local iteration (auto-detects arm64/x64)
- Update installer to optionally install Command Palette extension
- Add molty2.png to README
- Update CI to bundle Command Palette in installer
 2026-01-28 23:33:32 -08:00
Scott Hanselman
bd5b27f135 feat: Add last check timestamp, about info, UI polish 
- Add last check timestamp to tray tooltip (culture-aware)
- Add 'Made with lobster love' to Settings dialog (clickable → GitHub)
- Add tooltip on menu header with attribution
- Fix Settings dialog size (no scrollbars)
- Update parity table (now 10/12 features)
- Add attribution to README
 2026-01-28 22:41:11 -08:00
Scott Hanselman
b1c1e5edeb docs: Fix parity table to compare against actual Mac app 
Compare against magimetal/moltbot-menubar (the real macOS app).
Mac app is simpler - focused on gateway management.
Windows app has many additional features not in Mac version.
 2026-01-28 22:29:15 -08:00
Scott Hanselman
fb6260ea84 docs: Fix parity table with accurate status 
- Deep links now fully tested ✅
- Menu bar icon variants: ⚠️ (pixel art vs template)
- Native share sheet: ❌ (Mac-specific)
- Keyboard navigation: ❌ (not implemented)
- Activity animation: ⚠️ (badge dot only)
 2026-01-28 22:27:41 -08:00
Scott Hanselman
de558fa245 feat: Complete deep link support with IPC forwarding 
- Add IPC (named pipe) to forward deep links to running instance
- Add toast notifications for sent message confirmation
- moltbot://settings - Opens Settings dialog
- moltbot://chat - Opens Web Chat window
- moltbot://dashboard[/path] - Opens Dashboard in browser
- moltbot://send?message=... - Opens Quick Send with prefilled text
- moltbot://agent?message=... - Sends directly with confirmation prompt
- Document deep links in README
- Update Mac parity table (deep links now complete)
 2026-01-28 22:26:07 -08:00
Scott Hanselman
40a68ec100 Modern Windows 11 UI overhaul with Mac parity
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build-extension (x64) (push) Has been cancelled
Details
Build and Test / build (win-arm64) (push) Has been cancelled
Details
Build and Test / build (win-x64) (push) Has been cancelled
Details
Build and Test / build-extension (arm64) (push) Has been cancelled
Details
Build and Test / release (push) Has been cancelled
Details 
- New ModernTrayMenu: Windows 11-style flyout replacing ContextMenuStrip
  - Dark/light mode auto-detection
  - Lobster branding header with accent colors
  - Clickable channel toggles (start/stop Telegram/WhatsApp)
  - Sessions link to /sessions, Cron Jobs to /cron
  - Status badges with color coding (READY/IDLE/ON/OFF)

- New ModernForm base class for all dialogs
  - Rounded corners via DWM APIs
  - Consistent theming across Settings, QuickSend, WebChat, etc.
  - Accent color support

- New WelcomeDialog for first-run experience
  - Guides users to get API token
  - Links to docs.molt.bot documentation
  - Opens Settings after onboarding

- Channel status parity: unified READY status for linked channels
- Service Health menu item (replaces Run Health Check)
- Test Notification button in Settings
- Various DPI and spacing fixes

- Updated README with screenshot and expanded feature list
 2026-01-28 22:15:59 -08:00
Scott Hanselman
5a6a0646f2 Update README: .NET 10, correct hotkey, fix log path, add auto-update feature 2026-01-28 20:45:38 -08:00
Scott Hanselman
db54ae50b2 Initial commit: Moltbot Windows Hub monorepo 
- Moltbot.Tray: Windows system tray companion
- Moltbot.Shared: Gateway client library
- Moltbot.CommandPalette: PowerToys Command Palette extension

Consolidated from clawdbot-windows-tray, clawdbot-shared, and moltbot-commandpalette-extension repos.
 2026-01-28 17:34:44 -08:00