---
summary: 'Refactor tool results so agents can show rich, human-readable summaries'
read_when:
  - 'planning tool/agent runtime work'
  - 'touching ToolResponse or formatter plumbing'
---

# Tool Result Metadata Refactor Plan
_Status: Archived · Focus: richer ToolResponse formatting and summaries._

## Current Status
- `ToolEventSummary` struct + helpers live in `ToolEventSummary.swift`; pointer direction math handled in `PointerDirection.swift`.
- Tachikoma MCP adapter now preserves `meta` so summaries flow from tools to CLI/Mac renderers.
- Core UI/system tools (click/drag/move/swipe/scroll/see/shell/sleep/type/hotkey/app/menu/dialog/dock/list/window) populate summaries with human-readable labels instead of internal IDs.
- Permission/Image/Analyze/Space tool paths updated to emit contextual summaries (app name, capture source, question text, etc.).
- MCPAgentTool now emits summaries for session listings and agent runs, completing MCP tool coverage.
- CLI `AgentOutputDelegate` consumes `ToolEventSummary` data, strips legacy `[ok]` glyphs, and falls back to sanitized formatter output only when necessary.
- Mac tool formatter bridge + registry now prioritize `ToolEventSummary` data so timeline rows show the same human-readable summaries as the CLI.
- Added Swift Testing coverage (`ToolEventSummaryTests`, `ToolSummaryEmissionTests`) so shell/sleep summaries and short-description helpers are locked in.
- Streaming pipeline now injects a top-level `summary_text` field into tool completion payloads, giving JSON consumers the same human-readable copy without parsing nested meta blobs.
- Agent output formatters still contain legacy fallbacks; `[ok]` badges remain until we finish Phase 3.

## Next Steps
- Capture CLI/Mac golden transcripts once formatter cleanup lands in CI so we can detect regressions automatically.

## Goals
- Preserve structured context (app name, element label, pointer geometry, shell command, etc.) for every tool call.
- Render concise, human-readable summaries in the CLI/Mac agent views without exposing internal IDs or glyph tokens.
- Eliminate the success `[ok]` badge for normal completions; only show badges/flags on warnings or errors.
- Keep completion tools (`task_completed`, `need_more_information`, `need_info`) flowing through their existing "state" UI without extra summary lines.

## Constraints & Challenges
- `ToolResponse.meta` is currently dropped when converting to `AnyAgentToolValue`; formatters only see whatever plain text the tool returned.
- MCP tools live in `PeekabooAgentRuntime` while the agent runtime/CLI sits elsewhere, so the metadata schema must be shared via Tachikoma types.
- We must not break existing MCP integrations; the new summary data needs a backwards-compatible wire format.

## Phase 1 – Plumbing
1. Introduce a typed `ToolEventSummary` struct (in Tachikoma) with optional fields for app/window, element, coordinates, scroll/move vectors, command strings, durations, etc.
2. Extend `ToolResponse` to carry an optional `summary: ToolEventSummary` (or replace `meta` entirely) and ensure the MCP adapter serializes/deserializes it.
3. Update the agent streaming pipeline (`PeekabooAgentService+Streaming`, `AnyAgentToolValue`, CLI event payloads) so the summary is delivered alongside the existing text result.

## Phase 2 – Tool Implementations
1. Audit every MCP tool (click/type/scroll/see/shell/sleep/window/app/menu/dialog/drag/move/swipe/list/etc.).
2. For each tool, populate `ToolEventSummary` using the context it already has:
   - UI tools: `targetApp`, `windowTitle`, `elementLabel`, `elementRole`, `humanizedPosition`.
   - Pointer tools: `direction`, `distancePx`, `profile`, `durationMs`.
   - Vision tools: `captureApp`, `windowTitle`, `sessionId` (for internal tracing only if we still need it), element counts.
   - System tools: `shellCommand`, `workingDirectory`, `sleepMs`, `reason`.
3. Remove raw element IDs (`elem_153`) and replace them with user-facing labels.

## Phase 3 – Formatting & UX
1. Update `ToolFormatter` (and specialized subclasses) to prefer the new summary fields when generating compact/result summaries.
2. Teach `AgentOutputDelegate` to:
   - Drop the green `[ok]` marker on success.
   - Render geometry in natural language (e.g., `1280×720 anchored top-left on Display 1`).
   - Continue showing badges only for warnings/errors.
3. Verify the Mac UI timeline consumes the same summary strings.

## Phase 4 – Verification
- Add unit tests for representative tools ensuring they emit the expected `ToolEventSummary`.
- Record CLI golden outputs (before/after) to confirm we now print sentences like `Click – Chrome · Button "Sign In with Email"`.
- Dogfood on Grindr/Wingman workflow to ensure the motivation scenarios look correct end-to-end.

## Open Questions
- Should we completely remove `meta`, or keep it for third-party MCP clients that expect arbitrary dictionaries?
- Do we want localized summaries, or is English-only acceptable for now?
- How do we expose the same summaries via API (e.g., JSON streaming) for downstream automation/telemetry?
