644dc662df
- Removed all test images and screenshots from project root - Ensured all tests use temporary directories for file creation - Added .serialized trait to Swift tests that interact with OS resources - Updated AXorcist import statements to use AXorcistLib - Configured Vitest for serial test execution to avoid conflicts Note: Swift compilation errors due to AXorcist API changes need to be fixed separately 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
3.7 KiB
3.7 KiB
Menu Extraction Implementation
Overview
We've successfully implemented comprehensive menu bar extraction in Peekaboo using pure accessibility APIs, without any clicking or UI disruption.
Key Features
1. Pure Accessibility-Based Extraction
- Extracts entire menu hierarchy using the accessibility tree
- No clicking or menu opening required
- Preserves user's current UI state
- Works with nested submenus
2. Available Commands
menu list --app AppName
Lists all menus for a specific application with full hierarchy:
peekaboo menu list --app Calculator
menu list-all
Lists menus for the frontmost application (system-wide enumeration limited by macOS):
peekaboo menu list-all
peekaboo menu list-all --json-output
3. Data Structure
The menu extraction provides:
- Menu titles
- Enabled/disabled state
- Keyboard shortcuts
- Full submenu hierarchy
- Menu item count
Example JSON structure:
{
"app_name": "Calculator",
"menus": [
{
"title": "View",
"enabled": true,
"items": [
{
"title": "Scientific",
"enabled": true,
"shortcut": "⌘2"
},
{
"title": "Programmer",
"enabled": true,
"shortcut": "⌘3"
}
]
}
]
}
4. Agent Integration
The AI agent can now:
- Use
menu(app="AppName", subcommand="list")to discover all menus - Navigate complex menu structures
- Make informed decisions about available options
- Execute menu commands without trial and error
5. Implementation Details
Menu Extraction Functions
extractFullMenu(): Extracts complete menu structure from menu bar itemsextractMenuItems(): Recursively extracts menu items and submenus- Uses AXorcist's Element wrapper for clean API access
Key Insights
- Menu bars are per-application, not system-wide
- Menu items have children that represent submenus
- Accessibility provides full menu structure without activation
- Status items require special handling via AXGroup elements
6. Technical Architecture
// Extract menu without clicking
if let menuBar = app.menuBar() {
for menuBarItem in menuBar.children() {
if menuBarItem.role() == AXRoleNames.kAXMenuBarItemRole {
// This menu bar item has the full menu structure as children
let menuData = extractFullMenu(from: menuBarItem)
}
}
}
7. Benefits
- Non-Disruptive: No visual changes or menu activation
- Complete Information: Full menu hierarchy in one call
- Performance: Fast extraction without UI interaction
- Reliability: Works consistently across applications
- Agent-Friendly: Structured data for AI decision making
Future Enhancements
- Multi-App Enumeration: Iterate through all running applications
- Menu Search: Find menu items by partial text match
- AXorcist Integration: Move functionality to AXorcist library
- Menu State Monitoring: Track menu changes over time
- Contextual Menus: Support for right-click menus
Usage Examples
Basic Usage
# List all menus for an app
peekaboo menu list --app Safari
# Get frontmost app's menus
peekaboo menu list-all
# JSON output for programmatic access
peekaboo menu list --app Finder --json-output
Agent Usage
# Discover menus
peekaboo agent "List all menus in Calculator"
# Navigate menus
peekaboo agent "Switch Calculator to Scientific mode using the menu"
# Complex tasks
peekaboo agent "Find and use the export function in the current app"
This implementation provides a solid foundation for menu automation while respecting macOS security and user experience constraints.