DesktopCtl vs open-computer-use

DesktopCtl is a macOS launcher and command-line interface for AI agents that observe and control a computer through its screen, mouse, and keyboard. It works with any desktop application without app integrations or APIs, and accepts any CLI-capable agent and AI model, including local LLMs. Local GPU-accelerated text recognition and computer vision capture interface content and extract structured information. Deterministic CLI actions include clicking, typing, and waiting. The keyboard-first launcher shows active-window context and can run or resume sessions with multiple local agents. The maker says recognition runs locally, screenshots remain local by default, and a visual indicator appears during screen, mouse, or keyboard access. Operating-system permissions are granted only to DesktopCtl. It is open source under the MIT License, deployed locally, free to use, and currently supports macOS on Apple Silicon only. Release builds use an ad-hoc signature and are not Developer ID signed or notarized.

open-computer-use is a free, MIT-licensed local MCP server that lets AI agents inspect and operate desktop applications through accessibility interfaces. Its nine core tools cover app discovery, accessibility-tree and screenshot capture, clicking, scrolling, dragging, typing, key presses, secondary actions, and control-value changes. The project says it runs entirely on the user’s machine and uses the accessibility layer for core app actions without taking over the physical mouse and keyboard. Setup documentation covers Codex CLI and App, Claude Code, Gemini CLI, and opencode, with manual configuration for other MCP clients. A ZCode plugin installs the skill and connects an MCP server. macOS 14 or later requires Accessibility and Screen Recording permissions. Windows and Linux use a signed-in desktop session, and Linux requires AT-SPI2. Additional window-level tools are currently available on Windows, while macOS and Linux support is in progress. Password managers are refused; app launching, focus changes, and global input injection require explicit environment-variable gates. The project warns that high-privilege automation should be used only on authorized machines.