elophanto/EloPhanto v2026.03.04.1 on GitHub

Desktop GUI Control — Control Any Application Like a Human

The agent can now see your screen and control any desktop application — clicking buttons, typing text, scrolling, dragging — at pixel coordinates, just like a human sitting at your computer.

Two Modes

Local mode — control your own machine directly via pyautogui. No VM needed. Set desktop.mode: local and say "open Excel and make me a chart"
Remote mode — connect to a VM running the OSWorld HTTP server for sandboxed environments and benchmarks. Set desktop.mode: remote with a vm_ip

9 New Tools

Tool	What it does
`desktop_connect`	Connect to local desktop or remote VM
`desktop_screenshot`	Capture current screen as PNG
`desktop_click`	Click at x,y (left/right/double-click)
`desktop_type`	Type text, press key, or send hotkey
`desktop_scroll`	Scroll mouse wheel
`desktop_drag`	Drag from one position to another
`desktop_cursor`	Move cursor without clicking
`desktop_shell`	Run shell command on target machine
`desktop_file`	Download file from target for inspection

How It Works

User: "Open Excel and create a Q1 revenue spreadsheet"

→ desktop_connect(mode="local")     # Connect to this PC
→ desktop_screenshot                # See the desktop
→ desktop_click(52, 1060)           # Click Excel in Dock
→ desktop_screenshot                # Verify Excel opened
→ desktop_click(85, 200)            # Click cell A1
→ desktop_type("Month")             # Type header
→ desktop_type(key="tab")           # Next cell
→ desktop_type("Revenue")           # Type header
→ ...                               # Fill data, add chart
→ desktop_type(hotkey=["cmd", "s"]) # Save

Not just browsers. Any app — Excel, Photoshop, Terminal, Finder, anything with a GUI.

Architecture

core/desktop_controller.py — BaseDesktopController ABC with LocalDesktopController (pyautogui) and DesktopController (async HTTP to VM)
tools/desktop/tools.py — 9 tool classes extending BaseTool
Full integration: config, planner prompt, agent injection, feature badge, setup.sh

Also in this release

Agent Commune — social network for AI agents (7 tools)
Web deployment — deploy to Vercel/Railway + Supabase databases (3 tools)
Agent organization — self-cloning specialist agents (5 tools)
Gateway session fix — fixed UNIQUE constraint race on concurrent CLI+Telegram connects

Quick Start

# config.yaml
desktop:
  enabled: true
  mode: local    # or remote + vm_ip

uv pip install -e '.[desktop]'   # install pyautogui

Then tell the agent: "Connect to my desktop and take a screenshot"

Full Changelog: v2026.02.28.1...v2026.03.04.1

elophanto/EloPhanto v2026.03.04.1 v2026.03.04.1 — Desktop GUI Control on GitHub